-----============= acceptance-small: sanity-flr ============----- Tue Apr 1 07:51:52 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg314-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:51:57 (1743508317) === oleg314-client.virtnet: executing check_config_client /mnt/lustre oleg314-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg314-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800aa6aa000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800aa6aa000.idle_timeout=debug disable quota as required oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:52:05 (1743508325) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanity-flr test 0a: lfs mirror create with -N option == 07:52:05 (1743508325) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:52:08 (1743508328) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:52:09 (1743508329) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:52:10 (1743508330) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:52:14 (1743508334) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:52:15 (1743508335) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 07:52:16 (1743508336) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800aa6aa000.stats=clear osc.lustre-OST0001-osc-ffff8800aa6aa000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:52:18 (1743508338) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:52:20 (1743508340) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:52:22 (1743508342) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:52:23 (1743508343) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 07:52:25 (1743508345) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:52:27 (1743508347) PASS 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:52:28 (1743508348) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:52:30 (1743508350) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:52:32 (1743508352) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:52:34 (1743508354) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.293442 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023779 s, 44.1 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:52:36 (1743508356) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 0.135926 s, 131 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:52:38 (1743508358) Stopping /mnt/lustre-ost1 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800aa6aa000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aa6aa000.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg314-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800aa6aa000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800aa6aa000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg314-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:53:00 (1743508380) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.148776 s, 70.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800aa6aa000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aa6aa000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg314-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:53:08 (1743508388) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800aa6aa000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aa6aa000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg314-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800aa6aa000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800aa6aa000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg314-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:53:29 (1743508409) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.862859 s, 60.8 MB/s -rw-r--r-- 1 root root 52428800 Apr 1 07:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.226s user 0m0.000s sys 0m0.150s Stopping /mnt/lustre-ost1 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800aa6aa000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aa6aa000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.225s user 0m0.001s sys 0m0.156s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg314-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg314-server oleg314-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800aa6aa000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800aa6aa000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.248s user 0m0.000s sys 0m0.116s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg314-server: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg314-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800aa6aa000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800aa6aa000.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:53:51 (1743508431) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:53:52 (1743508432) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:53:53 (1743508433) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 07:53:53 (1743508433) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 07:53:55 (1743508435) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.133464 s, 94.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:53:58 (1743508438) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105208 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00975287 s, 108 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:53:59 (1743508439) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:54:02 (1743508442) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000475651 s, 2.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000630341 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000534495 s, 1.9 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000626659 s, 1.6 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 07:54:04 (1743508444) create test files with size 15737524 .. 86440f7803bcfe9efac86eafca8eab3b - 6bbc1a45b8f16605652bf503fa236f3f - 83f814edda21b6da96dd56733b283786 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg314-client.virtnet /mnt/lustre (opts:) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 07:54:09 (1743508449) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:54:11 (1743508451) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:54:12 (1743508452) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139246 s, 75.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00874537 s, 120 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 07:54:14 (1743508454) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0253617 s, 82.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0839896 s, 49.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0354693 s, 59.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 07:54:17 (1743508457) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 07:54:18 (1743508458) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:54:18 (1743508458) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120855 s, 86.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 07:54:20 (1743508460) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0867798 s, 36.2 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:54:22 (1743508462) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:54:24 (1743508464) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0628874 s, 167 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg314-client.virtnet /mnt/lustre (opts:) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:54:47 (1743508487) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b5928800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0807997 s, 130 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg314-client.virtnet /mnt/lustre (opts:) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012b71f800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:54:50 (1743508490) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:54:58 (1743508498) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:55:05 (1743508505) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 07:55:06 (1743508506) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0724932 s, 43.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0514643 s, 61.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 784173c8d19a85724f38a69186901ff2 - mirror 1 checksum: 741704bf4beac4f2c08af3b04cff3de5 - mirror 2 checksum: 784173c8d19a85724f38a69186901ff2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 1 07:55:06 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4b:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 1 07:55:06 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 784173c8d19a85724f38a69186901ff2 - mirror 1 checksum: 741704bf4beac4f2c08af3b04cff3de5 - mirror 2 checksum: 741704bf4beac4f2c08af3b04cff3de5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 1 07:55:06 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4b:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 1 07:55:06 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:55:08 (1743508508) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00997631 s, 105 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (1s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:55:09 (1743508509) Failing mds1 on oleg314-server Stopping /mnt/lustre-mds1 (opts:) on oleg314-server 07:55:17 (1743508517) shut down facet: mds1 facet_host: oleg314-server facet_failover_host: oleg314-server Failover mds1 to oleg314-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Started lustre-MDT0000 07:55:30 (1743508530) targets are mounted 07:55:30 (1743508530) facet_failover done oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00836274 s, 125 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:55:37 (1743508537) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00875951 s, 2.3 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00532244 s, 3.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:55:40 (1743508540) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00955948 s, 2.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00367523 s, 5.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:55:42 (1743508542) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00228137 s, 7.2 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:55:43 (1743508543) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0102624 s, 2.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:55:45 (1743508545) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00194575 s, 2.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00802913 s, 510 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:55:47 (1743508547) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:55:50 (1743508550) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:56:08 (1743508568) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:56:24 (1743508584) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:56:41 (1743508601) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 07:56:43 (1743508603) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26262 mirror split pid 26263 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7567: 26263 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26262 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 07:57:44 (1743508664) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 8665 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fallocating to largest ever: 0x3676b 000001 1743508665.149918 fallocate 0x027e6d thru 0x03676a (0xe8fe bytes) 000002 1743508665.153016 mapread 0x013775 thru 0x02290a (0xf196 bytes) 000003 1743508665.161004 read 0x02b7ea thru 0x02ff37 (0x474e bytes) 000004 1743508665.163332 mapwrite 0x024d29 thru 0x034834 (0xfb0c bytes) 000005 1743508665.179061 MIRROR_EXTEND to 2 mirrors 000006 1743508665.273053 write 0x00f8ae thru 0x0187ec (0x8f3f bytes) 000010 1743508665.285006 fallocate 0x02144f thru 0x027a59 (0x660b bytes) 000011 1743508665.286365 punch from 0x0023b0 to 0x009ece (0x7b1f bytes) 000012 1743508665.287623 mapread 0x010833 thru 0x019376 (0x8b44 bytes) 000014 1743508665.290342 mapwrite 0x013d8b thru 0x019698 (0x590e bytes) 000015 1743508665.296050 mapwrite 0x009341 thru 0x01773e (0xe3fe bytes) 000016 1743508665.304401 read 0x025457 thru 0x02eb57 (0x9701 bytes) 000017 1743508665.306896 read 0x00436a thru 0x00f4d1 (0xb168 bytes) 000018 1743508665.307355 read 0x02194a thru 0x02a463 (0x8b1a bytes) 000020 1743508665.310060 mapread 0x00c344 thru 0x012a1e (0x66db bytes) 000021 1743508665.310642 write 0x02aab2 thru 0x0356e6 (0xac35 bytes) 000023 1743508665.311942 mapread 0x023ca3 thru 0x028332 (0x4690 bytes) 000024 1743508665.312437 write 0x00cb5e thru 0x016a3f (0x9ee2 bytes) 000025 1743508665.313150 mapwrite 0x0248c9 thru 0x02a68a (0x5dc2 bytes) 000026 1743508665.320281 punch from 0x010289 to 0x01af64 (0xacdc bytes) 000027 1743508665.321901 punch from 0x01fbac to 0x02554d (0x59a2 bytes) 000029 1743508665.324056 write 0x02c0b6 thru 0x034956 (0x88a1 bytes) 000030 1743508665.325033 mapwrite 0x00e4dd thru 0x0163e2 (0x7f06 bytes) 000031 1743508665.340216 mapread 0x01c325 thru 0x01d21e (0xefa bytes) 000032 1743508665.340942 read 0x00c5cb thru 0x00c7f9 (0x22f bytes) 000033 1743508665.341392 mapread 0x02a4ab thru 0x03676a (0xc2c0 bytes) fallocating to largest ever: 0x40000 000034 1743508665.342148 fallocate 0x03adb8 thru 0x03ffff (0x5248 bytes) 000036 1743508665.345233 read 0x035570 thru 0x03ed21 (0x97b2 bytes) 000037 1743508665.348215 mapread 0x00d2f8 thru 0x019bd2 (0xc8db bytes) 000038 1743508665.348861 mapwrite 0x02c7bf thru 0x02c8da (0x11c bytes) 000039 1743508665.352545 trunc from 0x040000 to 0x03547e (0xab81 bytes) 000041 1743508665.361770 write 0x01a162 thru 0x01d505 (0x33a4 bytes) 000042 1743508665.362384 mapwrite 0x03948e thru 0x03ffff (0x6b72 bytes) 000043 1743508665.380448 mapread 0x03ff33 thru 0x03ffff (0xcd bytes) 000044 1743508665.381172 read 0x0322a3 thru 0x03df54 (0xbcb2 bytes) 000046 1743508665.383141 punch from 0x0219be to 0x0288f8 (0x6f3b bytes) 000047 1743508665.384724 read 0x03c576 thru 0x03fe93 (0x391e bytes) 000049 1743508665.385441 mapwrite 0x0120b0 thru 0x01fdee (0xdd3f bytes) 000050 1743508665.391676 mapread 0x0001a8 thru 0x00c3c0 (0xc219 bytes) 000051 1743508665.392195 write 0x0093a1 thru 0x016e48 (0xdaa8 bytes) 000052 1743508665.393058 trunc from 0x040000 to 0x020910 (0x1f6ef bytes) 000054 1743508665.397355 read 0x00f0ec thru 0x01bd37 (0xcc4c bytes) 000055 1743508665.397700 fallocate 0x03ef39 thru 0x03ffff (0x10c7 bytes) 000056 1743508665.399465 trunc from 0x040000 to 0x01933d (0x26cc2 bytes) 000057 1743508665.403297 punch from 0x01263b to 0x01933d (0x6d03 bytes) 000058 1743508665.407102 fallocate 0x003cf0 thru 0x006f83 (0x3294 bytes) 000059 1743508665.408584 mapwrite 0x00a727 thru 0x01386c (0x9146 bytes) 000060 1743508665.417066 punch from 0x001288 to 0x009ea6 (0x8c1f bytes) 000061 1743508665.418554 read 0x0091e8 thru 0x0121f0 (0x9009 bytes) 000063 1743508665.420976 mapwrite 0x0316d1 thru 0x03ffff (0xe92f bytes) 000064 1743508665.433301 mapread 0x002049 thru 0x00d031 (0xafe9 bytes) 000065 1743508665.433810 read 0x01c066 thru 0x0239b1 (0x794c bytes) 000066 1743508665.434153 punch from 0x03bef1 to 0x03ffff (0x410f bytes) 000067 1743508665.435693 mapread 0x01eab2 thru 0x02e18d (0xf6dc bytes) 000069 1743508665.437523 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000070 1743508665.472055 trunc from 0x040000 to 0x02df6c (0x12093 bytes) 000071 1743508665.476962 trunc from 0x02df6d to 0x03d2b4 (0xf348 bytes) 000072 1743508665.482194 MIRROR_EXTEND to 2 mirrors 000073 1743508665.596971 punch from 0x034942 to 0x03d2b4 (0x8973 bytes) 000074 1743508665.605700 write 0x024259 thru 0x025d4c (0x1af4 bytes) 000075 1743508665.609220 read 0x022058 thru 0x02b768 (0x9711 bytes) 000076 1743508665.613268 punch from 0x01d013 to 0x029dad (0xcd9b bytes) 000077 1743508665.616491 punch from 0x0053e4 to 0x00cc26 (0x7843 bytes) 000078 1743508665.618054 fallocate 0x03ec68 thru 0x03ffff (0x1398 bytes) 000080 1743508665.621234 MIRROR_EXTEND to 3 mirrors 000081 1743508665.735986 read 0x00941b thru 0x011a0d (0x85f3 bytes) 000083 1743508665.743059 MIRROR_RESYNC 3 mirrors 000084 1743508665.857108 read 0x023a70 thru 0x032830 (0xedc1 bytes) 000085 1743508665.859482 punch from 0x0340a8 to 0x038586 (0x44df bytes) 000086 1743508665.868020 mapread 0x004644 thru 0x00d248 (0x8c05 bytes) 000088 1743508665.873079 read 0x02f3e7 thru 0x03d2b4 (0xdece bytes) 000089 1743508665.876235 read 0x003c34 thru 0x006f8b (0x3358 bytes) 000091 1743508665.876967 trunc from 0x03d2b5 to 0x007cd7 (0x355dd bytes) 000092 1743508665.886846 mapwrite 0x03ccf4 thru 0x03ffff (0x330c bytes) 000093 1743508665.901495 MIRROR_EXTEND to 4 mirrors 000094 1743508666.007786 MIRROR_EXTEND to 5 mirrors 000096 1743508666.122187 MIRROR_RESYNC 5 mirrors 000097 1743508666.262755 trunc from 0x040000 to 0x03679b (0x9864 bytes) 000098 1743508666.271441 fallocate 0x033e0e thru 0x03ffff (0xc1f2 bytes) 000099 1743508666.272741 read 0x0234d0 thru 0x02ea63 (0xb594 bytes) 000100 1743508666.275947 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000101 1743508666.323919 mapwrite 0x03e4f0 thru 0x03e8c3 (0x3d4 bytes) 000102 1743508666.335384 write 0x03d63f thru 0x03ffff (0x29c1 bytes) 000103 1743508666.337285 mapwrite 0x008eff thru 0x0169e0 (0xdae2 bytes) 000104 1743508666.347697 mapwrite 0x00db9e thru 0x01572f (0x7b92 bytes) 000105 1743508666.353358 mapread 0x01b7e9 thru 0x027192 (0xb9aa bytes) 000106 1743508666.357730 trunc from 0x040000 to 0x016d30 (0x292cf bytes) 000107 1743508666.363161 MIRROR_RESYNC 4 mirrors 000108 1743508666.496999 mapwrite 0x018eb3 thru 0x024d04 (0xbe52 bytes) 000109 1743508666.522306 trunc from 0x024d05 to 0x00f29f (0x15a65 bytes) 000110 1743508666.526998 mapwrite 0x019cef thru 0x01a370 (0x682 bytes) 000111 1743508666.536637 read 0x00fdca thru 0x0132b1 (0x34e8 bytes) 000112 1743508666.539489 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000113 1743508666.588851 write 0x025c80 thru 0x026ce9 (0x106a bytes) 000114 1743508666.590730 trunc from 0x026cea to 0x017043 (0xfca6 bytes) 000115 1743508666.594366 punch from 0x003a7a to 0x00816a (0x46f1 bytes) 000116 1743508666.597158 read 0x0055b4 thru 0x00db26 (0x8573 bytes) 000117 1743508666.599527 fallocate 0x036835 thru 0x038c44 (0x2410 bytes) 000118 1743508666.601077 mapwrite 0x027cf8 thru 0x02a26a (0x2573 bytes) 000119 1743508666.610632 punch from 0x01848e to 0x01f3b6 (0x6f29 bytes) 000120 1743508666.612044 read 0x033756 thru 0x038c44 (0x54ef bytes) 000122 1743508666.615119 punch from 0x017ffc to 0x027f96 (0xff9b bytes) 000123 1743508666.616587 read 0x00aa9e thru 0x0188d8 (0xde3b bytes) 000124 1743508666.619079 fallocate 0x03c04e thru 0x03ffff (0x3fb2 bytes) 000126 1743508666.622467 MIRROR_RESYNC 3 mirrors 000127 1743508666.776235 mapread 0x01cd49 thru 0x02824e (0xb506 bytes) 000128 1743508666.782577 punch from 0x0215b4 to 0x02f1d9 (0xdc26 bytes) 000129 1743508666.791133 write 0x0164d9 thru 0x01e2e6 (0x7e0e bytes) 000131 1743508666.794236 mapwrite 0x01a7e6 thru 0x01dcf3 (0x350e bytes) 000132 1743508666.801155 mapwrite 0x0296cf thru 0x032a1c (0x934e bytes) 000133 1743508666.811878 fallocate 0x0195a3 thru 0x019fdb (0xa39 bytes) 000134 1743508666.814954 MIRROR_EXTEND to 4 mirrors 000136 1743508666.943909 MIRROR_RESYNC 4 mirrors 000139 1743508667.135048 mapwrite 0x0121dd thru 0x01db26 (0xb94a bytes) 000141 1743508667.171550 read 0x0061b4 thru 0x011c2a (0xba77 bytes) 000142 1743508667.174134 write 0x0092c3 thru 0x017e8a (0xebc8 bytes) 000143 1743508667.176293 MIRROR_RESYNC 4 mirrors 000144 1743508667.382330 read 0x01e4b5 thru 0x028792 (0xa2de bytes) 000145 1743508667.384612 fallocate 0x02447e thru 0x0303b0 (0xbf33 bytes) 000146 1743508667.394104 write 0x03c051 thru 0x03ffff (0x3faf bytes) 000147 1743508667.396088 fallocate 0x023106 thru 0x024813 (0x170e bytes) 000148 1743508667.397701 mapread 0x007bd1 thru 0x014233 (0xc663 bytes) 000149 1743508667.403673 write 0x024299 thru 0x030f1f (0xcc87 bytes) 000150 1743508667.408134 mapread 0x03bc7d thru 0x03ffff (0x4383 bytes) 000151 1743508667.410803 write 0x034f20 thru 0x03ffff (0xb0e0 bytes) 000152 1743508667.411765 write 0x00e18c thru 0x00ed40 (0xbb5 bytes) 000153 1743508667.412201 read 0x037e27 thru 0x03a7f3 (0x29cd bytes) 000154 1743508667.412494 punch from 0x03ba44 to 0x03ffff (0x45bc bytes) 000156 1743508667.417426 write 0x02348f thru 0x028d1f (0x5891 bytes) 000160 1743508667.419552 mapwrite 0x00bace thru 0x00f048 (0x357b bytes) 000161 1743508667.428037 read 0x00fc67 thru 0x019c97 (0xa031 bytes) 000163 1743508667.428581 trunc from 0x040000 to 0x01b883 (0x2477c bytes) 000165 1743508667.438910 trunc from 0x01b884 to 0x00afd5 (0x108ae bytes) 000166 1743508667.443198 fallocate 0x004cde thru 0x00f6c0 (0xa9e3 bytes) 000167 1743508667.444794 mapread 0x0091cd thru 0x00f6c0 (0x64f4 bytes) 000168 1743508667.447444 punch from 0x000bb4 to 0x00b1b0 (0xa5fd bytes) 000169 1743508667.449009 mapread 0x006b68 thru 0x00f6c0 (0x8b59 bytes) 000173 1743508667.456519 mapwrite 0x029ec1 thru 0x0375a0 (0xd6e0 bytes) 000174 1743508667.469114 punch from 0x00909f to 0x00e159 (0x50bb bytes) 000175 1743508667.470564 fallocate 0x00ec7e thru 0x01706c (0x83ef bytes) 000176 1743508667.472226 punch from 0x03082c to 0x0375a0 (0x6d75 bytes) 000177 1743508667.475016 MIRROR_RESYNC 4 mirrors 000178 1743508667.651968 punch from 0x02fa23 to 0x036054 (0x6632 bytes) 000179 1743508667.660811 punch from 0x032c2f to 0x033579 (0x94b bytes) 000180 1743508667.662004 read 0x02fe63 thru 0x0375a0 (0x773e bytes) 000182 1743508667.664759 write 0x036485 thru 0x03ac41 (0x47bd bytes) 000183 1743508667.666485 MIRROR_RESYNC 4 mirrors 000184 1743508667.851987 mapwrite 0x02ab7a thru 0x032f72 (0x83f9 bytes) 000185 1743508667.883248 write 0x03a1a9 thru 0x03ffff (0x5e57 bytes) 000186 1743508667.885168 fallocate 0x02558e thru 0x02d96e (0x83e1 bytes) 000187 1743508667.886583 mapread 0x01f812 thru 0x02e93d (0xf12c bytes) 000188 1743508667.890735 punch from 0x02a2fd to 0x038684 (0xe388 bytes) 000190 1743508667.892868 mapread 0x010f5d thru 0x0161d3 (0x5277 bytes) 000191 1743508667.895274 punch from 0x01cd8f to 0x028420 (0xb692 bytes) 000193 1743508667.897274 read 0x01d6f0 thru 0x022607 (0x4f18 bytes) 000194 1743508667.899526 trunc from 0x040000 to 0x026eda (0x19125 bytes) 000195 1743508667.909307 MIRROR_EXTEND to 5 mirrors 000197 1743508668.011981 fallocate 0x02bb8e thru 0x02bdc2 (0x235 bytes) 000198 1743508668.021848 write 0x02dc99 thru 0x02e944 (0xcac bytes) 000199 1743508668.023882 write 0x00eca1 thru 0x018b41 (0x9ea1 bytes) 000200 1743508668.027656 fallocate 0x021255 thru 0x02da6d (0xc819 bytes) 000201 1743508668.030717 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000202 1743508668.084649 MIRROR_RESYNC 4 mirrors 000203 1743508668.240130 MIRROR_RESYNC 4 mirrors 000204 1743508668.280023 read 0x00e7cd thru 0x012b41 (0x4375 bytes) 000205 1743508668.281859 trunc from 0x02e945 to 0x03c787 (0xde43 bytes) 000206 1743508668.291112 punch from 0x008c8d to 0x0099a3 (0xd17 bytes) 000207 1743508668.292144 mapread 0x009ee2 thru 0x016a82 (0xcba1 bytes) 000208 1743508668.297450 fallocate 0x030ca6 thru 0x03945f (0x87ba bytes) 000209 1743508668.298801 punch from 0x02d838 to 0x035e38 (0x8601 bytes) 000210 1743508668.301130 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000211 1743508668.350683 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000212 1743508668.396230 write 0x0108dd thru 0x010f18 (0x63c bytes) 000215 1743508668.399647 punch from 0x010f81 to 0x016352 (0x53d2 bytes) 000217 1743508668.404522 write 0x0091db thru 0x00e7f4 (0x561a bytes) 000218 1743508668.407561 mapwrite 0x019053 thru 0x02662e (0xd5dc bytes) 000219 1743508668.418239 read 0x033160 thru 0x035d61 (0x2c02 bytes) 000220 1743508668.419874 mapread 0x015fc9 thru 0x01d037 (0x706f bytes) 000221 1743508668.420361 punch from 0x022325 to 0x02f513 (0xd1ef bytes) 000223 1743508668.422454 mapread 0x0064a5 thru 0x00ed6d (0x88c9 bytes) 000224 1743508668.425645 trunc from 0x03c788 to 0x00c4cc (0x302bb bytes) 000225 1743508668.429521 punch from 0x000f66 to 0x00c4cc (0xb567 bytes) 000226 1743508668.431032 mapread 0x003b22 thru 0x00c4cc (0x89ab bytes) 000227 1743508668.434407 write 0x020247 thru 0x02abca (0xa984 bytes) 000228 1743508668.435258 fallocate 0x038d1d thru 0x03ffff (0x72e3 bytes) 000229 1743508668.438075 fallocate 0x0181c6 thru 0x0208b3 (0x86ee bytes) 000230 1743508668.439686 read 0x0000be thru 0x00d708 (0xd64b bytes) 000231 1743508668.441973 fallocate 0x0039b1 thru 0x007a4b (0x409b bytes) 000233 1743508668.443614 mapread 0x0099f1 thru 0x00f33b (0x594b bytes) 000234 1743508668.444026 mapwrite 0x02aa4d thru 0x02f1a1 (0x4755 bytes) 000235 1743508668.454825 fallocate 0x0337cc thru 0x03ffff (0xc834 bytes) 000236 1743508668.456455 read 0x008b15 thru 0x014ffc (0xc4e8 bytes) 000237 1743508668.457009 trunc from 0x040000 to 0x013bff (0x2c400 bytes) 000238 1743508668.460943 trunc from 0x013c00 to 0x03bc5c (0x2805d bytes) 000239 1743508668.464704 write 0x031c60 thru 0x03900e (0x73af bytes) 000240 1743508668.468170 write 0x03321b thru 0x039d50 (0x6b36 bytes) 000241 1743508668.468975 write 0x004d37 thru 0x008ac9 (0x3d93 bytes) 000242 1743508668.471027 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000243 1743508668.506758 trunc from 0x03bc5d to 0x0039c8 (0x38294 bytes) 000244 1743508668.512028 trunc from 0x0039c9 to 0x01b6d9 (0x17d11 bytes) 000245 1743508668.517424 MIRROR_EXTEND to 2 mirrors 000246 1743508668.618589 punch from 0x015133 to 0x01b6d9 (0x65a7 bytes) 000247 1743508668.627490 punch from 0x0070da to 0x00896b (0x1892 bytes) 000248 1743508668.628930 mapread 0x0197d3 thru 0x01b6d9 (0x1f07 bytes) 000249 1743508668.634417 write 0x0070f0 thru 0x010b3e (0x9a4f bytes) 000250 1743508668.638394 mapwrite 0x02a64f thru 0x0333ee (0x8da0 bytes) 000251 1743508668.659303 MIRROR_EXTEND to 3 mirrors 000252 1743508668.773768 punch from 0x014898 to 0x0191ee (0x4957 bytes) 000254 1743508668.782271 mapread 0x030ef5 thru 0x0333ee (0x24fa bytes) 000255 1743508668.787120 read 0x02e4fc thru 0x0333ee (0x4ef3 bytes) 000256 1743508668.787457 trunc from 0x0333ef to 0x00a7ff (0x28bef bytes) 000257 1743508668.796520 trunc from 0x00a800 to 0x022867 (0x18068 bytes) 000258 1743508668.800040 mapread 0x011c29 thru 0x01502f (0x3407 bytes) 000260 1743508668.803901 read 0x00db79 thru 0x01a5e9 (0xca71 bytes) 000261 1743508668.805975 mapread 0x0037b9 thru 0x00a51b (0x6d63 bytes) 000262 1743508668.806451 read 0x020b7a thru 0x022867 (0x1cee bytes) 000264 1743508668.808186 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000265 1743508668.856025 trunc from 0x022868 to 0x02469b (0x1e34 bytes) 000268 1743508668.861313 write 0x0303d9 thru 0x03a2b1 (0x9ed9 bytes) 000270 1743508668.862990 punch from 0x033265 to 0x03a2b1 (0x704d bytes) 000271 1743508668.867094 read 0x00792a thru 0x010b77 (0x924e bytes) 000272 1743508668.869876 punch from 0x00986d to 0x00e1cc (0x4960 bytes) 000273 1743508668.871584 fallocate 0x01f0b3 thru 0x0296dd (0xa62b bytes) 000274 1743508668.873271 trunc from 0x03a2b2 to 0x010f6a (0x29347 bytes) 000275 1743508668.877381 mapread 0x003ec9 thru 0x009587 (0x56bf bytes) 000277 1743508668.882655 trunc from 0x010f6b to 0x0291fc (0x18292 bytes) 000278 1743508668.887966 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000280 1743508668.922483 read 0x011404 thru 0x01d2fe (0xbefb bytes) 000281 1743508668.925698 read 0x023c00 thru 0x027478 (0x3879 bytes) 000283 1743508668.929259 mapread 0x02853f thru 0x0291fc (0xcbe bytes) 000284 1743508668.929730 fallocate 0x015a0c thru 0x016a26 (0x101b bytes) 000286 1743508668.933015 MIRROR_EXTEND to 2 mirrors 000287 1743508669.054897 MIRROR_EXTEND to 3 mirrors 000289 1743508669.180958 mapread 0x00d618 thru 0x019848 (0xc231 bytes) 000290 1743508669.191415 fallocate 0x01ca75 thru 0x01cca7 (0x233 bytes) 000291 1743508669.201162 read 0x026bc9 thru 0x0291fc (0x2634 bytes) 000292 1743508669.202972 read 0x0040de thru 0x0045f1 (0x514 bytes) 000293 1743508669.206103 trunc from 0x0291fd to 0x01bddc (0xd420 bytes) 000294 1743508669.212013 write 0x00367d thru 0x009f63 (0x68e7 bytes) 000295 1743508669.212640 mapread 0x007ad7 thru 0x00ce35 (0x535f bytes) 000296 1743508669.213107 trunc from 0x01bddd to 0x030734 (0x14958 bytes) 000297 1743508669.217653 MIRROR_RESYNC 3 mirrors 000298 1743508669.341073 mapread 0x00dc1c thru 0x00ea04 (0xde9 bytes) 000299 1743508669.343799 mapread 0x0122d2 thru 0x01af0e (0x8c3d bytes) 000300 1743508669.347647 mapwrite 0x034643 thru 0x036a0f (0x23cd bytes) 000301 1743508669.368758 read 0x0054da thru 0x00feff (0xaa26 bytes) 000302 1743508669.371156 fallocate 0x03fc46 thru 0x03ffff (0x3ba bytes) 000303 1743508669.372534 punch from 0x022be0 to 0x02df48 (0xb369 bytes) 000305 1743508669.374211 trunc from 0x040000 to 0x0309a5 (0xf65a bytes) 000306 1743508669.378351 punch from 0x00e2df to 0x016c3b (0x895d bytes) 000307 1743508669.380937 MIRROR_RESYNC 3 mirrors 000308 1743508669.520849 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000309 1743508669.573652 MIRROR_RESYNC 2 mirrors 000310 1743508669.613269 MIRROR_EXTEND to 3 mirrors 000311 1743508669.722851 read 0x005001 thru 0x006355 (0x1355 bytes) 000312 1743508669.725841 MIRROR_EXTEND to 4 mirrors 000313 1743508669.855294 read 0x0046fc thru 0x00782e (0x3133 bytes) 000314 1743508669.859642 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000317 1743508669.949016 fallocate 0x000ea1 thru 0x00349e (0x25fe bytes) 000318 1743508669.959471 punch from 0x0253e2 to 0x02eae7 (0x9706 bytes) 000319 1743508669.960878 read 0x014c2f thru 0x01ef45 (0xa317 bytes) 000320 1743508669.963412 mapread 0x01287e thru 0x016d1d (0x44a0 bytes) 000321 1743508669.966917 read 0x01a3b4 thru 0x020ac1 (0x670e bytes) 000322 1743508669.970006 punch from 0x028c47 to 0x0309a5 (0x7d5f bytes) 000323 1743508669.971801 trunc from 0x0309a6 to 0x01583e (0x1b167 bytes) 000324 1743508669.981444 trunc from 0x01583f to 0x02d31a (0x17adc bytes) 000326 1743508669.985426 punch from 0x025dca to 0x02624b (0x482 bytes) 000327 1743508669.987682 trunc from 0x02d31b to 0x037b83 (0xa869 bytes) 000329 1743508669.991243 mapwrite 0x01ca9a thru 0x023667 (0x6bce bytes) 000330 1743508670.008567 MIRROR_RESYNC 3 mirrors 000331 1743508670.142846 write 0x0178e4 thru 0x02275c (0xae79 bytes) 000334 1743508670.153705 read 0x01cb98 thru 0x0281ee (0xb657 bytes) 000335 1743508670.155864 fallocate 0x0127e2 thru 0x021426 (0xec45 bytes) 000337 1743508670.157966 fallocate 0x0026e5 thru 0x01025d (0xdb79 bytes) 000338 1743508670.159194 fallocate 0x01e08c thru 0x027e90 (0x9e05 bytes) 000340 1743508670.160741 write 0x00e35a thru 0x01c25c (0xdf03 bytes) 000341 1743508670.163867 punch from 0x00235a to 0x003a49 (0x16f0 bytes) 000342 1743508670.165308 write 0x034159 thru 0x03d514 (0x93bc bytes) 000343 1743508670.167970 trunc from 0x03d515 to 0x03ab01 (0x2a13 bytes) 000345 1743508670.173391 mapwrite 0x039c97 thru 0x03ffff (0x6369 bytes) 000346 1743508670.185172 mapwrite 0x0132ee thru 0x01ac16 (0x7929 bytes) 000347 1743508670.189723 punch from 0x00c68c to 0x017127 (0xaa9c bytes) 000348 1743508670.191264 write 0x015970 thru 0x023607 (0xdc98 bytes) 000349 1743508670.193665 read 0x018a06 thru 0x01cd17 (0x4312 bytes) 000350 1743508670.195215 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000352 1743508670.247000 mapwrite 0x02cebd thru 0x03125b (0x439f bytes) 000353 1743508670.257371 trunc from 0x040000 to 0x035796 (0xa869 bytes) 000354 1743508670.262135 mapread 0x007553 thru 0x0081a9 (0xc57 bytes) 000356 1743508670.266122 MIRROR_RESYNC 2 mirrors 000358 1743508670.379144 trunc from 0x035797 to 0x02e2ef (0x74a7 bytes) 000360 1743508670.392176 mapread 0x01dcc4 thru 0x022287 (0x45c4 bytes) 000361 1743508670.396953 read 0x02d909 thru 0x02e2ef (0x9e7 bytes) 000362 1743508670.398753 trunc from 0x02e2f0 to 0x026b53 (0x779c bytes) 000363 1743508670.403945 write 0x00cf26 thru 0x01c754 (0xf82f bytes) 000366 1743508670.407392 mapread 0x01fbb4 thru 0x026b53 (0x6fa0 bytes) 000367 1743508670.407905 mapread 0x0216ff thru 0x026b53 (0x5455 bytes) 000368 1743508670.408258 punch from 0x01e988 to 0x026b53 (0x81cc bytes) 000369 1743508670.410986 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000370 1743508670.444932 read 0x01ad3c thru 0x026b53 (0xbe18 bytes) 000371 1743508670.447505 mapread 0x018f36 thru 0x026b53 (0xdc1e bytes) 000375 1743508670.453162 MIRROR_EXTEND to 2 mirrors 000376 1743508670.551405 punch from 0x003420 to 0x00cf18 (0x9af9 bytes) 000377 1743508670.559924 mapread 0x007369 thru 0x0073a4 (0x3c bytes) 000379 1743508670.561920 punch from 0x0194e2 to 0x019686 (0x1a5 bytes) 000380 1743508670.563122 write 0x01860b thru 0x01c30b (0x3d01 bytes) 000381 1743508670.567198 fallocate 0x03c0f3 thru 0x03ea19 (0x2927 bytes) 000382 1743508670.570182 MIRROR_EXTEND to 3 mirrors 000383 1743508670.693750 mapread 0x002ec9 thru 0x00fda8 (0xcee0 bytes) 000384 1743508670.705991 mapread 0x007306 thru 0x00ca5f (0x575a bytes) 000385 1743508670.709903 read 0x03d537 thru 0x03ea19 (0x14e3 bytes) 000386 1743508670.713876 fallocate 0x00a18c thru 0x01512b (0xafa0 bytes) 000387 1743508670.722851 trunc from 0x03ea1a to 0x007893 (0x37186 bytes) 000388 1743508670.731903 fallocate 0x02cb77 thru 0x03c551 (0xf9db bytes) 000389 1743508670.733674 write 0x02f635 thru 0x033899 (0x4265 bytes) 000391 1743508670.734891 read 0x02c747 thru 0x033899 (0x7153 bytes) 000393 1743508670.739218 trunc from 0x03389a to 0x018731 (0x1b168 bytes) 000394 1743508670.743800 trunc from 0x018732 to 0x02e1bc (0x15a8b bytes) 000395 1743508670.747310 trunc from 0x02e1bd to 0x02be7b (0x2341 bytes) 000396 1743508670.750656 punch from 0x02a43c to 0x02be7b (0x1a40 bytes) 000397 1743508670.752094 fallocate 0x034fcf thru 0x03ffff (0xb031 bytes) 000398 1743508670.753516 punch from 0x02a06b to 0x02be7b (0x1e11 bytes) 000399 1743508670.754669 punch from 0x02ae07 to 0x02be7b (0x1075 bytes) 000400 1743508670.755848 trunc from 0x02be7c to 0x02d853 (0x19d8 bytes) 000402 1743508670.759485 mapread 0x000218 thru 0x001298 (0x1081 bytes) 000403 1743508670.759964 read 0x00e168 thru 0x01cfc4 (0xee5d bytes) 000404 1743508670.763651 MIRROR_EXTEND to 4 mirrors 000405 1743508670.861447 read 0x00a0b0 thru 0x010364 (0x62b5 bytes) 000406 1743508670.865929 MIRROR_RESYNC 4 mirrors 000407 1743508670.968044 trunc from 0x02d854 to 0x02018d (0xd6c6 bytes) 000408 1743508670.982531 mapwrite 0x0067d3 thru 0x00dcfe (0x752c bytes) 000409 1743508670.993821 mapwrite 0x002e49 thru 0x0101d6 (0xd38e bytes) 000410 1743508671.000503 punch from 0x01bfc2 to 0x01cd03 (0xd42 bytes) 000411 1743508671.001943 mapread 0x00a6ad thru 0x014344 (0x9c98 bytes) 000412 1743508671.004621 write 0x018f9f thru 0x0258ce (0xc930 bytes) 000413 1743508671.005390 fallocate 0x021a3e thru 0x031188 (0xf74b bytes) 000414 1743508671.007229 mapread 0x0233d9 thru 0x0258ce (0x24f6 bytes) 000415 1743508671.007928 trunc from 0x0258cf to 0x01d0c4 (0x880a bytes) 000416 1743508671.013058 trunc from 0x01d0c5 to 0x016f48 (0x617c bytes) 000417 1743508671.017070 read 0x010041 thru 0x016f48 (0x6f08 bytes) 000418 1743508671.017609 trunc from 0x016f49 to 0x00981a (0xd72e bytes) 000419 1743508671.021426 read 0x000d72 thru 0x00229d (0x152c bytes) 000420 1743508671.023030 trunc from 0x00981b to 0x028e1c (0x1f602 bytes) 000421 1743508671.026895 mapwrite 0x034029 thru 0x037de2 (0x3dba bytes) 000423 1743508671.039543 fallocate 0x0215e1 thru 0x02d536 (0xbf56 bytes) 000424 1743508671.041466 punch from 0x024d2e to 0x0329ff (0xdcd2 bytes) 000425 1743508671.042990 mapread 0x01c73b thru 0x020ef4 (0x47ba bytes) 000426 1743508671.043461 trunc from 0x037de3 to 0x01f463 (0x1897f bytes) 000427 1743508671.047519 mapread 0x008784 thru 0x00c4b1 (0x3d2e bytes) 000428 1743508671.048240 mapread 0x00b2a6 thru 0x01441f (0x917a bytes) 000430 1743508671.049328 mapread 0x00c9a0 thru 0x014d53 (0x83b4 bytes) 000432 1743508671.050384 trunc from 0x01f464 to 0x0303b3 (0x10f50 bytes) 000433 1743508671.054411 fallocate 0x038eaf thru 0x03b73e (0x2890 bytes) 000434 1743508671.056040 fallocate 0x030c7a thru 0x03d170 (0xc4f7 bytes) 000435 1743508671.057676 punch from 0x028fcd to 0x0297b2 (0x7e6 bytes) 000436 1743508671.059053 read 0x026123 thru 0x026e5b (0xd39 bytes) 000437 1743508671.061960 read 0x0215d7 thru 0x022ba5 (0x15cf bytes) 000438 1743508671.062451 read 0x01dcbb thru 0x028c96 (0xafdc bytes) 000439 1743508671.063006 write 0x026cca thru 0x03124c (0xa583 bytes) 000440 1743508671.064075 read 0x019ca0 thru 0x01b049 (0x13aa bytes) 000441 1743508671.064525 fallocate 0x00b756 thru 0x016d9b (0xb646 bytes) 000442 1743508671.066786 read 0x0225ca thru 0x02275d (0x194 bytes) 000443 1743508671.068895 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000444 1743508671.120398 trunc from 0x03124d to 0x01c9af (0x1489d bytes) 000445 1743508671.124934 fallocate 0x010dd0 thru 0x012b34 (0x1d65 bytes) 000446 1743508671.126462 read 0x010358 thru 0x0199db (0x9684 bytes) 000447 1743508671.130674 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000448 1743508671.178012 write 0x0245c7 thru 0x02bd78 (0x77b2 bytes) 000449 1743508671.179794 fallocate 0x035e63 thru 0x03e02d (0x81cb bytes) 000451 1743508671.181685 trunc from 0x03e02e to 0x03c7c5 (0x1868 bytes) 000453 1743508671.185958 trunc from 0x03c7c6 to 0x02bf06 (0x108bf bytes) 000454 1743508671.190014 punch from 0x01a777 to 0x0204e5 (0x5d6f bytes) 000456 1743508671.191748 mapwrite 0x03ce80 thru 0x03f2a7 (0x2428 bytes) 000457 1743508671.206401 read 0x02bfc5 thru 0x03be09 (0xfe45 bytes) 000458 1743508671.209205 write 0x0038f2 thru 0x006b22 (0x3231 bytes) 000459 1743508671.211797 fallocate 0x0223f5 thru 0x02293f (0x54b bytes) 000460 1743508671.213397 mapwrite 0x03e375 thru 0x03ffff (0x1c8b bytes) 000461 1743508671.221784 punch from 0x023c95 to 0x02a4fa (0x6866 bytes) 000462 1743508671.223572 punch from 0x000cc4 to 0x00eddd (0xe11a bytes) 000463 1743508671.225274 fallocate 0x028aa7 thru 0x031910 (0x8e6a bytes) 000464 1743508671.229942 MIRROR_EXTEND to 3 mirrors 000465 1743508671.341297 write 0x03a666 thru 0x03ffff (0x599a bytes) 000466 1743508671.352162 MIRROR_EXTEND to 4 mirrors 000468 1743508671.474406 MIRROR_RESYNC 4 mirrors 000469 1743508671.620099 read 0x00aeb7 thru 0x012e93 (0x7fdd bytes) 000470 1743508671.622168 mapread 0x03eed0 thru 0x03ffff (0x1130 bytes) 000471 1743508671.624323 write 0x018c25 thru 0x02383a (0xac16 bytes) 000472 1743508671.635680 fallocate 0x015d38 thru 0x01a72c (0x49f5 bytes) 000473 1743508671.637007 trunc from 0x040000 to 0x00020e (0x3fdf1 bytes) 000474 1743508671.646634 read 0x0001f5 thru 0x00020e (0x1a bytes) 000475 1743508671.648058 fallocate 0x005cd4 thru 0x014750 (0xea7d bytes) 000476 1743508671.649469 trunc from 0x014751 to 0x03453c (0x1fdec bytes) 000477 1743508671.653875 punch from 0x021e14 to 0x030e2e (0xf01b bytes) 000478 1743508671.655170 mapread 0x017371 thru 0x0233e8 (0xc078 bytes) 000479 1743508671.658707 trunc from 0x03453d to 0x00c276 (0x282c6 bytes) 000480 1743508671.663314 write 0x01f2da thru 0x02038a (0x10b1 bytes) 000483 1743508671.664780 mapread 0x013150 thru 0x015322 (0x21d3 bytes) 000484 1743508671.668424 write 0x001f48 thru 0x00a79a (0x8853 bytes) 000485 1743508671.669361 fallocate 0x01bab2 thru 0x026dd4 (0xb323 bytes) 000486 1743508671.671116 fallocate 0x018097 thru 0x01fa83 (0x79ed bytes) 000487 1743508671.673667 mapread 0x01116f thru 0x017ef0 (0x6d82 bytes) 000488 1743508671.678112 write 0x03fde1 thru 0x03ffff (0x21f bytes) 000489 1743508671.679777 mapwrite 0x010606 thru 0x0149ab (0x43a6 bytes) 000490 1743508671.687256 mapread 0x030f2e thru 0x03ffff (0xf0d2 bytes) 000491 1743508671.692687 mapread 0x025f75 thru 0x02f918 (0x99a4 bytes) 000492 1743508671.704163 mapwrite 0x010aac thru 0x0176e3 (0x6c38 bytes) 000493 1743508671.709367 mapread 0x0040e0 thru 0x0085ce (0x44ef bytes) 000494 1743508671.709859 write 0x022c14 thru 0x029a9e (0x6e8b bytes) 000495 1743508671.712671 MIRROR_RESYNC 4 mirrors 000498 1743508671.895633 write 0x03df49 thru 0x03ffff (0x20b7 bytes) 000499 1743508671.905580 trunc from 0x040000 to 0x028aff (0x17500 bytes) 000500 1743508671.913960 punch from 0x00e027 to 0x013621 (0x55fb bytes) 000501 1743508671.915515 punch from 0x017c57 to 0x018a9d (0xe47 bytes) 000502 1743508671.916934 trunc from 0x028b00 to 0x0274a7 (0x1658 bytes) 000503 1743508671.921579 mapwrite 0x02e1b9 thru 0x03ce57 (0xec9f bytes) 000504 1743508671.937885 trunc from 0x03ce58 to 0x0355a2 (0x78b5 bytes) 000505 1743508671.941638 read 0x00c626 thru 0x0104ac (0x3e87 bytes) 000506 1743508671.943923 mapread 0x009382 thru 0x014e77 (0xbaf6 bytes) 000507 1743508671.947929 fallocate 0x00a5a9 thru 0x016062 (0xbaba bytes) 000508 1743508671.949365 read 0x022f8a thru 0x02a111 (0x7188 bytes) 000509 1743508671.953327 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000510 1743508672.000427 trunc from 0x0355a3 to 0x00aeff (0x2a6a3 bytes) 000511 1743508672.005778 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000512 1743508672.054084 read 0x004c5d thru 0x00aeff (0x62a3 bytes) 000513 1743508672.056403 punch from 0x00513d to 0x00aeff (0x5dc3 bytes) 000514 1743508672.057991 fallocate 0x033d5a thru 0x03e087 (0xa32e bytes) 000515 1743508672.059656 mapread 0x02941e thru 0x02e408 (0x4feb bytes) 000518 1743508672.064918 mapread 0x02187f thru 0x03103a (0xf7bc bytes) 000519 1743508672.067298 mapwrite 0x03f10a thru 0x03ffff (0xef6 bytes) 000520 1743508672.078789 MIRROR_RESYNC 2 mirrors 000522 1743508672.178393 mapwrite 0x0184eb thru 0x0240ee (0xbc04 bytes) 000523 1743508672.210064 write 0x02ee97 thru 0x03d297 (0xe401 bytes) 000524 1743508672.214930 MIRROR_RESYNC 2 mirrors 000525 1743508672.333055 read 0x025f9f thru 0x033bc1 (0xdc23 bytes) 000528 1743508672.336173 mapread 0x023bcd thru 0x02fd46 (0xc17a bytes) 000529 1743508672.338803 read 0x0239d9 thru 0x02adf1 (0x7419 bytes) 000530 1743508672.339122 read 0x01f729 thru 0x029a1b (0xa2f3 bytes) 000531 1743508672.341992 punch from 0x0081c4 to 0x00be7f (0x3cbc bytes) 000532 1743508672.351390 mapwrite 0x003ff4 thru 0x0068d4 (0x28e1 bytes) 000533 1743508672.363842 MIRROR_EXTEND to 3 mirrors 000534 1743508672.480315 mapread 0x0185ef thru 0x022ff4 (0xaa06 bytes) 000535 1743508672.489211 mapread 0x00189e thru 0x0087f3 (0x6f56 bytes) 000536 1743508672.494339 mapread 0x0085d4 thru 0x015605 (0xd032 bytes) 000537 1743508672.497968 trunc from 0x040000 to 0x03b10b (0x4ef4 bytes) 000538 1743508672.512723 MIRROR_EXTEND to 4 mirrors 000540 1743508672.621546 punch from 0x01f51b to 0x02646e (0x6f54 bytes) 000541 1743508672.630163 MIRROR_RESYNC 4 mirrors 000542 1743508672.786901 fallocate 0x018972 thru 0x01b6ca (0x2d59 bytes) 000543 1743508672.794915 write 0x02d43d thru 0x02fb7b (0x273f bytes) 000544 1743508672.797498 write 0x00d65d thru 0x00f5fd (0x1fa1 bytes) 000545 1743508672.801286 MIRROR_RESYNC 4 mirrors 000546 1743508672.981709 fallocate 0x0034dc thru 0x00ad61 (0x7886 bytes) 000547 1743508672.991144 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000548 1743508673.036915 punch from 0x027b1c to 0x02c309 (0x47ee bytes) 000549 1743508673.038406 mapread 0x023f2a thru 0x02d339 (0x9410 bytes) 000550 1743508673.045629 fallocate 0x03c332 thru 0x03ffff (0x3cce bytes) 000551 1743508673.047408 mapread 0x0074c9 thru 0x016e6e (0xf9a6 bytes) 000552 1743508673.051802 read 0x007c97 thru 0x00acae (0x3018 bytes) 000554 1743508673.053798 MIRROR_EXTEND to 4 mirrors 000555 1743508673.179008 trunc from 0x040000 to 0x02ebd3 (0x1142c bytes) 000556 1743508673.190936 mapwrite 0x020e35 thru 0x0214a8 (0x674 bytes) 000557 1743508673.200040 MIRROR_EXTEND to 5 mirrors 000559 1743508673.308941 read 0x007217 thru 0x014a1d (0xd807 bytes) 000560 1743508673.312614 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000562 1743508673.385104 write 0x01305c thru 0x01b6a5 (0x864a bytes) 000563 1743508673.396170 write 0x02508f thru 0x0343ee (0xf360 bytes) 000564 1743508673.398886 mapread 0x021ac6 thru 0x02c2c4 (0xa7ff bytes) 000565 1743508673.399386 punch from 0x003a08 to 0x0047b5 (0xdae bytes) 000566 1743508673.400847 trunc from 0x0343ef to 0x0241ac (0x10242 bytes) 000567 1743508673.409738 mapwrite 0x0300d3 thru 0x03ea63 (0xe991 bytes) 000568 1743508673.426383 read 0x01323e thru 0x01e9fe (0xb7c1 bytes) 000570 1743508673.429628 MIRROR_EXTEND to 5 mirrors 000571 1743508673.536250 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000572 1743508673.605453 read 0x01f6c4 thru 0x02d9bd (0xe2fa bytes) 000573 1743508673.611198 mapwrite 0x033c41 thru 0x03ded3 (0xa293 bytes) 000575 1743508673.647719 write 0x01a265 thru 0x01a72a (0x4c6 bytes) 000576 1743508673.651942 write 0x033b30 thru 0x03ffff (0xc4d0 bytes) 000577 1743508673.653824 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000578 1743508673.707017 read 0x037929 thru 0x037c06 (0x2de bytes) 000579 1743508673.708799 fallocate 0x016c94 thru 0x022a4b (0xbdb8 bytes) 000580 1743508673.710319 mapwrite 0x00af13 thru 0x010146 (0x5234 bytes) 000581 1743508673.719960 mapread 0x016a36 thru 0x0182ef (0x18ba bytes) 000583 1743508673.722766 write 0x030c43 thru 0x031e8e (0x124c bytes) 000584 1743508673.726498 trunc from 0x040000 to 0x016902 (0x296fd bytes) 000585 1743508673.735541 MIRROR_RESYNC 3 mirrors 000586 1743508673.857731 write 0x0087ad thru 0x010404 (0x7c58 bytes) 000587 1743508673.867945 trunc from 0x016903 to 0x01c187 (0x5885 bytes) 000588 1743508673.875992 mapwrite 0x036758 thru 0x039dd4 (0x367d bytes) 000592 1743508673.887921 mapwrite 0x00e1a4 thru 0x01bbdd (0xda3a bytes) 000595 1743508673.897993 fallocate 0x00e940 thru 0x0150f9 (0x67ba bytes) 000597 1743508673.899556 write 0x022f93 thru 0x02b025 (0x8093 bytes) 000599 1743508673.903078 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000600 1743508673.952986 write 0x00d18f thru 0x01a33b (0xd1ad bytes) 000601 1743508673.957023 trunc from 0x039dd5 to 0x02f041 (0xad93 bytes) 000602 1743508673.961841 punch from 0x02d52a to 0x02f041 (0x1b18 bytes) 000603 1743508673.963562 read 0x014676 thru 0x018906 (0x4291 bytes) 000604 1743508673.965554 MIRROR_EXTEND to 3 mirrors 000605 1743508674.091306 MIRROR_EXTEND to 4 mirrors 000606 1743508674.200363 trunc from 0x02f042 to 0x00b3ee (0x23c53 bytes) 000607 1743508674.211743 MIRROR_RESYNC 4 mirrors 000608 1743508674.313502 trunc from 0x00b3ef to 0x00f2eb (0x3efd bytes) 000609 1743508674.323776 mapwrite 0x023077 thru 0x02cddc (0x9d66 bytes) 000611 1743508674.342045 punch from 0x00ea44 to 0x01a224 (0xb7e1 bytes) 000612 1743508674.344088 trunc from 0x02cddd to 0x0053de (0x279fe bytes) 000613 1743508674.348948 fallocate 0x02116d thru 0x025187 (0x401b bytes) 000616 1743508674.351474 punch from 0x001564 to 0x0053de (0x3e7b bytes) 000618 1743508674.354999 MIRROR_RESYNC 4 mirrors 000620 1743508674.479393 MIRROR_EXTEND to 5 mirrors 000621 1743508674.576816 write 0x021403 thru 0x02b451 (0xa04f bytes) 000622 1743508674.585686 read 0x026ebf thru 0x02b451 (0x4593 bytes) 000623 1743508674.586009 mapread 0x020bb3 thru 0x0222f5 (0x1743 bytes) 000624 1743508674.587706 trunc from 0x02b452 to 0x03cb32 (0x116e1 bytes) 000625 1743508674.592847 read 0x000b51 thru 0x00c1e5 (0xb695 bytes) 000626 1743508674.596059 fallocate 0x02e8ef thru 0x02ed8b (0x49d bytes) 000627 1743508674.597424 punch from 0x014e88 to 0x018c66 (0x3ddf bytes) 000628 1743508674.598666 mapread 0x00cff7 thru 0x00e8c1 (0x18cb bytes) 000629 1743508674.599142 punch from 0x03a5f2 to 0x03cb32 (0x2541 bytes) 000630 1743508674.600439 fallocate 0x02c6bf thru 0x03aa77 (0xe3b9 bytes) 000631 1743508674.601973 punch from 0x029d0d to 0x0335fd (0x98f1 bytes) 000632 1743508674.605577 fallocate 0x03fb5e thru 0x03ffff (0x4a2 bytes) 000633 1743508674.607164 fallocate 0x0081bb thru 0x0119fd (0x9843 bytes) 000634 1743508674.608829 mapread 0x02bfc6 thru 0x02e855 (0x2890 bytes) 000635 1743508674.613943 mapwrite 0x00e9bb thru 0x0137f6 (0x4e3c bytes) 000636 1743508674.619222 trunc from 0x040000 to 0x0182d2 (0x27d2d bytes) 000638 1743508674.623314 trunc from 0x0182d3 to 0x033212 (0x1af40 bytes) 000639 1743508674.626721 punch from 0x0018ad to 0x00de44 (0xc598 bytes) 000641 1743508674.628555 mapread 0x018be8 thru 0x01ecf9 (0x6112 bytes) 000642 1743508674.637288 trunc from 0x033213 to 0x03029f (0x2f73 bytes) 000643 1743508674.642658 MIRROR_RESYNC 5 mirrors 000644 1743508674.808323 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000646 1743508674.870760 punch from 0x010aa5 to 0x01fd22 (0xf27e bytes) 000648 1743508674.880486 MIRROR_EXTEND to 5 mirrors 000649 1743508674.981553 write 0x015deb thru 0x021a4a (0xbc60 bytes) 000650 1743508674.993206 punch from 0x022c8c to 0x0240b1 (0x1426 bytes) 000651 1743508674.994774 mapread 0x024d41 thru 0x02d515 (0x87d5 bytes) 000652 1743508674.998721 punch from 0x021083 to 0x02c9ae (0xb92c bytes) 000654 1743508675.002500 mapread 0x0059fd thru 0x00b607 (0x5c0b bytes) 000655 1743508675.004949 mapread 0x016905 thru 0x01dd60 (0x745c bytes) 000656 1743508675.005403 mapread 0x008d26 thru 0x012828 (0x9b03 bytes) 000657 1743508675.005836 mapread 0x0205d9 thru 0x022252 (0x1c7a bytes) 000658 1743508675.008019 read 0x01f200 thru 0x029669 (0xa46a bytes) 000660 1743508675.008495 trunc from 0x0302a0 to 0x02db24 (0x277b bytes) 000661 1743508675.016911 mapread 0x01c536 thru 0x0295a0 (0xd06b bytes) 000663 1743508675.017631 read 0x011714 thru 0x01483c (0x3129 bytes) 000665 1743508675.019901 punch from 0x00aa22 to 0x017d14 (0xd2f3 bytes) 000666 1743508675.021415 mapread 0x0115ec thru 0x018115 (0x6b2a bytes) 000667 1743508675.029171 mapread 0x022f1d thru 0x02c6c0 (0x97a4 bytes) 000668 1743508675.029651 write 0x02bcbc thru 0x030445 (0x478a bytes) 000669 1743508675.030189 fallocate 0x00505b thru 0x006d19 (0x1cbf bytes) 000670 1743508675.032774 MIRROR_EXTEND to 6 mirrors 000671 1743508675.133196 mapwrite 0x017a0a thru 0x017dcb (0x3c2 bytes) 000672 1743508675.161431 write 0x003dd6 thru 0x00f5df (0xb80a bytes) 000673 1743508675.165205 mapwrite 0x03a7b0 thru 0x03ffff (0x5850 bytes) 000674 1743508675.182488 punch from 0x00f70b to 0x016724 (0x701a bytes) 000675 1743508675.184107 write 0x00fe62 thru 0x015556 (0x56f5 bytes) 000676 1743508675.187613 MIRROR_RESYNC 6 mirrors 000677 1743508675.491848 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000678 1743508675.558110 mapwrite 0x020318 thru 0x02ef1a (0xec03 bytes) 000679 1743508675.590907 write 0x01b809 thru 0x025ff6 (0xa7ee bytes) 000680 1743508675.592779 write 0x011365 thru 0x01a19b (0x8e37 bytes) 000681 1743508675.595207 mapwrite 0x012299 thru 0x014c7b (0x29e3 bytes) 000682 1743508675.600225 fallocate 0x00437e thru 0x00955f (0x51e2 bytes) 000683 1743508675.602863 MIRROR_RESYNC 5 mirrors 000684 1743508675.786668 mapread 0x008c5a thru 0x0117a2 (0x8b49 bytes) 000685 1743508675.791120 punch from 0x01ac1a to 0x0211a9 (0x6590 bytes) 000686 1743508675.802088 mapread 0x00c3e1 thru 0x0150ea (0x8d0a bytes) 000687 1743508675.807176 mapread 0x03c0d4 thru 0x03ffff (0x3f2c bytes) 000688 1743508675.810997 MIRROR_EXTEND to 6 mirrors 000689 1743508675.932829 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000691 1743508675.990116 fallocate 0x02297e thru 0x028c0a (0x628d bytes) 000693 1743508675.999447 trunc from 0x040000 to 0x0266a3 (0x1995c bytes) 000694 1743508676.009753 fallocate 0x00f271 thru 0x0122d7 (0x3067 bytes) 000695 1743508676.011249 fallocate 0x027971 thru 0x030259 (0x88e9 bytes) 000696 1743508676.012849 fallocate 0x02edf1 thru 0x032f4d (0x415d bytes) 000697 1743508676.014356 punch from 0x008420 to 0x013c1d (0xb7fe bytes) 000698 1743508676.015873 write 0x030627 thru 0x030bf6 (0x5d0 bytes) 000700 1743508676.017556 mapwrite 0x006caa thru 0x00aff9 (0x4350 bytes) 000701 1743508676.028402 read 0x03045b thru 0x032f4d (0x2af3 bytes) 000702 1743508676.030304 punch from 0x0144a0 to 0x01dc59 (0x97ba bytes) 000705 1743508676.032701 write 0x037f93 thru 0x03ffff (0x806d bytes) 000706 1743508676.033842 mapread 0x010f73 thru 0x013635 (0x26c3 bytes) 000707 1743508676.036547 mapread 0x026306 thru 0x032663 (0xc35e bytes) 000708 1743508676.041264 mapwrite 0x03150d thru 0x03908a (0x7b7e bytes) 000709 1743508676.047308 mapwrite 0x034e54 thru 0x038968 (0x3b15 bytes) 000710 1743508676.052255 fallocate 0x01458c thru 0x016e62 (0x28d7 bytes) 000711 1743508676.054131 mapwrite 0x00d5a7 thru 0x019b91 (0xc5eb bytes) 000712 1743508676.063507 MIRROR_EXTEND to 6 mirrors 000713 1743508676.192815 mapread 0x034bdc thru 0x03ffff (0xb424 bytes) 000715 1743508676.203226 mapwrite 0x031cca thru 0x03af54 (0x928b bytes) 000716 1743508676.233231 punch from 0x03744d to 0x03d7df (0x6393 bytes) 000717 1743508676.234975 fallocate 0x0053e1 thru 0x007c07 (0x2827 bytes) 000718 1743508676.237964 MIRROR_RESYNC 6 mirrors 000720 1743508676.538967 read 0x03a565 thru 0x03dfc3 (0x3a5f bytes) 000721 1743508676.540549 mapwrite 0x02db0f thru 0x035a62 (0x7f54 bytes) 000722 1743508676.573782 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000723 1743508676.624830 mapread 0x020fc1 thru 0x0299da (0x8a1a bytes) 000724 1743508676.630288 trunc from 0x040000 to 0x027421 (0x18bde bytes) 000725 1743508676.639952 write 0x02f80f thru 0x034268 (0x4a5a bytes) 000726 1743508676.640893 read 0x00e088 thru 0x016788 (0x8701 bytes) 000727 1743508676.643335 trunc from 0x034269 to 0x012dda (0x2148e bytes) 000731 1743508676.648771 read 0x00ab54 thru 0x00c322 (0x17cf bytes) 000732 1743508676.651247 trunc from 0x012ddb to 0x00c0ce (0x6d0c bytes) 000733 1743508676.654866 trunc from 0x00c0cf to 0x02706c (0x1af9e bytes) 000734 1743508676.658523 mapwrite 0x02924b thru 0x029269 (0x1f bytes) 000735 1743508676.668213 fallocate 0x00ee22 thru 0x00f603 (0x7e2 bytes) 000736 1743508676.670705 MIRROR_EXTEND to 6 mirrors 000738 1743508676.771507 write 0x006191 thru 0x014438 (0xe2a8 bytes) 000739 1743508676.781791 fallocate 0x0004f6 thru 0x00c5e8 (0xc0f3 bytes) 000740 1743508676.783052 mapread 0x00d02a thru 0x011fb0 (0x4f87 bytes) 000743 1743508676.783954 fallocate 0x00d95f thru 0x01b869 (0xdf0b bytes) 000745 1743508676.785441 fallocate 0x00f508 thru 0x01c62d (0xd126 bytes) 000747 1743508676.787215 read 0x009450 thru 0x00ec80 (0x5831 bytes) 000748 1743508676.787481 punch from 0x013543 to 0x01d2ba (0x9d78 bytes) 000749 1743508676.790907 trunc from 0x02926a to 0x0103d8 (0x18e91 bytes) 000750 1743508676.798996 write 0x02c4cf thru 0x0361aa (0x9cdc bytes) 000751 1743508676.802131 MIRROR_EXTEND to 7 mirrors 000752 1743508676.901269 read 0x022399 thru 0x03054c (0xe1b4 bytes) 000753 1743508676.903640 mapread 0x004814 thru 0x006b80 (0x236d bytes) 000754 1743508676.908526 write 0x03b474 thru 0x03b635 (0x1c2 bytes) 000756 1743508676.917930 write 0x02c38c thru 0x03145f (0x50d4 bytes) 000757 1743508676.920232 punch from 0x01fab3 to 0x029dde (0xa32c bytes) 000758 1743508676.921756 trunc from 0x03b636 to 0x0026f8 (0x38f3d bytes) 000759 1743508676.931563 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000761 1743508676.986797 mapread 0x00131a thru 0x0026f8 (0x13df bytes) 000762 1743508676.989485 trunc from 0x0026f9 to 0x01abc1 (0x184c9 bytes) 000763 1743508676.993874 trunc from 0x01abc2 to 0x012e2e (0x7d93 bytes) 000764 1743508676.997268 read 0x0003c4 thru 0x009e6f (0x9aac bytes) 000765 1743508676.999526 read 0x00c4b4 thru 0x012e2e (0x697b bytes) 000766 1743508677.002637 MIRROR_EXTEND to 7 mirrors 000768 1743508677.105486 read 0x00b84c thru 0x012e2e (0x75e3 bytes) 000769 1743508677.108890 MIRROR_RESYNC 7 mirrors 000770 1743508677.255694 fallocate 0x0278ec thru 0x02879c (0xeb1 bytes) 000771 1743508677.265133 MIRROR_RESYNC 7 mirrors 000772 1743508677.418691 trunc from 0x012e2f to 0x01c822 (0x99f4 bytes) 000775 1743508677.429491 MIRROR_RESYNC 7 mirrors 000776 1743508677.579948 mapread 0x000a3f thru 0x010834 (0xfdf6 bytes) 000777 1743508677.588732 mapread 0x01c4dc thru 0x01c822 (0x347 bytes) 000778 1743508677.589121 trunc from 0x01c823 to 0x03d596 (0x20d74 bytes) 000779 1743508677.605611 MIRROR_RESYNC 7 mirrors 000780 1743508677.767984 mapread 0x022355 thru 0x02c283 (0x9f2f bytes) 000782 1743508677.775400 MIRROR_RESYNC 7 mirrors 000783 1743508677.830995 mapread 0x001a80 thru 0x002b53 (0x10d4 bytes) 000785 1743508677.834945 MIRROR_EXTEND to 8 mirrors 000786 1743508677.943337 fallocate 0x009d9a thru 0x00a8d2 (0xb39 bytes) 000787 1743508677.951706 mapwrite 0x003846 thru 0x00e06b (0xa826 bytes) 000789 1743508677.963136 read 0x02ccc9 thru 0x02e431 (0x1769 bytes) 000790 1743508677.966062 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000791 1743508678.013220 mapread 0x029edc thru 0x02bd02 (0x1e27 bytes) 000792 1743508678.017938 trunc from 0x03d597 to 0x01ceed (0x206a9 bytes) 000793 1743508678.026334 write 0x01f8e1 thru 0x028723 (0x8e43 bytes) 000794 1743508678.027028 read 0x028063 thru 0x028723 (0x6c1 bytes) 000795 1743508678.027365 mapwrite 0x0286f8 thru 0x02fdd9 (0x76e2 bytes) 000796 1743508678.038504 read 0x0293aa thru 0x02fdd9 (0x6a30 bytes) 000797 1743508678.038978 fallocate 0x00e0e1 thru 0x018c19 (0xab39 bytes) 000798 1743508678.042076 MIRROR_EXTEND to 8 mirrors 000799 1743508678.148564 fallocate 0x006d72 thru 0x0101a8 (0x9437 bytes) 000801 1743508678.157892 write 0x015a07 thru 0x01c0fc (0x66f6 bytes) 000802 1743508678.161988 trunc from 0x02fdda to 0x024baa (0xb22f bytes) 000803 1743508678.171414 read 0x00af36 thru 0x015ece (0xaf99 bytes) 000804 1743508678.174040 fallocate 0x00596e thru 0x012ab1 (0xd144 bytes) 000806 1743508678.175795 trunc from 0x024bab to 0x019b0c (0xb09e bytes) 000807 1743508678.179888 mapwrite 0x02c52d thru 0x037f50 (0xba24 bytes) 000808 1743508678.194120 MIRROR_RESYNC 8 mirrors 000809 1743508678.462966 read 0x0085dd thru 0x0136d0 (0xb0f4 bytes) 000811 1743508678.465557 mapread 0x010a88 thru 0x013624 (0x2b9d bytes) 000812 1743508678.465944 punch from 0x00ba5d to 0x01b39a (0xf93e bytes) 000813 1743508678.475077 write 0x018945 thru 0x01fe26 (0x74e2 bytes) 000814 1743508678.479125 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000815 1743508678.528834 mapread 0x0240a0 thru 0x02b7b9 (0x771a bytes) 000816 1743508678.533791 fallocate 0x03ec52 thru 0x03ffff (0x13ae bytes) 000817 1743508678.535364 mapread 0x029a97 thru 0x03392a (0x9e94 bytes) 000819 1743508678.538039 mapread 0x02242d thru 0x0262d0 (0x3ea4 bytes) 000820 1743508678.539836 MIRROR_EXTEND to 8 mirrors 000821 1743508678.641639 punch from 0x036daa to 0x037f50 (0x11a7 bytes) 000822 1743508678.650873 MIRROR_EXTEND to 9 mirrors 000823 1743508678.753402 trunc from 0x037f51 to 0x01cc9b (0x1b2b5 bytes) 000824 1743508678.767664 trunc from 0x01cc9c to 0x018984 (0x4317 bytes) 000825 1743508678.772128 trunc from 0x018985 to 0x029b4e (0x111ca bytes) 000826 1743508678.775675 trunc from 0x029b4f to 0x02cf73 (0x3425 bytes) 000828 1743508678.779323 punch from 0x01c67c to 0x0270ca (0xaa4f bytes) 000829 1743508678.780654 read 0x02c6a5 thru 0x02cf73 (0x8cf bytes) 000830 1743508678.782112 write 0x0264a7 thru 0x02d50c (0x7066 bytes) 000831 1743508678.784062 punch from 0x02be0c to 0x02d50c (0x1701 bytes) 000832 1743508678.787522 fallocate 0x019483 thru 0x01d3d1 (0x3f4f bytes) 000833 1743508678.788964 punch from 0x01f66a to 0x028fbc (0x9953 bytes) 000834 1743508678.791659 MIRROR_EXTEND to 10 mirrors 000835 1743508678.882944 trunc from 0x02d50d to 0x03b7a5 (0xe299 bytes) 000836 1743508678.897198 MIRROR_EXTEND to 11 mirrors 000837 1743508679.005627 trunc from 0x03b7a6 to 0x02ad02 (0x10aa3 bytes) 000838 1743508679.019332 punch from 0x000fee to 0x009788 (0x879b bytes) 000840 1743508679.020786 punch from 0x025aa4 to 0x02ad02 (0x525f bytes) 000841 1743508679.022031 read 0x00b452 thru 0x010f5a (0x5b09 bytes) 000842 1743508679.024047 mapwrite 0x00ed84 thru 0x01d02e (0xe2ab bytes) 000844 1743508679.036333 mapwrite 0x012c0c thru 0x01a8db (0x7cd0 bytes) 000846 1743508679.043187 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000847 1743508679.088819 fallocate 0x0376aa thru 0x03ffff (0x8956 bytes) 000848 1743508679.090581 fallocate 0x017652 thru 0x0239a3 (0xc352 bytes) 000850 1743508679.092597 punch from 0x027229 to 0x02a670 (0x3448 bytes) 000852 1743508679.094656 punch from 0x01a073 to 0x0267cb (0xc759 bytes) 000853 1743508679.096994 write 0x021020 thru 0x0276ac (0x668d bytes) 000854 1743508679.100994 fallocate 0x024009 thru 0x02d9c2 (0x99ba bytes) 000855 1743508679.102503 write 0x00b94f thru 0x00fb85 (0x4237 bytes) 000856 1743508679.105025 trunc from 0x02d9c3 to 0x03fc6c (0x122aa bytes) 000857 1743508679.108861 mapwrite 0x039a60 thru 0x03ffff (0x65a0 bytes) 000858 1743508679.119900 write 0x036d51 thru 0x03ffff (0x92af bytes) 000859 1743508679.120540 trunc from 0x040000 to 0x03eb41 (0x14be bytes) 000860 1743508679.124120 punch from 0x00806c to 0x015c01 (0xdb96 bytes) 000861 1743508679.125574 fallocate 0x02357c thru 0x02913d (0x5bc2 bytes) 000862 1743508679.126952 write 0x03a6f2 thru 0x03ffff (0x590e bytes) 000864 1743508679.127822 punch from 0x02f82d to 0x03dce7 (0xe4bb bytes) 000865 1743508679.131227 punch from 0x037d87 to 0x03ffff (0x8279 bytes) 000866 1743508679.133630 MIRROR_EXTEND to 11 mirrors 000867 1743508679.238259 punch from 0x01f9cb to 0x0282e5 (0x891b bytes) 000868 1743508679.246293 fallocate 0x01bb37 thru 0x028c21 (0xd0eb bytes) 000869 1743508679.247547 read 0x0297bd thru 0x0368d3 (0xd117 bytes) 000870 1743508679.250068 mapread 0x013018 thru 0x01fed4 (0xcebd bytes) 000871 1743508679.257117 read 0x0242e3 thru 0x026263 (0x1f81 bytes) 000872 1743508679.259282 read 0x0258d8 thru 0x025f2a (0x653 bytes) 000874 1743508679.259781 punch from 0x02ce0f to 0x0345c6 (0x77b8 bytes) 000876 1743508679.261153 punch from 0x024c36 to 0x034595 (0xf960 bytes) 000878 1743508679.264018 MIRROR_RESYNC 11 mirrors 000881 1743508679.687504 read 0x025690 thru 0x033c21 (0xe592 bytes) 000882 1743508679.690166 read 0x0091da thru 0x00beb1 (0x2cd8 bytes) 000883 1743508679.693066 punch from 0x038ef1 to 0x03b5b3 (0x26c3 bytes) 000884 1743508679.702868 trunc from 0x040000 to 0x012e24 (0x2d1db bytes) 000885 1743508679.711117 punch from 0x00c94c to 0x0112d7 (0x498c bytes) 000887 1743508679.712936 mapwrite 0x029b22 thru 0x02dfaf (0x448e bytes) 000888 1743508679.729349 MIRROR_EXTEND to 12 mirrors 000889 1743508679.834809 read 0x01e531 thru 0x0266a5 (0x8175 bytes) 000890 1743508679.839805 punch from 0x00728a to 0x00c41b (0x5192 bytes) 000891 1743508679.850228 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000892 1743508679.911153 punch from 0x026420 to 0x02dfaf (0x7b90 bytes) 000893 1743508679.913808 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000894 1743508679.967575 trunc from 0x02dfb0 to 0x022c2c (0xb383 bytes) 000895 1743508679.975797 trunc from 0x022c2d to 0x002bb3 (0x20079 bytes) 000896 1743508679.979308 trunc from 0x002bb4 to 0x00f8d2 (0xcd1f bytes) 000897 1743508679.983808 write 0x027a01 thru 0x02bfd5 (0x45d5 bytes) 000898 1743508679.984668 mapread 0x029ddc thru 0x02bfd5 (0x21fa bytes) 000899 1743508679.986528 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000903 1743508680.038094 read 0x018096 thru 0x018c7c (0xbe7 bytes) 000907 1743508680.040638 mapread 0x004d2b thru 0x013f3e (0xf214 bytes) 000908 1743508680.049590 MIRROR_EXTEND to 10 mirrors 000909 1743508680.158892 write 0x014afa thru 0x0177ab (0x2cb2 bytes) 000910 1743508680.171314 write 0x0011c3 thru 0x00f773 (0xe5b1 bytes) 000911 1743508680.174909 fallocate 0x0083e1 thru 0x016748 (0xe368 bytes) 000912 1743508680.176848 mapwrite 0x00b933 thru 0x00e5f4 (0x2cc2 bytes) 000913 1743508680.183395 mapwrite 0x02b928 thru 0x032ef4 (0x75cd bytes) 000915 1743508680.197634 fallocate 0x038e5f thru 0x0393be (0x560 bytes) 000916 1743508680.199428 write 0x00f40a thru 0x0110d1 (0x1cc8 bytes) 000917 1743508680.201718 mapread 0x01bd5c thru 0x01e87d (0x2b22 bytes) 000918 1743508680.204027 trunc from 0x032ef5 to 0x0094ea (0x29a0a bytes) 000919 1743508680.208669 write 0x00c1e9 thru 0x010333 (0x414b bytes) 000920 1743508680.209575 mapwrite 0x039c10 thru 0x03bd07 (0x20f8 bytes) 000922 1743508680.221488 mapwrite 0x0144bf thru 0x021632 (0xd174 bytes) 000924 1743508680.232199 mapwrite 0x026d88 thru 0x02f0b3 (0x832c bytes) 000925 1743508680.242303 mapread 0x02f47f thru 0x03640d (0x6f8f bytes) 000926 1743508680.242935 write 0x00d7da thru 0x017555 (0x9d7c bytes) 000927 1743508680.243767 trunc from 0x03bd08 to 0x0331a9 (0x8b5e bytes) 000929 1743508680.248238 fallocate 0x0106a5 thru 0x018a12 (0x836e bytes) 000930 1743508680.251030 MIRROR_RESYNC 10 mirrors 000931 1743508680.637989 MIRROR_EXTEND to 11 mirrors 000932 1743508680.784730 mapread 0x02a7b4 thru 0x02bfb1 (0x17fe bytes) 000933 1743508680.787721 read 0x0327e8 thru 0x0331a9 (0x9c2 bytes) 000934 1743508680.789179 write 0x03e184 thru 0x03ffff (0x1e7c bytes) 000937 1743508680.800727 trunc from 0x040000 to 0x03667c (0x9983 bytes) 000938 1743508680.809997 punch from 0x00f1fb to 0x0100dd (0xee3 bytes) 000939 1743508680.811585 fallocate 0x01efa4 thru 0x02523b (0x6298 bytes) 000940 1743508680.813069 punch from 0x034332 to 0x03667c (0x234b bytes) 000941 1743508680.814311 trunc from 0x03667d to 0x00914f (0x2d52d bytes) 000942 1743508680.819148 write 0x02b09f thru 0x02bfb3 (0xf15 bytes) 000943 1743508680.819713 fallocate 0x00945b thru 0x017180 (0xdd26 bytes) 000944 1743508680.821445 punch from 0x0107eb to 0x01a8c0 (0xa0d6 bytes) 000945 1743508680.822981 punch from 0x01757a to 0x024213 (0xcc9a bytes) 000946 1743508680.824541 write 0x03381b thru 0x03ffff (0xc7e5 bytes) 000947 1743508680.828373 MIRROR_EXTEND to 12 mirrors 000948 1743508680.957687 fallocate 0x008c02 thru 0x00dea0 (0x529f bytes) 000949 1743508680.967969 mapwrite 0x02a32a thru 0x0347d4 (0xa4ab bytes) 000950 1743508680.979588 punch from 0x03ad89 to 0x03ffff (0x5277 bytes) 000951 1743508680.981102 punch from 0x00768e to 0x0162a1 (0xec14 bytes) 000952 1743508680.982626 punch from 0x027ff6 to 0x029b82 (0x1b8d bytes) 000954 1743508680.984145 write 0x01ef83 thru 0x022c12 (0x3c90 bytes) 000955 1743508680.987532 punch from 0x017ba2 to 0x01bac7 (0x3f26 bytes) 000956 1743508680.988923 mapread 0x03ccbb thru 0x03ffff (0x3345 bytes) 000958 1743508680.991711 write 0x00068d thru 0x00eabc (0xe430 bytes) 000959 1743508680.994673 trunc from 0x040000 to 0x01c5fd (0x23a02 bytes) 000960 1743508681.005013 trunc from 0x01c5fe to 0x029612 (0xd015 bytes) 000961 1743508681.009306 mapwrite 0x024832 thru 0x02edef (0xa5be bytes) 000962 1743508681.023375 read 0x0091eb thru 0x0135ae (0xa3c4 bytes) 000963 1743508681.025259 mapwrite 0x010e48 thru 0x01c5cb (0xb784 bytes) 000965 1743508681.033647 write 0x007fd9 thru 0x008942 (0x96a bytes) 000966 1743508681.034344 fallocate 0x02d4a1 thru 0x03768d (0xa1ed bytes) 000967 1743508681.035978 fallocate 0x0073da thru 0x008618 (0x123f bytes) 000969 1743508681.037509 punch from 0x0203cd to 0x022ec7 (0x2afb bytes) 000970 1743508681.038682 mapwrite 0x01b03d thru 0x01db09 (0x2acd bytes) 000972 1743508681.043757 mapwrite 0x03c8ba thru 0x03ffff (0x3746 bytes) 000973 1743508681.054946 write 0x02cdf0 thru 0x03cc5d (0xfe6e bytes) 000974 1743508681.056381 write 0x019084 thru 0x026169 (0xd0e6 bytes) 000975 1743508681.058820 MIRROR_EXTEND to 13 mirrors 000976 1743508681.180017 MIRROR_RESYNC 13 mirrors 000978 1743508681.501062 write 0x03e0dd thru 0x03ffff (0x1f23 bytes) 000979 1743508681.510911 trunc from 0x040000 to 0x01aea9 (0x25156 bytes) 000980 1743508681.519787 fallocate 0x036c21 thru 0x03b9ae (0x4d8e bytes) 000981 1743508681.521120 punch from 0x00f4b3 to 0x016b49 (0x7697 bytes) 000982 1743508681.522528 mapwrite 0x03cd85 thru 0x03ffff (0x327b bytes) 000983 1743508681.535094 read 0x024b56 thru 0x02901d (0x44c8 bytes) 000984 1743508681.537531 fallocate 0x028816 thru 0x03703f (0xe82a bytes) 000985 1743508681.539073 fallocate 0x006942 thru 0x00ca02 (0x60c1 bytes) 000986 1743508681.540750 punch from 0x019a99 to 0x01d2c9 (0x3831 bytes) 000987 1743508681.542895 fallocate 0x03fa44 thru 0x03ffff (0x5bc bytes) 000988 1743508681.544565 trunc from 0x040000 to 0x01c7d8 (0x23827 bytes) 000989 1743508681.549742 punch from 0x002dc4 to 0x00dbf1 (0xae2e bytes) 000990 1743508681.551421 punch from 0x00db3f to 0x01b651 (0xdb13 bytes) 000992 1743508681.553444 mapwrite 0x03a706 thru 0x03c95e (0x2259 bytes) 000994 1743508681.567910 MIRROR_EXTEND to 14 mirrors 000996 1743508681.681350 MIRROR_RESYNC 14 mirrors 000997 1743508682.135973 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) 000998 1743508682.243863 punch from 0x031fb3 to 0x03c95e (0xa9ac bytes) 000999 1743508682.254271 trunc from 0x03c95f to 0x0225c8 (0x1a396 bytes) 001000 1743508682.260938 write 0x00b28c thru 0x00fe08 (0x4b7d bytes) All operations completed A-OK! PASS 70a (19s) == sanity-flr test 71: check mirror extend parent fid ==== 07:58:03 (1743508683) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0xa14:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (2s) == sanity-flr test 200a: stress test ===================== 07:58:05 (1743508685) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre2 Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8638880 .. Extending file size to 3813216 .. done Extending file size to 4114848 .. Extending file size to 7520256 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8299520 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8840896 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9103104 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8900448 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9039264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9204384 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9226656 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 3984 3985 3987 3988 3990 192.168.203.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) 192.168.203.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg314-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:59:08 (1743508748) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre2 Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8136000 .. Extending file size to 1444384 .. done Extending file size to 8905056 .. Extending file size to 5178368 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7277248 .. failed Extending file size to 8595520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9216352 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9221920 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8963808 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9003200 .. failed Extending file size to 9315872 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 8386 8387 8389 8390 8393 192.168.203.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) 192.168.203.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg314-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:00:12 (1743508812) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4048 1283640 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2148 1285540 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14872 3592148 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23012 3584008 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37884 7176156 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4048 1283640 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2148 1285540 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14872 3592148 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23012 3584008 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37884 7176156 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0390477 s, 53.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.051175 s, 41.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:00:16 (1743508816) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0272099 s, 77.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 08:00:18 (1743508818) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa20:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xee6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xee6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xee7:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec7:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:00:20 (1743508820) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0257074 s, 81.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x65:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x66:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:00:21 (1743508821) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0271944 s, 77.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xee8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xee9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xee8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xee9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec9:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:00:23 (1743508823) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0219679 s, 95.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecb:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:00:46 (1743508846) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg314-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013639 s, 76.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeeb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeeb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeec:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:01:20 (1743508880) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg314-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0273275 s, 76.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xece:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xece:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:01:55 (1743508915) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg314-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg314-server: Pool lustre.test_204f created Adding targets to pool oleg314-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg314-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0233617 s, 89.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xecf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg314-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg314-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg314-server: Pool lustre.test_204f destroyed PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:02:42 (1743508962) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:02:43 (1743508963) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:02:45 (1743508965) oleg314-server: Pool lustre.test_206 created Adding targets to pool oleg314-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg314-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg314-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg314-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg314-server: Pool lustre.test_206 destroyed PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:02:57 (1743508977) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192945 s, 54.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00830496 s, 126 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x1f0789c4 Mirror 3: 0x1f0789c4 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:02:59 (1743508979) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:03:00 (1743508980) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:03:01 (1743508981) Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00884931 s, 0.5 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x970000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2610000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2610000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2610000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1780000 Read this (4 bytes): BBBB 192.168.203.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg314-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:03:14 (1743508994) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00235288 s, 3.5 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:03:15 (1743508995) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100472 s, 104 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa43 lmm_fid: [0x200000405:0xa43:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa43 lmm_fid: [0x200000405:0xa43:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed9:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339613763 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-01 08:03:16.000000000 -0400 Modify: 2025-04-01 08:03:16.000000000 -0400 Change: 2025-04-01 08:03:16.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:03:17 (1743508997) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111532 s, 94.0 MB/s oleg314-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg314-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg314-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg314-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg314-server: O/280000401/d26/efa: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:03:27 (1743509007) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0222618 s, 1.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00396312 s, 1.0 MB/s size after second write -rw-r--r-- 1 root root 4096 Apr 1 08:03 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa48:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 1 08:03 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 697 sec ============ 08:03:29 (1743509009) === sanity-flr: start cleanup 08:03:29 (1743509009) === === sanity-flr: finish cleanup 08:03:31 (1743509011) ===