-----============= acceptance-small: sanity-flr ============----- Mon Jul 14 12:17:40 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg350-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:17:49 (1752509869) === oleg350-client.virtnet: executing check_config_client /mnt/lustre oleg350-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg350-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b43600cd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b43600cd000.idle_timeout=debug disable quota as required oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:18:07 (1752509887) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8204] == sanity-flr test 0a: lfs mirror create with -N option == 12:18:08 (1752509888) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:18:16 (1752509896) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:18:18 (1752509898) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:18:19 (1752509899) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:18:31 (1752509911) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:18:32 (1752509912) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 12:18:34 (1752509914) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b43600cd000.stats=clear osc.lustre-OST0001-osc-ffff9b43600cd000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:18:37 (1752509917) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:18:42 (1752509922) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:18:45 (1752509925) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:18:47 (1752509927) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 12:18:51 (1752509931) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:18:54 (1752509934) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:18:58 (1752509938) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:19:01 (1752509941) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:19:06 (1752509946) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:19:09 (1752509949) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0577355 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369144 s, 28.4 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:19:13 (1752509953) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.207047 s, 91.2 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:19:16 (1752509956) Stopping /mnt/lustre-ost1 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b43600cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b43600cd000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg350-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b43600cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b43600cd000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg350-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (34s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:19:50 (1752509990) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0539836 s, 38.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b43600cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b43600cd000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg350-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:20:06 (1752510006) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b43600cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b43600cd000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg350-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b43600cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b43600cd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg350-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (29s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:20:35 (1752510035) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.0434 s, 50.2 MB/s -rw-r--r-- 1 root root 52428800 Jul 14 12:20 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.304s user 0m0.002s sys 0m0.153s Stopping /mnt/lustre-ost1 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b43600cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b43600cd000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.345s user 0m0.000s sys 0m0.156s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg350-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg350-server oleg350-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b43600cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b43600cd000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.462s user 0m0.001s sys 0m0.170s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg350-server: oleg350-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg350-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b43600cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b43600cd000.ost_server_uuid in FULL state after 0 sec PASS 33b (38s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:21:13 (1752510073) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:21:14 (1752510074) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:21:15 (1752510075) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 12:21:17 (1752510077) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 12:21:20 (1752510080) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.194585 s, 64.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:21:24 (1752510084) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156079 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160691 s, 65.3 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:21:27 (1752510087) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:21:31 (1752510091) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000841148 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000937658 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000631771 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000710121 s, 1.4 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 12:21:34 (1752510094) create test files with size 15746392 .. 85f716adf266902be6e7a1975fa75f51 - 52684a6d9c13b234711826ee086e16cc - d91dc4a6ea990cbaeb1332d473b043bc - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 12:21:42 (1752510102) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:21:47 (1752510107) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:21:49 (1752510109) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411491 s, 25.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137849 s, 76.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 12:21:53 (1752510113) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0395294 s, 53.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.122672 s, 34.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0745705 s, 28.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 12:21:57 (1752510117) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 12:21:58 (1752510118) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:21:59 (1752510119) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143656 s, 73.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 12:22:02 (1752510122) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0973841 s, 32.3 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:22:06 (1752510126) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:22:09 (1752510129) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.109964 s, 95.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:22:27 (1752510147) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b4343ac8000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.110377 s, 95.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg350-client.virtnet /mnt/lustre (opts:) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b43511c9000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:22:32 (1752510152) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:22:47 (1752510167) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:23:02 (1752510182) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 12:23:03 (1752510183) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0884358 s, 35.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0636387 s, 49.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: d069671912ad9877bfc75cb23b7c34cf - mirror 1 checksum: a21958e1bc4dd345ed8c791605dff04a - mirror 2 checksum: d069671912ad9877bfc75cb23b7c34cf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jul 14 12:23:04 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jul 14 12:23:04 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d069671912ad9877bfc75cb23b7c34cf - mirror 1 checksum: a21958e1bc4dd345ed8c791605dff04a - mirror 2 checksum: a21958e1bc4dd345ed8c791605dff04a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jul 14 12:23:04 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jul 14 12:23:04 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:23:06 (1752510186) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137048 s, 76.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:23:09 (1752510189) Failing mds1 on oleg350-server Stopping /mnt/lustre-mds1 (opts:) on oleg350-server 12:23:16 (1752510196) shut down facet: mds1 facet_host: oleg350-server facet_failover_host: oleg350-server Failover mds1 to oleg350-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 12:23:32 (1752510212) targets are mounted 12:23:32 (1752510212) facet_failover done oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019194 s, 54.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:23:40 (1752510220) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0162537 s, 1.3 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00922697 s, 2.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:23:44 (1752510224) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0167138 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0048899 s, 4.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:23:47 (1752510227) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0047063 s, 3.5 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:23:50 (1752510230) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0201405 s, 1.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:23:53 (1752510233) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00261005 s, 1.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00855117 s, 479 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:23:56 (1752510236) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:24:02 (1752510242) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:24:22 (1752510262) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:24:41 (1752510281) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:24:59 (1752510299) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 12:25:01 (1752510301) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55863 mirror split pid 55864 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55863 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55864 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 12:26:04 (1752510364) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 365 skipping zero length punch hole skipping zero length punch hole fallocating to largest ever: 0x31511 000003 1752510365.313838 fallocate 0x02c1ec thru 0x031510 (0x5325 bytes) 000004 1752510365.317140 fallocate 0x029806 thru 0x02b129 (0x1924 bytes) 000005 1752510365.322830 trunc from 0x00fe7a to 0x011a21 (0x1ba8 bytes) 000009 1752510365.326863 mapwrite 0x03310d thru 0x03c5ca (0x94be bytes) 000010 1752510365.360627 trunc from 0x03c5cb to 0x020b38 (0x1ba92 bytes) 000011 1752510365.370848 trunc from 0x020b39 to 0x02be3d (0xb305 bytes) 000013 1752510365.380562 read 0x004ad1 thru 0x00b99e (0x6ece bytes) 000014 1752510365.388213 MIRROR_EXTEND to 2 mirrors 000015 1752510365.566476 punch from 0x00556e to 0x006b2f (0x15c2 bytes) 000017 1752510365.586695 fallocate 0x0108de thru 0x014692 (0x3db5 bytes) 000018 1752510365.590617 trunc from 0x02be3e to 0x00ed63 (0x1d0da bytes) 000019 1752510365.612309 MIRROR_EXTEND to 3 mirrors 000020 1752510365.791896 mapread 0x00e0f4 thru 0x00ed63 (0xc70 bytes) fallocating to largest ever: 0x40000 000021 1752510365.795432 fallocate 0x03768d thru 0x03ffff (0x8973 bytes) 000022 1752510365.814073 MIRROR_RESYNC 3 mirrors 000023 1752510366.011085 trunc from 0x040000 to 0x035bab (0xa454 bytes) 000026 1752510366.047325 mapwrite 0x02d396 thru 0x02f231 (0x1e9c bytes) 000028 1752510366.068468 read 0x01d030 thru 0x0215e4 (0x45b5 bytes) 000030 1752510366.074684 read 0x00466c thru 0x0135a1 (0xef36 bytes) 000031 1752510366.081531 punch from 0x03106c to 0x035bab (0x4b40 bytes) 000032 1752510366.087433 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000034 1752510366.187214 trunc from 0x035bac to 0x00adef (0x2adbc bytes) 000035 1752510366.197313 trunc from 0x0071cb to 0x009740 (0x2576 bytes) 000036 1752510366.201060 write 0x0066bf thru 0x008afa (0x243c bytes) 000037 1752510366.206658 read 0x0079c7 thru 0x00adef (0x3429 bytes) 000038 1752510366.210080 fallocate 0x0096b7 thru 0x01115a (0x7aa4 bytes) 000039 1752510366.213760 punch from 0x001519 to 0x007406 (0x5eee bytes) 000040 1752510366.223472 mapwrite 0x0180f9 thru 0x025a47 (0xd94f bytes) 000041 1752510366.261751 trunc from 0x025a48 to 0x02b7f6 (0x5daf bytes) 000043 1752510366.273206 write 0x03af91 thru 0x03ffff (0x506f bytes) 000044 1752510366.274380 fallocate 0x014569 thru 0x023838 (0xf2d0 bytes) 000046 1752510366.281259 MIRROR_EXTEND to 3 mirrors 000049 1752510366.511120 read 0x00c2b2 thru 0x00e156 (0x1ea5 bytes) 000050 1752510366.514877 trunc from 0x036c8e to 0x03ffff (0x9372 bytes) 000051 1752510366.535444 punch from 0x02f865 to 0x038666 (0x8e02 bytes) 000052 1752510366.538413 mapwrite 0x016cce thru 0x023cd8 (0xd00b bytes) 000053 1752510366.566268 read 0x03b6d5 thru 0x03e7f5 (0x3121 bytes) 000054 1752510366.572009 fallocate 0x00471e thru 0x004b59 (0x43c bytes) 000055 1752510366.575388 punch from 0x00d80b to 0x0171cd (0x99c3 bytes) 000056 1752510366.578861 trunc from 0x03e42f to 0x03ffff (0x1bd1 bytes) 000057 1752510366.582359 punch from 0x03bd21 to 0x03ffff (0x42df bytes) 000059 1752510366.586441 punch from 0x02ac31 to 0x02cba5 (0x1f75 bytes) 000060 1752510366.590255 trunc from 0x00427e to 0x011a2d (0xd7b0 bytes) 000061 1752510366.594191 fallocate 0x02c01a thru 0x03bc98 (0xfc7f bytes) 000062 1752510366.597599 fallocate 0x02bd27 thru 0x0385f4 (0xc8ce bytes) 000064 1752510366.604054 MIRROR_RESYNC 3 mirrors 000065 1752510366.897325 trunc from 0x039083 to 0x03ffff (0x6f7d bytes) 000066 1752510366.916370 write 0x039969 thru 0x03ffff (0x6697 bytes) 000067 1752510366.920295 write 0x00fa0f thru 0x01865a (0x8c4c bytes) 000068 1752510366.929839 MIRROR_EXTEND to 4 mirrors 000069 1752510367.203324 mapread 0x02f0e5 thru 0x039475 (0xa391 bytes) 000070 1752510367.223951 mapwrite 0x019900 thru 0x027f0f (0xe610 bytes) 000071 1752510367.300153 trunc from 0x040000 to 0x01f24a (0x20db5 bytes) 000072 1752510367.313095 punch from 0x00bb76 to 0x00d5ca (0x1a55 bytes) 000074 1752510367.316700 mapread 0x00c03d thru 0x01210d (0x60d1 bytes) 000075 1752510367.321744 mapread 0x012bae thru 0x01f24a (0xc69d bytes) 000076 1752510367.322558 read 0x015ca5 thru 0x01f24a (0x95a6 bytes) 000077 1752510367.323100 mapwrite 0x02a847 thru 0x030aaa (0x6264 bytes) 000078 1752510367.349352 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000079 1752510367.447994 mapwrite 0x02c856 thru 0x0324f8 (0x5ca3 bytes) 000080 1752510367.479171 fallocate 0x00253d thru 0x00fc85 (0xd749 bytes) 000081 1752510367.482789 punch from 0x02da40 to 0x0324f8 (0x4ab9 bytes) 000082 1752510367.486174 write 0x030edd thru 0x036863 (0x5987 bytes) 000083 1752510367.491680 trunc from 0x036864 to 0x03734d (0xaea bytes) 000084 1752510367.505620 MIRROR_EXTEND to 4 mirrors 000085 1752510367.780397 write 0x0288e5 thru 0x02f8ed (0x7009 bytes) 000086 1752510367.803637 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000087 1752510367.905261 trunc from 0x031692 to 0x033b97 (0x2506 bytes) 000088 1752510367.908405 read 0x0260a4 thru 0x0281fd (0x215a bytes) 000089 1752510367.911613 write 0x022404 thru 0x02938e (0x6f8b bytes) 000090 1752510367.921903 MIRROR_EXTEND to 4 mirrors 000091 1752510368.193016 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000092 1752510368.295057 mapread 0x004229 thru 0x012ead (0xec85 bytes) 000093 1752510368.309934 punch from 0x01a598 to 0x01c97a (0x23e3 bytes) 000094 1752510368.317577 mapread 0x007b03 thru 0x013ccb (0xc1c9 bytes) 000095 1752510368.329095 read 0x000cd3 thru 0x002188 (0x14b6 bytes) 000096 1752510368.333437 mapwrite 0x0362a8 thru 0x03ffff (0x9d58 bytes) 000098 1752510368.366887 read 0x02c824 thru 0x034479 (0x7c56 bytes) 000099 1752510368.375087 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000100 1752510368.476845 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000101 1752510368.535790 punch from 0x03e20b to 0x03ffff (0x1df5 bytes) 000102 1752510368.538313 trunc from 0x040000 to 0x03385b (0xc7a4 bytes) 000104 1752510368.547655 trunc from 0x03385c to 0x03a448 (0x6bed bytes) 000105 1752510368.554679 write 0x039ff6 thru 0x03ffff (0x600a bytes) 000106 1752510368.558147 write 0x00475d thru 0x00f8eb (0xb18f bytes) 000107 1752510368.564116 mapread 0x022eee thru 0x0307d8 (0xd8eb bytes) 000108 1752510368.573537 write 0x00ebf1 thru 0x01d921 (0xed31 bytes) 000109 1752510368.574666 fallocate 0x0263ac thru 0x03477a (0xe3cf bytes) 000110 1752510368.577779 trunc from 0x040000 to 0x02ea4a (0x115b5 bytes) 000111 1752510368.587567 write 0x03b7ff thru 0x03ffff (0x4801 bytes) 000112 1752510368.588899 trunc from 0x0258d7 to 0x029a7b (0x41a5 bytes) 000113 1752510368.592213 mapread 0x01b0a1 thru 0x0239a7 (0x8907 bytes) 000114 1752510368.593133 write 0x02203f thru 0x02dc12 (0xbbd4 bytes) 000115 1752510368.596858 MIRROR_EXTEND to 2 mirrors 000116 1752510368.853425 fallocate 0x00959a thru 0x009b94 (0x5fb bytes) 000117 1752510368.874447 MIRROR_RESYNC 2 mirrors 000118 1752510369.161047 read 0x01dcc9 thru 0x02683c (0x8b74 bytes) 000119 1752510369.165648 trunc from 0x0212a6 to 0x02d489 (0xc1e4 bytes) 000120 1752510369.188976 trunc from 0x040000 to 0x03a666 (0x5999 bytes) 000121 1752510369.200299 trunc from 0x005025 to 0x008abd (0x3a99 bytes) 000122 1752510369.204329 mapread 0x035c9f thru 0x035dac (0x10e bytes) 000123 1752510369.208237 trunc from 0x0367cf to 0x03a666 (0x3e98 bytes) 000124 1752510369.211400 write 0x02dd0b thru 0x0357e5 (0x7adb bytes) 000125 1752510369.216102 mapwrite 0x02f75b thru 0x033d41 (0x45e7 bytes) 000126 1752510369.229376 read 0x0319f2 thru 0x03a666 (0x8c75 bytes) 000127 1752510369.235677 read 0x036992 thru 0x037b2b (0x119a bytes) 000128 1752510369.236454 trunc from 0x03a667 to 0x02106d (0x195f9 bytes) 000129 1752510369.245258 mapread 0x01ba9b thru 0x02106d (0x55d3 bytes) 000130 1752510369.246354 trunc from 0x02106e to 0x014665 (0xca08 bytes) 000132 1752510369.255073 write 0x033fd8 thru 0x0354c7 (0x14f0 bytes) 000133 1752510369.258756 MIRROR_RESYNC 2 mirrors 000134 1752510369.456259 trunc from 0x021103 to 0x026e98 (0x5d96 bytes) 000135 1752510369.476026 MIRROR_EXTEND to 3 mirrors 000136 1752510369.677623 mapread 0x02c7cd thru 0x0354c7 (0x8cfb bytes) 000137 1752510369.691288 read 0x027c5e thru 0x0354c7 (0xd86a bytes) 000138 1752510369.691761 trunc from 0x01cc45 to 0x0200cc (0x3488 bytes) 000139 1752510369.712431 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000140 1752510369.809041 punch from 0x005b36 to 0x0068e1 (0xdac bytes) 000141 1752510369.812073 read 0x024cbf thru 0x02a431 (0x5773 bytes) 000142 1752510369.816620 mapwrite 0x039b30 thru 0x03d4d9 (0x39aa bytes) 000143 1752510369.869804 write 0x027715 thru 0x02ccf7 (0x55e3 bytes) 000144 1752510369.873746 write 0x0351b4 thru 0x03ffff (0xae4c bytes) 000145 1752510369.875011 fallocate 0x03b0a9 thru 0x03ffff (0x4f57 bytes) 000146 1752510369.878547 punch from 0x021dc6 to 0x02426d (0x24a8 bytes) 000147 1752510369.881398 mapread 0x00a6aa thru 0x010d23 (0x667a bytes) 000148 1752510369.889151 mapwrite 0x026e73 thru 0x027cff (0xe8d bytes) 000149 1752510369.899499 punch from 0x03603c to 0x036116 (0xdb bytes) 000150 1752510369.902626 mapwrite 0x00cddd thru 0x01ab2d (0xdd51 bytes) 000151 1752510369.919060 punch from 0x029457 to 0x02e27b (0x4e25 bytes) 000152 1752510369.921645 trunc from 0x020fdf to 0x030c34 (0xfc56 bytes) 000153 1752510369.924637 trunc from 0x0188b2 to 0x0228ee (0xa03d bytes) 000155 1752510369.928673 fallocate 0x020f59 thru 0x02b66d (0xa715 bytes) 000156 1752510369.932340 fallocate 0x03c64b thru 0x03ffff (0x39b5 bytes) 000158 1752510369.936979 mapwrite 0x00ac37 thru 0x0116db (0x6aa5 bytes) 000159 1752510369.947063 punch from 0x00360f to 0x0111d4 (0xdbc6 bytes) 000160 1752510369.952960 MIRROR_EXTEND to 3 mirrors 000161 1752510370.205092 read 0x037a87 thru 0x03ffff (0x8579 bytes) 000163 1752510370.211339 write 0x027447 thru 0x02bb83 (0x473d bytes) 000164 1752510370.238601 write 0x026f28 thru 0x02d673 (0x674c bytes) 000165 1752510370.243935 trunc from 0x00c618 to 0x013a9b (0x7484 bytes) 000166 1752510370.247675 fallocate 0x0022c9 thru 0x00c9d4 (0xa70c bytes) 000167 1752510370.252314 write 0x0377f6 thru 0x03e719 (0x6f24 bytes) 000168 1752510370.258070 fallocate 0x00ffc4 thru 0x0165c7 (0x6604 bytes) 000169 1752510370.261520 trunc from 0x040000 to 0x0369b5 (0x964a bytes) 000170 1752510370.271054 read 0x01eadf thru 0x02771d (0x8c3f bytes) 000171 1752510370.276683 read 0x004bbc thru 0x013bb0 (0xeff5 bytes) 000172 1752510370.277300 write 0x006cca thru 0x00716a (0x4a1 bytes) 000173 1752510370.278128 read 0x02d886 thru 0x0369b5 (0x9130 bytes) 000174 1752510370.278738 trunc from 0x0369b6 to 0x01265c (0x24359 bytes) 000176 1752510370.288830 fallocate 0x00db2d thru 0x0168cc (0x8da0 bytes) 000177 1752510370.291792 punch from 0x008ec5 to 0x00903c (0x178 bytes) 000178 1752510370.300168 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000179 1752510370.412450 fallocate 0x033b28 thru 0x0399fc (0x5ed5 bytes) 000180 1752510370.416056 mapread 0x0144f3 thru 0x01d0f8 (0x8c06 bytes) 000181 1752510370.427022 write 0x0294c5 thru 0x033ba4 (0xa6e0 bytes) 000182 1752510370.437350 trunc from 0x02b006 to 0x039997 (0xe992 bytes) 000184 1752510370.451468 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000185 1752510370.525172 MIRROR_EXTEND to 2 mirrors 000187 1752510370.768325 mapwrite 0x011029 thru 0x0112d4 (0x2ac bytes) 000189 1752510370.823902 read 0x02719a thru 0x0340ff (0xcf66 bytes) 000190 1752510370.829785 mapread 0x017c3c thru 0x0274cd (0xf892 bytes) 000191 1752510370.840502 punch from 0x03998e to 0x0399fc (0x6f bytes) 000192 1752510370.843553 trunc from 0x0399fd to 0x00e861 (0x2b19b bytes) 000193 1752510370.866566 trunc from 0x00e862 to 0x003b09 (0xad58 bytes) 000194 1752510370.875007 trunc from 0x002e5d to 0x003b09 (0xcad bytes) 000195 1752510370.878495 read 0x001c19 thru 0x003b09 (0x1ef1 bytes) 000196 1752510370.883949 fallocate 0x00a707 thru 0x01293e (0x8238 bytes) 000198 1752510370.888601 trunc from 0x002460 to 0x003b09 (0x16aa bytes) 000199 1752510370.891546 mapwrite 0x02409a thru 0x026969 (0x28d0 bytes) 000200 1752510370.917082 fallocate 0x008f5a thru 0x00ae89 (0x1f30 bytes) 000201 1752510370.920550 trunc from 0x022a8d to 0x026969 (0x3edd bytes) 000202 1752510370.923520 trunc from 0x02696a to 0x01e073 (0x88f6 bytes) 000203 1752510370.935510 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000204 1752510370.998240 mapwrite 0x027ae2 thru 0x02aa13 (0x2f32 bytes) 000205 1752510371.027501 fallocate 0x032c7d thru 0x03ffff (0xd383 bytes) 000206 1752510371.030620 mapread 0x01690f thru 0x01abd4 (0x42c6 bytes) 000207 1752510371.035124 punch from 0x00b1da to 0x01178c (0x65b3 bytes) 000209 1752510371.038511 trunc from 0x040000 to 0x0120de (0x2df21 bytes) 000210 1752510371.046357 read 0x00a8cf thru 0x0120de (0x7810 bytes) 000211 1752510371.051860 read 0x00f390 thru 0x0120de (0x2d4f bytes) 000212 1752510371.052446 mapread 0x0030bd thru 0x00885c (0x57a0 bytes) 000213 1752510371.057200 mapwrite 0x007490 thru 0x0077f9 (0x36a bytes) 000214 1752510371.066226 mapread 0x00df95 thru 0x0120de (0x414a bytes) 000215 1752510371.066864 read 0x003288 thru 0x00cf84 (0x9cfd bytes) 000216 1752510371.067363 write 0x01df26 thru 0x0253a7 (0x7482 bytes) 000217 1752510371.068590 mapread 0x021452 thru 0x0249ad (0x355c bytes) 000219 1752510371.069831 trunc from 0x0253a8 to 0x0285b4 (0x320d bytes) 000221 1752510371.078777 mapread 0x00c911 thru 0x016449 (0x9b39 bytes) 000222 1752510371.084111 mapread 0x028563 thru 0x0285b4 (0x52 bytes) 000223 1752510371.084906 read 0x010009 thru 0x01866d (0x8665 bytes) 000224 1752510371.085417 trunc from 0x0285b5 to 0x024d7b (0x3839 bytes) 000225 1752510371.092975 trunc from 0x0159b6 to 0x024d7b (0xf3c6 bytes) 000226 1752510371.100416 read 0x00eeca thru 0x01c01b (0xd152 bytes) 000227 1752510371.104658 mapwrite 0x02866c thru 0x028d74 (0x709 bytes) 000228 1752510371.124474 mapwrite 0x01b36f thru 0x02ab91 (0xf823 bytes) 000229 1752510371.158242 fallocate 0x03b555 thru 0x03ffff (0x4aab bytes) 000230 1752510371.164363 MIRROR_EXTEND to 2 mirrors 000231 1752510371.366097 mapread 0x02a794 thru 0x02ab91 (0x3fe bytes) 000232 1752510371.379671 punch from 0x0116d5 to 0x012432 (0xd5e bytes) 000233 1752510371.396439 MIRROR_RESYNC 2 mirrors 000234 1752510371.554170 mapread 0x00cac5 thru 0x011568 (0x4aa4 bytes) 000235 1752510371.564225 mapwrite 0x036c9a thru 0x03ffff (0x9366 bytes) 000236 1752510371.615866 MIRROR_EXTEND to 3 mirrors 000237 1752510371.853264 read 0x027cd0 thru 0x02cf2e (0x525f bytes) 000238 1752510371.858872 trunc from 0x01aa15 to 0x01b69e (0xc8a bytes) 000240 1752510371.882737 punch from 0x011a64 to 0x01d846 (0xbde3 bytes) 000241 1752510371.888668 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000242 1752510371.985823 punch from 0x02b08c to 0x02cf2d (0x1ea2 bytes) 000244 1752510371.988899 trunc from 0x036171 to 0x03ffff (0x9e8f bytes) 000245 1752510371.991969 punch from 0x03291b to 0x03a014 (0x76fa bytes) 000246 1752510371.994758 mapwrite 0x01b284 thru 0x01c3d8 (0x1155 bytes) 000247 1752510372.008952 mapwrite 0x00da55 thru 0x01676a (0x8d16 bytes) 000248 1752510372.035214 MIRROR_EXTEND to 3 mirrors 000249 1752510372.283110 fallocate 0x0007c7 thru 0x005930 (0x516a bytes) 000250 1752510372.308288 write 0x011e03 thru 0x01f497 (0xd695 bytes) 000251 1752510372.323367 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000252 1752510372.432941 punch from 0x028ebd to 0x035d38 (0xce7c bytes) 000253 1752510372.439927 MIRROR_EXTEND to 3 mirrors 000254 1752510372.646635 punch from 0x01dbf6 to 0x02ab81 (0xcf8c bytes) 000255 1752510372.663846 trunc from 0x040000 to 0x00b444 (0x34bbb bytes) 000256 1752510372.687345 trunc from 0x00b445 to 0x01f38e (0x13f4a bytes) 000257 1752510372.694996 trunc from 0x0050d3 to 0x010b6d (0xba9b bytes) 000258 1752510372.698292 mapwrite 0x01ac28 thru 0x026d1f (0xc0f8 bytes) 000259 1752510372.731187 trunc from 0x026d20 to 0x00c4b4 (0x1a86b bytes) 000260 1752510372.738451 fallocate 0x01c0ad thru 0x0203bc (0x4310 bytes) 000261 1752510372.741189 read 0x015935 thru 0x01ed83 (0x944f bytes) 000262 1752510372.745861 read 0x01d720 thru 0x0203bc (0x2c9d bytes) 000263 1752510372.751267 write 0x00e852 thru 0x017f69 (0x9718 bytes) 000264 1752510372.752834 mapread 0x00b34e thru 0x00fd17 (0x49ca bytes) 000266 1752510372.754152 write 0x01813d thru 0x019b6c (0x1a30 bytes) 000267 1752510372.757984 MIRROR_RESYNC 3 mirrors 000268 1752510372.985600 punch from 0x006d2a to 0x00e598 (0x786f bytes) 000269 1752510373.003035 mapread 0x0084dd thru 0x015019 (0xcb3d bytes) 000271 1752510373.018993 read 0x018a9f thru 0x01d40f (0x4971 bytes) 000272 1752510373.022637 write 0x03ba6f thru 0x03ffff (0x4591 bytes) 000273 1752510373.023836 mapwrite 0x005e1f thru 0x005eac (0x8e bytes) 000274 1752510373.034810 mapwrite 0x022c24 thru 0x02e1db (0xb5b8 bytes) 000276 1752510373.057334 write 0x026524 thru 0x02ed7e (0x885b bytes) 000277 1752510373.059613 read 0x00e754 thru 0x01bab4 (0xd361 bytes) 000278 1752510373.061199 punch from 0x0011ba to 0x009258 (0x809f bytes) 000279 1752510373.064310 read 0x0334a2 thru 0x03732d (0x3e8c bytes) 000280 1752510373.069180 mapread 0x037b2d thru 0x03ffff (0x84d3 bytes) 000281 1752510373.070013 write 0x0001e9 thru 0x00aa0b (0xa823 bytes) 000282 1752510373.071214 write 0x0100ce thru 0x01623b (0x616e bytes) 000283 1752510373.072299 write 0x028a47 thru 0x032a7f (0xa039 bytes) 000284 1752510373.073480 mapread 0x0277ff thru 0x02f762 (0x7f64 bytes) 000285 1752510373.074217 mapwrite 0x004521 thru 0x00b563 (0x7043 bytes) 000287 1752510373.088760 fallocate 0x00f846 thru 0x01bc26 (0xc3e1 bytes) 000288 1752510373.094743 punch from 0x03b64b to 0x03d9c2 (0x2378 bytes) 000290 1752510373.099255 trunc from 0x034988 to 0x03c677 (0x7cf0 bytes) 000291 1752510373.102686 mapwrite 0x029d7a thru 0x02f618 (0x589f bytes) 000292 1752510373.113948 read 0x02e80a thru 0x034118 (0x590f bytes) 000293 1752510373.118635 trunc from 0x040000 to 0x03f75a (0x8a5 bytes) 000294 1752510373.138691 trunc from 0x03f75b to 0x01b918 (0x23e42 bytes) 000295 1752510373.147363 fallocate 0x00c8f2 thru 0x01c721 (0xfe30 bytes) 000296 1752510373.150310 trunc from 0x01c722 to 0x012328 (0xa3f9 bytes) 000297 1752510373.157922 mapwrite 0x039a75 thru 0x03adf4 (0x1380 bytes) 000298 1752510373.179140 mapread 0x003add thru 0x0078fb (0x3e1f bytes) 000299 1752510373.182039 MIRROR_RESYNC 3 mirrors 000300 1752510373.446172 mapread 0x01b94a thru 0x01dfb8 (0x266f bytes) 000301 1752510373.456808 mapwrite 0x02f7e0 thru 0x02fc65 (0x486 bytes) 000302 1752510373.506176 punch from 0x03325a to 0x038e81 (0x5c28 bytes) 000303 1752510373.509331 trunc from 0x03adf5 to 0x03fba7 (0x4db3 bytes) 000304 1752510373.528616 mapwrite 0x01df5c thru 0x0249ef (0x6a94 bytes) 000305 1752510373.548876 mapwrite 0x031936 thru 0x03bf4e (0xa619 bytes) 000306 1752510373.564662 read 0x0003c6 thru 0x00930b (0x8f46 bytes) 000307 1752510373.569225 mapread 0x0074e4 thru 0x00f65b (0x8178 bytes) 000308 1752510373.574267 fallocate 0x0005d4 thru 0x00ca65 (0xc492 bytes) 000309 1752510373.577126 write 0x034745 thru 0x03d182 (0x8a3e bytes) 000310 1752510373.580493 trunc from 0x0125a5 to 0x0219f0 (0xf44c bytes) 000311 1752510373.583877 mapwrite 0x00b589 thru 0x015364 (0x9ddc bytes) 000313 1752510373.603285 mapread 0x02b0b7 thru 0x02d2ca (0x2214 bytes) 000314 1752510373.604184 mapread 0x00b4d6 thru 0x00d3c1 (0x1eec bytes) 000315 1752510373.604939 write 0x032cd3 thru 0x036b13 (0x3e41 bytes) 000316 1752510373.605934 punch from 0x0331dd to 0x03ec58 (0xba7c bytes) 000317 1752510373.614393 mapread 0x0262cf thru 0x0303f4 (0xa126 bytes) 000318 1752510373.619840 read 0x01d425 thru 0x0298ac (0xc488 bytes) 000319 1752510373.625693 punch from 0x003136 to 0x00f5ad (0xc478 bytes) 000320 1752510373.629172 trunc from 0x03aa7c to 0x03fba7 (0x512c bytes) 000321 1752510373.632196 trunc from 0x03b275 to 0x03fba7 (0x4933 bytes) 000322 1752510373.635065 read 0x01eb34 thru 0x023065 (0x4532 bytes) 000323 1752510373.635661 write 0x02075f thru 0x0212ca (0xb6c bytes) 000324 1752510373.636510 mapread 0x035ccf thru 0x037a39 (0x1d6b bytes) 000326 1752510373.640458 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000327 1752510373.748822 mapread 0x02738e thru 0x033530 (0xc1a3 bytes) 000328 1752510373.764514 mapread 0x03121f thru 0x037c5e (0x6a40 bytes) 000329 1752510373.765312 read 0x038c81 thru 0x03ccd4 (0x4054 bytes) 000330 1752510373.769177 read 0x01e1ac thru 0x02b6d5 (0xd52a bytes) 000331 1752510373.780772 MIRROR_RESYNC 2 mirrors 000332 1752510374.091950 trunc from 0x03fba8 to 0x016a7f (0x29128 bytes) 000333 1752510374.111558 trunc from 0x016a80 to 0x02b5a3 (0x14b24 bytes) 000334 1752510374.122392 mapread 0x01fbe2 thru 0x02738c (0x77ab bytes) 000336 1752510374.136473 mapread 0x01e077 thru 0x0269f6 (0x8980 bytes) 000337 1752510374.137240 trunc from 0x01eed8 to 0x02831a (0x9443 bytes) 000338 1752510374.140396 write 0x0333ec thru 0x0379d4 (0x45e9 bytes) 000339 1752510374.141497 mapread 0x016d29 thru 0x01b923 (0x4bfb bytes) 000340 1752510374.146577 trunc from 0x011f2e to 0x015474 (0x3547 bytes) 000341 1752510374.151133 read 0x02e1c1 thru 0x0379d4 (0x9814 bytes) 000342 1752510374.157591 read 0x01bfec thru 0x027791 (0xb7a6 bytes) 000343 1752510374.164774 punch from 0x00c073 to 0x01275a (0x66e8 bytes) 000344 1752510374.168956 trunc from 0x0379d5 to 0x03f22b (0x7857 bytes) 000345 1752510374.178602 trunc from 0x032257 to 0x03dfa8 (0xbd52 bytes) 000346 1752510374.194445 write 0x023391 thru 0x028ff7 (0x5c67 bytes) 000347 1752510374.195510 mapwrite 0x0335d2 thru 0x03ffff (0xca2e bytes) 000348 1752510374.234618 mapwrite 0x03c1e3 thru 0x03ffff (0x3e1d bytes) 000349 1752510374.242498 read 0x03150a thru 0x03dc3b (0xc732 bytes) 000350 1752510374.247439 mapread 0x004af8 thru 0x004f15 (0x41e bytes) 000351 1752510374.248258 trunc from 0x040000 to 0x0380d4 (0x7f2b bytes) 000352 1752510374.256025 trunc from 0x0380d5 to 0x0354b2 (0x2c22 bytes) 000353 1752510374.263342 write 0x0062e9 thru 0x011678 (0xb390 bytes) 000354 1752510374.264645 write 0x02a6a4 thru 0x02b948 (0x12a5 bytes) 000357 1752510374.266241 read 0x0343ec thru 0x0354b2 (0x10c7 bytes) 000358 1752510374.266744 trunc from 0x0354b3 to 0x01adea (0x1a6c8 bytes) 000359 1752510374.275074 trunc from 0x01449c to 0x01599c (0x1501 bytes) 000361 1752510374.279168 trunc from 0x015b37 to 0x0170bf (0x1589 bytes) 000362 1752510374.282355 mapread 0x000af0 thru 0x00f273 (0xe784 bytes) 000363 1752510374.283217 mapread 0x019204 thru 0x01adea (0x1be7 bytes) 000364 1752510374.284023 trunc from 0x01adeb to 0x014d13 (0x60d7 bytes) 000365 1752510374.292951 read 0x012615 thru 0x014d13 (0x26ff bytes) 000366 1752510374.299327 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000367 1752510374.366759 trunc from 0x014d14 to 0x036eee (0x221db bytes) 000368 1752510374.380177 MIRROR_EXTEND to 2 mirrors 000369 1752510374.632484 write 0x0316a4 thru 0x03362f (0x1f8c bytes) 000370 1752510374.656951 punch from 0x022641 to 0x024203 (0x1bc3 bytes) 000371 1752510374.659856 punch from 0x012f55 to 0x01d136 (0xa1e2 bytes) 000372 1752510374.662764 trunc from 0x036eef to 0x016e03 (0x200eb bytes) 000373 1752510374.680056 read 0x00f2d0 thru 0x016e03 (0x7b34 bytes) 000374 1752510374.684806 read 0x01549b thru 0x016e03 (0x1969 bytes) 000375 1752510374.685346 read 0x0104d5 thru 0x01358f (0x30bb bytes) 000376 1752510374.685864 write 0x0167a9 thru 0x019c33 (0x348b bytes) 000377 1752510374.686814 fallocate 0x00e150 thru 0x013489 (0x533a bytes) 000378 1752510374.689874 read 0x0184b0 thru 0x0188bf (0x410 bytes) 000380 1752510374.690768 fallocate 0x002d12 thru 0x0071a5 (0x4494 bytes) 000381 1752510374.693821 mapwrite 0x0229a7 thru 0x027109 (0x4763 bytes) 000382 1752510374.721268 punch from 0x0118e8 to 0x014ee7 (0x3600 bytes) 000383 1752510374.724831 mapwrite 0x00133b thru 0x003765 (0x242b bytes) 000384 1752510374.733522 read 0x024da1 thru 0x027109 (0x2369 bytes) 000385 1752510374.734127 read 0x012a01 thru 0x0164b1 (0x3ab1 bytes) 000386 1752510374.737669 trunc from 0x02710a to 0x0278b2 (0x7a9 bytes) 000387 1752510374.746176 trunc from 0x0278b3 to 0x02a0fe (0x284c bytes) 000388 1752510374.754422 trunc from 0x01e283 to 0x02a0fe (0xbe7c bytes) 000389 1752510374.758208 write 0x009dd4 thru 0x00da5f (0x3c8c bytes) 000390 1752510374.759155 trunc from 0x02a0ff to 0x02c4bb (0x23bd bytes) 000391 1752510374.766631 punch from 0x0186a3 to 0x023b5c (0xb4ba bytes) 000392 1752510374.769494 trunc from 0x02c4bc to 0x0256d4 (0x6de7 bytes) 000393 1752510374.776292 punch from 0x00e2dd to 0x019898 (0xb5bc bytes) 000394 1752510374.779446 trunc from 0x01da7b to 0x01e84f (0xdd5 bytes) 000395 1752510374.782429 trunc from 0x0256d5 to 0x026aed (0x1419 bytes) 000396 1752510374.790628 read 0x022bec thru 0x026aed (0x3f02 bytes) 000397 1752510374.795672 mapwrite 0x00e30a thru 0x018941 (0xa638 bytes) 000398 1752510374.813757 read 0x02459d thru 0x026aed (0x2551 bytes) 000399 1752510374.814271 trunc from 0x026aee to 0x03dd5b (0x1726e bytes) 000400 1752510374.823492 punch from 0x02967f to 0x0310f6 (0x7a78 bytes) 000401 1752510374.829999 MIRROR_EXTEND to 3 mirrors 000402 1752510375.107335 write 0x035a2e thru 0x03ffff (0xa5d2 bytes) 000403 1752510375.132163 fallocate 0x01eba2 thru 0x02c3cb (0xd82a bytes) 000404 1752510375.135686 read 0x01310b thru 0x019665 (0x655b bytes) 000405 1752510375.140416 write 0x00c3f6 thru 0x012653 (0x625e bytes) 000406 1752510375.147516 read 0x02e600 thru 0x033cde (0x56df bytes) 000407 1752510375.153096 trunc from 0x040000 to 0x0093cd (0x36c32 bytes) 000409 1752510375.167258 MIRROR_EXTEND to 4 mirrors 000410 1752510375.396118 trunc from 0x0093ce to 0x03246a (0x2909d bytes) 000411 1752510375.437865 trunc from 0x03246b to 0x02b5c2 (0x6ea8 bytes) 000413 1752510375.452654 write 0x03c6f3 thru 0x03ffff (0x390d bytes) 000414 1752510375.453697 write 0x00a909 thru 0x01594c (0xb044 bytes) 000415 1752510375.462875 mapwrite 0x02a68e thru 0x02ce80 (0x27f3 bytes) 000416 1752510375.481222 trunc from 0x020c2d to 0x0284a9 (0x787d bytes) 000417 1752510375.484552 punch from 0x018a16 to 0x0239ce (0xafb9 bytes) 000418 1752510375.488075 fallocate 0x01b201 thru 0x01d873 (0x2673 bytes) 000419 1752510375.491566 trunc from 0x040000 to 0x005020 (0x3afdf bytes) 000420 1752510375.501769 read 0x001db8 thru 0x005020 (0x3269 bytes) 000421 1752510375.508829 MIRROR_RESYNC 4 mirrors 000422 1752510375.779978 trunc from 0x00486b to 0x005020 (0x7b6 bytes) 000423 1752510375.799957 punch from 0x00486c to 0x004c28 (0x3bd bytes) 000424 1752510375.803358 trunc from 0x00025f to 0x000fb5 (0xd57 bytes) 000425 1752510375.807700 punch from 0x000911 to 0x005020 (0x4710 bytes) 000426 1752510375.812407 mapread 0x004b3a thru 0x005020 (0x4e7 bytes) 000427 1752510375.820264 read 0x004b3b thru 0x005020 (0x4e6 bytes) 000428 1752510375.821001 mapread 0x0020ab thru 0x005020 (0x2f76 bytes) 000429 1752510375.825353 mapread 0x0038b6 thru 0x005020 (0x176b bytes) 000430 1752510375.827678 trunc from 0x005021 to 0x0258af (0x2088f bytes) 000431 1752510375.853380 mapwrite 0x00ddc0 thru 0x01ba39 (0xdc7a bytes) 000432 1752510375.877321 fallocate 0x038496 thru 0x03f6ed (0x7258 bytes) 000433 1752510375.880826 punch from 0x011acc to 0x020a03 (0xef38 bytes) 000434 1752510375.884352 write 0x012f5f thru 0x0142bf (0x1361 bytes) 000435 1752510375.889764 punch from 0x0088f6 to 0x0162ab (0xd9b6 bytes) 000436 1752510375.896397 fallocate 0x02a20d thru 0x02bf32 (0x1d26 bytes) 000437 1752510375.900082 trunc from 0x0132ea to 0x023044 (0xfd5b bytes) 000440 1752510375.904555 trunc from 0x017f56 to 0x01fd79 (0x7e24 bytes) 000441 1752510375.907882 punch from 0x0252d7 to 0x0258af (0x5d9 bytes) 000442 1752510375.912775 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000443 1752510376.027606 fallocate 0x015759 thru 0x01f710 (0x9fb8 bytes) 000444 1752510376.031089 read 0x023016 thru 0x0258af (0x289a bytes) 000445 1752510376.034514 write 0x01dccb thru 0x0295ac (0xb8e2 bytes) 000447 1752510376.043310 MIRROR_EXTEND to 4 mirrors 000448 1752510376.267681 trunc from 0x01c4d9 to 0x01ec2d (0x2755 bytes) 000449 1752510376.284857 mapwrite 0x00c86a thru 0x019014 (0xc7ab bytes) 000450 1752510376.312972 fallocate 0x026b7c thru 0x0291f0 (0x2675 bytes) 000451 1752510376.317348 write 0x03f32d thru 0x03ffff (0xcd3 bytes) 000452 1752510376.321190 punch from 0x026705 to 0x02b471 (0x4d6d bytes) 000453 1752510376.324763 punch from 0x039f7c to 0x03ffff (0x6084 bytes) 000454 1752510376.333473 mapread 0x01b9ad thru 0x02a99e (0xeff2 bytes) 000455 1752510376.344402 mapread 0x03cae6 thru 0x03e1c4 (0x16df bytes) 000456 1752510376.351200 write 0x009cf4 thru 0x00ae92 (0x119f bytes) 000457 1752510376.352049 mapwrite 0x012ec0 thru 0x016611 (0x3752 bytes) 000458 1752510376.362174 mapwrite 0x01b4ba thru 0x0229eb (0x7532 bytes) 000459 1752510376.373528 punch from 0x0187f7 to 0x020269 (0x7a73 bytes) 000460 1752510376.378431 trunc from 0x040000 to 0x009373 (0x36c8c bytes) 000461 1752510376.399842 trunc from 0x009374 to 0x018cd7 (0xf964 bytes) 000462 1752510376.408528 write 0x03b61f thru 0x03d6c5 (0x20a7 bytes) 000463 1752510376.412190 MIRROR_EXTEND to 5 mirrors 000464 1752510376.650170 write 0x03c964 thru 0x03ffff (0x369c bytes) 000465 1752510376.672199 mapwrite 0x03fa7d thru 0x03ffff (0x583 bytes) 000466 1752510376.683407 trunc from 0x036191 to 0x03ffff (0x9e6f bytes) 000467 1752510376.686650 mapread 0x02a510 thru 0x035d32 (0xb823 bytes) 000468 1752510376.703897 write 0x034322 thru 0x0346bd (0x39c bytes) 000470 1752510376.705164 mapwrite 0x001a32 thru 0x003f61 (0x2530 bytes) 000471 1752510376.721361 punch from 0x029a47 to 0x030990 (0x6f4a bytes) 000472 1752510376.726142 trunc from 0x03db85 to 0x03ffff (0x247b bytes) 000473 1752510376.729310 mapread 0x02dd1a thru 0x031ccc (0x3fb3 bytes) 000474 1752510376.735331 read 0x032039 thru 0x03bdf9 (0x9dc1 bytes) 000475 1752510376.736025 trunc from 0x02e40f to 0x035baa (0x779c bytes) 000476 1752510376.740569 read 0x019d52 thru 0x026888 (0xcb37 bytes) 000477 1752510376.747092 fallocate 0x01f29e thru 0x027fbb (0x8d1e bytes) 000478 1752510376.751302 fallocate 0x02de06 thru 0x0360f6 (0x82f1 bytes) 000479 1752510376.755735 read 0x01382a thru 0x02145b (0xdc32 bytes) 000480 1752510376.756321 trunc from 0x040000 to 0x02643b (0x19bc4 bytes) 000481 1752510376.780284 write 0x02828e thru 0x03189c (0x960f bytes) 000482 1752510376.781649 punch from 0x01839e to 0x020762 (0x83c5 bytes) 000483 1752510376.790946 MIRROR_RESYNC 5 mirrors 000484 1752510377.297861 read 0x02a0fe thru 0x03189c (0x779f bytes) 000485 1752510377.305452 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000486 1752510377.444957 trunc from 0x01429f to 0x01f3b5 (0xb117 bytes) 000487 1752510377.463570 trunc from 0x03189d to 0x02aede (0x69be bytes) 000488 1752510377.475500 write 0x021374 thru 0x023e52 (0x2adf bytes) 000489 1752510377.482931 write 0x03bd39 thru 0x03ffff (0x42c7 bytes) 000490 1752510377.484054 trunc from 0x01612c to 0x01ce6a (0x6d3f bytes) 000491 1752510377.488371 read 0x03c980 thru 0x03f3bf (0x2a40 bytes) 000492 1752510377.491030 MIRROR_RESYNC 4 mirrors 000493 1752510378.023667 trunc from 0x040000 to 0x03e58c (0x1a73 bytes) 000494 1752510378.052170 trunc from 0x02595b to 0x03310e (0xd7b4 bytes) 000495 1752510378.055069 fallocate 0x01276a thru 0x0217c2 (0xf059 bytes) 000496 1752510378.060639 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000497 1752510378.158497 fallocate 0x01660e thru 0x01d04c (0x6a3f bytes) 000498 1752510378.161677 mapwrite 0x0354bc thru 0x039faf (0x4af4 bytes) 000499 1752510378.186970 trunc from 0x02165e to 0x02af12 (0x98b5 bytes) 000500 1752510378.191058 trunc from 0x03e58d to 0x018471 (0x2611b bytes) 000502 1752510378.207571 read 0x015ccd thru 0x017163 (0x1497 bytes) 000503 1752510378.211558 fallocate 0x032d7e thru 0x03453a (0x17bd bytes) 000504 1752510378.216247 punch from 0x002e87 to 0x00394d (0xac7 bytes) 000505 1752510378.222308 mapread 0x01ca16 thru 0x01f103 (0x26ee bytes) 000506 1752510378.228012 read 0x01f1c0 thru 0x027765 (0x85a6 bytes) 000507 1752510378.236677 mapwrite 0x035be6 thru 0x03cfcb (0x73e6 bytes) 000508 1752510378.267272 fallocate 0x01a2d3 thru 0x020ce2 (0x6a10 bytes) 000509 1752510378.271375 punch from 0x01c94e to 0x01f97c (0x302f bytes) 000510 1752510378.275986 trunc from 0x03cfcc to 0x033fb3 (0x9018 bytes) 000511 1752510378.285839 mapwrite 0x016d7c thru 0x01821a (0x149f bytes) 000512 1752510378.300552 fallocate 0x011ee6 thru 0x01afcf (0x90ea bytes) 000514 1752510378.304835 read 0x0123d8 thru 0x012dde (0xa07 bytes) 000515 1752510378.305388 write 0x008b8e thru 0x012dae (0xa221 bytes) 000516 1752510378.306548 read 0x016422 thru 0x0245ec (0xe1cb bytes) 000517 1752510378.311308 trunc from 0x033fb4 to 0x0291de (0xadd5 bytes) 000518 1752510378.321997 mapread 0x024b5e thru 0x0291de (0x4681 bytes) 000519 1752510378.322987 mapwrite 0x00e63a thru 0x01cfe1 (0xe9a8 bytes) 000520 1752510378.340922 fallocate 0x008717 thru 0x00f600 (0x6eea bytes) 000521 1752510378.344127 read 0x00a9bd thru 0x00d0bb (0x26ff bytes) 000523 1752510378.345084 mapread 0x025328 thru 0x0291de (0x3eb7 bytes) 000524 1752510378.345910 trunc from 0x00a77f to 0x019a23 (0xf2a5 bytes) 000525 1752510378.351635 fallocate 0x03a649 thru 0x03ffff (0x59b7 bytes) 000526 1752510378.355471 write 0x00dd7f thru 0x01ca5e (0xece0 bytes) 000527 1752510378.362839 punch from 0x00757d to 0x01603c (0xeac0 bytes) 000528 1752510378.379018 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000529 1752510378.485554 MIRROR_EXTEND to 3 mirrors 000530 1752510378.745103 fallocate 0x03438c thru 0x038b85 (0x47fa bytes) 000531 1752510378.767684 write 0x03678d thru 0x03df19 (0x778d bytes) 000532 1752510378.771799 fallocate 0x01d67a thru 0x026ee1 (0x9868 bytes) 000533 1752510378.775283 write 0x029fe5 thru 0x02f26f (0x528b bytes) 000534 1752510378.781317 mapwrite 0x02fabd thru 0x03a599 (0xaadd bytes) 000535 1752510378.806046 read 0x039566 thru 0x03dac0 (0x455b bytes) 000536 1752510378.806805 write 0x03b35c thru 0x03b610 (0x2b5 bytes) 000537 1752510378.807692 trunc from 0x03df1a to 0x005a3b (0x384de bytes) 000538 1752510378.838691 write 0x01b858 thru 0x0228cb (0x7074 bytes) 000539 1752510378.840223 mapwrite 0x032453 thru 0x03ffff (0xdbad bytes) 000540 1752510378.902857 write 0x001835 thru 0x00edf4 (0xd5c0 bytes) 000541 1752510378.917848 MIRROR_RESYNC 3 mirrors 000542 1752510379.336891 mapread 0x03e714 thru 0x03ffff (0x18ec bytes) 000543 1752510379.344195 punch from 0x00e3f7 to 0x011d83 (0x398d bytes) 000544 1752510379.370127 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000545 1752510379.495224 MIRROR_EXTEND to 3 mirrors 000546 1752510379.801896 punch from 0x01f7a8 to 0x02d695 (0xdeee bytes) 000547 1752510379.823838 trunc from 0x032256 to 0x03e10d (0xbeb8 bytes) 000548 1752510379.827329 read 0x018c9a thru 0x024e33 (0xc19a bytes) 000549 1752510379.834934 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000550 1752510379.938551 trunc from 0x010f4f to 0x01f51c (0xe5ce bytes) 000551 1752510379.942554 read 0x0142d7 thru 0x01ce14 (0x8b3e bytes) 000553 1752510379.948805 write 0x00293f thru 0x00bc75 (0x9337 bytes) 000554 1752510379.959118 mapwrite 0x015443 thru 0x01efe2 (0x9ba0 bytes) 000555 1752510379.979776 fallocate 0x0311cc thru 0x03ffff (0xee34 bytes) 000556 1752510379.983146 mapread 0x035d4b thru 0x03ffff (0xa2b5 bytes) 000557 1752510379.991919 trunc from 0x00dd4c to 0x0187b3 (0xaa68 bytes) 000558 1752510379.996195 fallocate 0x0277a1 thru 0x02c024 (0x4884 bytes) 000559 1752510379.999857 mapread 0x02de27 thru 0x038dda (0xafb4 bytes) 000560 1752510380.004769 mapwrite 0x031a6e thru 0x03f092 (0xd625 bytes) 000561 1752510380.018019 trunc from 0x040000 to 0x01ead2 (0x2152d bytes) 000562 1752510380.028025 read 0x00a449 thru 0x0109a6 (0x655e bytes) 000563 1752510380.034647 read 0x015148 thru 0x01ead2 (0x998b bytes) 000565 1752510380.035792 mapwrite 0x015cfa thru 0x015f4e (0x255 bytes) 000566 1752510380.051436 MIRROR_EXTEND to 3 mirrors 000567 1752510380.288001 mapwrite 0x032876 thru 0x03467b (0x1e06 bytes) 000568 1752510380.361461 read 0x00445c thru 0x013568 (0xf10d bytes) 000569 1752510380.369189 punch from 0x02ab81 to 0x03467b (0x9afb bytes) 000570 1752510380.373646 mapwrite 0x008ed7 thru 0x0143e8 (0xb512 bytes) 000571 1752510380.395048 trunc from 0x03467c to 0x029954 (0xad27 bytes) 000573 1752510380.411062 mapwrite 0x015bda thru 0x020bf2 (0xb019 bytes) 000574 1752510380.434374 punch from 0x01c3f4 to 0x026009 (0x9c16 bytes) 000575 1752510380.443767 MIRROR_EXTEND to 4 mirrors 000576 1752510380.750834 MIRROR_RESYNC 4 mirrors 000577 1752510381.027178 mapread 0x00dbc1 thru 0x0185d6 (0xaa16 bytes) 000578 1752510381.050792 mapread 0x023dd0 thru 0x029954 (0x5b85 bytes) 000579 1752510381.057453 trunc from 0x029955 to 0x002684 (0x272d0 bytes) 000580 1752510381.100295 write 0x02d68e thru 0x0307a7 (0x311a bytes) 000581 1752510381.101802 mapread 0x003046 thru 0x00e78f (0xb74a bytes) 000582 1752510381.108290 trunc from 0x019119 to 0x01d0bb (0x3fa3 bytes) 000583 1752510381.114966 mapwrite 0x00aa20 thru 0x01a067 (0xf648 bytes) 000584 1752510381.142733 mapread 0x019ee5 thru 0x01b305 (0x1421 bytes) 000585 1752510381.143571 fallocate 0x00f411 thru 0x018051 (0x8c41 bytes) 000586 1752510381.147087 mapwrite 0x02d137 thru 0x02d968 (0x832 bytes) 000587 1752510381.154500 punch from 0x001ca0 to 0x00e42e (0xc78f bytes) 000588 1752510381.158501 mapread 0x024ba9 thru 0x02bdcf (0x7227 bytes) 000589 1752510381.166090 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000590 1752510381.273696 fallocate 0x0380b8 thru 0x039e0f (0x1d58 bytes) 000591 1752510381.281312 MIRROR_RESYNC 3 mirrors 000592 1752510381.644802 punch from 0x01ddb1 to 0x02209c (0x42ec bytes) 000593 1752510381.666618 punch from 0x0195bc to 0x0229a2 (0x93e7 bytes) 000594 1752510381.669389 trunc from 0x00609e to 0x007729 (0x168c bytes) 000595 1752510381.674780 MIRROR_RESYNC 3 mirrors 000596 1752510382.002176 write 0x00e6b5 thru 0x01994f (0xb29b bytes) 000597 1752510382.032964 punch from 0x004e86 to 0x00c98b (0x7b06 bytes) 000598 1752510382.038172 read 0x01107d thru 0x0181ae (0x7132 bytes) 000599 1752510382.038925 trunc from 0x0064f3 to 0x012baf (0xc6bd bytes) 000600 1752510382.050239 punch from 0x00f508 to 0x014214 (0x4d0d bytes) 000601 1752510382.055045 trunc from 0x02ead4 to 0x0307a7 (0x1cd4 bytes) 000602 1752510382.059152 fallocate 0x01e6a0 thru 0x02012e (0x1a8f bytes) 000603 1752510382.065118 punch from 0x030311 to 0x0307a7 (0x497 bytes) 000605 1752510382.077643 MIRROR_EXTEND to 4 mirrors 000607 1752510382.376520 read 0x030594 thru 0x0307a7 (0x214 bytes) 000608 1752510382.387444 mapwrite 0x030bd8 thru 0x038a61 (0x7e8a bytes) 000610 1752510382.447193 read 0x026393 thru 0x02c4bd (0x612b bytes) 000611 1752510382.453687 mapwrite 0x02d602 thru 0x031b3c (0x453b bytes) 000612 1752510382.471645 punch from 0x00ea3a to 0x0194e1 (0xaaa8 bytes) 000613 1752510382.475337 write 0x028351 thru 0x03222b (0x9edb bytes) 000615 1752510382.477217 trunc from 0x02d28e to 0x038a61 (0xb7d4 bytes) 000616 1752510382.485684 punch from 0x01b9f5 to 0x023796 (0x7da2 bytes) 000617 1752510382.490361 punch from 0x03863c to 0x038a61 (0x426 bytes) 000619 1752510382.494173 fallocate 0x02baa0 thru 0x03987a (0xdddb bytes) 000620 1752510382.497494 write 0x02d1c0 thru 0x02f810 (0x2651 bytes) 000621 1752510382.505837 mapwrite 0x02955a thru 0x02da43 (0x44ea bytes) 000622 1752510382.521225 mapwrite 0x00c88b thru 0x011e2c (0x55a2 bytes) 000623 1752510382.548394 trunc from 0x038a62 to 0x011a6e (0x26ff3 bytes) 000624 1752510382.559102 mapwrite 0x03c7ac thru 0x03cc89 (0x4de bytes) 000625 1752510382.582082 read 0x030e2e thru 0x03cc89 (0xbe5c bytes) 000626 1752510382.593692 MIRROR_EXTEND to 5 mirrors 000627 1752510382.853116 trunc from 0x02690e to 0x030190 (0x9883 bytes) 000628 1752510382.871811 fallocate 0x010c1f thru 0x0174f0 (0x68d2 bytes) 000629 1752510382.875313 mapwrite 0x00dec6 thru 0x0154e9 (0x7624 bytes) 000630 1752510382.905137 trunc from 0x03a9f7 to 0x03c31e (0x1928 bytes) 000631 1752510382.908551 fallocate 0x03a3e7 thru 0x03ffff (0x5c19 bytes) 000633 1752510382.913013 mapwrite 0x01b099 thru 0x02a761 (0xf6c9 bytes) 000634 1752510382.939807 fallocate 0x010d97 thru 0x01d8a7 (0xcb11 bytes) 000636 1752510382.944839 trunc from 0x034cb5 to 0x03b09c (0x63e8 bytes) 000637 1752510382.949315 read 0x029afa thru 0x02e37c (0x4883 bytes) 000638 1752510382.956543 read 0x00c001 thru 0x019f88 (0xdf88 bytes) 000639 1752510382.957196 trunc from 0x03cc8a to 0x0280e7 (0x14ba2 bytes) 000640 1752510382.982172 mapwrite 0x00ede0 thru 0x018ff0 (0xa211 bytes) 000642 1752510383.002375 punch from 0x011d19 to 0x01414d (0x2435 bytes) 000643 1752510383.008358 mapread 0x00a7aa thru 0x00f2d8 (0x4b2f bytes) 000644 1752510383.010223 trunc from 0x0280e8 to 0x006145 (0x21fa2 bytes) 000645 1752510383.027372 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000646 1752510383.149287 punch from 0x002eeb to 0x006145 (0x325b bytes) 000647 1752510383.152445 punch from 0x0040e8 to 0x0058ee (0x1807 bytes) 000648 1752510383.155557 punch from 0x002cc3 to 0x006077 (0x33b5 bytes) 000649 1752510383.160753 fallocate 0x033d20 thru 0x03ffff (0xc2e0 bytes) 000650 1752510383.166559 trunc from 0x006146 to 0x023966 (0x1d821 bytes) 000651 1752510383.186302 punch from 0x00735d to 0x00edfe (0x7aa2 bytes) 000652 1752510383.190383 mapread 0x010ade thru 0x016811 (0x5d34 bytes) 000653 1752510383.206118 MIRROR_EXTEND to 5 mirrors 000654 1752510383.511537 trunc from 0x01f50b to 0x023966 (0x445c bytes) 000655 1752510383.532142 trunc from 0x023967 to 0x0003df (0x23587 bytes) 000656 1752510383.557570 trunc from 0x0003e0 to 0x006de0 (0x6a01 bytes) 000657 1752510383.570514 trunc from 0x005b22 to 0x006de0 (0x12bf bytes) 000658 1752510383.574840 trunc from 0x005f31 to 0x006de0 (0xeb0 bytes) 000659 1752510383.583873 write 0x00f2b9 thru 0x01471e (0x5466 bytes) 000660 1752510383.585136 mapread 0x00cd67 thru 0x00e29f (0x1539 bytes) 000661 1752510383.597411 trunc from 0x01471f to 0x02055c (0xbe3e bytes) 000662 1752510383.607725 mapread 0x0162db thru 0x018bde (0x2904 bytes) 000663 1752510383.614244 write 0x03ba3f thru 0x03ffff (0x45c1 bytes) 000664 1752510383.615525 punch from 0x003233 to 0x00487a (0x1648 bytes) 000665 1752510383.623647 MIRROR_EXTEND to 6 mirrors 000666 1752510383.925303 trunc from 0x023c1d to 0x02d139 (0x951d bytes) 000667 1752510383.955964 mapwrite 0x039779 thru 0x03ffff (0x6887 bytes) 000669 1752510383.993978 read 0x0354a4 thru 0x03ffff (0xab5c bytes) 000670 1752510383.994483 trunc from 0x00d7b1 to 0x0162dc (0x8b2c bytes) 000671 1752510383.998278 fallocate 0x02e837 thru 0x03aaf0 (0xc2ba bytes) 000673 1752510384.002124 punch from 0x02c508 to 0x038b37 (0xc630 bytes) 000674 1752510384.006991 trunc from 0x022e57 to 0x032085 (0xf22f bytes) 000675 1752510384.010876 trunc from 0x040000 to 0x02ade8 (0x15217 bytes) 000676 1752510384.035926 fallocate 0x03a5c2 thru 0x03d667 (0x30a6 bytes) 000677 1752510384.038568 punch from 0x02a0c7 to 0x03277c (0x86b6 bytes) 000678 1752510384.041311 mapwrite 0x0210cc thru 0x02862f (0x7564 bytes) 000679 1752510384.065802 punch from 0x006d1d to 0x0168bc (0xfba0 bytes) 000680 1752510384.070125 trunc from 0x02f6b8 to 0x037fad (0x88f6 bytes) 000681 1752510384.073128 trunc from 0x03d668 to 0x00b781 (0x31ee6 bytes) 000683 1752510384.084456 read 0x00b234 thru 0x00b781 (0x54e bytes) 000684 1752510384.089264 trunc from 0x00b782 to 0x014b88 (0x9407 bytes) 000685 1752510384.102247 MIRROR_RESYNC 6 mirrors 000686 1752510384.524983 MIRROR_EXTEND to 7 mirrors 000687 1752510384.837945 mapread 0x002c5f thru 0x006338 (0x36da bytes) 000688 1752510384.861034 trunc from 0x014b89 to 0x00c199 (0x89ef bytes) 000689 1752510384.905665 trunc from 0x00957e to 0x00c199 (0x2c1c bytes) 000690 1752510384.909262 punch from 0x0037cc to 0x0099e3 (0x6218 bytes) 000691 1752510384.912646 trunc from 0x0076b2 to 0x00c199 (0x4ae8 bytes) 000692 1752510384.916356 punch from 0x003c1b to 0x00a10b (0x64f1 bytes) 000693 1752510384.919339 write 0x03b36f thru 0x03bfc0 (0xc52 bytes) 000694 1752510384.920307 trunc from 0x03bfc1 to 0x033577 (0x8a49 bytes) 000696 1752510384.933134 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000697 1752510385.038927 trunc from 0x02d90c to 0x033577 (0x5c6c bytes) 000698 1752510385.048374 mapwrite 0x00c5f4 thru 0x019a75 (0xd482 bytes) 000699 1752510385.094808 fallocate 0x01f291 thru 0x020626 (0x1396 bytes) 000700 1752510385.099170 fallocate 0x00f94b thru 0x01bb8f (0xc245 bytes) 000701 1752510385.103151 mapwrite 0x03ca3c thru 0x03ffff (0x35c4 bytes) 000702 1752510385.134087 punch from 0x02e9f9 to 0x03b665 (0xcc6d bytes) 000703 1752510385.141143 MIRROR_EXTEND to 7 mirrors 000704 1752510385.439546 read 0x0262eb thru 0x0326f7 (0xc40d bytes) 000705 1752510385.452089 write 0x0044f4 thru 0x010955 (0xc462 bytes) 000706 1752510385.485067 punch from 0x01e4eb to 0x0241f9 (0x5d0f bytes) 000707 1752510385.489349 mapread 0x03f35f thru 0x03ffff (0xca1 bytes) 000708 1752510385.495821 mapread 0x001b0c thru 0x00df15 (0xc40a bytes) 000709 1752510385.500242 fallocate 0x03190d thru 0x03ac06 (0x92fa bytes) 000710 1752510385.504175 mapread 0x038c42 thru 0x03a5d1 (0x1990 bytes) 000711 1752510385.505004 read 0x0088e3 thru 0x015f7e (0xd69c bytes) 000712 1752510385.511400 write 0x01b1e9 thru 0x01d77a (0x2592 bytes) 000714 1752510385.516036 MIRROR_EXTEND to 8 mirrors 000715 1752510385.794585 read 0x01d6b3 thru 0x0260bc (0x8a0a bytes) 000716 1752510385.811034 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000717 1752510385.948173 MIRROR_RESYNC 7 mirrors 000718 1752510386.734997 punch from 0x036bd8 to 0x03ffff (0x9428 bytes) 000719 1752510386.751258 write 0x02779a thru 0x028c8b (0x14f2 bytes) 000720 1752510386.756072 mapwrite 0x03a49e thru 0x03ffff (0x5b62 bytes) 000721 1752510386.773959 write 0x00a413 thru 0x012c49 (0x8837 bytes) 000722 1752510386.779152 mapread 0x012a5e thru 0x01364f (0xbf2 bytes) 000723 1752510386.783491 trunc from 0x03ad80 to 0x03ffff (0x5280 bytes) 000724 1752510386.786390 fallocate 0x03d3fb thru 0x03ffff (0x2c05 bytes) 000725 1752510386.789576 punch from 0x0138ab to 0x014c39 (0x138f bytes) 000726 1752510386.792607 mapwrite 0x002b98 thru 0x01022a (0xd693 bytes) 000727 1752510386.812437 punch from 0x00c5a1 to 0x00ef45 (0x29a5 bytes) 000728 1752510386.819294 MIRROR_RESYNC 7 mirrors 000729 1752510388.011691 punch from 0x019c16 to 0x01a2d6 (0x6c1 bytes) 000730 1752510388.032878 trunc from 0x040000 to 0x005497 (0x3ab68 bytes) 000731 1752510388.048201 mapread 0x003462 thru 0x005497 (0x2036 bytes) 000732 1752510388.058115 fallocate 0x03366c thru 0x03ffff (0xc994 bytes) 000733 1752510388.061512 mapwrite 0x025d62 thru 0x025eb2 (0x151 bytes) 000734 1752510388.087346 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000735 1752510388.222098 MIRROR_RESYNC 6 mirrors 000736 1752510388.729178 mapread 0x00a320 thru 0x011a06 (0x76e7 bytes) 000737 1752510388.747196 punch from 0x00be39 to 0x00eebf (0x3087 bytes) 000738 1752510388.767467 mapread 0x017c2a thru 0x01b569 (0x3940 bytes) 000739 1752510388.773170 trunc from 0x025eb3 to 0x00805f (0x1de53 bytes) 000740 1752510388.786245 mapread 0x005cc1 thru 0x00805f (0x239f bytes) 000741 1752510388.793249 trunc from 0x008060 to 0x01c31e (0x142bf bytes) 000742 1752510388.804156 trunc from 0x0074d7 to 0x01240d (0xaf37 bytes) 000743 1752510388.809083 mapread 0x003fb1 thru 0x00c80e (0x885e bytes) 000744 1752510388.815907 punch from 0x01783a to 0x01c1cf (0x4996 bytes) 000745 1752510388.818814 trunc from 0x01c31f to 0x01d99e (0x1680 bytes) 000746 1752510388.827932 punch from 0x013f9d to 0x019d96 (0x5dfa bytes) 000747 1752510388.831182 mapread 0x00174c thru 0x0072f8 (0x5bad bytes) 000748 1752510388.832114 trunc from 0x007334 to 0x012cd8 (0xb9a5 bytes) 000749 1752510388.837468 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000750 1752510388.949313 mapwrite 0x0111a8 thru 0x01e98e (0xd7e7 bytes) 000751 1752510388.996412 mapread 0x01b7d0 thru 0x01e98e (0x31bf bytes) 000752 1752510388.997415 trunc from 0x01e98f to 0x0065d3 (0x183bb bytes) 000753 1752510389.008074 write 0x026f22 thru 0x02b7a2 (0x4881 bytes) 000754 1752510389.010097 mapread 0x005e49 thru 0x012f38 (0xd0f0 bytes) 000756 1752510389.019515 punch from 0x0273f0 to 0x02b7a2 (0x43b3 bytes) 000757 1752510389.028829 trunc from 0x012db9 to 0x01c1a2 (0x93ea bytes) 000760 1752510389.033548 fallocate 0x023583 thru 0x0264bd (0x2f3b bytes) 000761 1752510389.037498 punch from 0x0206cc to 0x02b7a2 (0xb0d7 bytes) 000762 1752510389.040344 write 0x0015a9 thru 0x004a14 (0x346c bytes) 000763 1752510389.041400 trunc from 0x02b7a3 to 0x0193f8 (0x123aa bytes) 000765 1752510389.052888 MIRROR_RESYNC 5 mirrors 000766 1752510389.401796 punch from 0x009778 to 0x0149c1 (0xb24a bytes) 000767 1752510389.420886 write 0x03a994 thru 0x03ffff (0x566c bytes) 000768 1752510389.424941 MIRROR_RESYNC 5 mirrors 000769 1752510389.792789 fallocate 0x03cb05 thru 0x03ffff (0x34fb bytes) 000770 1752510389.808586 write 0x01e3cb thru 0x02703a (0x8c70 bytes) 000771 1752510389.814674 mapread 0x00fdfa thru 0x014cca (0x4ed1 bytes) 000772 1752510389.824505 MIRROR_RESYNC 5 mirrors 000773 1752510390.259441 punch from 0x020eb1 to 0x027195 (0x62e5 bytes) 000774 1752510390.289141 MIRROR_RESYNC 5 mirrors 000775 1752510390.773563 MIRROR_EXTEND to 6 mirrors 000776 1752510391.039146 MIRROR_EXTEND to 7 mirrors 000777 1752510391.394034 trunc from 0x00b2d4 to 0x00f181 (0x3eae bytes) 000778 1752510391.416047 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000780 1752510391.517514 read 0x03468c thru 0x038eca (0x483f bytes) 000781 1752510391.522850 punch from 0x0084a1 to 0x008a87 (0x5e7 bytes) 000782 1752510391.525629 fallocate 0x0334dd thru 0x0371e6 (0x3d0a bytes) 000783 1752510391.529757 punch from 0x016229 to 0x016fac (0xd84 bytes) 000784 1752510391.532569 write 0x02c332 thru 0x0365b6 (0xa285 bytes) 000785 1752510391.539044 punch from 0x028071 to 0x033fd0 (0xbf60 bytes) 000786 1752510391.548051 trunc from 0x040000 to 0x02c65c (0x139a3 bytes) 000787 1752510391.555417 write 0x028116 thru 0x03540b (0xd2f6 bytes) 000788 1752510391.559874 fallocate 0x03fe8e thru 0x03ffff (0x172 bytes) 000789 1752510391.564403 read 0x01bc9a thru 0x026eda (0xb241 bytes) 000790 1752510391.569876 trunc from 0x040000 to 0x028667 (0x17998 bytes) 000791 1752510391.580190 MIRROR_RESYNC 6 mirrors 000792 1752510392.196134 fallocate 0x0192bc thru 0x022c29 (0x996e bytes) 000793 1752510392.215413 trunc from 0x01b5a6 to 0x01c313 (0xd6e bytes) 000794 1752510392.219688 punch from 0x001a3b to 0x00f983 (0xdf49 bytes) 000795 1752510392.224294 trunc from 0x006d11 to 0x011cbf (0xafaf bytes) 000796 1752510392.230290 trunc from 0x00cc63 to 0x0160e2 (0x9480 bytes) 000797 1752510392.233295 trunc from 0x003866 to 0x0123f6 (0xeb91 bytes) 000798 1752510392.239097 write 0x023fe8 thru 0x033d0f (0xfd28 bytes) 000799 1752510392.246406 punch from 0x021484 to 0x027cb3 (0x6830 bytes) 000800 1752510392.257205 trunc from 0x033d10 to 0x020ad0 (0x1323f bytes) 000801 1752510392.273928 trunc from 0x020ad1 to 0x014820 (0xc2b0 bytes) 000803 1752510392.287519 read 0x000b1b thru 0x008c3b (0x8121 bytes) 000805 1752510392.298557 MIRROR_EXTEND to 7 mirrors 000806 1752510392.560444 fallocate 0x01fd73 thru 0x02f80d (0xfa9b bytes) 000807 1752510392.587508 trunc from 0x02f80e to 0x02c879 (0x2f94 bytes) 000808 1752510392.607936 mapread 0x009990 thru 0x00a9c6 (0x1037 bytes) 000809 1752510392.614863 read 0x018d52 thru 0x023d31 (0xafe0 bytes) 000810 1752510392.622235 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000811 1752510392.724272 fallocate 0x026058 thru 0x026a0e (0x9b7 bytes) 000812 1752510392.727802 fallocate 0x0289a4 thru 0x031cbb (0x9318 bytes) 000813 1752510392.730839 punch from 0x01502d to 0x0196dc (0x46b0 bytes) 000814 1752510392.733556 mapwrite 0x0094e0 thru 0x00ee31 (0x5952 bytes) 000815 1752510392.758143 read 0x00aa2d thru 0x017a9b (0xd06f bytes) 000816 1752510392.762241 fallocate 0x01511f thru 0x01cc18 (0x7afa bytes) 000817 1752510392.766221 read 0x007b57 thru 0x011aad (0x9f57 bytes) 000818 1752510392.767263 trunc from 0x00488c to 0x0105a0 (0xbd15 bytes) 000819 1752510392.771382 read 0x00e7d1 thru 0x01ce8e (0xe6be bytes) 000820 1752510392.778697 trunc from 0x01b105 to 0x01d108 (0x2004 bytes) 000821 1752510392.782274 mapwrite 0x02f66c thru 0x03dafa (0xe48f bytes) 000822 1752510392.824025 punch from 0x00de6b to 0x0162c9 (0x845f bytes) 000823 1752510392.827377 trunc from 0x03dafb to 0x00eae5 (0x2f015 bytes) 000824 1752510392.836996 mapwrite 0x013e99 thru 0x017fdb (0x4143 bytes) 000825 1752510392.876660 punch from 0x002679 to 0x01183d (0xf1c5 bytes) 000826 1752510392.880374 mapread 0x0164a9 thru 0x017fdb (0x1b33 bytes) 000827 1752510392.881260 mapwrite 0x006d20 thru 0x010fbb (0xa29c bytes) 000828 1752510392.930461 write 0x02b3ea thru 0x033d0d (0x8924 bytes) 000831 1752510392.932887 punch from 0x02159f to 0x031407 (0xfe69 bytes) 000833 1752510392.944890 trunc from 0x03093f to 0x033d0d (0x33cf bytes) 000834 1752510392.948802 trunc from 0x00481f to 0x00a0e6 (0x58c8 bytes) 000835 1752510392.951856 trunc from 0x012e23 to 0x021324 (0xe502 bytes) 000836 1752510392.955143 write 0x02962a thru 0x02de33 (0x480a bytes) 000837 1752510392.962035 write 0x00b949 thru 0x014859 (0x8f11 bytes) 000838 1752510392.969008 MIRROR_EXTEND to 7 mirrors 000839 1752510393.235597 write 0x020d56 thru 0x023392 (0x263d bytes) 000840 1752510393.256049 trunc from 0x033d0e to 0x01e7b7 (0x15556 bytes) 000841 1752510393.266040 punch from 0x010d00 to 0x010e75 (0x176 bytes) 000842 1752510393.269471 read 0x00b827 thru 0x014f4d (0x9727 bytes) 000844 1752510393.274586 fallocate 0x03538d thru 0x038069 (0x2cdd bytes) 000845 1752510393.277938 fallocate 0x0085ea thru 0x00fefa (0x7911 bytes) 000846 1752510393.281008 write 0x0169eb thru 0x01df3e (0x7554 bytes) 000847 1752510393.285938 punch from 0x00ae58 to 0x015186 (0xa32f bytes) 000848 1752510393.291543 mapwrite 0x022f6c thru 0x0232bc (0x351 bytes) 000849 1752510393.328093 MIRROR_EXTEND to 8 mirrors 000851 1752510393.559840 write 0x001328 thru 0x00dc21 (0xc8fa bytes) 000852 1752510393.583336 punch from 0x00403f to 0x0046b0 (0x672 bytes) 000853 1752510393.590922 trunc from 0x0232bd to 0x010a8e (0x1282e bytes) 000854 1752510393.610684 punch from 0x00cfe6 to 0x00dd67 (0xd82 bytes) 000855 1752510393.615424 fallocate 0x029e20 thru 0x034fb3 (0xb194 bytes) 000856 1752510393.622863 MIRROR_RESYNC 8 mirrors 000857 1752510394.107019 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000858 1752510394.257767 mapread 0x0252e7 thru 0x030531 (0xb24b bytes) 000859 1752510394.271499 fallocate 0x006fb1 thru 0x015248 (0xe298 bytes) 000860 1752510394.290692 punch from 0x0200c7 to 0x02ffec (0xff26 bytes) 000861 1752510394.293990 write 0x021fec thru 0x027e24 (0x5e39 bytes) 000862 1752510394.302894 trunc from 0x034fb4 to 0x02c1d8 (0x8ddb bytes) 000863 1752510394.325893 fallocate 0x03947d thru 0x03ffff (0x6b83 bytes) 000864 1752510394.330358 fallocate 0x03ee5e thru 0x03ffff (0x11a2 bytes) 000865 1752510394.334744 mapread 0x009e72 thru 0x015ca7 (0xbe36 bytes) 000866 1752510394.346535 read 0x001bf1 thru 0x008b98 (0x6fa8 bytes) 000868 1752510394.351650 trunc from 0x02d311 to 0x03a07a (0xcd6a bytes) 000869 1752510394.355362 trunc from 0x03a659 to 0x03ffff (0x59a7 bytes) 000870 1752510394.358943 read 0x02fb98 thru 0x03e011 (0xe47a bytes) 000872 1752510394.367118 mapread 0x022f42 thru 0x031048 (0xe107 bytes) 000873 1752510394.368108 punch from 0x01dbe0 to 0x0257f5 (0x7c16 bytes) 000874 1752510394.381449 trunc from 0x040000 to 0x00297a (0x3d685 bytes) 000875 1752510394.390208 read 0x0020be thru 0x00297a (0x8bd bytes) 000876 1752510394.390839 mapwrite 0x02ae1a thru 0x035065 (0xa24c bytes) 000878 1752510394.427034 MIRROR_EXTEND to 8 mirrors 000879 1752510394.641238 fallocate 0x002ee8 thru 0x00cab0 (0x9bc9 bytes) 000880 1752510394.659745 fallocate 0x009891 thru 0x00eb11 (0x5281 bytes) 000881 1752510394.662798 write 0x03e744 thru 0x03ffff (0x18bc bytes) 000882 1752510394.665981 trunc from 0x03dacb to 0x03ffff (0x2535 bytes) 000883 1752510394.673270 trunc from 0x040000 to 0x03dde4 (0x221b bytes) 000884 1752510394.693107 mapwrite 0x03b58a thru 0x03ffff (0x4a76 bytes) 000885 1752510394.724645 mapwrite 0x028e1f thru 0x028e67 (0x49 bytes) 000886 1752510394.736079 mapwrite 0x0028ec thru 0x00fdb2 (0xd4c7 bytes) 000887 1752510394.751963 trunc from 0x040000 to 0x00c0c5 (0x33f3a bytes) 000888 1752510394.759783 trunc from 0x00a8ed to 0x00c0c5 (0x17d9 bytes) 000889 1752510394.763108 punch from 0x0006d3 to 0x004de2 (0x4710 bytes) 000890 1752510394.765743 trunc from 0x008547 to 0x00c0c5 (0x3b7f bytes) 000891 1752510394.769295 punch from 0x0067f0 to 0x00b0fa (0x490b bytes) 000892 1752510394.775660 MIRROR_RESYNC 8 mirrors 000893 1752510395.302836 mapwrite 0x020172 thru 0x02ba1b (0xb8aa bytes) 000894 1752510395.344123 fallocate 0x007db9 thru 0x0154c1 (0xd709 bytes) 000896 1752510395.347512 punch from 0x002f2c to 0x004c1f (0x1cf4 bytes) 000897 1752510395.352031 MIRROR_EXTEND to 9 mirrors 000899 1752510395.593694 MIRROR_RESYNC 9 mirrors 000900 1752510396.317442 punch from 0x027319 to 0x0294ec (0x21d4 bytes) 000901 1752510396.334216 fallocate 0x00ac19 thru 0x018f9a (0xe382 bytes) 000902 1752510396.336998 trunc from 0x02ba1c to 0x0063a2 (0x25679 bytes) 000903 1752510396.358156 punch from 0x00437f to 0x0063a2 (0x2024 bytes) 000904 1752510396.362028 fallocate 0x037ed8 thru 0x03f716 (0x783f bytes) 000905 1752510396.365250 fallocate 0x00dc28 thru 0x0142e4 (0x66bd bytes) 000906 1752510396.369184 trunc from 0x0063a3 to 0x00e6fb (0x8359 bytes) 000907 1752510396.376691 trunc from 0x004f72 to 0x007af5 (0x2b84 bytes) 000908 1752510396.383030 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000909 1752510396.474129 mapread 0x00a829 thru 0x00e6fb (0x3ed3 bytes) 000910 1752510396.486067 mapwrite 0x02687a thru 0x02ae3a (0x45c1 bytes) 000912 1752510396.510331 read 0x011bc0 thru 0x01b5f6 (0x9a37 bytes) 000913 1752510396.514987 mapread 0x0147c3 thru 0x023943 (0xf181 bytes) 000914 1752510396.518916 read 0x00ddab thru 0x010e04 (0x305a bytes) 000915 1752510396.524222 read 0x007529 thru 0x011dc5 (0xa89d bytes) 000916 1752510396.524873 mapread 0x028101 thru 0x02ae3a (0x2d3a bytes) 000917 1752510396.528483 MIRROR_EXTEND to 9 mirrors 000918 1752510396.746926 write 0x01a16d thru 0x01c8ef (0x2783 bytes) 000919 1752510396.768349 trunc from 0x02ae3b to 0x013833 (0x17607 bytes) 000920 1752510396.787666 write 0x003f6f thru 0x00d479 (0x950b bytes) 000922 1752510396.792856 fallocate 0x0135fd thru 0x01be0d (0x8811 bytes) 000923 1752510396.796386 mapread 0x00427b thru 0x00f4c4 (0xb24a bytes) 000924 1752510396.799547 MIRROR_EXTEND to 10 mirrors 000925 1752510397.019084 write 0x004a9d thru 0x012954 (0xdeb8 bytes) 000926 1752510397.042012 trunc from 0x018ff7 to 0x01be0d (0x2e17 bytes) 000929 1752510397.045930 mapwrite 0x0304b6 thru 0x034450 (0x3f9b bytes) 000930 1752510397.078046 mapwrite 0x00626a thru 0x00d952 (0x76e9 bytes) 000931 1752510397.087563 mapread 0x00bc58 thru 0x00c39e (0x747 bytes) 000932 1752510397.088288 trunc from 0x02ff28 to 0x034450 (0x4529 bytes) 000934 1752510397.091940 fallocate 0x027183 thru 0x02e98b (0x7809 bytes) 000935 1752510397.095545 trunc from 0x034451 to 0x03b154 (0x6d04 bytes) 000936 1752510397.103650 punch from 0x026c2e to 0x034c7f (0xe052 bytes) 000937 1752510397.107187 punch from 0x01c858 to 0x02b790 (0xef39 bytes) 000938 1752510397.110098 punch from 0x0096cf to 0x00b66a (0x1f9c bytes) 000939 1752510397.113183 trunc from 0x02ed8b to 0x03a8b1 (0xbb27 bytes) 000940 1752510397.119484 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000943 1752510397.215374 fallocate 0x010bb4 thru 0x013a34 (0x2e81 bytes) 000944 1752510397.223200 write 0x01c1ff thru 0x029714 (0xd516 bytes) 000945 1752510397.233216 trunc from 0x01a70b to 0x01df9e (0x3894 bytes) 000946 1752510397.244894 MIRROR_EXTEND to 10 mirrors 000948 1752510397.547409 read 0x009861 thru 0x014261 (0xaa01 bytes) 000949 1752510397.556165 punch from 0x028340 to 0x02f0ff (0x6dc0 bytes) 000950 1752510397.575436 mapwrite 0x011fc9 thru 0x01fd3a (0xdd72 bytes) 000951 1752510397.614472 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000952 1752510397.728466 punch from 0x02cee9 to 0x031950 (0x4a68 bytes) 000953 1752510397.731943 write 0x004533 thru 0x00b2fe (0x6dcc bytes) 000954 1752510397.740042 mapwrite 0x03919c thru 0x03e0f1 (0x4f56 bytes) 000955 1752510397.771819 read 0x0238ca thru 0x02dbf4 (0xa32b bytes) 000956 1752510397.777489 write 0x0188a5 thru 0x021bed (0x9349 bytes) 000957 1752510397.786551 write 0x00df1e thru 0x015087 (0x716a bytes) 000958 1752510397.793176 fallocate 0x03ca22 thru 0x03ffff (0x35de bytes) 000959 1752510397.796273 mapwrite 0x000f42 thru 0x00d5fd (0xc6bc bytes) 000960 1752510397.819823 MIRROR_RESYNC 9 mirrors 000961 1752510398.329092 fallocate 0x03dce9 thru 0x03ffff (0x2317 bytes) 000962 1752510398.344164 trunc from 0x03ce0b to 0x03cec7 (0xbd bytes) 000963 1752510398.347616 read 0x00b576 thru 0x0178c8 (0xc353 bytes) 000964 1752510398.352466 read 0x013733 thru 0x02162b (0xdef9 bytes) 000965 1752510398.359115 mapread 0x020816 thru 0x0278dc (0x70c7 bytes) 000966 1752510398.360036 trunc from 0x040000 to 0x028da7 (0x17258 bytes) 000967 1752510398.371102 trunc from 0x028da8 to 0x026803 (0x25a4 bytes) 000968 1752510398.379962 trunc from 0x026804 to 0x00c174 (0x1a68f bytes) 000970 1752510398.389613 mapread 0x00368d thru 0x00c174 (0x8ae8 bytes) 000971 1752510398.392934 MIRROR_RESYNC 9 mirrors 000973 1752510398.694841 mapwrite 0x0288a1 thru 0x02b306 (0x2a66 bytes) 000974 1752510398.739519 mapwrite 0x006dd2 thru 0x0139b0 (0xcbdf bytes) 000975 1752510398.760082 mapwrite 0x03c336 thru 0x03ffff (0x3cca bytes) 000976 1752510398.784348 punch from 0x0344c3 to 0x03ffff (0xbb3d bytes) 000977 1752510398.787491 read 0x01ab3b thru 0x01e8e2 (0x3da8 bytes) 000978 1752510398.794762 MIRROR_EXTEND to 10 mirrors 000979 1752510399.045484 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53d:0x0]: Invalid argument (22) 000980 1752510399.202237 mapwrite 0x00cc7c thru 0x014b1e (0x7ea3 bytes) 000981 1752510399.280984 read 0x033898 thru 0x03b1f0 (0x7959 bytes) 000982 1752510399.287195 mapwrite 0x0374d3 thru 0x03ffff (0x8b2d bytes) 000983 1752510399.310219 trunc from 0x040000 to 0x027412 (0x18bed bytes) 000984 1752510399.347110 mapread 0x020efb thru 0x027412 (0x6518 bytes) 000985 1752510399.355514 read 0x021725 thru 0x027412 (0x5cee bytes) 000986 1752510399.356045 trunc from 0x010eb0 to 0x01f6a6 (0xe7f7 bytes) 000987 1752510399.360863 mapwrite 0x00b4d7 thru 0x00ed4d (0x3877 bytes) 000988 1752510399.375081 punch from 0x01cd9c to 0x027412 (0xa677 bytes) 000989 1752510399.379255 fallocate 0x014770 thru 0x01573c (0xfcd bytes) 000990 1752510399.383254 mapwrite 0x0386b5 thru 0x03bc08 (0x3554 bytes) 000991 1752510399.407945 mapwrite 0x027014 thru 0x02e0e0 (0x70cd bytes) 000992 1752510399.418329 write 0x00f8f4 thru 0x010dfc (0x1509 bytes) 000993 1752510399.419343 write 0x02cb4a thru 0x0359b3 (0x8e6a bytes) 000994 1752510399.424810 MIRROR_RESYNC 9 mirrors 000995 1752510400.164156 mapread 0x030223 thru 0x03620b (0x5fe9 bytes) 000996 1752510400.175877 mapread 0x027188 thru 0x02a5fb (0x3474 bytes) 000997 1752510400.181149 write 0x036f4a thru 0x03ffff (0x90b6 bytes) 000998 1752510400.201375 trunc from 0x040000 to 0x00ff71 (0x3008e bytes) 000999 1752510400.212110 trunc from 0x00d7fa to 0x00f71b (0x1f22 bytes) 001000 1752510400.215083 trunc from 0x00ff72 to 0x0135d3 (0x3662 bytes) All operations completed A-OK! PASS 70a (38s) == sanity-flr test 71: check mirror extend parent fid ==== 12:26:42 (1752510402) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x58c:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 200a: stress test ===================== 12:26:46 (1752510406) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7015360 .. Extending file size to 1385376 .. done Extending file size to 8522496 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7592320 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7864384 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8481056 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9034656 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9039136 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8622016 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8639200 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9066016 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62058 62059 62060 62062 62064 done 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) 192.168.203.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:27:54 (1752510474) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1625408 .. Extending file size to 113728 .. failed Extending file size to 7858528 .. Extending file size to 4678048 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5404832 .. failed Extending file size to 8540576 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8798080 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9233792 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8806944 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9292160 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 65582 65583 65585 65586 65589 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) 192.168.203.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:28:59 (1752510539) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2588 1285100 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2852 1284836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14824 3592196 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22944 3584076 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37768 7176272 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2588 1285100 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2852 1284836 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14824 3592196 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22944 3584076 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37768 7176272 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0450762 s, 46.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.07543 s, 27.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:29:06 (1752510546) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0393979 s, 53.2 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 12:29:09 (1752510549) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:29:11 (1752510551) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0395712 s, 53.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:29:14 (1752510554) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0421222 s, 49.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:29:17 (1752510557) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.033205 s, 63.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:29:42 (1752510582) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg350-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215357 s, 48.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } PASS 204d (39s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:30:22 (1752510622) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg350-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0330962 s, 63.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (37s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:30:58 (1752510658) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg350-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg350-server: Pool lustre.test_204f created Adding targets to pool oleg350-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg350-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0319967 s, 65.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg350-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg350-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg350-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:31:47 (1752510707) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:31:50 (1752510710) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:31:52 (1752510712) oleg350-server: Pool lustre.test_206 created Adding targets to pool oleg350-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg350-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg350-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg350-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg350-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:32:10 (1752510730) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360629 s, 29.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193985 s, 54.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xd75b0b6c Mirror 3: 0xd75b0b6c PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:32:14 (1752510734) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:32:16 (1752510736) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:32:17 (1752510737) Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0157187 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2420000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1cc0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1cc0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1cc0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe80000 Read this (4 bytes): BBBB 192.168.203.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg350-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:32:35 (1752510755) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00734863 s, 1.1 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:32:38 (1752510758) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203326 s, 51.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611269 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-14 12:32:39.000000000 -0400 Modify: 2025-07-14 12:32:39.000000000 -0400 Change: 2025-07-14 12:32:39.000000000 -0400 Birth: 2025-07-14 12:32:39.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:32:41 (1752510761) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189433 s, 55.4 MB/s oleg350-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg350-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg350-server: O/280000401/d30/be: File not found by ext2_lookup PASS 210b (15s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:32:56 (1752510776) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0477118 s, 858 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00769992 s, 532 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 14 12:32 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 14 12:32 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 918 sec ============ 12:32:59 (1752510779) === sanity-flr: start cleanup 12:33:00 (1752510780) === === sanity-flr: finish cleanup 12:33:03 (1752510783) ===