-----============= acceptance-small: sanity-flr ============----- Wed Jul 16 13:44:02 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg309-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:44:09 (1752687849) === oleg309-client.virtnet: executing check_config_client /mnt/lustre oleg309-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg309-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9bb090bf1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9bb090bf1000.idle_timeout=debug disable quota as required oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:44:20 (1752687860) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8199] == sanity-flr test 0a: lfs mirror create with -N option == 13:44:21 (1752687861) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:44:26 (1752687866) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:44:27 (1752687867) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:44:28 (1752687868) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:44:33 (1752687873) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:44:34 (1752687874) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 13:44:36 (1752687876) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9bb090bf1000.stats=clear osc.lustre-OST0001-osc-ffff9bb090bf1000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:44:38 (1752687878) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:44:41 (1752687881) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:44:44 (1752687884) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:44:45 (1752687885) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 13:44:47 (1752687887) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:44:49 (1752687889) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:44:52 (1752687892) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:44:54 (1752687894) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:44:57 (1752687897) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:44:59 (1752687899) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.258375 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276776 s, 37.9 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:45:02 (1752687902) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.111842 s, 84.4 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:45:05 (1752687905) Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bb090bf1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bb090bf1000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bb090bf1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bb090bf1000.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:45:34 (1752687934) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.144265 s, 50.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bb090bf1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bb090bf1000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:45:48 (1752687948) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bb090bf1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bb090bf1000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bb090bf1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bb090bf1000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:46:14 (1752687974) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05117 s, 49.9 MB/s -rw-r--r-- 1 root root 52428800 Jul 16 13:46 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.276s user 0m0.000s sys 0m0.161s Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9bb090bf1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9bb090bf1000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.251s user 0m0.001s sys 0m0.144s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9bb090bf1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bb090bf1000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.291s user 0m0.001s sys 0m0.139s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg309-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bb090bf1000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9bb090bf1000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:46:42 (1752688002) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:46:43 (1752688003) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:46:44 (1752688004) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 13:46:45 (1752688005) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 13:46:47 (1752688007) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.152275 s, 82.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:46:51 (1752688011) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103282 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116861 s, 89.7 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:46:53 (1752688013) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:46:56 (1752688016) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000700115 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000596037 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00073385 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000640836 s, 1.6 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 13:46:59 (1752688019) create test files with size 15734955 .. a4865cc8f2aaa9590a8b391880c0a5f7 - 76a26fd367a2914e5f0179ec17ee3057 - 56cdc50370a1a1ae85223517baef42d8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 13:47:05 (1752688025) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:47:08 (1752688028) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:47:11 (1752688031) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390367 s, 26.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012278 s, 85.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 13:47:13 (1752688033) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0311923 s, 67.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.106059 s, 39.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.041331 s, 50.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 13:47:17 (1752688037) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 13:47:18 (1752688038) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:47:19 (1752688039) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119983 s, 87.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 13:47:22 (1752688042) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.107902 s, 29.2 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:47:25 (1752688045) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x45:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:47:28 (1752688048) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.113806 s, 92.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:47:47 (1752688067) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9bb088846800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.124715 s, 84.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9bb0889f0800.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:47:53 (1752688073) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (24s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:48:17 (1752688097) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (27s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:48:44 (1752688124) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 13:48:46 (1752688126) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.177341 s, 17.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0965266 s, 32.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 078db89de610f066a80f31bfc744b9cd - mirror 1 checksum: 584a42c6713cf1b066aac4049d8488d9 - mirror 2 checksum: 078db89de610f066a80f31bfc744b9cd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jul 16 13:48:49 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Jul 16 13:48:49 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 078db89de610f066a80f31bfc744b9cd - mirror 1 checksum: 584a42c6713cf1b066aac4049d8488d9 - mirror 2 checksum: 584a42c6713cf1b066aac4049d8488d9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jul 16 13:48:49 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Jul 16 13:48:49 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 48 (6s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:48:52 (1752688132) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176285 s, 59.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:48:55 (1752688135) Failing mds1 on oleg309-server Stopping /mnt/lustre-mds1 (opts:) on oleg309-server 13:49:05 (1752688145) shut down facet: mds1 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds1 to oleg309-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 13:49:21 (1752688161) targets are mounted 13:49:21 (1752688161) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206422 s, 50.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 50A (36s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:49:31 (1752688171) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0377797 s, 542 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0127557 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:49:35 (1752688175) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0189459 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00874932 s, 2.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:49:39 (1752688179) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00419762 s, 3.9 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:49:43 (1752688183) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0205846 s, 995 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:49:46 (1752688186) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00310386 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0177643 s, 231 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:49:49 (1752688189) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:49:59 (1752688199) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:50:19 (1752688219) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:50:38 (1752688238) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:50:57 (1752688257) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 13:51:01 (1752688261) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55871 mirror split pid 55872 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55871 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55872 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 13:52:04 (1752688324) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 8325 fallocating to largest ever: 0x1309b 000001 1752688325.363933 fallocate 0x00f4ab thru 0x01309a (0x3bf0 bytes) 000002 1752688325.370664 write 0x004780 thru 0x00fe43 (0xb6c4 bytes) 000003 1752688325.377155 mapwrite 0x03c092 thru 0x03ffff (0x3f6e bytes) 000004 1752688325.409509 punch from 0x03ca03 to 0x03f085 (0x2683 bytes) fallocating to largest ever: 0x40000 000006 1752688325.414206 fallocate 0x03c44d thru 0x03ffff (0x3bb3 bytes) 000007 1752688325.418057 fallocate 0x032433 thru 0x03ffff (0xdbcd bytes) 000008 1752688325.422552 mapread 0x022b9c thru 0x02327b (0x6e0 bytes) 000009 1752688325.427975 write 0x020bb2 thru 0x026b03 (0x5f52 bytes) 000010 1752688325.429200 trunc from 0x040000 to 0x02cf44 (0x130bb bytes) 000011 1752688325.438002 mapwrite 0x00ae88 thru 0x018d9c (0xdf15 bytes) 000012 1752688325.461443 trunc from 0x016f1a to 0x017968 (0xa4f bytes) 000013 1752688325.467403 mapread 0x020d96 thru 0x02209b (0x1306 bytes) 000014 1752688325.468310 mapread 0x027b76 thru 0x02cf44 (0x53cf bytes) 000015 1752688325.473924 punch from 0x00749c to 0x010000 (0x8b65 bytes) 000016 1752688325.480115 mapread 0x0236c4 thru 0x023782 (0xbf bytes) 000017 1752688325.480883 mapread 0x00bad4 thru 0x018041 (0xc56e bytes) 000018 1752688325.495799 punch from 0x008430 to 0x010cf4 (0x88c5 bytes) 000019 1752688325.500803 trunc from 0x01a181 to 0x026fb7 (0xce37 bytes) 000020 1752688325.504825 write 0x0171e0 thru 0x01b975 (0x4796 bytes) 000021 1752688325.514620 MIRROR_EXTEND to 2 mirrors 000022 1752688325.852907 mapread 0x01e4d3 thru 0x022c7e (0x47ac bytes) 000023 1752688325.871053 trunc from 0x023140 to 0x028ff1 (0x5eb2 bytes) 000024 1752688325.899920 write 0x02d3a9 thru 0x030941 (0x3599 bytes) 000025 1752688325.904293 write 0x03491c thru 0x03be9a (0x757f bytes) 000026 1752688325.905608 write 0x020ad5 thru 0x02ea1d (0xdf49 bytes) 000027 1752688325.913403 fallocate 0x02be1f thru 0x038131 (0xc313 bytes) 000028 1752688325.918598 mapwrite 0x03ba85 thru 0x03e0e8 (0x2664 bytes) 000029 1752688325.963726 mapwrite 0x00b845 thru 0x01b5d6 (0xfd92 bytes) 000030 1752688325.998007 punch from 0x035274 to 0x03e0e8 (0x8e75 bytes) 000031 1752688326.005214 MIRROR_EXTEND to 3 mirrors 000032 1752688326.407845 mapread 0x01c7e5 thru 0x028b7a (0xc396 bytes) 000033 1752688326.444396 fallocate 0x032a81 thru 0x03ffff (0xd57f bytes) 000036 1752688326.464832 fallocate 0x015b72 thru 0x01f12c (0x95bb bytes) 000038 1752688326.469277 trunc from 0x037c4b to 0x03ffff (0x83b5 bytes) 000039 1752688326.473102 fallocate 0x00b57b thru 0x01850c (0xcf92 bytes) 000040 1752688326.476912 mapwrite 0x020411 thru 0x02a8b3 (0xa4a3 bytes) 000041 1752688326.514929 trunc from 0x016f89 to 0x021ddf (0xae57 bytes) 000042 1752688326.526827 MIRROR_RESYNC 3 mirrors 000044 1752688327.262961 read 0x032206 thru 0x0328f3 (0x6ee bytes) 000046 1752688327.267250 punch from 0x014478 to 0x020429 (0xbfb2 bytes) 000047 1752688327.289779 fallocate 0x01eba8 thru 0x029449 (0xa8a2 bytes) 000048 1752688327.293712 mapwrite 0x022583 thru 0x023299 (0xd17 bytes) 000049 1752688327.314584 mapread 0x016f05 thru 0x021a06 (0xab02 bytes) 000050 1752688327.331087 trunc from 0x040000 to 0x01cf8f (0x23070 bytes) 000051 1752688327.349440 punch from 0x001d3d to 0x01147f (0xf743 bytes) 000052 1752688327.354193 punch from 0x018a81 to 0x0192c3 (0x843 bytes) 000054 1752688327.362680 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000055 1752688327.496170 trunc from 0x014b6f to 0x017290 (0x2722 bytes) 000056 1752688327.503360 read 0x0192eb thru 0x01cf8f (0x3ca5 bytes) 000057 1752688327.509437 write 0x00e40d thru 0x016d77 (0x896b bytes) 000058 1752688327.527275 MIRROR_EXTEND to 3 mirrors 000060 1752688327.843717 write 0x014428 thru 0x019e84 (0x5a5d bytes) 000061 1752688327.876983 fallocate 0x0035ed thru 0x0068e7 (0x32fb bytes) 000062 1752688327.882084 read 0x00ef40 thru 0x0111ef (0x22b0 bytes) 000064 1752688327.887924 write 0x026882 thru 0x027482 (0xc01 bytes) 000065 1752688327.888933 trunc from 0x027483 to 0x0235f7 (0x3e8b bytes) 000066 1752688327.917869 read 0x00787c thru 0x0133d4 (0xbb59 bytes) 000067 1752688327.927450 trunc from 0x01cc2e to 0x0235f7 (0x69ca bytes) 000068 1752688327.931693 trunc from 0x0235f8 to 0x035572 (0x11f7b bytes) 000071 1752688327.942665 fallocate 0x003319 thru 0x004b30 (0x1818 bytes) 000072 1752688327.947539 trunc from 0x035573 to 0x00d0c0 (0x284b2 bytes) 000073 1752688327.956935 trunc from 0x009142 to 0x00d0c0 (0x3f7f bytes) 000074 1752688327.963973 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000075 1752688328.111804 mapread 0x006e42 thru 0x00d0c0 (0x627f bytes) 000076 1752688328.129545 mapread 0x00bed3 thru 0x00d0c0 (0x11ee bytes) 000077 1752688328.130245 mapread 0x0021c0 thru 0x00889b (0x66dc bytes) 000078 1752688328.130807 fallocate 0x02896f thru 0x034738 (0xbdca bytes) 000079 1752688328.136071 read 0x005a27 thru 0x00d0c0 (0x769a bytes) 000080 1752688328.137004 trunc from 0x00d0c1 to 0x00f82c (0x276c bytes) 000081 1752688328.159364 punch from 0x00a87e to 0x00f82c (0x4faf bytes) 000082 1752688328.164087 punch from 0x00666c to 0x00c094 (0x5a29 bytes) 000083 1752688328.167749 fallocate 0x036c6d thru 0x03e244 (0x75d8 bytes) 000085 1752688328.173419 punch from 0x000379 to 0x00c6d0 (0xc358 bytes) 000086 1752688328.181214 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000088 1752688328.276106 mapwrite 0x021671 thru 0x03049b (0xee2b bytes) 000089 1752688328.327563 mapread 0x007b34 thru 0x013942 (0xbe0f bytes) 000090 1752688328.341248 mapwrite 0x0244d8 thru 0x02df7c (0x9aa5 bytes) 000091 1752688328.355027 fallocate 0x02550f thru 0x02947f (0x3f71 bytes) 000092 1752688328.358458 fallocate 0x003bb3 thru 0x00cf39 (0x9387 bytes) 000093 1752688328.362102 mapread 0x02a869 thru 0x03049b (0x5c33 bytes) 000094 1752688328.362904 trunc from 0x021b8d to 0x026551 (0x49c5 bytes) 000096 1752688328.367728 trunc from 0x03049c to 0x03a6b3 (0xa218 bytes) 000098 1752688328.383926 mapwrite 0x020f85 thru 0x02a91c (0x9998 bytes) 000099 1752688328.405176 write 0x03785f thru 0x03f127 (0x78c9 bytes) 000100 1752688328.411913 write 0x02dbad thru 0x03c051 (0xe4a5 bytes) 000101 1752688328.413302 trunc from 0x01497e to 0x01c010 (0x7693 bytes) 000103 1752688328.418421 trunc from 0x005eeb to 0x00a572 (0x4688 bytes) 000104 1752688328.423067 mapwrite 0x00165a thru 0x00a0e7 (0x8a8e bytes) 000105 1752688328.444889 MIRROR_EXTEND to 2 mirrors 000106 1752688328.709975 trunc from 0x03f128 to 0x02d719 (0x11a0e bytes) 000107 1752688328.752420 punch from 0x00d5bd to 0x01ca54 (0xf498 bytes) 000108 1752688328.758038 mapread 0x01ceed thru 0x01fccd (0x2de1 bytes) 000109 1752688328.771080 punch from 0x019f0f to 0x01febf (0x5fb1 bytes) 000111 1752688328.779475 MIRROR_RESYNC 2 mirrors 000112 1752688329.025392 trunc from 0x02d71a to 0x03966a (0xbf51 bytes) 000113 1752688329.056572 read 0x01bf9b thru 0x0207b7 (0x481d bytes) 000114 1752688329.062197 mapread 0x009d2e thru 0x01759d (0xd870 bytes) 000115 1752688329.081499 write 0x02732a thru 0x0296e2 (0x23b9 bytes) 000116 1752688329.087794 read 0x018737 thru 0x021fcf (0x9899 bytes) 000117 1752688329.091561 MIRROR_EXTEND to 3 mirrors 000119 1752688329.393979 trunc from 0x00b767 to 0x018df0 (0xd68a bytes) 000120 1752688329.414782 trunc from 0x03966b to 0x0318ee (0x7d7c bytes) 000121 1752688329.437033 write 0x00f875 thru 0x01b1c5 (0xb951 bytes) 000122 1752688329.444949 read 0x028de3 thru 0x0318ee (0x8b0c bytes) 000123 1752688329.449536 read 0x010eb6 thru 0x01aed3 (0xa01e bytes) 000124 1752688329.450115 write 0x0130d8 thru 0x02055b (0xd484 bytes) 000125 1752688329.457540 punch from 0x0272ab to 0x02b9cb (0x4721 bytes) 000126 1752688329.463487 mapread 0x02ef3b thru 0x0318ee (0x29b4 bytes) 000127 1752688329.464340 write 0x03a55b thru 0x03ffff (0x5aa5 bytes) 000128 1752688329.465292 trunc from 0x0048cb to 0x00fa8d (0xb1c3 bytes) 000129 1752688329.479549 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000130 1752688329.622531 write 0x01d552 thru 0x024e94 (0x7943 bytes) 000131 1752688329.633102 mapread 0x01ddf9 thru 0x0204e3 (0x26eb bytes) 000132 1752688329.640785 trunc from 0x000c90 to 0x009905 (0x8c76 bytes) 000133 1752688329.644958 write 0x02ab65 thru 0x02da33 (0x2ecf bytes) 000135 1752688329.651554 trunc from 0x040000 to 0x012be1 (0x2d41e bytes) 000136 1752688329.664000 fallocate 0x03bedf thru 0x03bffe (0x120 bytes) 000137 1752688329.667518 punch from 0x005e3d to 0x012be1 (0xcda5 bytes) 000138 1752688329.671176 mapwrite 0x0362d0 thru 0x03ceb3 (0x6be4 bytes) 000139 1752688329.701878 read 0x01c7d1 thru 0x01f14f (0x297f bytes) 000140 1752688329.709824 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000142 1752688329.779617 trunc from 0x03ceb4 to 0x03f128 (0x2275 bytes) 000143 1752688329.791765 fallocate 0x020e82 thru 0x024ff3 (0x4172 bytes) 000144 1752688329.797727 read 0x011755 thru 0x020326 (0xebd2 bytes) 000145 1752688329.806367 fallocate 0x01ab7e thru 0x025b1b (0xaf9e bytes) 000146 1752688329.810412 read 0x016dbb thru 0x022ab2 (0xbcf8 bytes) 000147 1752688329.821016 read 0x01130b thru 0x01d3d0 (0xc0c6 bytes) 000148 1752688329.821673 fallocate 0x033d69 thru 0x03ffff (0xc297 bytes) 000149 1752688329.826398 punch from 0x02a17d to 0x03523f (0xb0c3 bytes) 000150 1752688329.830836 trunc from 0x03a555 to 0x03ffff (0x5aab bytes) 000151 1752688329.838952 MIRROR_EXTEND to 2 mirrors 000152 1752688330.074766 trunc from 0x01938e to 0x01f0f4 (0x5d67 bytes) 000153 1752688330.092673 mapwrite 0x009517 thru 0x00d089 (0x3b73 bytes) 000154 1752688330.112081 trunc from 0x040000 to 0x0072ab (0x38d54 bytes) 000155 1752688330.136737 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000156 1752688330.206803 fallocate 0x035e60 thru 0x03ffff (0xa1a0 bytes) 000157 1752688330.210472 mapread 0x005db7 thru 0x008a3f (0x2c89 bytes) 000158 1752688330.221449 punch from 0x014bb6 to 0x01717b (0x25c6 bytes) 000159 1752688330.227066 MIRROR_EXTEND to 2 mirrors 000160 1752688330.455809 fallocate 0x026296 thru 0x02a235 (0x3fa0 bytes) 000161 1752688330.491737 punch from 0x018f16 to 0x0198c4 (0x9af bytes) 000163 1752688330.498075 mapwrite 0x035c01 thru 0x03d0f4 (0x74f4 bytes) 000164 1752688330.531408 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000165 1752688330.643586 read 0x0073a9 thru 0x008183 (0xddb bytes) 000166 1752688330.648478 read 0x030592 thru 0x038388 (0x7df7 bytes) 000167 1752688330.655526 punch from 0x0155b7 to 0x020932 (0xb37c bytes) 000168 1752688330.660757 trunc from 0x03f529 to 0x03ffff (0xad7 bytes) 000169 1752688330.664117 mapwrite 0x02d251 thru 0x033030 (0x5de0 bytes) 000170 1752688330.677598 trunc from 0x040000 to 0x010ead (0x2f152 bytes) 000172 1752688330.695160 trunc from 0x010eae to 0x0079f9 (0x94b4 bytes) 000176 1752688330.711086 trunc from 0x004544 to 0x0079f9 (0x34b6 bytes) 000178 1752688330.715385 trunc from 0x0079fa to 0x0295b2 (0x21bb9 bytes) 000179 1752688330.725429 trunc from 0x02007c to 0x02350f (0x3494 bytes) 000180 1752688330.728990 read 0x020c4d thru 0x0295b2 (0x8966 bytes) 000183 1752688330.736007 write 0x037579 thru 0x03ffff (0x8a87 bytes) 000184 1752688330.737606 read 0x03b195 thru 0x03ffff (0x4e6b bytes) 000185 1752688330.738150 trunc from 0x021b29 to 0x0269f1 (0x4ec9 bytes) 000186 1752688330.741872 punch from 0x00d0dd to 0x0128d8 (0x57fc bytes) 000187 1752688330.745417 fallocate 0x039137 thru 0x03ef34 (0x5dfe bytes) 000189 1752688330.749291 write 0x020def thru 0x02f92c (0xeb3e bytes) 000190 1752688330.756048 punch from 0x0357b0 to 0x03ec50 (0x94a1 bytes) 000192 1752688330.766713 read 0x03dde6 thru 0x03ffff (0x221a bytes) 000193 1752688330.773710 write 0x026686 thru 0x0286a7 (0x2022 bytes) 000194 1752688330.774643 mapread 0x03114d thru 0x03afa9 (0x9e5d bytes) 000195 1752688330.775544 punch from 0x02efb9 to 0x0364b2 (0x74fa bytes) 000196 1752688330.779274 fallocate 0x00adbe thru 0x01685f (0xbaa2 bytes) 000197 1752688330.782593 trunc from 0x040000 to 0x0000fa (0x3ff05 bytes) 000198 1752688330.791683 trunc from 0x0000fb to 0x0117a0 (0x116a6 bytes) 000201 1752688330.806071 read 0x000da3 thru 0x005545 (0x47a3 bytes) 000202 1752688330.812451 fallocate 0x01248c thru 0x019cb0 (0x7825 bytes) 000203 1752688330.815977 trunc from 0x01143b to 0x0117a0 (0x366 bytes) 000204 1752688330.819830 fallocate 0x0245ab thru 0x02ef04 (0xa95a bytes) 000205 1752688330.823673 write 0x000358 thru 0x0094e8 (0x9191 bytes) 000206 1752688330.824885 fallocate 0x01fc27 thru 0x02e15d (0xe537 bytes) 000207 1752688330.828622 trunc from 0x015d9e to 0x017aae (0x1d11 bytes) 000208 1752688330.832169 write 0x0396de thru 0x03ffff (0x6922 bytes) 000209 1752688330.833305 trunc from 0x00afb7 to 0x015ed7 (0xaf21 bytes) 000210 1752688330.836541 write 0x02eafa thru 0x03a28a (0xb791 bytes) 000211 1752688330.842984 fallocate 0x032215 thru 0x03bb65 (0x9951 bytes) 000212 1752688330.846854 trunc from 0x034e10 to 0x03dea9 (0x909a bytes) 000213 1752688330.857028 trunc from 0x00aaf2 to 0x00af40 (0x44f bytes) 000214 1752688330.860761 punch from 0x01a2f6 to 0x01f634 (0x533f bytes) 000215 1752688330.864173 mapwrite 0x012876 thru 0x01b512 (0x8c9d bytes) 000216 1752688330.887740 punch from 0x026793 to 0x02d2d8 (0x6b46 bytes) 000217 1752688330.891507 write 0x007501 thru 0x015afa (0xe5fa bytes) 000219 1752688330.893717 punch from 0x015d1f to 0x016444 (0x726 bytes) 000221 1752688330.903912 mapwrite 0x021b1a thru 0x02467f (0x2b66 bytes) 000222 1752688330.914022 mapwrite 0x0204f0 thru 0x020827 (0x338 bytes) 000223 1752688330.925274 read 0x028981 thru 0x036b08 (0xe188 bytes) 000224 1752688330.932404 fallocate 0x02f26b thru 0x036db2 (0x7b48 bytes) 000226 1752688330.937625 fallocate 0x01f4cb thru 0x0260da (0x6c10 bytes) 000227 1752688330.941802 mapread 0x00a98d thru 0x00acd0 (0x344 bytes) 000228 1752688330.943320 read 0x0332ae thru 0x03f5a3 (0xc2f6 bytes) 000229 1752688330.943906 fallocate 0x01daab thru 0x026620 (0x8b76 bytes) 000230 1752688330.950532 MIRROR_EXTEND to 2 mirrors 000231 1752688331.247836 fallocate 0x0194bf thru 0x020b3c (0x767e bytes) 000232 1752688331.271926 read 0x00e83c thru 0x011ea2 (0x3667 bytes) 000233 1752688331.277092 punch from 0x00324f to 0x00b988 (0x873a bytes) 000234 1752688331.286921 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000235 1752688331.380617 MIRROR_EXTEND to 2 mirrors 000236 1752688331.687691 mapwrite 0x00ba6f thru 0x015242 (0x97d4 bytes) 000237 1752688331.747615 write 0x012980 thru 0x017570 (0x4bf1 bytes) 000238 1752688331.751592 fallocate 0x00300f thru 0x01016b (0xd15d bytes) 000239 1752688331.757031 mapwrite 0x03472d thru 0x03f2e0 (0xabb4 bytes) 000240 1752688331.779760 fallocate 0x02f2c7 thru 0x0301aa (0xee4 bytes) 000241 1752688331.786244 MIRROR_RESYNC 2 mirrors 000242 1752688332.039308 write 0x009a6c thru 0x00ac1d (0x11b2 bytes) 000243 1752688332.056799 read 0x00e69f thru 0x010755 (0x20b7 bytes) 000244 1752688332.059919 trunc from 0x040000 to 0x00b20f (0x34df0 bytes) 000245 1752688332.078438 fallocate 0x03d5c7 thru 0x03ffff (0x2a39 bytes) 000246 1752688332.081402 write 0x0104a6 thru 0x016dc8 (0x6923 bytes) 000247 1752688332.082426 punch from 0x003519 to 0x013376 (0xfe5e bytes) 000248 1752688332.090703 fallocate 0x015dad thru 0x01f70c (0x9960 bytes) 000249 1752688332.094520 fallocate 0x025b26 thru 0x034079 (0xe554 bytes) 000250 1752688332.098111 punch from 0x029e19 to 0x02ea78 (0x4c60 bytes) 000251 1752688332.101345 punch from 0x02464a to 0x02d9e1 (0x9398 bytes) 000252 1752688332.105029 read 0x01df06 thru 0x02c9ed (0xeae8 bytes) 000253 1752688332.112723 trunc from 0x02a5c2 to 0x034079 (0x9ab8 bytes) 000254 1752688332.119870 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000255 1752688332.186249 fallocate 0x02288a thru 0x02f9d8 (0xd14f bytes) 000256 1752688332.189664 read 0x010974 thru 0x010ab9 (0x146 bytes) 000257 1752688332.192456 fallocate 0x0238ea thru 0x0315bc (0xdcd3 bytes) 000258 1752688332.195554 trunc from 0x03407a to 0x03fe60 (0xbde7 bytes) 000259 1752688332.203989 trunc from 0x00b300 to 0x00badf (0x7e0 bytes) 000260 1752688332.206744 mapwrite 0x0078df thru 0x00c757 (0x4e79 bytes) 000262 1752688332.228777 write 0x03cf56 thru 0x03ffff (0x30aa bytes) 000263 1752688332.234186 trunc from 0x00d71f to 0x01ad88 (0xd66a bytes) 000264 1752688332.237805 mapwrite 0x0152ad thru 0x0158ef (0x643 bytes) 000265 1752688332.254584 read 0x023685 thru 0x025137 (0x1ab3 bytes) 000266 1752688332.260940 mapwrite 0x018c21 thru 0x01de0e (0x51ee bytes) 000267 1752688332.273687 trunc from 0x020687 to 0x029dbc (0x9736 bytes) 000268 1752688332.277456 trunc from 0x0112fb to 0x01cd37 (0xba3d bytes) 000270 1752688332.281039 write 0x02c00c thru 0x03b9b8 (0xf9ad bytes) 000271 1752688332.282459 punch from 0x00bc90 to 0x019486 (0xd7f7 bytes) 000272 1752688332.287467 punch from 0x0113cb to 0x0135ed (0x2223 bytes) 000273 1752688332.292641 trunc from 0x040000 to 0x035b1b (0xa4e4 bytes) 000275 1752688332.305741 trunc from 0x0354f2 to 0x035b1b (0x62a bytes) 000277 1752688332.317418 fallocate 0x00cd8a thru 0x013ee0 (0x7157 bytes) 000278 1752688332.322875 write 0x00c136 thru 0x0196bb (0xd586 bytes) 000280 1752688332.331842 read 0x011382 thru 0x01d929 (0xc5a8 bytes) 000281 1752688332.339364 punch from 0x026738 to 0x035835 (0xf0fe bytes) 000282 1752688332.344115 trunc from 0x035b1c to 0x00df96 (0x27b85 bytes) 000283 1752688332.357692 mapread 0x00c1fd thru 0x00df96 (0x1d9a bytes) 000284 1752688332.358584 fallocate 0x02c7d2 thru 0x037f64 (0xb793 bytes) 000285 1752688332.363878 mapread 0x00fdb1 thru 0x016044 (0x6294 bytes) 000286 1752688332.396866 trunc from 0x0320df to 0x036f51 (0x4e73 bytes) 000287 1752688332.404242 MIRROR_EXTEND to 2 mirrors 000288 1752688332.637097 trunc from 0x037f65 to 0x01d0cb (0x1ae99 bytes) 000289 1752688332.665007 fallocate 0x02c139 thru 0x037ca5 (0xbb6d bytes) 000290 1752688332.668480 mapread 0x015621 thru 0x01978b (0x416b bytes) 000291 1752688332.677248 read 0x017cd5 thru 0x01cf69 (0x5295 bytes) 000292 1752688332.680384 MIRROR_RESYNC 2 mirrors 000293 1752688332.842952 write 0x01105a thru 0x01a5de (0x9585 bytes) 000294 1752688332.867278 mapread 0x007948 thru 0x0092c6 (0x197f bytes) 000295 1752688332.873541 trunc from 0x037ca6 to 0x0360ce (0x1bd7 bytes) 000296 1752688332.892039 fallocate 0x025dc0 thru 0x02fd2c (0x9f6d bytes) 000297 1752688332.895800 fallocate 0x023d63 thru 0x02a706 (0x69a4 bytes) 000298 1752688332.899670 read 0x0072a9 thru 0x00f856 (0x85ae bytes) 000299 1752688332.900285 mapwrite 0x03c20b thru 0x03ffff (0x3df5 bytes) 000300 1752688332.927314 read 0x032328 thru 0x03ef64 (0xcc3d bytes) 000301 1752688332.932634 mapread 0x009b78 thru 0x00f940 (0x5dc9 bytes) 000302 1752688332.936531 MIRROR_EXTEND to 3 mirrors 000303 1752688333.192246 trunc from 0x01fb3a to 0x024645 (0x4b0c bytes) 000304 1752688333.210424 read 0x0058fb thru 0x015586 (0xfc8c bytes) 000305 1752688333.214994 punch from 0x02e988 to 0x0306cd (0x1d46 bytes) 000306 1752688333.220093 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000307 1752688333.329355 punch from 0x005acb to 0x01369a (0xdbd0 bytes) 000308 1752688333.332746 punch from 0x02dadc to 0x034dd8 (0x72fd bytes) 000309 1752688333.337584 read 0x003c0c thru 0x00b4eb (0x78e0 bytes) 000310 1752688333.342856 punch from 0x012db7 to 0x013f80 (0x11ca bytes) 000311 1752688333.347875 fallocate 0x029d09 thru 0x032b1f (0x8e17 bytes) 000312 1752688333.353261 trunc from 0x01eb91 to 0x021bd8 (0x3048 bytes) 000313 1752688333.357836 mapread 0x038785 thru 0x03ffff (0x787b bytes) 000314 1752688333.369380 mapwrite 0x01d988 thru 0x02d289 (0xf902 bytes) 000315 1752688333.394187 mapread 0x03af4d thru 0x03d598 (0x264c bytes) 000316 1752688333.394984 mapwrite 0x005f1d thru 0x0113e6 (0xb4ca bytes) 000317 1752688333.416270 MIRROR_RESYNC 2 mirrors 000318 1752688333.790350 mapread 0x00b20c thru 0x013f66 (0x8d5b bytes) 000319 1752688333.803890 punch from 0x00eb90 to 0x01a6fe (0xbb6f bytes) 000320 1752688333.830144 fallocate 0x01549f thru 0x02072d (0xb28f bytes) 000321 1752688333.835045 trunc from 0x040000 to 0x03b34d (0x4cb2 bytes) 000323 1752688333.861387 write 0x00084c thru 0x009358 (0x8b0d bytes) 000324 1752688333.867919 fallocate 0x032eb2 thru 0x03c8cb (0x9a1a bytes) 000325 1752688333.871872 fallocate 0x037ebb thru 0x03ffff (0x8145 bytes) 000326 1752688333.876601 fallocate 0x0157d5 thru 0x018449 (0x2c75 bytes) 000327 1752688333.880455 mapread 0x037508 thru 0x03ffff (0x8af8 bytes) 000328 1752688333.887113 write 0x0142f5 thru 0x0168aa (0x25b6 bytes) 000329 1752688333.894151 read 0x01586d thru 0x022a57 (0xd1eb bytes) 000330 1752688333.901059 write 0x00e145 thru 0x0182c1 (0xa17d bytes) 000331 1752688333.902809 mapwrite 0x00fad8 thru 0x014d86 (0x52af bytes) 000334 1752688333.919428 trunc from 0x0142b8 to 0x01ad67 (0x6ab0 bytes) 000335 1752688333.923710 mapwrite 0x010efe thru 0x011b3c (0xc3f bytes) 000336 1752688333.933143 trunc from 0x040000 to 0x0188db (0x27724 bytes) 000338 1752688333.944951 mapwrite 0x00f586 thru 0x018150 (0x8bcb bytes) 000339 1752688333.964490 read 0x005402 thru 0x00faf6 (0xa6f5 bytes) 000340 1752688333.965181 trunc from 0x0048ba to 0x00a37c (0x5ac3 bytes) 000341 1752688333.968931 punch from 0x0143c1 to 0x01453d (0x17d bytes) 000342 1752688333.972036 trunc from 0x00b309 to 0x015ae8 (0xa7e0 bytes) 000344 1752688333.978652 MIRROR_EXTEND to 3 mirrors 000345 1752688334.307472 write 0x023d2a thru 0x02bec6 (0x819d bytes) 000346 1752688334.327743 trunc from 0x02bec7 to 0x0113b9 (0x1ab0d bytes) 000347 1752688334.338715 fallocate 0x01835e thru 0x022a9c (0xa73f bytes) 000348 1752688334.342699 trunc from 0x00fe28 to 0x0129bc (0x2b95 bytes) 000349 1752688334.345981 read 0x01d505 thru 0x022a9c (0x5598 bytes) 000350 1752688334.351552 punch from 0x0058e0 to 0x007633 (0x1d54 bytes) 000351 1752688334.355379 write 0x03969b thru 0x03ffff (0x6965 bytes) 000352 1752688334.356598 mapwrite 0x016d14 thru 0x01f566 (0x8853 bytes) 000353 1752688334.384960 punch from 0x03586c to 0x0381a9 (0x293e bytes) 000354 1752688334.391103 mapread 0x03e0de thru 0x03ffff (0x1f22 bytes) 000356 1752688334.393612 mapwrite 0x027cc3 thru 0x0282a3 (0x5e1 bytes) 000357 1752688334.413808 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000358 1752688334.535864 trunc from 0x03191a to 0x037681 (0x5d68 bytes) 000359 1752688334.539399 read 0x03ee97 thru 0x03ffff (0x1169 bytes) 000360 1752688334.544872 mapwrite 0x0072da thru 0x00b1b7 (0x3ede bytes) 000361 1752688334.565621 trunc from 0x040000 to 0x00c774 (0x3388b bytes) 000362 1752688334.575094 trunc from 0x0008ea to 0x00c774 (0xbe8b bytes) 000363 1752688334.581305 MIRROR_RESYNC 2 mirrors 000364 1752688334.753412 trunc from 0x009339 to 0x00c774 (0x343c bytes) 000365 1752688334.778838 trunc from 0x00c775 to 0x01a8eb (0xe177 bytes) 000366 1752688334.807746 trunc from 0x0147ed to 0x01a8eb (0x60ff bytes) 000367 1752688334.811362 read 0x007e64 thru 0x017773 (0xf910 bytes) 000368 1752688334.818832 fallocate 0x0371cb thru 0x03ffff (0x8e35 bytes) 000369 1752688334.828379 MIRROR_RESYNC 2 mirrors 000370 1752688335.030662 trunc from 0x03374b to 0x0357e4 (0x209a bytes) 000371 1752688335.049733 read 0x007698 thru 0x00b82e (0x4197 bytes) 000372 1752688335.054890 read 0x033852 thru 0x035a0f (0x21be bytes) 000373 1752688335.065297 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000374 1752688335.161255 trunc from 0x040000 to 0x030991 (0xf66e bytes) 000375 1752688335.196012 trunc from 0x030992 to 0x027847 (0x914a bytes) 000376 1752688335.213852 MIRROR_EXTEND to 2 mirrors 000377 1752688335.521962 MIRROR_RESYNC 2 mirrors 000378 1752688335.609529 write 0x00d575 thru 0x0198fd (0xc389 bytes) 000382 1752688335.639715 mapread 0x0055b9 thru 0x00a35a (0x4da2 bytes) 000383 1752688335.647889 punch from 0x002894 to 0x00bb32 (0x929f bytes) 000384 1752688335.651270 mapwrite 0x02113c thru 0x02965c (0x8521 bytes) 000385 1752688335.701228 punch from 0x01db2f to 0x01e6a4 (0xb76 bytes) 000386 1752688335.704421 trunc from 0x01e575 to 0x025059 (0x6ae5 bytes) 000388 1752688335.709315 punch from 0x021ce1 to 0x02965c (0x797c bytes) 000389 1752688335.715472 MIRROR_RESYNC 2 mirrors 000390 1752688335.946724 trunc from 0x02965d to 0x029585 (0xd7 bytes) 000391 1752688335.989892 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000392 1752688336.078903 MIRROR_EXTEND to 2 mirrors 000394 1752688336.356781 write 0x02a0b8 thru 0x02ee64 (0x4dad bytes) 000395 1752688336.377865 mapwrite 0x02a718 thru 0x03390f (0x91f8 bytes) 000396 1752688336.415698 punch from 0x017662 to 0x021a4c (0xa3eb bytes) 000397 1752688336.421643 mapwrite 0x01c554 thru 0x02480b (0x82b8 bytes) 000398 1752688336.439368 mapread 0x02254d thru 0x02eb5a (0xc60e bytes) 000399 1752688336.445007 write 0x025dfc thru 0x02eef6 (0x90fb bytes) 000400 1752688336.448472 MIRROR_EXTEND to 3 mirrors 000401 1752688336.698174 MIRROR_EXTEND to 4 mirrors 000402 1752688336.982874 MIRROR_EXTEND to 5 mirrors 000403 1752688337.352621 mapread 0x0141fe thru 0x02199d (0xd7a0 bytes) 000404 1752688337.374050 punch from 0x00f07d to 0x015377 (0x62fb bytes) 000405 1752688337.402638 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000406 1752688337.515773 mapread 0x029d61 thru 0x03390f (0x9baf bytes) 000407 1752688337.530433 write 0x0342b0 thru 0x03daeb (0x983c bytes) 000408 1752688337.536261 write 0x005aa4 thru 0x0155ae (0xfb0b bytes) 000409 1752688337.548664 mapwrite 0x02976b thru 0x032a02 (0x9298 bytes) 000410 1752688337.567483 write 0x02566d thru 0x029dd9 (0x476d bytes) 000411 1752688337.572066 MIRROR_EXTEND to 5 mirrors 000412 1752688337.825510 trunc from 0x03daec to 0x0195d5 (0x24516 bytes) 000414 1752688337.866425 read 0x0134a4 thru 0x01572b (0x2288 bytes) 000415 1752688337.870463 read 0x0033d2 thru 0x007dc8 (0x49f7 bytes) 000416 1752688337.877068 mapread 0x008400 thru 0x00d330 (0x4f31 bytes) 000417 1752688337.878081 fallocate 0x030938 thru 0x03f046 (0xe70f bytes) 000418 1752688337.882373 mapwrite 0x02204e thru 0x025a09 (0x39bc bytes) 000419 1752688337.900353 read 0x00b34c thru 0x00c41f (0x10d4 bytes) 000420 1752688337.901206 mapread 0x022fe0 thru 0x02cfbc (0x9fdd bytes) 000421 1752688337.902226 mapread 0x00313e thru 0x005b3f (0x2a02 bytes) 000422 1752688337.902980 write 0x02eff0 thru 0x031bb1 (0x2bc2 bytes) 000423 1752688337.906763 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000424 1752688338.053734 trunc from 0x03f047 to 0x03111b (0xdf2b bytes) 000425 1752688338.069207 read 0x017a63 thru 0x01e7e6 (0x6d84 bytes) 000426 1752688338.075703 read 0x006e24 thru 0x01244f (0xb62c bytes) 000427 1752688338.086410 trunc from 0x03111c to 0x007561 (0x29bba bytes) 000428 1752688338.104060 MIRROR_EXTEND to 5 mirrors 000429 1752688338.339218 MIRROR_RESYNC 5 mirrors 000430 1752688338.633396 trunc from 0x0047a7 to 0x007561 (0x2dbb bytes) 000431 1752688338.658481 MIRROR_RESYNC 5 mirrors 000432 1752688338.998438 mapwrite 0x035299 thru 0x03af0e (0x5c76 bytes) 000433 1752688339.050477 mapwrite 0x00fd19 thru 0x0135c4 (0x38ac bytes) 000434 1752688339.068261 trunc from 0x016aef to 0x0187cd (0x1cdf bytes) 000435 1752688339.072400 write 0x0052bf thru 0x009a9a (0x47dc bytes) 000436 1752688339.079891 fallocate 0x001aa2 thru 0x006a9b (0x4ffa bytes) 000437 1752688339.084407 read 0x0317a8 thru 0x0360cf (0x4928 bytes) 000438 1752688339.084984 punch from 0x01e3f4 to 0x029f51 (0xbb5e bytes) 000440 1752688339.088813 fallocate 0x032bf3 thru 0x03f930 (0xcd3e bytes) 000442 1752688339.093331 punch from 0x003575 to 0x00c1ff (0x8c8b bytes) 000443 1752688339.102718 write 0x03615d thru 0x03ffff (0x9ea3 bytes) 000444 1752688339.103997 trunc from 0x03cce7 to 0x03ffff (0x3319 bytes) 000445 1752688339.112043 mapread 0x008690 thru 0x011ea0 (0x9811 bytes) 000446 1752688339.118486 read 0x02720f thru 0x03685e (0xf650 bytes) 000447 1752688339.124476 mapread 0x0186e1 thru 0x01df25 (0x5845 bytes) 000448 1752688339.125325 mapwrite 0x0338f4 thru 0x03e834 (0xaf41 bytes) 000449 1752688339.137807 mapread 0x01233d thru 0x019468 (0x712c bytes) 000450 1752688339.138775 punch from 0x017333 to 0x0251a7 (0xde75 bytes) 000451 1752688339.142470 trunc from 0x00db7e to 0x01b0c0 (0xd543 bytes) 000452 1752688339.146251 fallocate 0x0296d0 thru 0x02c4ca (0x2dfb bytes) 000453 1752688339.150385 mapread 0x0189a7 thru 0x021cab (0x9305 bytes) 000454 1752688339.180407 write 0x019947 thru 0x021e4f (0x8509 bytes) 000455 1752688339.181785 trunc from 0x040000 to 0x026616 (0x199e9 bytes) 000456 1752688339.194485 trunc from 0x026617 to 0x007716 (0x1ef00 bytes) 000457 1752688339.203203 fallocate 0x0266df thru 0x02a998 (0x42ba bytes) 000458 1752688339.206506 trunc from 0x006a63 to 0x007716 (0xcb4 bytes) 000459 1752688339.209392 write 0x027162 thru 0x03460e (0xd4ad bytes) 000460 1752688339.210948 fallocate 0x024469 thru 0x0297be (0x5356 bytes) 000462 1752688339.215028 mapread 0x02333e thru 0x02705a (0x3d1d bytes) 000463 1752688339.227966 punch from 0x00ca24 to 0x013532 (0x6b0f bytes) 000464 1752688339.231827 fallocate 0x023751 thru 0x028da2 (0x5652 bytes) 000465 1752688339.235522 trunc from 0x03460f to 0x013cc2 (0x2094c bytes) 000467 1752688339.245270 mapwrite 0x01df07 thru 0x023153 (0x524d bytes) 000468 1752688339.290989 mapwrite 0x02f6d8 thru 0x032c10 (0x3539 bytes) 000470 1752688339.329223 read 0x006f87 thru 0x011618 (0xa692 bytes) 000471 1752688339.334967 trunc from 0x032c11 to 0x000559 (0x326b7 bytes) 000473 1752688339.346075 mapwrite 0x012ff2 thru 0x01e518 (0xb527 bytes) 000474 1752688339.418208 trunc from 0x01e519 to 0x0372cf (0x18db7 bytes) 000475 1752688339.427964 punch from 0x0171ef to 0x0245a3 (0xd3b5 bytes) 000476 1752688339.431933 punch from 0x01e7a3 to 0x02b6cf (0xcf2d bytes) 000477 1752688339.435285 mapwrite 0x0152e1 thru 0x021b52 (0xc872 bytes) 000478 1752688339.486935 MIRROR_RESYNC 5 mirrors 000479 1752688339.897746 write 0x01070e thru 0x018c24 (0x8517 bytes) 000480 1752688339.936214 read 0x016b0c thru 0x01c9f3 (0x5ee8 bytes) 000481 1752688339.940999 trunc from 0x0372d0 to 0x0092d2 (0x2dffd bytes) 000482 1752688339.966781 trunc from 0x0089f9 to 0x0092d2 (0x8da bytes) 000483 1752688339.973551 MIRROR_RESYNC 5 mirrors 000484 1752688340.324801 mapwrite 0x03102d thru 0x03ffff (0xefd3 bytes) 000485 1752688340.396994 trunc from 0x0269f6 to 0x034790 (0xdd9b bytes) 000486 1752688340.400993 mapread 0x015bee thru 0x02225a (0xc66d bytes) 000487 1752688340.411694 punch from 0x00143b to 0x0104b8 (0xf07e bytes) 000488 1752688340.420063 MIRROR_EXTEND to 6 mirrors 000490 1752688340.672746 trunc from 0x040000 to 0x02293f (0x1d6c0 bytes) 000491 1752688340.714282 trunc from 0x01f599 to 0x02293f (0x33a7 bytes) 000492 1752688340.718656 write 0x0385ea thru 0x03ffff (0x7a16 bytes) 000493 1752688340.719939 write 0x00cd53 thru 0x019ae8 (0xcd96 bytes) 000494 1752688340.730501 mapread 0x02929c thru 0x032cbf (0x9a24 bytes) 000495 1752688340.760894 mapwrite 0x018eb3 thru 0x023304 (0xa452 bytes) 000496 1752688340.785246 trunc from 0x00909e to 0x00ad0f (0x1c72 bytes) 000497 1752688340.790726 trunc from 0x03cab6 to 0x03ffff (0x354a bytes) 000498 1752688340.795844 trunc from 0x0014a4 to 0x00734f (0x5eac bytes) 000499 1752688340.805429 fallocate 0x03eed6 thru 0x03ffff (0x112a bytes) 000500 1752688340.814292 MIRROR_EXTEND to 7 mirrors 000501 1752688341.198729 trunc from 0x03bd71 to 0x03ffff (0x428f bytes) 000502 1752688341.226171 mapwrite 0x025da8 thru 0x032aab (0xcd04 bytes) 000503 1752688341.259325 trunc from 0x019a0e to 0x01eb3d (0x5130 bytes) 000504 1752688341.262868 mapread 0x035319 thru 0x03573c (0x424 bytes) 000505 1752688341.263840 write 0x005c49 thru 0x00ab05 (0x4ebd bytes) 000506 1752688341.271720 mapread 0x00d98c thru 0x01634c (0x89c1 bytes) 000507 1752688341.277911 mapread 0x00ead4 thru 0x0151cd (0x66fa bytes) 000508 1752688341.278907 fallocate 0x03dbb9 thru 0x03ffff (0x2447 bytes) 000509 1752688341.282747 read 0x0171db thru 0x01786c (0x692 bytes) 000510 1752688341.283312 write 0x0350cc thru 0x03f4df (0xa414 bytes) 000511 1752688341.288073 mapwrite 0x03d43d thru 0x03ffff (0x2bc3 bytes) 000512 1752688341.305610 trunc from 0x017dcb to 0x02681a (0xea50 bytes) 000513 1752688341.311582 mapread 0x02fb27 thru 0x038fc4 (0x949e bytes) 000514 1752688341.312616 mapread 0x029640 thru 0x02d9cc (0x438d bytes) 000515 1752688341.313478 trunc from 0x040000 to 0x028bac (0x17453 bytes) 000516 1752688341.332862 write 0x007e04 thru 0x00ed9f (0x6f9c bytes) 000517 1752688341.334283 punch from 0x0224cd to 0x0238d1 (0x1405 bytes) 000518 1752688341.339894 trunc from 0x028bad to 0x0009d8 (0x281d4 bytes) 000519 1752688341.351564 fallocate 0x01d60f thru 0x025c4b (0x863d bytes) 000520 1752688341.356252 mapread 0x0002c7 thru 0x0009d8 (0x712 bytes) 000521 1752688341.361631 read 0x000141 thru 0x0009d8 (0x898 bytes) 000522 1752688341.362276 trunc from 0x0009d9 to 0x007223 (0x684b bytes) 000524 1752688341.374544 mapread 0x0017af thru 0x007223 (0x5a75 bytes) 000525 1752688341.381981 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000526 1752688341.515272 trunc from 0x007224 to 0x01744a (0x10227 bytes) 000528 1752688341.537566 trunc from 0x01277d to 0x01744a (0x4cce bytes) 000529 1752688341.543745 fallocate 0x022f70 thru 0x02917e (0x620f bytes) 000531 1752688341.550553 trunc from 0x02917f to 0x02e296 (0x5118 bytes) 000532 1752688341.564452 mapwrite 0x003476 thru 0x00d71c (0xa2a7 bytes) 000533 1752688341.595511 fallocate 0x00c4b1 thru 0x01031c (0x3e6c bytes) 000534 1752688341.601965 trunc from 0x02e297 to 0x015789 (0x18b0d bytes) 000535 1752688341.617617 MIRROR_RESYNC 6 mirrors 000536 1752688342.336489 mapwrite 0x03db96 thru 0x03ffff (0x246a bytes) 000537 1752688342.411355 trunc from 0x0366ba to 0x03ffff (0x9946 bytes) 000538 1752688342.418226 write 0x01d051 thru 0x01f1bd (0x216d bytes) 000539 1752688342.430762 mapwrite 0x03ac0e thru 0x03ffff (0x53f2 bytes) 000540 1752688342.482688 mapwrite 0x0369fb thru 0x03ffff (0x9605 bytes) 000541 1752688342.506799 fallocate 0x01eab2 thru 0x021c79 (0x31c8 bytes) 000542 1752688342.511362 read 0x00a6bb thru 0x017c27 (0xd56d bytes) 000543 1752688342.529371 mapread 0x033c1b thru 0x03c34b (0x8731 bytes) 000544 1752688342.530223 punch from 0x005352 to 0x00ceb1 (0x7b60 bytes) 000545 1752688342.543486 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000546 1752688342.676859 MIRROR_EXTEND to 6 mirrors 000547 1752688343.087181 trunc from 0x03ded6 to 0x03e793 (0x8be bytes) 000548 1752688343.122041 mapread 0x0167e0 thru 0x0198f8 (0x3119 bytes) 000549 1752688343.141244 mapwrite 0x014a20 thru 0x01625a (0x183b bytes) 000550 1752688343.163193 trunc from 0x040000 to 0x020064 (0x1ff9b bytes) 000551 1752688343.205881 punch from 0x007edd to 0x010295 (0x83b9 bytes) 000552 1752688343.209662 trunc from 0x020065 to 0x01468e (0xb9d6 bytes) 000554 1752688343.222522 fallocate 0x000c66 thru 0x00ca3f (0xbdda bytes) 000555 1752688343.226911 punch from 0x01344c to 0x01468e (0x1243 bytes) 000556 1752688343.230620 fallocate 0x0339b2 thru 0x03ce35 (0x9484 bytes) 000557 1752688343.235528 mapwrite 0x038f39 thru 0x03a5a7 (0x166f bytes) 000559 1752688343.254182 trunc from 0x03ce36 to 0x0262e2 (0x16b53 bytes) 000560 1752688343.263346 mapwrite 0x00ed9c thru 0x01a463 (0xb6c8 bytes) 000561 1752688343.289591 write 0x0028f2 thru 0x00e288 (0xb997 bytes) 000562 1752688343.295247 read 0x01d201 thru 0x020ea3 (0x3ca3 bytes) 000563 1752688343.299340 punch from 0x022a7c to 0x0262e2 (0x3867 bytes) 000564 1752688343.302415 punch from 0x004888 to 0x00b5f3 (0x6d6c bytes) 000565 1752688343.316834 mapread 0x011509 thru 0x01aeed (0x99e5 bytes) 000566 1752688343.317946 mapread 0x009d2a thru 0x0111c3 (0x749a bytes) 000567 1752688343.325050 fallocate 0x037c57 thru 0x03fab2 (0x7e5c bytes) 000568 1752688343.329735 mapread 0x023441 thru 0x032f46 (0xfb06 bytes) 000569 1752688343.346840 trunc from 0x00d3de to 0x010253 (0x2e76 bytes) 000571 1752688343.352030 fallocate 0x008c2c thru 0x0106c3 (0x7a98 bytes) 000572 1752688343.355613 mapwrite 0x00b5ad thru 0x017e15 (0xc869 bytes) 000573 1752688343.376794 read 0x00576e thru 0x0106c3 (0xaf56 bytes) 000574 1752688343.377609 mapread 0x015785 thru 0x01a72b (0x4fa7 bytes) 000575 1752688343.378520 read 0x0377e6 thru 0x03fab2 (0x82cd bytes) 000576 1752688343.386971 punch from 0x00277d to 0x007b17 (0x539b bytes) 000577 1752688343.391110 mapwrite 0x0039f8 thru 0x0137b4 (0xfdbd bytes) 000578 1752688343.431017 fallocate 0x03caa8 thru 0x03ffff (0x3558 bytes) 000580 1752688343.435753 read 0x0225fb thru 0x02d4ba (0xaec0 bytes) 000581 1752688343.436414 trunc from 0x040000 to 0x024498 (0x1bb67 bytes) 000582 1752688343.447966 read 0x017187 thru 0x01cfc2 (0x5e3c bytes) 000583 1752688343.448747 mapread 0x0018e7 thru 0x00da44 (0xc15e bytes) 000584 1752688343.453478 trunc from 0x000928 to 0x002277 (0x1950 bytes) 000585 1752688343.462054 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000586 1752688343.620435 punch from 0x002fc6 to 0x012da7 (0xfde2 bytes) 000587 1752688343.626232 write 0x003d84 thru 0x00530c (0x1589 bytes) 000588 1752688343.641958 read 0x00b6aa thru 0x00d4da (0x1e31 bytes) 000589 1752688343.648830 mapread 0x00e9bb thru 0x00fe24 (0x146a bytes) 000590 1752688343.659575 MIRROR_RESYNC 5 mirrors 000591 1752688344.355323 write 0x02ac6c thru 0x03001d (0x53b2 bytes) 000592 1752688344.376067 punch from 0x0093c1 to 0x0191f9 (0xfe39 bytes) 000593 1752688344.380442 fallocate 0x039490 thru 0x03ffff (0x6b70 bytes) 000594 1752688344.384942 fallocate 0x00e7d0 thru 0x01c596 (0xddc7 bytes) 000595 1752688344.389423 trunc from 0x029560 to 0x02d8cb (0x436c bytes) 000596 1752688344.400344 trunc from 0x03001e to 0x011e22 (0x1e1fb bytes) 000598 1752688344.418600 mapread 0x004e0a thru 0x0085fa (0x37f1 bytes) 000599 1752688344.437475 mapwrite 0x02ba2f thru 0x037f44 (0xc516 bytes) 000600 1752688344.480449 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000601 1752688344.604092 write 0x02de29 thru 0x03c6e8 (0xe8c0 bytes) 000602 1752688344.611813 trunc from 0x00a850 to 0x0171fd (0xc9ae bytes) 000603 1752688344.616206 punch from 0x02e298 to 0x030747 (0x24b0 bytes) 000604 1752688344.627624 punch from 0x02a283 to 0x030564 (0x62e2 bytes) 000605 1752688344.632102 mapread 0x028127 thru 0x031e44 (0x9d1e bytes) 000606 1752688344.648962 MIRROR_RESYNC 4 mirrors 000607 1752688345.185973 MIRROR_EXTEND to 5 mirrors 000608 1752688345.584304 mapread 0x02c606 thru 0x03c209 (0xfc04 bytes) 000609 1752688345.612468 fallocate 0x01c7d7 thru 0x0219f5 (0x521f bytes) 000610 1752688345.641965 write 0x032d08 thru 0x03a823 (0x7b1c bytes) 000612 1752688345.658961 mapread 0x0120d4 thru 0x0143db (0x2308 bytes) 000613 1752688345.666673 fallocate 0x010192 thru 0x01bc74 (0xbae3 bytes) 000614 1752688345.673276 trunc from 0x03c6e9 to 0x03ccf2 (0x60a bytes) 000615 1752688345.695573 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000616 1752688345.836862 mapread 0x003496 thru 0x00a62e (0x7199 bytes) 000617 1752688345.851294 MIRROR_RESYNC 4 mirrors 000618 1752688346.412482 write 0x00c61d thru 0x00eee3 (0x28c7 bytes) 000619 1752688346.433951 mapwrite 0x0087f2 thru 0x00cc06 (0x4415 bytes) 000620 1752688346.456931 trunc from 0x00cdfd to 0x00ef60 (0x2164 bytes) 000621 1752688346.460960 mapwrite 0x0158cd thru 0x01d265 (0x7999 bytes) 000622 1752688346.482065 write 0x01f1e9 thru 0x022826 (0x363e bytes) 000623 1752688346.490900 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000624 1752688346.645367 punch from 0x018dbc to 0x0191f4 (0x439 bytes) 000625 1752688346.648867 read 0x02dafe thru 0x035d50 (0x8253 bytes) 000626 1752688346.660756 mapwrite 0x009319 thru 0x0107cc (0x74b4 bytes) 000627 1752688346.700601 trunc from 0x03ccf3 to 0x004025 (0x38ccd bytes) 000628 1752688346.715353 trunc from 0x004026 to 0x02e83d (0x2a818 bytes) 000629 1752688346.728477 fallocate 0x00117e thru 0x0037ae (0x2631 bytes) 000631 1752688346.738027 fallocate 0x01f11f thru 0x01fd0d (0xbef bytes) 000632 1752688346.746609 MIRROR_RESYNC 3 mirrors 000633 1752688347.173253 mapwrite 0x03ba3a thru 0x03ffff (0x45c6 bytes) 000634 1752688347.236549 write 0x005637 thru 0x01453b (0xef05 bytes) 000635 1752688347.249920 read 0x02f0ef thru 0x02f1f4 (0x106 bytes) 000636 1752688347.254085 mapread 0x004779 thru 0x010cb7 (0xc53f bytes) 000637 1752688347.255058 trunc from 0x0055a0 to 0x00c059 (0x6aba bytes) 000638 1752688347.267908 write 0x036f10 thru 0x03d834 (0x6925 bytes) 000639 1752688347.269027 mapread 0x028a4f thru 0x03683c (0xddee bytes) 000641 1752688347.284434 write 0x00518f thru 0x00c668 (0x74da bytes) 000642 1752688347.291741 read 0x0253a4 thru 0x02b976 (0x65d3 bytes) 000643 1752688347.292540 mapread 0x01c178 thru 0x01e5b2 (0x243b bytes) 000644 1752688347.299600 fallocate 0x02ad28 thru 0x02d588 (0x2861 bytes) 000645 1752688347.305174 trunc from 0x02d8b5 to 0x032b22 (0x526e bytes) 000646 1752688347.309324 trunc from 0x02916b to 0x0350bf (0xbf55 bytes) 000647 1752688347.313022 trunc from 0x040000 to 0x016b11 (0x294ee bytes) 000648 1752688347.332820 punch from 0x0154cc to 0x016b11 (0x1646 bytes) 000649 1752688347.338511 fallocate 0x033c61 thru 0x03e332 (0xa6d2 bytes) 000650 1752688347.343535 mapwrite 0x00a042 thru 0x011e14 (0x7dd3 bytes) 000651 1752688347.367971 trunc from 0x03e333 to 0x014d4c (0x295e6 bytes) 000652 1752688347.384549 MIRROR_EXTEND to 4 mirrors 000653 1752688347.679240 punch from 0x013883 to 0x014d4c (0x14ca bytes) 000654 1752688347.710439 punch from 0x008813 to 0x00eaca (0x62b8 bytes) 000655 1752688347.714951 mapwrite 0x026840 thru 0x030d9d (0xa55e bytes) 000656 1752688347.779766 punch from 0x01dc1a to 0x028b08 (0xaeef bytes) 000657 1752688347.783693 write 0x026f6d thru 0x036883 (0xf917 bytes) 000658 1752688347.794672 MIRROR_EXTEND to 5 mirrors 000659 1752688348.114258 punch from 0x02f9f5 to 0x031e53 (0x245f bytes) 000660 1752688348.139540 mapread 0x002bfd thru 0x00c235 (0x9639 bytes) 000661 1752688348.155678 write 0x038eb0 thru 0x03b5a2 (0x26f3 bytes) 000662 1752688348.162831 MIRROR_RESYNC 5 mirrors 000663 1752688349.077460 MIRROR_RESYNC 5 mirrors 000664 1752688349.185461 read 0x010254 thru 0x016e42 (0x6bef bytes) 000665 1752688349.193788 mapread 0x02dd81 thru 0x03ab67 (0xcde7 bytes) 000666 1752688349.209297 trunc from 0x03b5a3 to 0x02cd3e (0xe864 bytes) 000667 1752688349.245184 trunc from 0x02cd3f to 0x006aac (0x26292 bytes) 000668 1752688349.258165 trunc from 0x006aad to 0x023618 (0x1cb6c bytes) 000669 1752688349.270886 write 0x016d2f thru 0x01c050 (0x5322 bytes) 000670 1752688349.279529 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000671 1752688349.412700 punch from 0x0127ad to 0x01a1d0 (0x7a24 bytes) 000672 1752688349.417036 fallocate 0x039e27 thru 0x03ffff (0x61d9 bytes) 000673 1752688349.421294 fallocate 0x02d8fb thru 0x0308c4 (0x2fca bytes) 000674 1752688349.427887 fallocate 0x008856 thru 0x012cb7 (0xa462 bytes) 000675 1752688349.431921 write 0x009d15 thru 0x017c81 (0xdf6d bytes) 000677 1752688349.443051 MIRROR_EXTEND to 5 mirrors 000678 1752688349.838999 read 0x02c743 thru 0x02f683 (0x2f41 bytes) 000679 1752688349.848613 mapwrite 0x011d9e thru 0x01a99d (0x8c00 bytes) 000680 1752688349.935486 write 0x00bec0 thru 0x00cab5 (0xbf6 bytes) 000681 1752688349.946353 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000682 1752688350.098675 fallocate 0x0044a6 thru 0x009617 (0x5172 bytes) 000683 1752688350.104328 fallocate 0x00d79e thru 0x011022 (0x3885 bytes) 000684 1752688350.107499 punch from 0x009c13 to 0x015468 (0xb856 bytes) 000685 1752688350.111687 mapwrite 0x025b13 thru 0x030bad (0xb09b bytes) 000686 1752688350.149227 mapwrite 0x03f983 thru 0x03ffff (0x67d bytes) 000687 1752688350.178515 write 0x0306fb thru 0x03ca15 (0xc31b bytes) 000688 1752688350.183180 trunc from 0x01513f to 0x01e003 (0x8ec5 bytes) 000689 1752688350.188957 fallocate 0x00b35d thru 0x00c924 (0x15c8 bytes) 000690 1752688350.195695 mapread 0x03e6b2 thru 0x03ffff (0x194e bytes) 000691 1752688350.199910 MIRROR_RESYNC 4 mirrors 000692 1752688350.808966 write 0x009521 thru 0x00d986 (0x4466 bytes) 000693 1752688350.832337 trunc from 0x0317ba to 0x035d01 (0x4548 bytes) 000694 1752688350.838259 write 0x01199a thru 0x01ec82 (0xd2e9 bytes) 000695 1752688350.849810 write 0x03a828 thru 0x03ffff (0x57d8 bytes) 000696 1752688350.855494 trunc from 0x02cfbd to 0x02edc4 (0x1e08 bytes) 000697 1752688350.859046 punch from 0x00b0ae to 0x01018b (0x50de bytes) 000698 1752688350.869991 trunc from 0x040000 to 0x03d9ba (0x2645 bytes) 000699 1752688350.895260 trunc from 0x03d9bb to 0x01cf4b (0x20a6f bytes) 000700 1752688350.909409 trunc from 0x01cf4c to 0x02a0cf (0xd184 bytes) 000701 1752688350.920197 write 0x01f223 thru 0x025da8 (0x6b86 bytes) 000702 1752688350.925943 fallocate 0x0348a7 thru 0x03ffff (0xb759 bytes) 000703 1752688350.929932 fallocate 0x023f3b thru 0x02464d (0x713 bytes) 000704 1752688350.934472 mapwrite 0x010777 thru 0x011d60 (0x15ea bytes) 000705 1752688350.960282 punch from 0x039867 to 0x03cd85 (0x351f bytes) 000706 1752688350.964793 mapread 0x0370e4 thru 0x03ffff (0x8f1c bytes) 000707 1752688350.971726 trunc from 0x026933 to 0x02f6a3 (0x8d71 bytes) 000708 1752688350.982954 MIRROR_RESYNC 4 mirrors 000709 1752688351.615688 punch from 0x030621 to 0x034056 (0x3a36 bytes) 000710 1752688351.637738 mapread 0x0255ef thru 0x02e6e2 (0x90f4 bytes) 000712 1752688351.653463 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000713 1752688351.755900 mapread 0x00ae16 thru 0x0191a5 (0xe390 bytes) 000714 1752688351.772173 fallocate 0x018bc6 thru 0x01d05b (0x4496 bytes) 000715 1752688351.776179 write 0x016b6e thru 0x01d6c3 (0x6b56 bytes) 000716 1752688351.780535 fallocate 0x031953 thru 0x039e6b (0x8519 bytes) 000717 1752688351.784955 read 0x010046 thru 0x01a5ce (0xa589 bytes) 000718 1752688351.785588 trunc from 0x002a3d to 0x003072 (0x636 bytes) 000719 1752688351.789905 trunc from 0x030324 to 0x031b47 (0x1824 bytes) 000720 1752688351.793585 mapwrite 0x02c3d8 thru 0x02f50a (0x3133 bytes) 000721 1752688351.814245 punch from 0x029655 to 0x03222a (0x8bd6 bytes) 000722 1752688351.819355 write 0x015d12 thru 0x01b4c5 (0x57b4 bytes) 000723 1752688351.820470 fallocate 0x00104f thru 0x00750a (0x64bc bytes) 000724 1752688351.825288 punch from 0x012468 to 0x01e460 (0xbff9 bytes) 000725 1752688351.838157 read 0x022ddb thru 0x024eef (0x2115 bytes) 000726 1752688351.845534 punch from 0x007183 to 0x00846c (0x12ea bytes) 000727 1752688351.851507 fallocate 0x02b077 thru 0x037afe (0xca88 bytes) 000728 1752688351.855235 write 0x02b5f5 thru 0x037a6d (0xc479 bytes) 000729 1752688351.856644 punch from 0x0269e6 to 0x03384d (0xce68 bytes) 000730 1752688351.870551 trunc from 0x040000 to 0x0237cd (0x1c832 bytes) 000731 1752688351.881024 trunc from 0x0237ce to 0x00bf3f (0x1788e bytes) 000732 1752688351.890486 read 0x00160e thru 0x00bf3f (0xa932 bytes) 000733 1752688351.895744 trunc from 0x006a5a to 0x00a97a (0x3f21 bytes) 000734 1752688351.899826 mapwrite 0x025804 thru 0x02aca9 (0x54a6 bytes) 000735 1752688351.931995 read 0x02651a thru 0x02aca9 (0x4790 bytes) 000736 1752688351.932815 fallocate 0x0378e6 thru 0x03e046 (0x6761 bytes) 000737 1752688351.936925 fallocate 0x0049df thru 0x005cd5 (0x12f7 bytes) 000738 1752688351.942787 fallocate 0x018a5b thru 0x019b1f (0x10c5 bytes) 000739 1752688351.946391 punch from 0x01a5aa to 0x01c43b (0x1e92 bytes) 000740 1752688351.951741 trunc from 0x02deda to 0x03d5f6 (0xf71d bytes) 000741 1752688351.955537 fallocate 0x024d0b thru 0x0335d3 (0xe8c9 bytes) 000742 1752688351.959228 read 0x0120bd thru 0x01d489 (0xb3cd bytes) 000743 1752688351.969081 write 0x00d482 thru 0x015904 (0x8483 bytes) 000745 1752688351.970921 trunc from 0x027a25 to 0x0368db (0xeeb7 bytes) 000746 1752688351.977967 trunc from 0x03e047 to 0x0337ca (0xa87c bytes) 000747 1752688351.992104 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000748 1752688352.124618 mapwrite 0x025c33 thru 0x029013 (0x33e1 bytes) 000749 1752688352.152874 fallocate 0x016aa8 thru 0x017c30 (0x1189 bytes) 000750 1752688352.158481 trunc from 0x006476 to 0x008613 (0x219e bytes) 000751 1752688352.163110 fallocate 0x01c290 thru 0x025f58 (0x9cc9 bytes) 000752 1752688352.166879 punch from 0x031439 to 0x0337ca (0x2392 bytes) 000753 1752688352.170637 punch from 0x025f2e to 0x02e8aa (0x897d bytes) 000754 1752688352.174586 fallocate 0x022c32 thru 0x032b71 (0xff40 bytes) 000755 1752688352.178043 trunc from 0x0337cb to 0x0016af (0x3211b bytes) 000756 1752688352.190960 fallocate 0x02e446 thru 0x0394b8 (0xb073 bytes) 000757 1752688352.195240 trunc from 0x0016b0 to 0x00656e (0x4ebf bytes) 000758 1752688352.208796 mapwrite 0x00ef36 thru 0x019c2b (0xacf6 bytes) 000760 1752688352.249516 MIRROR_EXTEND to 3 mirrors 000761 1752688352.510406 trunc from 0x019c2c to 0x0334b2 (0x19887 bytes) 000762 1752688352.572823 punch from 0x02e3d3 to 0x0317b7 (0x33e5 bytes) 000763 1752688352.579223 punch from 0x02952e to 0x0334b2 (0x9f85 bytes) 000764 1752688352.589206 mapread 0x004803 thru 0x00d599 (0x8d97 bytes) 000765 1752688352.605673 mapread 0x006677 thru 0x009781 (0x310b bytes) 000766 1752688352.606491 read 0x01bfa6 thru 0x024069 (0x80c4 bytes) 000767 1752688352.612077 punch from 0x00750d to 0x014f17 (0xda0b bytes) 000768 1752688352.616549 write 0x00df82 thru 0x01ae0d (0xce8c bytes) 000769 1752688352.627085 read 0x021e7c thru 0x028d0d (0x6e92 bytes) 000770 1752688352.634919 write 0x027f2b thru 0x0341e2 (0xc2b8 bytes) 000771 1752688352.637708 trunc from 0x0341e3 to 0x02abd8 (0x960a bytes) 000772 1752688352.652411 write 0x018e8c thru 0x01e85f (0x59d4 bytes) 000773 1752688352.653379 read 0x01b37d thru 0x0284aa (0xd12e bytes) 000774 1752688352.653909 fallocate 0x022c62 thru 0x02d8e1 (0xac80 bytes) 000775 1752688352.659999 MIRROR_EXTEND to 4 mirrors 000776 1752688352.990909 read 0x003c38 thru 0x0106d0 (0xca99 bytes) 000777 1752688352.999731 fallocate 0x01fd5e thru 0x02cff7 (0xd29a bytes) 000780 1752688353.023211 mapwrite 0x004c6f thru 0x011e0d (0xd19f bytes) 000781 1752688353.063221 mapread 0x0076b7 thru 0x01633a (0xec84 bytes) 000782 1752688353.070024 MIRROR_EXTEND to 5 mirrors 000783 1752688353.416322 mapread 0x029d92 thru 0x02cff7 (0x3266 bytes) 000784 1752688353.445882 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000785 1752688353.673395 read 0x01d7fd thru 0x02b465 (0xdc69 bytes) 000786 1752688353.691447 punch from 0x004957 to 0x00a218 (0x58c2 bytes) 000787 1752688353.718814 mapwrite 0x00afae thru 0x018640 (0xd693 bytes) 000788 1752688353.753513 fallocate 0x03868a thru 0x03ffff (0x7976 bytes) 000789 1752688353.758081 read 0x0270ce thru 0x02eb3b (0x7a6e bytes) 000790 1752688353.769664 mapread 0x01718b thru 0x022da0 (0xbc16 bytes) 000791 1752688353.770819 fallocate 0x011ae9 thru 0x0155c9 (0x3ae1 bytes) 000792 1752688353.775054 write 0x029c2e thru 0x0356a2 (0xba75 bytes) 000793 1752688353.783686 mapread 0x009274 thru 0x00df1d (0x4caa bytes) 000794 1752688353.784807 mapread 0x00d386 thru 0x00ed3d (0x19b8 bytes) 000795 1752688353.786419 mapwrite 0x019ef6 thru 0x01f0ea (0x51f5 bytes) 000796 1752688353.800417 read 0x01ed4a thru 0x026e8f (0x8146 bytes) 000797 1752688353.801104 trunc from 0x040000 to 0x0140d6 (0x2bf29 bytes) 000798 1752688353.832067 fallocate 0x01f9c5 thru 0x028407 (0x8a43 bytes) 000799 1752688353.836025 punch from 0x011ac1 to 0x0140d6 (0x2616 bytes) 000800 1752688353.840401 read 0x006d43 thru 0x0086cf (0x198d bytes) 000801 1752688353.840989 write 0x0154b6 thru 0x022c2e (0xd779 bytes) 000802 1752688353.842562 trunc from 0x0023be to 0x009c67 (0x78aa bytes) 000804 1752688353.846978 mapread 0x017da6 thru 0x01c588 (0x47e3 bytes) 000805 1752688353.848620 punch from 0x00b025 to 0x01784a (0xc826 bytes) 000806 1752688353.862772 trunc from 0x011cb4 to 0x013be3 (0x1f30 bytes) 000807 1752688353.869841 mapwrite 0x003659 thru 0x013429 (0xfdd1 bytes) 000808 1752688353.912508 read 0x0066ec thru 0x00cb8a (0x649f bytes) 000809 1752688353.913053 trunc from 0x005a61 to 0x01406b (0xe60b bytes) 000810 1752688353.918082 write 0x01453c thru 0x01beb7 (0x797c bytes) 000811 1752688353.924346 write 0x013e71 thru 0x01eac4 (0xac54 bytes) 000812 1752688353.925663 read 0x00959a thru 0x016795 (0xd1fc bytes) 000813 1752688353.926343 read 0x0207a3 thru 0x020d32 (0x590 bytes) 000814 1752688353.930039 MIRROR_RESYNC 4 mirrors 000815 1752688354.451365 mapwrite 0x001c35 thru 0x00d6a8 (0xba74 bytes) 000816 1752688354.518334 trunc from 0x022c2f to 0x02dd71 (0xb143 bytes) 000817 1752688354.528975 mapwrite 0x02ff1f thru 0x0384dd (0x85bf bytes) 000818 1752688354.568409 mapwrite 0x007a65 thru 0x011940 (0x9edc bytes) 000819 1752688354.592313 mapwrite 0x026567 thru 0x026f20 (0x9ba bytes) 000820 1752688354.609962 write 0x01b40a thru 0x0286d0 (0xd2c7 bytes) 000822 1752688354.621251 punch from 0x0059d4 to 0x015677 (0xfca4 bytes) 000823 1752688354.626173 punch from 0x02c81d to 0x0384dd (0xbcc1 bytes) 000824 1752688354.632367 mapwrite 0x024e3f thru 0x02e6a2 (0x9864 bytes) 000825 1752688354.659998 read 0x02fc48 thru 0x0384dd (0x8896 bytes) 000826 1752688354.667525 fallocate 0x0139a8 thru 0x01d201 (0x985a bytes) 000827 1752688354.672175 mapread 0x011804 thru 0x01984f (0x804c bytes) 000828 1752688354.681296 trunc from 0x0060bd to 0x011ce4 (0xbc28 bytes) 000829 1752688354.685271 punch from 0x02bf9b to 0x034d16 (0x8d7c bytes) 000830 1752688354.689255 read 0x01ab11 thru 0x0263f5 (0xb8e5 bytes) 000831 1752688354.689819 read 0x01b601 thru 0x024058 (0x8a58 bytes) 000832 1752688354.690392 write 0x01b740 thru 0x01fff2 (0x48b3 bytes) 000833 1752688354.691347 read 0x027042 thru 0x02a826 (0x37e5 bytes) 000834 1752688354.691744 trunc from 0x0384de to 0x02cb06 (0xb9d7 bytes) 000835 1752688354.700836 read 0x026eaf thru 0x027376 (0x4c8 bytes) 000837 1752688354.701875 trunc from 0x01b68d to 0x023685 (0x7ff9 bytes) 000840 1752688354.710646 trunc from 0x02cb07 to 0x01c2b8 (0x1084e bytes) 000841 1752688354.723686 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000842 1752688354.871160 trunc from 0x01c2b9 to 0x00d5b9 (0xecff bytes) 000843 1752688354.892597 fallocate 0x026c59 thru 0x032bb7 (0xbf5f bytes) 000844 1752688354.896639 mapread 0x0067d6 thru 0x01302e (0xc859 bytes) 000845 1752688354.925356 write 0x0178c1 thru 0x01c62c (0x4d6c bytes) 000846 1752688354.935953 read 0x0261a3 thru 0x0304f0 (0xa34e bytes) 000847 1752688354.948226 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000848 1752688355.061811 write 0x007f4e thru 0x016a9c (0xeb4f bytes) 000849 1752688355.072091 write 0x01704f thru 0x0200c0 (0x9072 bytes) 000850 1752688355.082705 trunc from 0x032bb8 to 0x01ca8f (0x16128 bytes) 000851 1752688355.092725 trunc from 0x01ca90 to 0x00a71e (0x12371 bytes) 000852 1752688355.102277 trunc from 0x00613e to 0x00a71e (0x45e1 bytes) 000853 1752688355.110388 punch from 0x00535b to 0x00a71e (0x53c4 bytes) 000854 1752688355.113948 read 0x0030ca thru 0x00a71e (0x7655 bytes) 000855 1752688355.118886 punch from 0x0079f4 to 0x00a71e (0x2d2b bytes) 000856 1752688355.122576 mapread 0x005d96 thru 0x00a71e (0x4989 bytes) 000857 1752688355.127950 write 0x02ef4b thru 0x037a87 (0x8b3d bytes) 000858 1752688355.129354 punch from 0x012365 to 0x02081f (0xe4bb bytes) 000859 1752688355.132511 trunc from 0x037a88 to 0x01d570 (0x1a517 bytes) 000860 1752688355.141287 trunc from 0x01d571 to 0x03ed3a (0x217ca bytes) 000861 1752688355.152208 MIRROR_EXTEND to 3 mirrors 000862 1752688355.468557 read 0x031ffc thru 0x03d74c (0xb751 bytes) 000863 1752688355.480115 trunc from 0x037c0c to 0x03ed3a (0x712f bytes) 000864 1752688355.500715 read 0x0095c1 thru 0x017ebe (0xe8fe bytes) 000865 1752688355.517326 mapread 0x0337be thru 0x036ac2 (0x3305 bytes) 000866 1752688355.518277 write 0x0176d4 thru 0x01cabb (0x53e8 bytes) 000867 1752688355.529539 MIRROR_EXTEND to 4 mirrors 000868 1752688355.855099 mapread 0x034090 thru 0x03ed3a (0xacab bytes) 000870 1752688355.882827 punch from 0x031552 to 0x03ed3a (0xd7e9 bytes) 000871 1752688355.904944 mapwrite 0x03aa6f thru 0x03e742 (0x3cd4 bytes) 000872 1752688355.924931 mapread 0x022407 thru 0x02822f (0x5e29 bytes) 000873 1752688355.936806 write 0x015d65 thru 0x017a23 (0x1cbf bytes) 000874 1752688355.949055 mapread 0x00a21c thru 0x01175c (0x7541 bytes) 000875 1752688355.955553 mapwrite 0x02fb27 thru 0x0337fe (0x3cd8 bytes) 000876 1752688355.983107 fallocate 0x030869 thru 0x03e7eb (0xdf83 bytes) 000877 1752688355.987863 fallocate 0x027450 thru 0x02a7d3 (0x3384 bytes) 000878 1752688355.992182 trunc from 0x03ed3b to 0x002c8a (0x3c0b0 bytes) 000879 1752688356.030171 trunc from 0x0020ba to 0x002c8a (0xbd1 bytes) 000880 1752688356.037809 mapread 0x002619 thru 0x002c8a (0x672 bytes) 000881 1752688356.042789 mapwrite 0x019250 thru 0x01e282 (0x5033 bytes) 000882 1752688356.088898 mapread 0x0083ed thru 0x00b5b4 (0x31c8 bytes) 000883 1752688356.096029 trunc from 0x01e283 to 0x03862b (0x1a3a9 bytes) 000884 1752688356.111477 mapwrite 0x03ad5a thru 0x03ffff (0x52a6 bytes) 000885 1752688356.147040 write 0x00de33 thru 0x015668 (0x7836 bytes) 000886 1752688356.148419 trunc from 0x028638 to 0x02ca87 (0x4450 bytes) 000887 1752688356.151388 trunc from 0x013d8f to 0x017fd7 (0x4249 bytes) 000888 1752688356.160929 fallocate 0x032545 thru 0x038373 (0x5e2f bytes) 000889 1752688356.167780 MIRROR_EXTEND to 5 mirrors 000893 1752688356.496106 trunc from 0x03eceb to 0x03ffff (0x1315 bytes) 000894 1752688356.518939 read 0x000030 thru 0x00e066 (0xe037 bytes) 000895 1752688356.526008 trunc from 0x040000 to 0x01258a (0x2da75 bytes) 000896 1752688356.552466 trunc from 0x01258b to 0x03f3da (0x2ce50 bytes) 000897 1752688356.563993 MIRROR_RESYNC 5 mirrors 000898 1752688357.031796 fallocate 0x0143a8 thru 0x01b389 (0x6fe2 bytes) 000899 1752688357.051965 fallocate 0x0334e4 thru 0x03ef47 (0xba64 bytes) 000900 1752688357.055499 read 0x022b2e thru 0x022b5b (0x2e bytes) 000901 1752688357.059467 trunc from 0x00755f to 0x010a20 (0x94c2 bytes) 000902 1752688357.064090 mapread 0x039715 thru 0x03c846 (0x3132 bytes) 000904 1752688357.081972 write 0x0063c1 thru 0x0067fb (0x43b bytes) 000906 1752688357.091252 fallocate 0x00ab03 thru 0x00cb4f (0x204d bytes) 000907 1752688357.095490 write 0x0042df thru 0x005704 (0x1426 bytes) 000908 1752688357.096365 read 0x02ca5e thru 0x02d4ac (0xa4f bytes) 000909 1752688357.105194 punch from 0x035a31 to 0x03c365 (0x6935 bytes) 000910 1752688357.110527 punch from 0x03cd07 to 0x03f3da (0x26d4 bytes) 000911 1752688357.115921 read 0x021877 thru 0x02bce2 (0xa46c bytes) 000912 1752688357.116650 trunc from 0x03f3db to 0x0333f6 (0xbfe4 bytes) 000913 1752688357.146518 read 0x0165bf thru 0x023713 (0xd155 bytes) 000914 1752688357.150380 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000915 1752688357.315655 trunc from 0x0333f7 to 0x01828b (0x1b16b bytes) 000916 1752688357.335254 trunc from 0x004ca3 to 0x00ea99 (0x9df7 bytes) 000918 1752688357.347060 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000919 1752688357.494163 trunc from 0x01828c to 0x0338ac (0x1b621 bytes) 000920 1752688357.511237 mapwrite 0x032ad8 thru 0x03b6d2 (0x8bfb bytes) 000921 1752688357.554535 punch from 0x037731 to 0x03b6d2 (0x3fa2 bytes) 000922 1752688357.559060 mapread 0x0072c9 thru 0x0143be (0xd0f6 bytes) 000923 1752688357.571694 trunc from 0x03b6d3 to 0x0323dd (0x92f5 bytes) 000924 1752688357.583205 fallocate 0x0306fa thru 0x0338cd (0x31d4 bytes) 000925 1752688357.587764 mapread 0x0059a4 thru 0x00c095 (0x66f2 bytes) 000926 1752688357.588731 write 0x0067f2 thru 0x014bf8 (0xe407 bytes) 000927 1752688357.590108 punch from 0x013cb4 to 0x017918 (0x3c65 bytes) 000928 1752688357.601978 read 0x008eea thru 0x0157e1 (0xc8f8 bytes) 000930 1752688357.612494 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x330:0x0]: Invalid argument (22) 000931 1752688357.754237 mapread 0x02ce9b thru 0x02d6af (0x815 bytes) 000933 1752688357.764021 fallocate 0x005b10 thru 0x00763b (0x1b2c bytes) 000934 1752688357.771351 write 0x0267a8 thru 0x0357e9 (0xf042 bytes) 000935 1752688357.782328 mapwrite 0x010b7b thru 0x01c284 (0xb70a bytes) 000936 1752688357.820402 mapwrite 0x03c654 thru 0x03ffff (0x39ac bytes) 000937 1752688357.856481 trunc from 0x02c107 to 0x031b9a (0x5a94 bytes) 000938 1752688357.861212 trunc from 0x016567 to 0x01cad2 (0x656c bytes) 000939 1752688357.866371 fallocate 0x02fccf thru 0x03cc44 (0xcf76 bytes) 000940 1752688357.871430 punch from 0x022d95 to 0x026e39 (0x40a5 bytes) 000941 1752688357.880217 MIRROR_RESYNC 2 mirrors 000942 1752688358.352054 trunc from 0x040000 to 0x00db44 (0x324bb bytes) 000943 1752688358.382643 punch from 0x0026f6 to 0x00db44 (0xb44f bytes) 000944 1752688358.390997 read 0x0044f9 thru 0x0045cb (0xd3 bytes) 000945 1752688358.399210 mapwrite 0x01868b thru 0x026fe2 (0xe958 bytes) 000946 1752688358.452278 mapread 0x00c32f thru 0x018065 (0xbd37 bytes) 000947 1752688358.461907 MIRROR_RESYNC 2 mirrors 000949 1752688358.693707 fallocate 0x0179e6 thru 0x019909 (0x1f24 bytes) 000950 1752688358.728019 read 0x012ef4 thru 0x013289 (0x396 bytes) 000951 1752688358.734194 fallocate 0x027602 thru 0x034349 (0xcd48 bytes) 000952 1752688358.743930 punch from 0x00d5a7 to 0x01142e (0x3e88 bytes) 000954 1752688358.748772 trunc from 0x03434a to 0x02536e (0xefdb bytes) 000955 1752688358.779572 mapwrite 0x01ff22 thru 0x024d8d (0x4e6c bytes) 000956 1752688358.814187 write 0x033f04 thru 0x0366fb (0x27f8 bytes) 000957 1752688358.815473 punch from 0x0206dc to 0x0255aa (0x4ecf bytes) 000958 1752688358.820595 fallocate 0x038258 thru 0x03ffff (0x7da8 bytes) 000959 1752688358.832785 write 0x0344be thru 0x03ffff (0xbb42 bytes) 000960 1752688358.834572 write 0x030b8e thru 0x034741 (0x3bb4 bytes) 000961 1752688358.851589 MIRROR_EXTEND to 3 mirrors 000962 1752688359.371528 read 0x012555 thru 0x01f901 (0xd3ad bytes) 000963 1752688359.387739 mapread 0x01e2df thru 0x01f3be (0x10e0 bytes) 000964 1752688359.396809 trunc from 0x040000 to 0x00164c (0x3e9b3 bytes) 000965 1752688359.428045 trunc from 0x00164d to 0x03c880 (0x3b234 bytes) 000966 1752688359.439861 mapread 0x014c3f thru 0x020537 (0xb8f9 bytes) 000967 1752688359.457159 write 0x0052f3 thru 0x006f77 (0x1c85 bytes) 000968 1752688359.465852 punch from 0x0047df to 0x012213 (0xda35 bytes) 000969 1752688359.473359 trunc from 0x03c881 to 0x0067ae (0x360d2 bytes) 000970 1752688359.483243 trunc from 0x005ddd to 0x0067ae (0x9d2 bytes) 000971 1752688359.489396 trunc from 0x00595e to 0x0067ae (0xe51 bytes) 000972 1752688359.492560 read 0x004f13 thru 0x0052ee (0x3dc bytes) 000974 1752688359.499532 trunc from 0x004738 to 0x0067ae (0x2077 bytes) 000975 1752688359.504851 write 0x00b5a0 thru 0x019520 (0xdf81 bytes) 000976 1752688359.506517 fallocate 0x005c66 thru 0x012bf1 (0xcf8c bytes) 000977 1752688359.511261 mapread 0x01223c thru 0x019520 (0x72e5 bytes) 000978 1752688359.512153 trunc from 0x013484 to 0x019520 (0x609d bytes) 000979 1752688359.525568 write 0x0215da thru 0x026286 (0x4cad bytes) 000980 1752688359.526799 fallocate 0x033303 thru 0x03ffff (0xccfd bytes) 000981 1752688359.532179 trunc from 0x040000 to 0x001767 (0x3e898 bytes) 000982 1752688359.544905 trunc from 0x000688 to 0x001767 (0x10e0 bytes) 000983 1752688359.549598 mapread 0x0016f8 thru 0x001767 (0x70 bytes) 000984 1752688359.553199 fallocate 0x02feb8 thru 0x03c9f9 (0xcb42 bytes) 000985 1752688359.556776 mapwrite 0x036c47 thru 0x03ffff (0x93b9 bytes) 000986 1752688359.593577 MIRROR_EXTEND to 4 mirrors 000987 1752688359.885490 trunc from 0x03d3aa to 0x03ffff (0x2c56 bytes) 000989 1752688359.911420 read 0x02b0e9 thru 0x0370d9 (0xbff1 bytes) 000990 1752688359.921714 MIRROR_RESYNC 4 mirrors 000991 1752688360.350368 trunc from 0x040000 to 0x017169 (0x28e96 bytes) 000992 1752688360.377434 trunc from 0x00b814 to 0x017169 (0xb956 bytes) 000993 1752688360.381977 trunc from 0x01716a to 0x0101e8 (0x6f81 bytes) 000994 1752688360.393545 mapwrite 0x022d32 thru 0x02d333 (0xa602 bytes) 000995 1752688360.439495 fallocate 0x0341f2 thru 0x036c46 (0x2a55 bytes) 000996 1752688360.445641 write 0x03985c thru 0x03ffff (0x67a4 bytes) 000997 1752688360.446754 read 0x00b3b7 thru 0x014531 (0x917b bytes) 000998 1752688360.452279 fallocate 0x02e591 thru 0x033174 (0x4be4 bytes) 000999 1752688360.456241 trunc from 0x00f351 to 0x01dd28 (0xe9d8 bytes) 001000 1752688360.461331 mapwrite 0x008b8d thru 0x00a036 (0x14aa bytes) All operations completed A-OK! PASS 70a (38s) == sanity-flr test 71: check mirror extend parent fid ==== 13:52:42 (1752688362) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x55:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 200a: stress test ===================== 13:52:47 (1752688367) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1653824 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3885312 .. Extending file size to 5679200 .. Extending file size to 6696800 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8939520 .. Extending file size to 7889536 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8284512 .. Extending file size to 8905888 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8921184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9144960 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9051136 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9187264 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9122144 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60364 60365 60366 60370 60371 done 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) 192.168.203.109@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:53:54 (1752688434) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2228480 .. Extending file size to 3076128 .. Extending file size to 8958080 .. Extending file size to 3864128 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6076896 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8622592 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9158880 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8976928 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9043712 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9295008 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63776 63777 63779 63781 63783 done 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) 192.168.203.109@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:55:02 (1752688502) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2556 1285132 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2516 1285172 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19996 3587024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17752 3589268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37748 7176292 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2556 1285132 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2516 1285172 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 19996 3587024 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17752 3589268 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37748 7176292 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.05026 s, 41.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0915356 s, 22.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:55:10 (1752688510) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0820938 s, 25.5 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 13:55:14 (1752688514) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:55:18 (1752688518) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0551821 s, 38.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:55:21 (1752688521) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0508134 s, 41.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b3:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:55:25 (1752688525) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0463763 s, 45.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:55:50 (1752688550) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221975 s, 47.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4b5:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:56:26 (1752688586) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0381153 s, 55.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (37s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:57:04 (1752688624) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg309-server: Pool lustre.test_204f created Adding targets to pool oleg309-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg309-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0343455 s, 61.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg309-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg309-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg309-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' Updated after 2s: want 'foo' got 'foo' PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:57:54 (1752688674) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:57:57 (1752688677) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:57:59 (1752688679) oleg309-server: Pool lustre.test_206 created Adding targets to pool oleg309-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg309-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg309-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg309-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg309-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:58:15 (1752688695) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358232 s, 29.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145409 s, 72.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x5ce6e20b Mirror 3: 0x5ce6e20b PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:58:19 (1752688699) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:58:20 (1752688700) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:58:22 (1752688702) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0141989 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1360000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x25e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x25e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x25e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf30000 Read this (4 bytes): BBBB 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:58:39 (1752688719) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0036647 s, 2.2 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:58:41 (1752688721) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178186 s, 58.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-16 13:58:42.000000000 -0400 Modify: 2025-07-16 13:58:42.000000000 -0400 Change: 2025-07-16 13:58:42.000000000 -0400 Birth: 2025-07-16 13:58:42.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:58:44 (1752688724) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164031 s, 63.9 MB/s oleg309-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg309-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg309-server: O/280000401/d28/bc: File not found by ext2_lookup PASS 210b (12s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:58:56 (1752688736) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0772206 s, 530 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0122733 s, 334 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 16 13:58 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 16 13:58 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 896 sec ============ 13:58:59 (1752688739) === sanity-flr: start cleanup 13:59:00 (1752688740) === === sanity-flr: finish cleanup 13:59:03 (1752688743) ===