-----============= acceptance-small: sanity-flr ============----- Tue Oct 7 02:53:13 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg217-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:53:19 (1759819999) === oleg217-client.virtnet: executing check_config_client /mnt/lustre oleg217-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg217-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ad243273000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ad243273000.idle_timeout=debug disable quota as required oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:53:31 (1759820011) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8171] == sanity-flr test 0a: lfs mirror create with -N option == 02:53:32 (1759820012) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:53:36 (1759820016) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:53:37 (1759820017) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:53:38 (1759820018) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:53:44 (1759820024) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:53:45 (1759820025) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 02:53:46 (1759820026) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ad243273000.stats=clear osc.lustre-OST0001-osc-ffff8ad243273000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:53:49 (1759820029) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:53:52 (1759820032) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:53:54 (1759820034) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:53:55 (1759820035) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 02:53:58 (1759820038) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:54:00 (1759820040) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:54:02 (1759820042) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:54:04 (1759820044) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:54:07 (1759820047) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:54:09 (1759820049) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0964192 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228286 s, 45.9 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:54:12 (1759820052) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.142782 s, 125 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:54:14 (1759820054) Stopping /mnt/lustre-ost1 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad243273000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad243273000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg217-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad243273000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad243273000.ost_server_uuid in DISCONN state after 3 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg217-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (34s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:54:48 (1759820088) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0792381 s, 39.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad243273000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad243273000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg217-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:55:02 (1759820102) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad243273000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad243273000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg217-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad243273000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad243273000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg217-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (30s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:55:32 (1759820132) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.31768 s, 39.8 MB/s -rw-r--r-- 1 root root 52428800 Oct 7 02:55 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.363s user 0m0.000s sys 0m0.228s Stopping /mnt/lustre-ost1 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ad243273000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ad243273000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.390s user 0m0.003s sys 0m0.228s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg217-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg217-server oleg217-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ad243273000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad243273000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.288s user 0m0.000s sys 0m0.142s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg217-server: oleg217-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg217-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg217-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ad243273000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ad243273000.ost_server_uuid in FULL state after 0 sec PASS 33b (33s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:56:05 (1759820165) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:56:06 (1759820166) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:56:07 (1759820167) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 02:56:08 (1759820168) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 02:56:10 (1759820170) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.146052 s, 86.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:56:14 (1759820174) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112869 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106859 s, 98.1 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:56:16 (1759820176) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:56:20 (1759820180) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000643794 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000457786 s, 2.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000489187 s, 2.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000488085 s, 2.1 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 02:56:22 (1759820182) create test files with size 15738102 .. b53d260a5fc22195f14feddb6cdd8bed - f5a616e2012d66e201650395b77bb519 - fc04af7b98accfa0ef8028121aa6c15d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 02:56:28 (1759820188) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:56:31 (1759820191) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:56:33 (1759820193) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214169 s, 49.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106615 s, 98.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 02:56:36 (1759820196) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0245541 s, 85.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0819581 s, 51.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0350618 s, 59.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 02:56:39 (1759820199) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 02:56:40 (1759820200) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:56:41 (1759820201) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010099 s, 104 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 02:56:43 (1759820203) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0704829 s, 44.6 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:56:46 (1759820206) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:56:49 (1759820209) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.096354 s, 109 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:57:09 (1759820229) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8ad25a659800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0773837 s, 136 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg217-client.virtnet /mnt/lustre (opts:) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8ad2588a5800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:57:13 (1759820233) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:57:25 (1759820245) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:57:36 (1759820256) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 02:57:37 (1759820257) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0800175 s, 39.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.049251 s, 63.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 202c8dff7116817f9ee4827492009041 - mirror 1 checksum: ddd1bf51034578a6d3b2b5388775e03a - mirror 2 checksum: 202c8dff7116817f9ee4827492009041 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Oct 7 02:57:38 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Oct 7 02:57:38 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 202c8dff7116817f9ee4827492009041 - mirror 1 checksum: ddd1bf51034578a6d3b2b5388775e03a - mirror 2 checksum: ddd1bf51034578a6d3b2b5388775e03a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Oct 7 02:57:38 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Oct 7 02:57:38 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:57:40 (1759820260) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111634 s, 93.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:57:43 (1759820263) Failing mds1 on oleg217-server Stopping /mnt/lustre-mds1 (opts:) on oleg217-server 02:57:51 (1759820271) shut down facet: mds1 facet_host: oleg217-server facet_failover_host: oleg217-server Failover mds1 to oleg217-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 02:58:06 (1759820286) targets are mounted 02:58:06 (1759820286) facet_failover done oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02269 s, 46.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:58:14 (1759820294) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00959005 s, 2.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0041638 s, 4.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:58:17 (1759820297) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0119934 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00412537 s, 5.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:58:21 (1759820301) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00380418 s, 4.3 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:58:25 (1759820305) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0103651 s, 2.0 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:58:28 (1759820308) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00269879 s, 1.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0121737 s, 336 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:58:30 (1759820310) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:58:37 (1759820317) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:58:57 (1759820337) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:59:15 (1759820355) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:59:34 (1759820374) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 02:59:37 (1759820377) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55851 mirror split pid 55852 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 55851 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 55852 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 03:00:39 (1759820439) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 440 000001 1759820440.638665 mapwrite 0x0153be thru 0x020b94 (0xb7d7 bytes) truncating to largest ever: 0x301b8 000002 1759820440.676060 trunc from 0x020b95 to 0x0301b7 (0xf623 bytes) 000003 1759820440.683909 mapwrite 0x039b77 thru 0x03cf64 (0x33ee bytes) 000004 1759820440.704393 trunc from 0x01c51e to 0x021c79 (0x575c bytes) 000005 1759820440.707657 mapwrite 0x018064 thru 0x025020 (0xcfbd bytes) 000006 1759820440.728281 punch from 0x001a4a to 0x009d1e (0x82d5 bytes) fallocating to largest ever: 0x40000 000007 1759820440.731238 fallocate 0x03d452 thru 0x03ffff (0x2bae bytes) 000008 1759820440.734671 write 0x02b47f thru 0x03b1e0 (0xfd62 bytes) 000011 1759820440.738721 mapread 0x00abae thru 0x01a367 (0xf7ba bytes) 000012 1759820440.743519 mapread 0x03d692 thru 0x03ffff (0x296e bytes) 000013 1759820440.746679 mapread 0x00d668 thru 0x01d24e (0xfbe7 bytes) 000014 1759820440.747907 trunc from 0x040000 to 0x011879 (0x2e786 bytes) 000015 1759820440.754396 write 0x00909b thru 0x0112b2 (0x8218 bytes) 000016 1759820440.755782 read 0x00be8d thru 0x010d48 (0x4ebc bytes) 000017 1759820440.756416 write 0x03e48f thru 0x03ffff (0x1b71 bytes) 000019 1759820440.758094 punch from 0x019502 to 0x01ba0f (0x250e bytes) 000020 1759820440.760121 trunc from 0x027704 to 0x02a404 (0x2d01 bytes) 000021 1759820440.762030 read 0x039001 thru 0x03ffff (0x6fff bytes) 000022 1759820440.765689 trunc from 0x0239c7 to 0x030b2b (0xd165 bytes) 000023 1759820440.767736 read 0x03f6f1 thru 0x03ffff (0x90f bytes) 000024 1759820440.768099 fallocate 0x0114bd thru 0x01c40e (0xaf52 bytes) 000025 1759820440.770119 punch from 0x020f70 to 0x023c8f (0x2d20 bytes) 000026 1759820440.771970 read 0x03f26d thru 0x03ffff (0xd93 bytes) 000027 1759820440.772323 read 0x01ac04 thru 0x01cf54 (0x2351 bytes) 000028 1759820440.772710 mapwrite 0x026154 thru 0x034183 (0xe030 bytes) 000029 1759820440.784259 mapwrite 0x016ae1 thru 0x021001 (0xa521 bytes) 000030 1759820440.792642 fallocate 0x019f05 thru 0x027925 (0xda21 bytes) 000031 1759820440.794923 mapwrite 0x03da73 thru 0x03ffff (0x258d bytes) 000032 1759820440.800823 fallocate 0x006d47 thru 0x00a610 (0x38ca bytes) 000035 1759820440.804565 mapwrite 0x0062ac thru 0x00d0bb (0x6e10 bytes) 000036 1759820440.813231 read 0x01f150 thru 0x020b02 (0x19b3 bytes) 000037 1759820440.813665 trunc from 0x001bb0 to 0x001de9 (0x23a bytes) 000038 1759820440.816225 trunc from 0x01aba5 to 0x0289ea (0xde46 bytes) 000039 1759820440.818625 trunc from 0x040000 to 0x007ec9 (0x38136 bytes) 000040 1759820440.826106 mapwrite 0x01f0a0 thru 0x021d8d (0x2cee bytes) 000041 1759820440.844179 mapwrite 0x025235 thru 0x031fbe (0xcd8a bytes) 000043 1759820440.866154 mapwrite 0x02573a thru 0x031c71 (0xc538 bytes) 000045 1759820440.877931 write 0x00ea44 thru 0x0150a1 (0x665e bytes) 000046 1759820440.878755 punch from 0x0011f3 to 0x00bf04 (0xad12 bytes) 000049 1759820440.882471 read 0x01e5b2 thru 0x028c85 (0xa6d4 bytes) 000050 1759820440.882909 mapwrite 0x022bda thru 0x02698c (0x3db3 bytes) 000052 1759820440.892462 trunc from 0x031fbf to 0x0317f0 (0x7ce bytes) 000053 1759820440.899526 trunc from 0x0256ec to 0x02fbf0 (0xa505 bytes) 000054 1759820440.902129 punch from 0x027307 to 0x0317f0 (0xa4ea bytes) 000055 1759820440.905012 read 0x00c557 thru 0x00dce2 (0x178c bytes) 000056 1759820440.905557 read 0x002855 thru 0x00a7d4 (0x7f80 bytes) 000057 1759820440.910184 punch from 0x02f633 to 0x0317f0 (0x21be bytes) 000060 1759820440.913753 mapwrite 0x003e89 thru 0x007ac2 (0x3c3a bytes) 000061 1759820440.923137 trunc from 0x0317f1 to 0x00e3dc (0x23414 bytes) 000062 1759820440.930656 punch from 0x0078c8 to 0x008012 (0x74b bytes) 000063 1759820440.933219 punch from 0x006bfd to 0x0087b9 (0x1bbd bytes) 000064 1759820440.935856 trunc from 0x00e3dd to 0x033766 (0x2538a bytes) 000065 1759820440.942848 trunc from 0x033767 to 0x02e0e0 (0x5686 bytes) 000067 1759820440.950026 trunc from 0x02e0e1 to 0x01465f (0x19a81 bytes) 000068 1759820440.956891 trunc from 0x005fa0 to 0x00a8e2 (0x4943 bytes) 000069 1759820440.959519 trunc from 0x014660 to 0x01c9b3 (0x8354 bytes) 000070 1759820440.966042 mapread 0x007bd5 thru 0x0141e7 (0xc613 bytes) 000071 1759820440.990269 write 0x027e3c thru 0x02d4de (0x56a3 bytes) 000072 1759820440.991186 fallocate 0x03e428 thru 0x03ffff (0x1bd8 bytes) 000073 1759820440.994043 read 0x0274b6 thru 0x02d4de (0x6029 bytes) 000074 1759820440.994531 trunc from 0x0089ce to 0x00dd38 (0x536b bytes) 000075 1759820440.997384 mapwrite 0x02254d thru 0x02c3df (0x9e93 bytes) 000076 1759820441.011491 write 0x017633 thru 0x019f6a (0x2938 bytes) 000078 1759820441.013027 trunc from 0x014ba9 to 0x01603e (0x1496 bytes) 000079 1759820441.015670 mapread 0x0069dc thru 0x016604 (0xfc29 bytes) 000080 1759820441.022971 write 0x01ce57 thru 0x01f33a (0x24e4 bytes) 000081 1759820441.023638 trunc from 0x02d4df to 0x0362dc (0x8dfe bytes) 000082 1759820441.032589 MIRROR_EXTEND to 2 mirrors 000083 1759820441.388950 read 0x01df40 thru 0x0251be (0x727f bytes) 000084 1759820441.398052 trunc from 0x0362dd to 0x00fa46 (0x26896 bytes) 000085 1759820441.422328 write 0x017bac thru 0x020bc9 (0x901e bytes) 000086 1759820441.425402 fallocate 0x029458 thru 0x037768 (0xe311 bytes) 000087 1759820441.429678 mapread 0x023d76 thru 0x02753c (0x37c7 bytes) 000088 1759820441.440519 mapread 0x01c4f1 thru 0x024eb9 (0x89c9 bytes) 000089 1759820441.442160 write 0x01f9a1 thru 0x02f182 (0xf7e2 bytes) 000090 1759820441.446632 fallocate 0x0374f2 thru 0x03ffff (0x8b0e bytes) 000092 1759820441.450869 mapread 0x009239 thru 0x00b6a4 (0x246c bytes) 000093 1759820441.457507 write 0x023a00 thru 0x02f806 (0xbe07 bytes) 000094 1759820441.459911 fallocate 0x02558e thru 0x034d91 (0xf804 bytes) 000095 1759820441.464173 mapwrite 0x037d44 thru 0x03ffff (0x82bc bytes) 000096 1759820441.487975 trunc from 0x0182f8 to 0x01d93c (0x5645 bytes) 000097 1759820441.490910 fallocate 0x03385c thru 0x033fa5 (0x74a bytes) 000098 1759820441.494073 punch from 0x026f02 to 0x0294de (0x25dd bytes) 000099 1759820441.497059 fallocate 0x0364df thru 0x03ffff (0x9b21 bytes) 000100 1759820441.500082 read 0x03e92d thru 0x03ffff (0x16d3 bytes) 000102 1759820441.501343 fallocate 0x03785d thru 0x03bfe0 (0x4784 bytes) 000103 1759820441.504354 fallocate 0x01d81e thru 0x0263a4 (0x8b87 bytes) 000104 1759820441.507571 trunc from 0x040000 to 0x0283b0 (0x17c4f bytes) 000105 1759820441.515010 trunc from 0x016cff to 0x01c93b (0x5c3d bytes) 000106 1759820441.517914 mapwrite 0x03d04b thru 0x03ea6b (0x1a21 bytes) 000108 1759820441.537382 read 0x015827 thru 0x01804b (0x2825 bytes) 000109 1759820441.538172 mapread 0x01df80 thru 0x02842a (0xa4ab bytes) 000110 1759820441.542175 MIRROR_EXTEND to 3 mirrors 000111 1759820441.752606 mapwrite 0x01ca78 thru 0x02b2fc (0xe885 bytes) 000112 1759820441.801565 write 0x02b4ef thru 0x039e02 (0xe914 bytes) 000113 1759820441.807046 trunc from 0x03ea6c to 0x0115c1 (0x2d4aa bytes) 000114 1759820441.824286 fallocate 0x0135e7 thru 0x019462 (0x5e7c bytes) 000115 1759820441.827280 mapread 0x0011ac thru 0x008588 (0x73dd bytes) 000116 1759820441.831525 mapwrite 0x002a72 thru 0x005d03 (0x3292 bytes) 000117 1759820441.840967 MIRROR_RESYNC 3 mirrors 000118 1759820442.166636 mapwrite 0x02c59a thru 0x0343ba (0x7e21 bytes) 000119 1759820442.202498 MIRROR_EXTEND to 4 mirrors 000120 1759820442.483319 trunc from 0x0275f9 to 0x02c293 (0x4c9b bytes) 000121 1759820442.498369 fallocate 0x014f8b thru 0x015f77 (0xfed bytes) 000124 1759820442.501563 punch from 0x0096eb to 0x00e4de (0x4df4 bytes) 000125 1759820442.503816 mapread 0x00cac4 thru 0x012d2b (0x6268 bytes) 000126 1759820442.511143 write 0x001be6 thru 0x00c6ba (0xaad5 bytes) 000127 1759820442.516798 mapwrite 0x038927 thru 0x03ffff (0x76d9 bytes) 000128 1759820442.543092 mapread 0x01645b thru 0x017ce8 (0x188e bytes) 000130 1759820442.548227 fallocate 0x034775 thru 0x036132 (0x19be bytes) 000131 1759820442.551380 mapread 0x0356cb thru 0x03fede (0xa814 bytes) 000132 1759820442.557049 fallocate 0x030757 thru 0x0312d3 (0xb7d bytes) 000133 1759820442.560050 punch from 0x02b611 to 0x03840b (0xcdfb bytes) 000134 1759820442.563206 mapread 0x006182 thru 0x01603c (0xfebb bytes) 000135 1759820442.564368 trunc from 0x00ddbd to 0x010a56 (0x2c9a bytes) 000136 1759820442.567176 write 0x02667f thru 0x029884 (0x3206 bytes) 000137 1759820442.571923 mapread 0x021e5c thru 0x02796f (0x5b14 bytes) 000138 1759820442.572978 trunc from 0x0351d5 to 0x03cbec (0x7a18 bytes) 000140 1759820442.576591 trunc from 0x040000 to 0x023edf (0x1c120 bytes) 000141 1759820442.584027 read 0x0002cd thru 0x002d57 (0x2a8b bytes) 000142 1759820442.587045 read 0x010837 thru 0x01303c (0x2806 bytes) 000143 1759820442.590225 fallocate 0x02335a thru 0x02ba8c (0x8733 bytes) 000144 1759820442.593194 punch from 0x028ff4 to 0x029e3a (0xe47 bytes) 000145 1759820442.596113 fallocate 0x024161 thru 0x034155 (0xfff5 bytes) 000146 1759820442.599135 mapread 0x0156fc thru 0x01d9d9 (0x82de bytes) 000147 1759820442.600216 write 0x005f95 thru 0x015ca8 (0xfd14 bytes) 000148 1759820442.601999 mapread 0x022d5d thru 0x02888a (0x5b2e bytes) 000149 1759820442.606641 mapwrite 0x00652e thru 0x014293 (0xdd66 bytes) 000150 1759820442.617610 read 0x0171dc thru 0x023108 (0xbf2d bytes) 000151 1759820442.618355 fallocate 0x009dd4 thru 0x00e658 (0x4885 bytes) 000152 1759820442.621201 trunc from 0x02ba8d to 0x02fcda (0x424e bytes) 000153 1759820442.627648 write 0x01a7f9 thru 0x01b382 (0xb8a bytes) 000154 1759820442.628752 punch from 0x0152d9 to 0x020a0e (0xb736 bytes) 000155 1759820442.633646 read 0x0283e2 thru 0x029ce6 (0x1905 bytes) 000156 1759820442.634342 mapread 0x00d385 thru 0x00f6da (0x2356 bytes) 000158 1759820442.636062 punch from 0x010c88 to 0x018daf (0x8128 bytes) 000159 1759820442.638838 write 0x01c23b thru 0x023572 (0x7338 bytes) 000160 1759820442.644158 fallocate 0x02eb29 thru 0x035d7b (0x7253 bytes) 000161 1759820442.646984 read 0x01af0d thru 0x023b85 (0x8c79 bytes) 000162 1759820442.647757 mapread 0x026212 thru 0x02fcda (0x9ac9 bytes) 000163 1759820442.648885 write 0x001db5 thru 0x0020f8 (0x344 bytes) 000165 1759820442.650492 write 0x01566b thru 0x025428 (0xfdbe bytes) 000166 1759820442.652387 write 0x035e6b thru 0x03ffff (0xa195 bytes) 000167 1759820442.654028 mapread 0x02a2a4 thru 0x02f170 (0x4ecd bytes) 000168 1759820442.655033 mapread 0x028fd6 thru 0x02eb46 (0x5b71 bytes) 000169 1759820442.656020 trunc from 0x040000 to 0x025f17 (0x1a0e8 bytes) 000170 1759820442.662774 trunc from 0x008aca to 0x00a6ed (0x1c24 bytes) 000171 1759820442.665333 mapwrite 0x01691f thru 0x023295 (0xc977 bytes) 000172 1759820442.676232 mapread 0x006e21 thru 0x00c3ae (0x558e bytes) 000173 1759820442.680047 trunc from 0x016f62 to 0x02495f (0xd9fe bytes) 000175 1759820442.683349 mapread 0x018b20 thru 0x019ed3 (0x13b4 bytes) 000176 1759820442.687858 trunc from 0x00e8de to 0x013345 (0x4a68 bytes) 000177 1759820442.690327 mapread 0x0141a7 thru 0x021853 (0xd6ad bytes) 000178 1759820442.709201 mapwrite 0x015562 thru 0x021358 (0xbdf7 bytes) 000179 1759820442.719276 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000180 1759820442.783565 trunc from 0x025e12 to 0x025f17 (0x106 bytes) 000181 1759820442.785996 trunc from 0x0075d6 to 0x012a7b (0xb4a6 bytes) 000182 1759820442.788047 write 0x02e417 thru 0x03585d (0x7447 bytes) 000183 1759820442.790829 mapread 0x0182bb thru 0x01b5a4 (0x32ea bytes) 000184 1759820442.797316 read 0x0323f5 thru 0x032ff5 (0xc01 bytes) 000186 1759820442.798346 mapwrite 0x033c71 thru 0x03a086 (0x6416 bytes) 000187 1759820442.813033 trunc from 0x03a087 to 0x038695 (0x19f1 bytes) 000188 1759820442.818668 write 0x0207ea thru 0x02e95a (0xe171 bytes) 000189 1759820442.821079 mapread 0x03504e thru 0x038695 (0x3648 bytes) 000190 1759820442.822154 write 0x00e0c6 thru 0x00f678 (0x15b3 bytes) 000191 1759820442.825962 read 0x034117 thru 0x036250 (0x213a bytes) 000192 1759820442.826397 mapwrite 0x01f24c thru 0x029b86 (0xa93b bytes) 000193 1759820442.834966 write 0x03f1c8 thru 0x03ffff (0xe38 bytes) 000194 1759820442.837859 MIRROR_EXTEND to 4 mirrors 000197 1759820443.093115 trunc from 0x0381bd to 0x039e88 (0x1ccc bytes) 000198 1759820443.109397 fallocate 0x01d263 thru 0x025428 (0x81c6 bytes) 000199 1759820443.113013 mapwrite 0x03983d thru 0x03ffff (0x67c3 bytes) 000201 1759820443.134399 MIRROR_EXTEND to 5 mirrors 000202 1759820443.410859 trunc from 0x040000 to 0x038ebc (0x7143 bytes) 000203 1759820443.429663 write 0x02238a thru 0x031ff7 (0xfc6e bytes) 000204 1759820443.433126 trunc from 0x038ebd to 0x036f88 (0x1f34 bytes) 000205 1759820443.438420 mapread 0x014485 thru 0x01d6fb (0x9277 bytes) 000206 1759820443.443890 trunc from 0x036f89 to 0x0357d7 (0x17b1 bytes) 000207 1759820443.448257 read 0x010397 thru 0x019e81 (0x9aeb bytes) 000208 1759820443.449971 MIRROR_RESYNC 5 mirrors 000209 1759820443.784607 MIRROR_RESYNC 5 mirrors 000213 1759820443.873992 write 0x00819d thru 0x0156b4 (0xd518 bytes) 000214 1759820443.893210 write 0x029ed7 thru 0x02f18c (0x52b6 bytes) 000215 1759820443.897399 fallocate 0x021922 thru 0x027b44 (0x6223 bytes) 000216 1759820443.900185 mapread 0x0337bd thru 0x0357d7 (0x201b bytes) 000217 1759820443.903235 trunc from 0x020392 to 0x02edde (0xea4d bytes) 000218 1759820443.910904 mapwrite 0x017bd9 thru 0x01db37 (0x5f5f bytes) 000219 1759820443.925705 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000221 1759820443.999042 trunc from 0x0029f6 to 0x00d0d6 (0xa6e1 bytes) 000222 1759820444.001709 trunc from 0x0357d8 to 0x00f1f9 (0x265de bytes) 000223 1759820444.018851 trunc from 0x00f1fa to 0x03164c (0x22453 bytes) 000224 1759820444.025518 mapread 0x00e091 thru 0x014d3c (0x6cac bytes) 000225 1759820444.033762 mapwrite 0x00823a thru 0x01630a (0xe0d1 bytes) 000226 1759820444.047292 trunc from 0x03164d to 0x014565 (0x1d0e7 bytes) 000227 1759820444.054468 write 0x00b7c1 thru 0x0111e7 (0x5a27 bytes) 000228 1759820444.055682 trunc from 0x014566 to 0x02bafc (0x17597 bytes) 000229 1759820444.062818 mapread 0x011b1e thru 0x013fea (0x24cd bytes) 000230 1759820444.063506 trunc from 0x00b7f3 to 0x00bc0b (0x419 bytes) 000231 1759820444.069618 trunc from 0x02bafd to 0x0007d4 (0x2b328 bytes) 000232 1759820444.076662 punch from 0x0004ca to 0x0007d4 (0x30b bytes) 000233 1759820444.078841 fallocate 0x008640 thru 0x016db3 (0xe774 bytes) 000234 1759820444.081497 write 0x006486 thru 0x007556 (0x10d1 bytes) 000235 1759820444.086013 read 0x007688 thru 0x01586c (0xe1e5 bytes) 000236 1759820444.090144 mapread 0x005db2 thru 0x0117ad (0xb9fc bytes) 000238 1759820444.091114 trunc from 0x008d45 to 0x00ba96 (0x2d52 bytes) 000239 1759820444.095740 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000240 1759820444.171546 trunc from 0x0164a7 to 0x016db3 (0x90d bytes) 000241 1759820444.174180 trunc from 0x00e58c to 0x016db3 (0x8828 bytes) 000242 1759820444.176599 mapwrite 0x03fbcc thru 0x03ffff (0x434 bytes) 000243 1759820444.194131 mapread 0x00feb7 thru 0x01aa70 (0xabba bytes) 000244 1759820444.204738 write 0x030b47 thru 0x03e7d9 (0xdc93 bytes) 000245 1759820444.211609 write 0x02270c thru 0x028109 (0x59fe bytes) 000246 1759820444.215854 read 0x00f96f thru 0x01012d (0x7bf bytes) 000247 1759820444.216238 trunc from 0x003829 to 0x00de21 (0xa5f9 bytes) 000248 1759820444.219074 mapwrite 0x033a8e thru 0x035685 (0x1bf8 bytes) 000249 1759820444.227389 mapread 0x01b9e4 thru 0x025495 (0x9ab2 bytes) 000250 1759820444.227978 trunc from 0x035286 to 0x03f010 (0x9d8b bytes) 000251 1759820444.230537 write 0x03aaf0 thru 0x03ffff (0x5510 bytes) 000252 1759820444.234760 fallocate 0x032f93 thru 0x03cf09 (0x9f77 bytes) 000253 1759820444.237273 trunc from 0x040000 to 0x003975 (0x3c68a bytes) 000254 1759820444.244125 mapwrite 0x038097 thru 0x03dfc5 (0x5f2f bytes) 000256 1759820444.266386 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000257 1759820444.340403 write 0x032304 thru 0x03f03a (0xcd37 bytes) 000258 1759820444.347639 MIRROR_EXTEND to 3 mirrors 000259 1759820444.561187 mapwrite 0x026fdb thru 0x0339c8 (0xc9ee bytes) 000260 1759820444.608053 fallocate 0x008973 thru 0x00daba (0x5148 bytes) 000261 1759820444.610261 punch from 0x01b593 to 0x021507 (0x5f75 bytes) 000262 1759820444.612178 fallocate 0x02c860 thru 0x0387b2 (0xbf53 bytes) 000263 1759820444.614146 fallocate 0x01baae thru 0x01e09f (0x25f2 bytes) 000264 1759820444.616026 fallocate 0x01aa3d thru 0x0255aa (0xab6e bytes) 000265 1759820444.617946 fallocate 0x02a826 thru 0x0363b2 (0xbb8d bytes) 000266 1759820444.619776 punch from 0x03c0f5 to 0x03da3d (0x1949 bytes) 000267 1759820444.622741 MIRROR_RESYNC 3 mirrors 000268 1759820444.785751 write 0x03bdee thru 0x03c1b4 (0x3c7 bytes) 000269 1759820444.796311 write 0x02c62e thru 0x035df5 (0x97c8 bytes) 000270 1759820444.800570 trunc from 0x00556c to 0x015071 (0xfb06 bytes) 000271 1759820444.803822 MIRROR_EXTEND to 4 mirrors 000272 1759820444.990363 mapwrite 0x004037 thru 0x006ba2 (0x2b6c bytes) 000273 1759820445.039340 trunc from 0x03f03b to 0x024ff3 (0x1a047 bytes) 000274 1759820445.057398 trunc from 0x024ff4 to 0x01169f (0x13954 bytes) 000275 1759820445.063904 write 0x028f1d thru 0x02f2fd (0x63e1 bytes) 000276 1759820445.064755 mapread 0x023097 thru 0x02dc6a (0xabd4 bytes) 000278 1759820445.068228 mapread 0x0000e5 thru 0x00b6ad (0xb5c9 bytes) 000279 1759820445.070930 write 0x025815 thru 0x030151 (0xa93d bytes) 000280 1759820445.071781 punch from 0x000cff to 0x007439 (0x673b bytes) 000281 1759820445.074330 mapwrite 0x007c1c thru 0x00974b (0x1b30 bytes) 000282 1759820445.084547 read 0x020d5f thru 0x025b7a (0x4e1c bytes) 000283 1759820445.084962 trunc from 0x002a85 to 0x00d903 (0xae7f bytes) 000285 1759820445.087529 trunc from 0x004c35 to 0x006958 (0x1d24 bytes) 000286 1759820445.089201 fallocate 0x00c2b2 thru 0x012065 (0x5db4 bytes) 000287 1759820445.091275 mapwrite 0x0011bf thru 0x006315 (0x5157 bytes) 000288 1759820445.101199 trunc from 0x030152 to 0x02e194 (0x1fbd bytes) 000289 1759820445.107027 mapwrite 0x011545 thru 0x01b553 (0xa00f bytes) 000290 1759820445.119104 mapwrite 0x036520 thru 0x03ffff (0x9ae0 bytes) 000291 1759820445.137973 fallocate 0x03cc3f thru 0x03ef3d (0x22ff bytes) 000293 1759820445.142456 MIRROR_EXTEND to 5 mirrors 000294 1759820445.354718 trunc from 0x040000 to 0x02a914 (0x156eb bytes) 000295 1759820445.379171 read 0x00d13e thru 0x012656 (0x5519 bytes) 000296 1759820445.382374 fallocate 0x01bfda thru 0x01e73a (0x2761 bytes) 000297 1759820445.384531 trunc from 0x01af9a to 0x01f4a4 (0x450b bytes) 000298 1759820445.386787 write 0x019b5e thru 0x01cd30 (0x31d3 bytes) 000299 1759820445.390686 mapread 0x012c88 thru 0x01cbe5 (0x9f5e bytes) 000300 1759820445.394142 punch from 0x003f11 to 0x0134a8 (0xf598 bytes) 000301 1759820445.396372 write 0x012390 thru 0x01ad0a (0x897b bytes) 000303 1759820445.400537 write 0x0133bc thru 0x014a01 (0x1646 bytes) 000304 1759820445.401151 mapread 0x00f7e7 thru 0x010399 (0xbb3 bytes) 000305 1759820445.403524 MIRROR_EXTEND to 6 mirrors 000306 1759820445.607362 trunc from 0x0176dd to 0x0194a3 (0x1dc7 bytes) 000307 1759820445.621824 mapread 0x004d29 thru 0x011961 (0xcc39 bytes) 000308 1759820445.633031 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000309 1759820445.703755 mapread 0x00cadf thru 0x0147bc (0x7cde bytes) 000310 1759820445.710527 mapwrite 0x004a63 thru 0x010f63 (0xc501 bytes) 000311 1759820445.722447 read 0x017bdb thru 0x01b726 (0x3b4c bytes) 000312 1759820445.725799 mapread 0x02046a thru 0x0254f3 (0x508a bytes) 000314 1759820445.729913 trunc from 0x02a915 to 0x03bc1f (0x1130b bytes) 000315 1759820445.744257 mapread 0x023243 thru 0x023816 (0x5d4 bytes) 000316 1759820445.744880 write 0x021284 thru 0x029058 (0x7dd5 bytes) 000317 1759820445.747239 fallocate 0x02e5de thru 0x033f42 (0x5965 bytes) 000318 1759820445.749475 fallocate 0x03ecda thru 0x03ffff (0x1326 bytes) 000319 1759820445.751582 mapread 0x00f218 thru 0x01a5f0 (0xb3d9 bytes) 000320 1759820445.752194 write 0x0322ec thru 0x033f40 (0x1c55 bytes) 000321 1759820445.756313 write 0x038bfd thru 0x03ffff (0x7403 bytes) 000322 1759820445.757104 mapread 0x00addd thru 0x0120b7 (0x72db bytes) 000323 1759820445.757758 mapwrite 0x016f1a thru 0x01e18b (0x7272 bytes) 000324 1759820445.766458 write 0x010bf4 thru 0x0142ec (0x36f9 bytes) 000325 1759820445.767313 read 0x00ce94 thru 0x0106f5 (0x3862 bytes) 000326 1759820445.767741 mapwrite 0x008b6f thru 0x0135fa (0xaa8c bytes) 000327 1759820445.776167 fallocate 0x0146e1 thru 0x01731a (0x2c3a bytes) 000329 1759820445.780957 MIRROR_EXTEND to 6 mirrors 000330 1759820446.010801 fallocate 0x001a81 thru 0x0060b3 (0x4633 bytes) 000331 1759820446.025071 MIRROR_EXTEND to 7 mirrors 000332 1759820446.267916 read 0x00bf14 thru 0x00fd1c (0x3e09 bytes) 000333 1759820446.272611 trunc from 0x020ed2 to 0x02a147 (0x9276 bytes) 000334 1759820446.283784 trunc from 0x040000 to 0x0129f1 (0x2d60e bytes) 000335 1759820446.291374 mapread 0x005d23 thru 0x0129f1 (0xcccf bytes) 000336 1759820446.298369 mapwrite 0x036503 thru 0x03f18f (0x8c8d bytes) 000337 1759820446.313506 trunc from 0x03f190 to 0x0088a3 (0x368ec bytes) 000339 1759820446.318324 fallocate 0x02367c thru 0x032542 (0xeec7 bytes) 000340 1759820446.321714 MIRROR_RESYNC 7 mirrors 000341 1759820446.591259 trunc from 0x0020bb to 0x0082b2 (0x61f8 bytes) 000342 1759820446.601235 trunc from 0x0071a2 to 0x0088a3 (0x1702 bytes) 000343 1759820446.602946 write 0x025440 thru 0x02e56d (0x912e bytes) 000344 1759820446.603893 write 0x0099c0 thru 0x010db3 (0x73f4 bytes) 000345 1759820446.607672 read 0x018a47 thru 0x01aece (0x2488 bytes) 000346 1759820446.611583 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000347 1759820446.678119 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000349 1759820446.733936 MIRROR_EXTEND to 6 mirrors 000351 1759820446.902932 fallocate 0x02397a thru 0x026a36 (0x30bd bytes) 000352 1759820446.914204 read 0x012d20 thru 0x01c90b (0x9bec bytes) 000353 1759820446.916958 trunc from 0x02e56e to 0x0046c4 (0x29ea9 bytes) 000354 1759820446.928781 fallocate 0x0323bd thru 0x033fd0 (0x1c14 bytes) 000355 1759820446.930613 fallocate 0x035400 thru 0x03ffff (0xac00 bytes) 000356 1759820446.933518 MIRROR_EXTEND to 7 mirrors 000357 1759820447.097728 write 0x03c1a7 thru 0x03ffff (0x3e59 bytes) 000358 1759820447.106751 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000359 1759820447.173818 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000360 1759820447.233254 punch from 0x007261 to 0x015a96 (0xe836 bytes) 000361 1759820447.235377 write 0x036521 thru 0x03ffff (0x9adf bytes) 000362 1759820447.239374 mapwrite 0x01e3f6 thru 0x028119 (0x9d24 bytes) 000363 1759820447.252608 trunc from 0x03f63e to 0x03ffff (0x9c2 bytes) 000364 1759820447.254198 trunc from 0x03f9b7 to 0x03ffff (0x649 bytes) 000365 1759820447.257021 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000366 1759820447.309260 mapread 0x02d07b thru 0x0370d3 (0xa059 bytes) 000367 1759820447.314340 mapwrite 0x01e717 thru 0x02e643 (0xff2d bytes) 000368 1759820447.325892 trunc from 0x040000 to 0x014d72 (0x2b28d bytes) 000369 1759820447.331460 mapwrite 0x03c9b7 thru 0x03ffff (0x3649 bytes) 000370 1759820447.342338 trunc from 0x040000 to 0x00ae9a (0x35165 bytes) 000371 1759820447.348051 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000373 1759820447.399400 mapread 0x0061c4 thru 0x00ae9a (0x4cd7 bytes) 000374 1759820447.405209 mapwrite 0x0077c8 thru 0x00b25e (0x3a97 bytes) 000375 1759820447.417400 mapread 0x00515d thru 0x005414 (0x2b8 bytes) 000376 1759820447.417935 trunc from 0x003dc2 to 0x00b25e (0x749d bytes) 000377 1759820447.419626 mapread 0x0054dd thru 0x00b25e (0x5d82 bytes) 000378 1759820447.422282 mapwrite 0x00ef7a thru 0x01a024 (0xb0ab bytes) 000379 1759820447.436932 trunc from 0x01a025 to 0x00ae38 (0xf1ec bytes) 000381 1759820447.442020 fallocate 0x00fe35 thru 0x019dff (0x9fcb bytes) 000382 1759820447.443852 punch from 0x0057b1 to 0x00ccd5 (0x7525 bytes) 000383 1759820447.445604 mapread 0x005076 thru 0x006ffd (0x1f88 bytes) 000384 1759820447.448590 fallocate 0x010593 thru 0x01d3b4 (0xce22 bytes) 000386 1759820447.450923 write 0x03d923 thru 0x03ffff (0x26dd bytes) 000388 1759820447.451826 read 0x01e36f thru 0x027273 (0x8f05 bytes) 000389 1759820447.454560 mapwrite 0x036491 thru 0x03ffff (0x9b6f bytes) 000390 1759820447.463112 trunc from 0x00e4f7 to 0x011855 (0x335f bytes) 000391 1759820447.464769 trunc from 0x040000 to 0x01d6e5 (0x2291a bytes) 000392 1759820447.470856 MIRROR_RESYNC 3 mirrors 000393 1759820447.661392 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000394 1759820447.759516 punch from 0x00a9ee to 0x00cb5c (0x216f bytes) 000395 1759820447.776023 punch from 0x018880 to 0x01d6e5 (0x4e66 bytes) 000396 1759820447.778695 read 0x015919 thru 0x01d6e5 (0x7dcd bytes) 000397 1759820447.783581 mapread 0x013838 thru 0x01d6e5 (0x9eae bytes) 000398 1759820447.789156 trunc from 0x01d6e6 to 0x00d47b (0x1026a bytes) 000399 1759820447.808940 trunc from 0x005a1c to 0x00d47b (0x7a60 bytes) 000400 1759820447.812056 fallocate 0x006f97 thru 0x015479 (0xe4e3 bytes) 000401 1759820447.817384 MIRROR_RESYNC 2 mirrors 000402 1759820448.030849 write 0x0352ac thru 0x03ffff (0xad54 bytes) 000403 1759820448.039557 write 0x0215d9 thru 0x022c1e (0x1646 bytes) 000404 1759820448.042529 punch from 0x00e8ee to 0x010adb (0x21ee bytes) 000405 1759820448.044211 fallocate 0x0157c2 thru 0x022add (0xd31c bytes) 000406 1759820448.045969 punch from 0x03d654 to 0x03ffff (0x29ac bytes) 000407 1759820448.049733 punch from 0x03c568 to 0x03ffff (0x3a98 bytes) 000408 1759820448.051232 trunc from 0x002a0c to 0x00712a (0x471f bytes) 000409 1759820448.052838 mapread 0x00a66f thru 0x00f141 (0x4ad3 bytes) 000410 1759820448.055621 trunc from 0x040000 to 0x023a47 (0x1c5b8 bytes) 000412 1759820448.065591 trunc from 0x023a48 to 0x01e153 (0x58f4 bytes) 000413 1759820448.071197 MIRROR_RESYNC 2 mirrors 000414 1759820448.248649 punch from 0x00d193 to 0x01cfb9 (0xfe27 bytes) 000415 1759820448.256275 punch from 0x006e10 to 0x00c6d3 (0x58c4 bytes) 000416 1759820448.257855 write 0x03735d thru 0x03f6b1 (0x8355 bytes) 000417 1759820448.258773 write 0x009a1c thru 0x00f3c2 (0x59a7 bytes) 000418 1759820448.262025 fallocate 0x005c44 thru 0x00e2ed (0x86aa bytes) 000419 1759820448.263829 read 0x008fad thru 0x00cc95 (0x3ce9 bytes) 000420 1759820448.266850 MIRROR_EXTEND to 3 mirrors 000421 1759820448.453174 mapread 0x00238f thru 0x005bd3 (0x3845 bytes) 000422 1759820448.459261 mapread 0x021d3f thru 0x031782 (0xfa44 bytes) 000423 1759820448.465977 read 0x020943 thru 0x023836 (0x2ef4 bytes) 000424 1759820448.466607 trunc from 0x03f6b2 to 0x00db22 (0x31b8f bytes) 000425 1759820448.493083 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000426 1759820448.556573 trunc from 0x00db23 to 0x021b50 (0x1402e bytes) 000427 1759820448.563528 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000428 1759820448.596278 mapread 0x01adf3 thru 0x021b50 (0x6d5e bytes) 000429 1759820448.601927 mapwrite 0x0084d8 thru 0x011bb0 (0x96d9 bytes) 000430 1759820448.611935 trunc from 0x006ab8 to 0x00a62f (0x3b78 bytes) 000432 1759820448.614418 mapwrite 0x022131 thru 0x02bcbb (0x9b8b bytes) 000433 1759820448.631589 trunc from 0x025e14 to 0x02bcbb (0x5ea8 bytes) 000434 1759820448.633814 mapwrite 0x03a8fa thru 0x03ffff (0x5706 bytes) 000435 1759820448.648518 mapread 0x0152af thru 0x018985 (0x36d7 bytes) 000436 1759820448.649206 trunc from 0x040000 to 0x02be99 (0x14166 bytes) 000437 1759820448.654266 punch from 0x01b02b to 0x021e97 (0x6e6d bytes) 000438 1759820448.656200 read 0x004734 thru 0x0060a3 (0x1970 bytes) 000441 1759820448.659796 fallocate 0x00aa91 thru 0x012eb5 (0x8425 bytes) 000442 1759820448.662119 mapwrite 0x02eabc thru 0x031297 (0x27dc bytes) 000443 1759820448.674409 trunc from 0x00223b to 0x00b67e (0x9444 bytes) 000444 1759820448.676328 trunc from 0x031298 to 0x01dfdf (0x132b8 bytes) 000445 1759820448.682435 write 0x00bd56 thru 0x01901d (0xd2c8 bytes) 000446 1759820448.686898 mapread 0x010463 thru 0x0118d8 (0x1476 bytes) 000448 1759820448.688286 punch from 0x00134d to 0x00bfca (0xac7e bytes) 000449 1759820448.694604 read 0x0139c6 thru 0x016e93 (0x34ce bytes) 000450 1759820448.695495 write 0x025d4b thru 0x030c90 (0xaf46 bytes) 000451 1759820448.697554 mapwrite 0x02d05c thru 0x035d43 (0x8ce8 bytes) 000452 1759820448.714751 read 0x02feb2 thru 0x035d43 (0x5e92 bytes) 000453 1759820448.715456 fallocate 0x003ec4 thru 0x00be3a (0x7f77 bytes) 000454 1759820448.718131 punch from 0x0057c6 to 0x00e053 (0x888e bytes) 000455 1759820448.720404 trunc from 0x035d44 to 0x02204e (0x13cf5 bytes) 000456 1759820448.725211 punch from 0x015963 to 0x01a9dd (0x507b bytes) 000458 1759820448.727463 punch from 0x013c84 to 0x016fac (0x3329 bytes) 000459 1759820448.729097 mapread 0x00959d thru 0x00e252 (0x4cb6 bytes) 000460 1759820448.731886 punch from 0x00addb to 0x015a1b (0xac41 bytes) 000461 1759820448.733306 read 0x01a23a thru 0x020f75 (0x6d3c bytes) 000462 1759820448.733710 mapread 0x00afde thru 0x00b1d7 (0x1fa bytes) 000463 1759820448.736158 mapwrite 0x03a493 thru 0x03ffff (0x5b6d bytes) 000464 1759820448.751188 write 0x00bb6f thru 0x01431c (0x87ae bytes) 000465 1759820448.752778 write 0x0170c7 thru 0x018d45 (0x1c7f bytes) 000466 1759820448.753692 read 0x03c25b thru 0x03ecd1 (0x2a77 bytes) 000467 1759820448.754303 mapwrite 0x005ed6 thru 0x01102f (0xb15a bytes) 000468 1759820448.764078 trunc from 0x00c3d4 to 0x00fe01 (0x3a2e bytes) 000470 1759820448.767163 write 0x007e88 thru 0x009320 (0x1499 bytes) 000471 1759820448.767998 trunc from 0x03a3a3 to 0x03ffff (0x5c5d bytes) 000472 1759820448.770411 mapwrite 0x00d5e5 thru 0x0151bc (0x7bd8 bytes) 000474 1759820448.781221 write 0x02e3d7 thru 0x036ef2 (0x8b1c bytes) 000475 1759820448.781980 mapwrite 0x003fc7 thru 0x005947 (0x1981 bytes) 000476 1759820448.788553 punch from 0x02a1c7 to 0x03377a (0x95b4 bytes) 000477 1759820448.790832 fallocate 0x0330d9 thru 0x039ad1 (0x69f9 bytes) 000478 1759820448.793476 read 0x026e96 thru 0x036d07 (0xfe72 bytes) 000479 1759820448.797660 mapread 0x011d9c thru 0x013121 (0x1386 bytes) 000480 1759820448.798403 punch from 0x02d74d to 0x039bab (0xc45f bytes) 000481 1759820448.800829 punch from 0x032b7e to 0x0360ec (0x356f bytes) 000483 1759820448.803485 mapwrite 0x039b0e thru 0x03b250 (0x1743 bytes) 000484 1759820448.813082 write 0x00e8e0 thru 0x01d576 (0xec97 bytes) 000485 1759820448.814516 mapread 0x01066f thru 0x01102b (0x9bd bytes) 000486 1759820448.815280 trunc from 0x019580 to 0x01a75f (0x11e0 bytes) 000487 1759820448.820845 fallocate 0x03ec42 thru 0x03ffff (0x13be bytes) 000489 1759820448.824539 trunc from 0x010ee6 to 0x018cc2 (0x7ddd bytes) 000490 1759820448.827320 trunc from 0x01ad4e to 0x0227ef (0x7aa2 bytes) 000491 1759820448.829826 write 0x0090b6 thru 0x00a5c6 (0x1511 bytes) 000492 1759820448.830820 trunc from 0x011bb6 to 0x0203b6 (0xe801 bytes) 000493 1759820448.833772 fallocate 0x002357 thru 0x00e938 (0xc5e2 bytes) 000494 1759820448.836764 mapwrite 0x032b7d thru 0x03a8a0 (0x7d24 bytes) 000495 1759820448.866120 mapwrite 0x03388d thru 0x03ffff (0xc773 bytes) 000496 1759820448.877059 trunc from 0x040000 to 0x023a71 (0x1c58e bytes) 000497 1759820448.884850 write 0x0328b5 thru 0x035c06 (0x3352 bytes) 000498 1759820448.886051 mapread 0x01b043 thru 0x01dee7 (0x2ea5 bytes) 000499 1759820448.892749 trunc from 0x035c07 to 0x013db0 (0x21e56 bytes) 000501 1759820448.899886 fallocate 0x012da4 thru 0x01534c (0x25a9 bytes) 000502 1759820448.902231 mapread 0x00b577 thru 0x0120da (0x6b64 bytes) 000503 1759820448.907601 punch from 0x00179c to 0x00acab (0x9510 bytes) 000504 1759820448.909558 mapwrite 0x0162df thru 0x01b12c (0x4e4e bytes) 000505 1759820448.928498 trunc from 0x01b12d to 0x004bcc (0x16560 bytes) 000506 1759820448.934788 punch from 0x000c25 to 0x004bcc (0x3fa8 bytes) 000507 1759820448.936728 trunc from 0x004bcd to 0x034f43 (0x30377 bytes) 000508 1759820448.941500 write 0x03063d thru 0x03aec5 (0xa889 bytes) 000510 1759820448.944583 punch from 0x018261 to 0x02179c (0x953c bytes) 000511 1759820448.946343 trunc from 0x03aec6 to 0x0143d9 (0x26aec bytes) 000512 1759820448.951229 trunc from 0x003a4b to 0x00aa7c (0x7032 bytes) 000513 1759820448.953070 mapwrite 0x024f70 thru 0x026067 (0x10f8 bytes) 000514 1759820448.966653 fallocate 0x00077c thru 0x00ce84 (0xc709 bytes) 000515 1759820448.968988 mapread 0x0038a5 thru 0x007439 (0x3b95 bytes) 000516 1759820448.977051 fallocate 0x0362d5 thru 0x03fd31 (0x9a5d bytes) 000517 1759820448.979014 mapread 0x00a37e thru 0x015000 (0xac83 bytes) 000518 1759820448.994676 read 0x0103f1 thru 0x019760 (0x9370 bytes) 000519 1759820448.997835 trunc from 0x03f1c8 to 0x03f5f2 (0x42b bytes) 000520 1759820448.999458 write 0x0335fa thru 0x03e04b (0xaa52 bytes) 000521 1759820449.000314 fallocate 0x03c8d9 thru 0x03ffff (0x3727 bytes) 000522 1759820449.001943 fallocate 0x01a9ea thru 0x0240bc (0x96d3 bytes) 000524 1759820449.004000 mapread 0x0255ec thru 0x026eca (0x18df bytes) 000525 1759820449.004541 punch from 0x027c1e to 0x02c9bc (0x4d9f bytes) 000527 1759820449.006359 mapwrite 0x019431 thru 0x0206a5 (0x7275 bytes) 000528 1759820449.012281 mapread 0x012eb9 thru 0x021b25 (0xec6d bytes) 000530 1759820449.013190 fallocate 0x03f498 thru 0x03f5ad (0x116 bytes) 000532 1759820449.015112 mapwrite 0x0206ce thru 0x02534d (0x4c80 bytes) 000533 1759820449.021923 MIRROR_EXTEND to 2 mirrors 000535 1759820449.225183 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000536 1759820449.273016 fallocate 0x01ba62 thru 0x029bb4 (0xe153 bytes) 000537 1759820449.278670 trunc from 0x03fd32 to 0x00d861 (0x324d0 bytes) 000538 1759820449.293712 mapread 0x000757 thru 0x008690 (0x7f3a bytes) 000540 1759820449.304804 MIRROR_EXTEND to 2 mirrors 000541 1759820449.486551 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000542 1759820449.524583 fallocate 0x019b69 thru 0x020d7f (0x7217 bytes) 000543 1759820449.528265 mapwrite 0x030c3e thru 0x03c0df (0xb4a2 bytes) 000544 1759820449.549421 fallocate 0x0328d0 thru 0x03d904 (0xb035 bytes) 000545 1759820449.551122 write 0x025239 thru 0x02ce9f (0x7c67 bytes) 000546 1759820449.554106 mapread 0x01d81a thru 0x024a14 (0x71fb bytes) 000547 1759820449.556826 trunc from 0x03d905 to 0x02fd66 (0xdb9e bytes) 000548 1759820449.562503 MIRROR_EXTEND to 2 mirrors 000550 1759820449.738323 mapread 0x02b923 thru 0x02fd66 (0x4444 bytes) 000551 1759820449.747284 trunc from 0x02fd67 to 0x030019 (0x2b3 bytes) 000552 1759820449.764947 mapwrite 0x03e786 thru 0x03ffff (0x187a bytes) 000554 1759820449.776794 mapwrite 0x030e7c thru 0x03694d (0x5ad2 bytes) 000556 1759820449.784781 fallocate 0x0223c5 thru 0x025d3a (0x3976 bytes) 000557 1759820449.786525 mapread 0x0219ea thru 0x027425 (0x5a3c bytes) 000558 1759820449.789261 trunc from 0x000657 to 0x00ad32 (0xa6dc bytes) 000559 1759820449.791030 mapread 0x029751 thru 0x0367ca (0xd07a bytes) 000560 1759820449.791700 write 0x009c60 thru 0x00e345 (0x46e6 bytes) 000561 1759820449.794685 trunc from 0x040000 to 0x0100d0 (0x2ff2f bytes) 000562 1759820449.800871 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000563 1759820449.838900 write 0x03273f thru 0x037a79 (0x533b bytes) 000564 1759820449.840956 trunc from 0x00c648 to 0x00e184 (0x1b3d bytes) 000565 1759820449.842602 write 0x011d05 thru 0x0182d3 (0x65cf bytes) 000566 1759820449.845670 punch from 0x002de1 to 0x003477 (0x697 bytes) 000567 1759820449.847220 write 0x006815 thru 0x01254a (0xbd36 bytes) 000568 1759820449.850323 mapwrite 0x034e19 thru 0x036875 (0x1a5d bytes) 000569 1759820449.855923 fallocate 0x014c35 thru 0x02433d (0xf709 bytes) 000570 1759820449.857717 trunc from 0x037a7a to 0x009b57 (0x2df22 bytes) 000571 1759820449.862239 read 0x005840 thru 0x007df3 (0x25b4 bytes) 000572 1759820449.864020 MIRROR_EXTEND to 2 mirrors 000573 1759820450.017943 trunc from 0x009b58 to 0x039f25 (0x303ce bytes) 000574 1759820450.033584 mapwrite 0x003a34 thru 0x0052f3 (0x18c0 bytes) 000575 1759820450.043288 mapread 0x0019d3 thru 0x006351 (0x497f bytes) 000577 1759820450.044145 trunc from 0x039f26 to 0x03b2d6 (0x13b1 bytes) 000578 1759820450.049790 fallocate 0x03461d thru 0x03bce1 (0x76c5 bytes) 000579 1759820450.051760 punch from 0x01b439 to 0x0252d8 (0x9ea0 bytes) 000580 1759820450.053341 write 0x021649 thru 0x02c419 (0xadd1 bytes) 000581 1759820450.057962 mapread 0x033758 thru 0x03b2d6 (0x7b7f bytes) 000582 1759820450.060829 mapwrite 0x00d82e thru 0x019dbc (0xc58f bytes) 000583 1759820450.070005 mapwrite 0x000d02 thru 0x003377 (0x2676 bytes) 000584 1759820450.076145 fallocate 0x00c714 thru 0x01437d (0x7c6a bytes) 000585 1759820450.077956 fallocate 0x034514 thru 0x0387f3 (0x42e0 bytes) 000586 1759820450.079669 mapread 0x014167 thru 0x01722b (0x30c5 bytes) 000587 1759820450.080191 punch from 0x00087a to 0x006d8c (0x6513 bytes) 000588 1759820450.081928 write 0x036aac thru 0x03c394 (0x58e9 bytes) 000589 1759820450.082686 mapwrite 0x03013d thru 0x0384da (0x839e bytes) 000590 1759820450.088549 write 0x03c676 thru 0x03ffff (0x398a bytes) 000591 1759820450.089144 trunc from 0x001ce3 to 0x008a5b (0x6d79 bytes) 000592 1759820450.090986 punch from 0x022fcb to 0x023de6 (0xe1c bytes) 000593 1759820450.092521 punch from 0x037357 to 0x03f7a6 (0x8450 bytes) 000594 1759820450.095279 trunc from 0x018958 to 0x0190a5 (0x74e bytes) 000595 1759820450.096850 trunc from 0x040000 to 0x01cd27 (0x232d8 bytes) 000596 1759820450.101465 trunc from 0x014883 to 0x01cd27 (0x84a5 bytes) 000597 1759820450.103002 write 0x00aac2 thru 0x00ade9 (0x328 bytes) 000598 1759820450.103543 mapread 0x0007e1 thru 0x00dbf1 (0xd411 bytes) 000600 1759820450.106537 mapread 0x005305 thru 0x009bdc (0x48d8 bytes) 000601 1759820450.107029 mapwrite 0x00fd1f thru 0x012423 (0x2705 bytes) 000602 1759820450.112002 punch from 0x0180d7 to 0x01cd27 (0x4c51 bytes) 000603 1759820450.113704 punch from 0x002849 to 0x00521c (0x29d4 bytes) 000605 1759820450.115655 mapread 0x006834 thru 0x01610c (0xf8d9 bytes) 000606 1759820450.118332 mapread 0x0146a2 thru 0x017168 (0x2ac7 bytes) 000607 1759820450.120173 MIRROR_RESYNC 2 mirrors 000608 1759820450.284388 mapwrite 0x02a750 thru 0x033d17 (0x95c8 bytes) 000609 1759820450.311728 read 0x001738 thru 0x003cac (0x2575 bytes) 000610 1759820450.313707 trunc from 0x033d18 to 0x009f4d (0x29dca bytes) 000612 1759820450.320005 mapread 0x008050 thru 0x009f4d (0x1efe bytes) 000613 1759820450.324256 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000614 1759820450.359749 mapwrite 0x00763a thru 0x00bebd (0x4884 bytes) 000615 1759820450.375083 fallocate 0x032304 thru 0x033cba (0x19b7 bytes) 000616 1759820450.376886 trunc from 0x003ade to 0x003b4b (0x6e bytes) 000617 1759820450.378465 write 0x005ce0 thru 0x009633 (0x3954 bytes) 000618 1759820450.379082 trunc from 0x00bebe to 0x02aeb6 (0x1eff9 bytes) 000619 1759820450.383311 mapread 0x000935 thru 0x00e6cf (0xdd9b bytes) 000620 1759820450.385973 fallocate 0x035888 thru 0x0363a3 (0xb1c bytes) 000621 1759820450.387785 mapwrite 0x032e5d thru 0x03a0b1 (0x7255 bytes) 000622 1759820450.400259 mapwrite 0x001301 thru 0x00bac4 (0xa7c4 bytes) 000623 1759820450.406575 fallocate 0x0349fb thru 0x03ffff (0xb605 bytes) 000625 1759820450.408907 mapread 0x038068 thru 0x03ffff (0x7f98 bytes) 000626 1759820450.411632 mapwrite 0x026b6e thru 0x034907 (0xdd9a bytes) 000627 1759820450.420656 trunc from 0x00006d to 0x005f78 (0x5f0c bytes) 000628 1759820450.422317 fallocate 0x009fe6 thru 0x012135 (0x8150 bytes) 000630 1759820450.424417 mapread 0x03aed8 thru 0x03ffff (0x5128 bytes) 000631 1759820450.424983 fallocate 0x0286d3 thru 0x02f423 (0x6d51 bytes) 000632 1759820450.426772 trunc from 0x022746 to 0x02a1a0 (0x7a5b bytes) 000633 1759820450.428456 read 0x0093ab thru 0x017a7c (0xe6d2 bytes) 000634 1759820450.430986 trunc from 0x040000 to 0x003273 (0x3cd8c bytes) 000635 1759820450.435582 mapwrite 0x01974f thru 0x02230c (0x8bbe bytes) 000637 1759820450.449407 trunc from 0x02230d to 0x02095e (0x19ae bytes) 000638 1759820450.453583 fallocate 0x0287b1 thru 0x02f21b (0x6a6b bytes) 000639 1759820450.456674 MIRROR_EXTEND to 2 mirrors 000640 1759820450.611131 trunc from 0x0219df to 0x02bbd7 (0xa1f9 bytes) 000641 1759820450.620937 mapwrite 0x01ba0b thru 0x022552 (0x6b48 bytes) 000642 1759820450.631735 fallocate 0x03e154 thru 0x03ffff (0x1eac bytes) 000643 1759820450.633668 trunc from 0x01b342 to 0x021e84 (0x6b43 bytes) 000644 1759820450.636707 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000647 1759820450.680689 read 0x018216 thru 0x020778 (0x8563 bytes) 000648 1759820450.683919 fallocate 0x016319 thru 0x020d15 (0xa9fd bytes) 000649 1759820450.685903 write 0x0265f4 thru 0x030ab8 (0xa4c5 bytes) 000650 1759820450.690076 punch from 0x019864 to 0x026686 (0xce23 bytes) 000651 1759820450.694685 fallocate 0x002b0f thru 0x007e50 (0x5342 bytes) 000652 1759820450.696687 read 0x001316 thru 0x00e929 (0xd614 bytes) 000653 1759820450.699883 punch from 0x008ae9 to 0x013b3b (0xb053 bytes) 000654 1759820450.701535 trunc from 0x01fa4e to 0x0235b6 (0x3b69 bytes) 000655 1759820450.703087 fallocate 0x0208e7 thru 0x023ec1 (0x35db bytes) 000656 1759820450.704872 read 0x01fb44 thru 0x02108b (0x1548 bytes) 000657 1759820450.709089 MIRROR_EXTEND to 2 mirrors 000658 1759820450.874271 punch from 0x0085c3 to 0x008b8e (0x5cc bytes) 000659 1759820450.886511 mapwrite 0x006d54 thru 0x0113b7 (0xa664 bytes) 000661 1759820450.902538 fallocate 0x01e4d6 thru 0x02c2b6 (0xdde1 bytes) 000662 1759820450.904580 read 0x00abef thru 0x00c32b (0x173d bytes) 000663 1759820450.904949 write 0x0335d5 thru 0x037114 (0x3b40 bytes) 000664 1759820450.907315 trunc from 0x037115 to 0x015e31 (0x212e3 bytes) 000665 1759820450.918161 trunc from 0x00068b to 0x007a44 (0x73ba bytes) 000666 1759820450.919953 mapread 0x004d61 thru 0x006c9e (0x1f3e bytes) 000667 1759820450.922794 trunc from 0x014e8a to 0x015e31 (0xfa8 bytes) 000669 1759820450.924810 trunc from 0x015e32 to 0x004a95 (0x1139c bytes) 000670 1759820450.929860 trunc from 0x0016ed to 0x004a95 (0x33a9 bytes) 000671 1759820450.931682 trunc from 0x002270 to 0x004a95 (0x2826 bytes) 000672 1759820450.933355 punch from 0x001b61 to 0x004a95 (0x2f35 bytes) 000673 1759820450.935035 punch from 0x001bd7 to 0x004a95 (0x2ebf bytes) 000674 1759820450.936557 mapwrite 0x03266b thru 0x032ee6 (0x87c bytes) 000675 1759820450.948112 write 0x00f1ca thru 0x0120e7 (0x2f1e bytes) 000676 1759820450.951453 trunc from 0x032ee7 to 0x00f6b6 (0x23830 bytes) 000677 1759820450.956116 write 0x000934 thru 0x003433 (0x2b00 bytes) 000678 1759820450.958934 fallocate 0x032ae3 thru 0x03f465 (0xc983 bytes) 000679 1759820450.960779 write 0x01cc94 thru 0x0212ce (0x463b bytes) 000680 1759820450.964056 read 0x002814 thru 0x0117ac (0xef99 bytes) 000681 1759820450.967618 mapread 0x004b5b thru 0x007d54 (0x31fa bytes) 000682 1759820450.968163 trunc from 0x00c293 to 0x0174ba (0xb228 bytes) 000683 1759820450.972802 write 0x01d935 thru 0x02cf52 (0xf61e bytes) 000684 1759820450.973683 fallocate 0x035186 thru 0x03ffff (0xae7a bytes) 000685 1759820450.975464 trunc from 0x040000 to 0x0315ee (0xea11 bytes) 000686 1759820450.981469 MIRROR_RESYNC 2 mirrors 000688 1759820451.148809 write 0x003919 thru 0x00832b (0x4a13 bytes) 000689 1759820451.160524 trunc from 0x009919 to 0x00ec58 (0x5340 bytes) 000690 1759820451.164278 MIRROR_EXTEND to 3 mirrors 000691 1759820451.367862 MIRROR_EXTEND to 4 mirrors 000692 1759820451.562259 write 0x033fea thru 0x03bf61 (0x7f78 bytes) 000693 1759820451.571329 trunc from 0x018bc6 to 0x01b1fc (0x2637 bytes) 000694 1759820451.573132 fallocate 0x017ca8 thru 0x01c110 (0x4469 bytes) 000695 1759820451.574844 trunc from 0x03bf62 to 0x01ca19 (0x1f548 bytes) 000696 1759820451.580810 write 0x00a015 thru 0x00f1cf (0x51bb bytes) 000697 1759820451.584153 fallocate 0x010984 thru 0x01192f (0xfac bytes) 000698 1759820451.586066 trunc from 0x001a43 to 0x0087d9 (0x6d97 bytes) 000699 1759820451.587821 punch from 0x0075e6 to 0x007706 (0x121 bytes) 000701 1759820451.589623 trunc from 0x01ca1a to 0x035601 (0x18be8 bytes) 000702 1759820451.594704 mapread 0x0035c1 thru 0x00c339 (0x8d79 bytes) 000703 1759820451.597818 write 0x0334e6 thru 0x03d1ac (0x9cc7 bytes) 000704 1759820451.600872 fallocate 0x03f71d thru 0x03ffff (0x8e3 bytes) 000705 1759820451.602699 read 0x00de0d thru 0x0120d4 (0x42c8 bytes) 000706 1759820451.604652 trunc from 0x040000 to 0x03d348 (0x2cb7 bytes) 000707 1759820451.610548 MIRROR_EXTEND to 5 mirrors 000708 1759820451.808294 trunc from 0x03d349 to 0x031605 (0xbd43 bytes) 000709 1759820451.825933 mapwrite 0x03ecee thru 0x03ffff (0x1312 bytes) 000710 1759820451.839200 punch from 0x03dc60 to 0x03ffff (0x23a0 bytes) 000711 1759820451.841017 trunc from 0x0189f0 to 0x01a749 (0x1d5a bytes) 000712 1759820451.842712 write 0x00e5a2 thru 0x015a01 (0x7460 bytes) 000713 1759820451.845795 trunc from 0x040000 to 0x0133c0 (0x2cc3f bytes) 000714 1759820451.850354 mapread 0x0021e8 thru 0x011997 (0xf7b0 bytes) 000715 1759820451.853274 punch from 0x001c92 to 0x005e50 (0x41bf bytes) 000716 1759820451.856630 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000717 1759820451.927023 mapwrite 0x00cf47 thru 0x00f431 (0x24eb bytes) 000718 1759820451.937969 mapread 0x009f3d thru 0x0133c0 (0x9484 bytes) 000719 1759820451.938621 punch from 0x00f1d4 to 0x00f1dc (0x9 bytes) 000720 1759820451.940413 punch from 0x00a7cc to 0x0133c0 (0x8bf5 bytes) 000721 1759820451.942362 trunc from 0x0116cf to 0x0133c0 (0x1cf2 bytes) 000722 1759820451.944583 trunc from 0x005b97 to 0x00c99f (0x6e09 bytes) 000723 1759820451.946907 fallocate 0x03a8e6 thru 0x03ffff (0x571a bytes) 000724 1759820451.949091 read 0x008d50 thru 0x0133c0 (0xa671 bytes) 000726 1759820451.952536 mapread 0x00522e thru 0x00ec36 (0x9a09 bytes) 000727 1759820451.955909 fallocate 0x022975 thru 0x0284d6 (0x5b62 bytes) 000728 1759820451.957975 trunc from 0x0133c1 to 0x00c3c5 (0x6ffb bytes) 000729 1759820451.965003 write 0x035cd8 thru 0x03c7e6 (0x6b0f bytes) 000731 1759820451.967645 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000732 1759820452.028993 mapread 0x02df0b thru 0x0357a0 (0x7896 bytes) 000733 1759820452.034812 write 0x038d73 thru 0x03ffff (0x728d bytes) 000734 1759820452.039044 write 0x03f0ec thru 0x03ffff (0xf14 bytes) 000735 1759820452.039477 trunc from 0x006843 to 0x010cca (0xa488 bytes) 000736 1759820452.042654 MIRROR_RESYNC 3 mirrors 000737 1759820452.247748 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000738 1759820452.368084 MIRROR_EXTEND to 3 mirrors 000739 1759820452.572717 mapread 0x023187 thru 0x026eda (0x3d54 bytes) 000741 1759820452.580295 fallocate 0x03bb0a thru 0x03c897 (0xd8e bytes) 000743 1759820452.595783 read 0x014495 thru 0x015079 (0xbe5 bytes) 000744 1759820452.598254 trunc from 0x018e5d to 0x021bf0 (0x8d94 bytes) 000745 1759820452.602603 MIRROR_EXTEND to 4 mirrors 000746 1759820452.796402 write 0x02334b thru 0x0329ca (0xf680 bytes) 000747 1759820452.811461 fallocate 0x03e6ce thru 0x03ffff (0x1932 bytes) 000748 1759820452.813508 trunc from 0x040000 to 0x006ae3 (0x3951c bytes) 000749 1759820452.826914 trunc from 0x003783 to 0x006ae3 (0x3361 bytes) 000750 1759820452.828975 fallocate 0x000ff1 thru 0x006c55 (0x5c65 bytes) 000751 1759820452.831337 trunc from 0x006c56 to 0x01c25c (0x15607 bytes) 000752 1759820452.838012 read 0x00b750 thru 0x018aac (0xd35d bytes) 000754 1759820452.842625 trunc from 0x0140b1 to 0x01c25c (0x81ac bytes) 000756 1759820452.845301 mapread 0x001397 thru 0x0068ab (0x5515 bytes) 000757 1759820452.849259 trunc from 0x000478 to 0x00ad36 (0xa8bf bytes) 000758 1759820452.851703 trunc from 0x01c25d to 0x002e95 (0x193c7 bytes) 000760 1759820452.858709 trunc from 0x001d8f to 0x002e95 (0x1107 bytes) 000761 1759820452.860797 mapread 0x002708 thru 0x002e95 (0x78e bytes) 000763 1759820452.864195 mapwrite 0x000dbd thru 0x009bd4 (0x8e18 bytes) 000764 1759820452.882908 punch from 0x0065f5 to 0x009bd4 (0x35e0 bytes) 000765 1759820452.885553 mapwrite 0x03a936 thru 0x03b073 (0x73e bytes) 000766 1759820452.903817 MIRROR_EXTEND to 5 mirrors 000767 1759820453.128453 read 0x01f105 thru 0x02660e (0x750a bytes) 000768 1759820453.134641 read 0x0398f0 thru 0x039cc5 (0x3d6 bytes) 000769 1759820453.141695 trunc from 0x03b074 to 0x008f2e (0x32145 bytes) 000770 1759820453.158184 mapwrite 0x03690f thru 0x03ffff (0x96f1 bytes) 000771 1759820453.179650 trunc from 0x022c8e to 0x022d81 (0xf4 bytes) 000772 1759820453.181200 read 0x01f48e thru 0x029229 (0x9d9c bytes) 000773 1759820453.184480 write 0x014931 thru 0x023fa5 (0xf675 bytes) 000774 1759820453.185329 write 0x02f6a2 thru 0x03b4b0 (0xbe0f bytes) 000775 1759820453.186138 punch from 0x03bba8 to 0x03ffff (0x4458 bytes) 000776 1759820453.190305 punch from 0x004462 to 0x011ce3 (0xd882 bytes) 000777 1759820453.192269 mapread 0x0137b2 thru 0x01f5ae (0xbdfd bytes) 000778 1759820453.192866 trunc from 0x018b6c to 0x0219a2 (0x8e37 bytes) 000779 1759820453.194558 fallocate 0x0080b7 thru 0x00b2f9 (0x3243 bytes) 000780 1759820453.196365 trunc from 0x040000 to 0x029ca5 (0x1635a bytes) 000782 1759820453.201030 read 0x0147e8 thru 0x016617 (0x1e30 bytes) 000783 1759820453.201375 read 0x00595c thru 0x00aca0 (0x5345 bytes) 000784 1759820453.204303 trunc from 0x029ca6 to 0x0036f7 (0x265ae bytes) 000785 1759820453.208907 trunc from 0x0036f8 to 0x008064 (0x496d bytes) 000786 1759820453.212909 trunc from 0x002fda to 0x008064 (0x508b bytes) 000787 1759820453.214829 write 0x0280bb thru 0x0361e4 (0xe12a bytes) 000788 1759820453.215898 mapread 0x00e092 thru 0x0198ec (0xb85b bytes) 000789 1759820453.232049 trunc from 0x0361e5 to 0x02fa2d (0x67b7 bytes) 000791 1759820453.236933 write 0x00dd8b thru 0x018470 (0xa6e6 bytes) 000792 1759820453.239186 trunc from 0x02fa2e to 0x001552 (0x2e4db bytes) 000793 1759820453.244992 mapread 0x000290 thru 0x001552 (0x12c3 bytes) 000794 1759820453.245567 punch from 0x000766 to 0x001552 (0xded bytes) 000795 1759820453.247233 write 0x005d43 thru 0x00e147 (0x8405 bytes) 000796 1759820453.248120 write 0x03db15 thru 0x03ffff (0x24eb bytes) 000797 1759820453.248766 mapwrite 0x03b948 thru 0x03ffff (0x46b8 bytes) 000798 1759820453.257977 write 0x039845 thru 0x03ffff (0x67bb bytes) 000799 1759820453.260336 MIRROR_RESYNC 5 mirrors 000800 1759820453.610544 punch from 0x01ea30 to 0x02060c (0x1bdd bytes) 000802 1759820453.621416 mapwrite 0x035733 thru 0x039fc0 (0x488e bytes) 000803 1759820453.632745 mapread 0x02cb8c thru 0x03013a (0x35af bytes) 000804 1759820453.635757 trunc from 0x018d87 to 0x01b5c8 (0x2842 bytes) 000805 1759820453.637614 fallocate 0x02680e thru 0x029cb0 (0x34a3 bytes) 000806 1759820453.639403 mapread 0x03185f thru 0x038f84 (0x7726 bytes) 000807 1759820453.640005 mapwrite 0x018bf1 thru 0x024261 (0xb671 bytes) 000808 1759820453.651178 trunc from 0x040000 to 0x023dca (0x1c235 bytes) 000809 1759820453.663225 mapwrite 0x02b05e thru 0x035371 (0xa314 bytes) 000810 1759820453.678897 trunc from 0x035372 to 0x01b86e (0x19b03 bytes) 000811 1759820453.683933 mapwrite 0x03d241 thru 0x03ffff (0x2dbf bytes) 000812 1759820453.696252 write 0x0129db thru 0x016f08 (0x452e bytes) 000813 1759820453.696957 write 0x005f1b thru 0x00bc4a (0x5d30 bytes) 000814 1759820453.700142 mapwrite 0x00ab57 thru 0x01873c (0xdbe6 bytes) 000816 1759820453.708268 read 0x03bb51 thru 0x03cf83 (0x1433 bytes) 000817 1759820453.708690 mapread 0x03a791 thru 0x03c7d3 (0x2043 bytes) 000818 1759820453.709168 trunc from 0x040000 to 0x0273c3 (0x18c3c bytes) 000819 1759820453.715019 read 0x00946e thru 0x0161f4 (0xcd87 bytes) 000821 1759820453.715697 mapread 0x022dec thru 0x0273c3 (0x45d8 bytes) 000822 1759820453.719187 write 0x01a655 thru 0x01c859 (0x2205 bytes) 000824 1759820453.720106 read 0x0259d9 thru 0x0273c3 (0x19eb bytes) 000826 1759820453.722505 MIRROR_RESYNC 5 mirrors 000827 1759820454.016893 write 0x019dd0 thru 0x02891d (0xeb4e bytes) 000828 1759820454.032835 punch from 0x01aa30 to 0x026a28 (0xbff9 bytes) 000829 1759820454.038759 trunc from 0x00e56f to 0x014b4a (0x65dc bytes) 000830 1759820454.040799 mapwrite 0x02c44e thru 0x03173f (0x52f2 bytes) 000831 1759820454.064761 trunc from 0x004c72 to 0x011e5d (0xd1ec bytes) 000832 1759820454.066810 write 0x02234d thru 0x030f7d (0xec31 bytes) 000833 1759820454.071615 MIRROR_EXTEND to 6 mirrors 000834 1759820454.239961 trunc from 0x0128c0 to 0x015b11 (0x3252 bytes) 000835 1759820454.250029 mapwrite 0x01546c thru 0x01dfb9 (0x8b4e bytes) 000836 1759820454.261751 punch from 0x02ea91 to 0x03173f (0x2caf bytes) 000838 1759820454.263900 write 0x01eb3b thru 0x02a9b0 (0xbe76 bytes) 000839 1759820454.267405 read 0x02772a thru 0x028836 (0x110d bytes) 000840 1759820454.267788 trunc from 0x020115 to 0x02e79d (0xe689 bytes) 000841 1759820454.271938 trunc from 0x031740 to 0x02e8e1 (0x2e5e bytes) 000842 1759820454.282606 punch from 0x012833 to 0x016111 (0x38df bytes) 000843 1759820454.284121 trunc from 0x01f54d to 0x023f40 (0x49f4 bytes) 000844 1759820454.287207 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000845 1759820454.344709 mapread 0x000bb2 thru 0x00c40b (0xb85a bytes) 000846 1759820454.351564 trunc from 0x02e8e2 to 0x022cdc (0xbc05 bytes) 000847 1759820454.358223 read 0x010246 thru 0x01723e (0x6ff9 bytes) 000848 1759820454.363128 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000849 1759820454.424012 trunc from 0x022cdd to 0x0335ca (0x108ee bytes) 000850 1759820454.430507 trunc from 0x0335cb to 0x0264f9 (0xd0d1 bytes) 000851 1759820454.434759 mapwrite 0x02733e thru 0x02e4b8 (0x717b bytes) 000852 1759820454.453256 read 0x022806 thru 0x027a71 (0x526c bytes) 000853 1759820454.453722 fallocate 0x00d351 thru 0x01b30e (0xdfbe bytes) 000854 1759820454.455832 write 0x01d14d thru 0x025e3b (0x8cef bytes) 000855 1759820454.459125 mapwrite 0x000b33 thru 0x00f40b (0xe8d9 bytes) 000858 1759820454.470839 trunc from 0x02e4b9 to 0x0236c4 (0xadf4 bytes) 000859 1759820454.476345 mapwrite 0x02e955 thru 0x039dfc (0xb4a8 bytes) 000862 1759820454.494780 trunc from 0x02973a to 0x02fa1f (0x62e6 bytes) 000864 1759820454.497112 punch from 0x014fc4 to 0x016981 (0x19be bytes) 000865 1759820454.500861 MIRROR_RESYNC 4 mirrors 000867 1759820454.753906 MIRROR_RESYNC 4 mirrors 000868 1759820454.831652 read 0x01510f thru 0x01a3c7 (0x52b9 bytes) 000869 1759820454.834341 trunc from 0x024af4 to 0x03127c (0xc789 bytes) 000870 1759820454.848171 trunc from 0x039dfd to 0x012eae (0x26f4e bytes) 000871 1759820454.861140 fallocate 0x010151 thru 0x01c46c (0xc31c bytes) 000872 1759820454.863145 trunc from 0x01c46d to 0x00e93a (0xdb32 bytes) 000873 1759820454.868886 read 0x008a99 thru 0x00e93a (0x5ea2 bytes) 000874 1759820454.874516 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000875 1759820454.943219 punch from 0x00bf44 to 0x00e93a (0x29f7 bytes) 000879 1759820454.946392 punch from 0x0063f0 to 0x009375 (0x2f86 bytes) 000880 1759820454.948530 trunc from 0x00e93b to 0x0365a2 (0x27c68 bytes) 000881 1759820454.958280 MIRROR_EXTEND to 4 mirrors 000882 1759820455.167732 fallocate 0x02ea26 thru 0x034c34 (0x620f bytes) 000883 1759820455.177146 mapwrite 0x037935 thru 0x03ea7e (0x714a bytes) 000884 1759820455.195940 mapwrite 0x002e4c thru 0x00d6b0 (0xa865 bytes) 000885 1759820455.205662 write 0x00a098 thru 0x0164fc (0xc465 bytes) 000886 1759820455.207914 write 0x00cd09 thru 0x00d762 (0xa5a bytes) 000888 1759820455.210273 MIRROR_RESYNC 4 mirrors 000889 1759820455.425371 trunc from 0x03ea7f to 0x028120 (0x1695e bytes) 000890 1759820455.444993 write 0x03290f thru 0x039bbb (0x72ad bytes) 000891 1759820455.445878 fallocate 0x029b9d thru 0x029e18 (0x27c bytes) 000892 1759820455.448010 write 0x0069b6 thru 0x01054a (0x9b95 bytes) 000893 1759820455.452050 trunc from 0x039bbc to 0x01c512 (0x1d6a9 bytes) 000895 1759820455.460194 write 0x00916e thru 0x017a7c (0xe90f bytes) 000896 1759820455.464612 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000897 1759820455.528467 mapwrite 0x0207c8 thru 0x0227cb (0x2004 bytes) 000898 1759820455.547336 trunc from 0x01a620 to 0x0227cb (0x81ac bytes) 000900 1759820455.550843 mapwrite 0x020098 thru 0x02a522 (0xa48b bytes) 000901 1759820455.573141 mapread 0x027bf2 thru 0x02a522 (0x2931 bytes) 000902 1759820455.575621 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000903 1759820455.649653 write 0x0386b3 thru 0x03ffff (0x794d bytes) 000905 1759820455.652941 write 0x03edf2 thru 0x03ffff (0x120e bytes) 000906 1759820455.653601 mapread 0x020c7e thru 0x02863c (0x79bf bytes) 000907 1759820455.662312 fallocate 0x02dbbf thru 0x03246e (0x48b0 bytes) 000908 1759820455.664877 trunc from 0x03a6a9 to 0x03fb86 (0x54de bytes) 000909 1759820455.671350 trunc from 0x01b82a to 0x0273ca (0xbba1 bytes) 000910 1759820455.673813 trunc from 0x040000 to 0x039922 (0x66dd bytes) 000911 1759820455.680692 fallocate 0x03d510 thru 0x03ffff (0x2af0 bytes) 000912 1759820455.683394 fallocate 0x0053fd thru 0x005979 (0x57d bytes) 000913 1759820455.685932 trunc from 0x040000 to 0x011ad2 (0x2e52d bytes) 000914 1759820455.692956 write 0x03940f thru 0x039788 (0x37a bytes) 000916 1759820455.693944 mapwrite 0x034a71 thru 0x03bb82 (0x7112 bytes) 000917 1759820455.714287 read 0x0342a4 thru 0x03493f (0x69c bytes) 000918 1759820455.714795 punch from 0x012b94 to 0x01c9d0 (0x9e3d bytes) 000919 1759820455.717329 read 0x00bea9 thru 0x01784d (0xb9a5 bytes) 000920 1759820455.722821 trunc from 0x03bb83 to 0x00afa7 (0x30bdb bytes) 000921 1759820455.730477 punch from 0x00a1b3 to 0x00afa7 (0xdf5 bytes) 000922 1759820455.733054 write 0x01d16b thru 0x02437c (0x7212 bytes) 000923 1759820455.733873 write 0x00c82a thru 0x014bc5 (0x839c bytes) 000924 1759820455.738618 write 0x001c14 thru 0x00430b (0x26f8 bytes) 000926 1759820455.739506 trunc from 0x02231d to 0x02437c (0x2060 bytes) 000928 1759820455.747298 fallocate 0x00831e thru 0x014624 (0xc307 bytes) 000929 1759820455.749815 read 0x01342c thru 0x0191e6 (0x5dbb bytes) 000931 1759820455.754412 trunc from 0x02437d to 0x03d9af (0x19633 bytes) 000933 1759820455.762151 trunc from 0x01127b to 0x01af2a (0x9cb0 bytes) 000934 1759820455.764824 punch from 0x00c23c to 0x01964e (0xd413 bytes) 000935 1759820455.767531 trunc from 0x02402e to 0x032465 (0xe438 bytes) 000936 1759820455.770207 trunc from 0x03bc7e to 0x03d9af (0x1d32 bytes) 000937 1759820455.772641 trunc from 0x00add5 to 0x017f9b (0xd1c7 bytes) 000938 1759820455.775363 read 0x02e4af thru 0x0307b3 (0x2305 bytes) 000939 1759820455.780491 trunc from 0x03d9b0 to 0x029488 (0x14527 bytes) 000940 1759820455.787858 write 0x02d9ac thru 0x035b72 (0x81c7 bytes) 000941 1759820455.788815 punch from 0x029faa to 0x03271f (0x8776 bytes) 000942 1759820455.797891 MIRROR_RESYNC 2 mirrors 000943 1759820456.017917 fallocate 0x03262a thru 0x035fb5 (0x398c bytes) 000944 1759820456.030022 trunc from 0x00968a to 0x015598 (0xbf0f bytes) 000945 1759820456.034063 MIRROR_EXTEND to 3 mirrors 000947 1759820456.274710 trunc from 0x00be1e to 0x019751 (0xd934 bytes) 000948 1759820456.292300 write 0x01ed17 thru 0x02c588 (0xd872 bytes) 000949 1759820456.301468 MIRROR_EXTEND to 4 mirrors 000950 1759820456.559456 trunc from 0x035fb6 to 0x028c70 (0xd345 bytes) 000951 1759820456.576965 trunc from 0x0063ce to 0x00ec4a (0x887d bytes) 000952 1759820456.579564 punch from 0x00ca06 to 0x01a3bf (0xd9ba bytes) 000953 1759820456.581955 read 0x027f6e thru 0x028c70 (0xd03 bytes) 000954 1759820456.584598 trunc from 0x028c71 to 0x013239 (0x15a37 bytes) 000955 1759820456.592773 fallocate 0x03f65c thru 0x03ffff (0x9a4 bytes) 000956 1759820456.597076 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x53b:0x0]: Invalid argument (22) 000957 1759820456.670306 MIRROR_RESYNC 3 mirrors 000958 1759820456.884763 trunc from 0x01323a to 0x030440 (0x1d207 bytes) 000959 1759820456.907462 mapread 0x015549 thru 0x01cbf2 (0x76aa bytes) 000960 1759820456.915150 punch from 0x01264b to 0x019248 (0x6bfe bytes) 000962 1759820456.917829 punch from 0x00bdac to 0x012549 (0x679e bytes) 000963 1759820456.920073 trunc from 0x030441 to 0x02cfd3 (0x346d bytes) 000964 1759820456.928378 write 0x017640 thru 0x0215f6 (0x9fb7 bytes) 000965 1759820456.934683 mapread 0x00a6b3 thru 0x00de9c (0x37ea bytes) 000966 1759820456.938401 mapread 0x01f585 thru 0x020131 (0xbad bytes) 000967 1759820456.938915 fallocate 0x02eabd thru 0x039452 (0xa996 bytes) 000968 1759820456.941169 write 0x0354d2 thru 0x03ffff (0xab2e bytes) 000969 1759820456.942080 punch from 0x02ddad to 0x02e088 (0x2dc bytes) 000970 1759820456.944006 mapread 0x02d693 thru 0x0306b9 (0x3027 bytes) 000971 1759820456.947271 mapwrite 0x03472d thru 0x035b60 (0x1434 bytes) 000972 1759820456.955283 punch from 0x0034b7 to 0x00b666 (0x81b0 bytes) 000973 1759820456.959790 MIRROR_EXTEND to 4 mirrors 000974 1759820457.197218 trunc from 0x0179c6 to 0x0269c7 (0xf002 bytes) 000975 1759820457.211634 trunc from 0x039839 to 0x03ffff (0x67c7 bytes) 000976 1759820457.213996 mapwrite 0x00eefe thru 0x011736 (0x2839 bytes) 000977 1759820457.228166 trunc from 0x015e6a to 0x022bf9 (0xcd90 bytes) 000978 1759820457.230718 trunc from 0x00c348 to 0x0108b9 (0x4572 bytes) 000979 1759820457.233178 punch from 0x03eff3 to 0x03ffff (0x100d bytes) 000980 1759820457.235570 mapwrite 0x0393d4 thru 0x03b433 (0x2060 bytes) 000981 1759820457.245707 trunc from 0x040000 to 0x011cad (0x2e352 bytes) 000982 1759820457.256946 mapread 0x00b150 thru 0x011cad (0x6b5e bytes) 000983 1759820457.261330 trunc from 0x011cae to 0x00acc0 (0x6fed bytes) 000984 1759820457.268130 trunc from 0x00acc1 to 0x0373a4 (0x2c6e4 bytes) 000985 1759820457.274522 punch from 0x028319 to 0x0354f1 (0xd1d9 bytes) 000986 1759820457.276924 trunc from 0x018f8a to 0x01ab0b (0x1b82 bytes) 000987 1759820457.279343 mapread 0x021de6 thru 0x029e46 (0x8061 bytes) 000988 1759820457.283738 fallocate 0x02eb1a thru 0x0335af (0x4a96 bytes) 000989 1759820457.286350 punch from 0x004672 to 0x01162d (0xcfbc bytes) 000990 1759820457.288749 read 0x01daaa thru 0x023e28 (0x637f bytes) 000991 1759820457.289123 read 0x031295 thru 0x0373a4 (0x6110 bytes) 000993 1759820457.293720 trunc from 0x0373a5 to 0x038919 (0x1575 bytes) 000995 1759820457.300892 write 0x007b43 thru 0x0133aa (0xb868 bytes) 000996 1759820457.301794 read 0x012893 thru 0x020f50 (0xe6be bytes) 000997 1759820457.302179 mapread 0x01db63 thru 0x02da7f (0xff1d bytes) 000998 1759820457.302797 fallocate 0x012b9f thru 0x01a156 (0x75b8 bytes) 000999 1759820457.305279 trunc from 0x03891a to 0x026adb (0x11e3e bytes) All operations completed A-OK! PASS 70a (20s) == sanity-flr test 71: check mirror extend parent fid ==== 03:00:59 (1759820459) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x55:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 03:01:02 (1759820462) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7945056 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2853408 .. failed Extending file size to 6190432 .. Extending file size to 8043872 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6821248 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8133408 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8844576 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8468672 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8591552 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9120192 .. Extending file size to 8706976 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9249856 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8974976 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61929 61930 61932 61935 61938 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) 192.168.202.117@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:02:08 (1759820528) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 3062560 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2181024 .. done Extending file size to 8016480 .. Extending file size to 2763424 .. Extending file size to 7720224 .. Extending file size to 8475968 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9170912 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9209408 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8489792 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8791872 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9390720 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 65959 65960 65962 65964 65966 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) 192.168.202.117@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:03:13 (1759820593) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2728 1284960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15948 3591072 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22088 3584932 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38036 7176004 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2632 1285056 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2728 1284960 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15948 3591072 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22088 3584932 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38036 7176004 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0491349 s, 42.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0687972 s, 30.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:03:22 (1759820602) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.059695 s, 35.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 03:03:25 (1759820605) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:03:27 (1759820607) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.028682 s, 73.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:03:30 (1759820610) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0254317 s, 82.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:03:34 (1759820614) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0408233 s, 51.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x92:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:03:59 (1759820639) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg217-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298815 s, 35.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x808:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x809:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x808:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x808:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x809:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x808:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x809:0x0] } PASS 204d (37s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:04:36 (1759820676) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg217-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0401413 s, 52.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x93:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:05:12 (1759820712) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg217-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg217-server: Pool lustre.test_204f created Adding targets to pool oleg217-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg217-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0513064 s, 40.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x94:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg217-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg217-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg217-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:06:01 (1759820761) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:06:04 (1759820764) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:06:08 (1759820768) oleg217-server: Pool lustre.test_206 created Adding targets to pool oleg217-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg217-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg217-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg217-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg217-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:06:26 (1759820786) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495102 s, 21.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238934 s, 43.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6b228c39 Mirror 3: 0x6b228c39 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:06:30 (1759820790) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:06:31 (1759820791) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:06:33 (1759820793) Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0142568 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2340000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x22e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x22e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x22e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ec0000 Read this (4 bytes): BBBB 192.168.202.117@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg217-client.virtnet /mnt/lustre2 (opts:) PASS 209a (19s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:06:52 (1759820812) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0062216 s, 1.3 MB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:06:56 (1759820816) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027469 s, 38.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611268 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-07 03:06:56.000000000 -0400 Modify: 2025-10-07 03:06:56.000000000 -0400 Change: 2025-10-07 03:06:57.000000000 -0400 Birth: 2025-10-07 03:06:57.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:06:59 (1759820819) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201582 s, 52.0 MB/s oleg217-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg217-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg217-server: O/280000401/d29/bd: File not found by ext2_lookup PASS 210b (17s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:07:16 (1759820836) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0272161 s, 1.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00917804 s, 446 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 7 03:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x89:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 7 03:07 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 846 sec ============ 03:07:19 (1759820839) === sanity-flr: start cleanup 03:07:20 (1759820840) === === sanity-flr: finish cleanup 03:07:23 (1759820843) ===