-----============= acceptance-small: sanity-flr ============----- Thu Jul 24 09:43:06 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg205-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:43:15 (1753364595) === oleg205-client.virtnet: executing check_config_client /mnt/lustre oleg205-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg205-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97d721907800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97d721907800.idle_timeout=debug disable quota as required oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:43:29 (1753364609) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8168] == sanity-flr test 0a: lfs mirror create with -N option == 09:43:30 (1753364610) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:43:37 (1753364617) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:43:38 (1753364618) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:43:40 (1753364620) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:43:48 (1753364628) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:43:50 (1753364630) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:43:51 (1753364631) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff97d721907800.stats=clear osc.lustre-OST0001-osc-ffff97d721907800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:43:54 (1753364634) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:43:58 (1753364638) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:44:01 (1753364641) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:44:02 (1753364642) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 09:44:05 (1753364645) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:44:08 (1753364648) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:44:11 (1753364651) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:44:13 (1753364653) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:44:16 (1753364656) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:44:19 (1753364659) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.191925 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357951 s, 29.3 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:44:22 (1753364662) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.136959 s, 84.2 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:44:26 (1753364666) Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97d721907800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97d721907800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97d721907800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97d721907800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:44:57 (1753364697) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.276306 s, 37.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97d721907800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97d721907800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:45:17 (1753364717) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97d721907800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97d721907800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97d721907800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97d721907800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (36s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:45:53 (1753364753) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2104 s, 43.3 MB/s -rw-r--r-- 1 root root 52428800 Jul 24 09:45 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.484s user 0m0.000s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97d721907800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97d721907800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.397s user 0m0.001s sys 0m0.182s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97d721907800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97d721907800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.427s user 0m0.002s sys 0m0.176s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97d721907800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97d721907800.ost_server_uuid in FULL state after 0 sec PASS 33b (36s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:46:30 (1753364790) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:46:31 (1753364791) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:46:32 (1753364792) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 09:46:34 (1753364794) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 09:46:36 (1753364796) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.246135 s, 51.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:46:41 (1753364801) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015024 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134821 s, 77.8 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:46:44 (1753364804) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:46:48 (1753364808) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000925588 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000773451 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000799407 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00093846 s, 1.1 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 09:46:52 (1753364812) create test files with size 15740435 .. c02c66b31820b584d722b816b362d111 - 372f46d41e6436ce977dfc326dd22ca8 - 746c67c89bdbb53599d1619b10c8dbbc - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 09:47:01 (1753364821) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:47:07 (1753364827) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:47:10 (1753364830) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418019 s, 25.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158708 s, 66.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 09:47:14 (1753364834) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0349835 s, 59.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.119755 s, 35.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0735734 s, 28.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 09:47:18 (1753364838) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:47:19 (1753364839) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:47:20 (1753364840) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127462 s, 82.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 09:47:23 (1753364843) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.120885 s, 26.0 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:47:27 (1753364847) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:47:31 (1753364851) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.114042 s, 91.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:47:48 (1753364868) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff97d71256f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.159865 s, 65.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff97d719065800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:47:53 (1753364873) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (17s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:48:10 (1753364890) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:48:25 (1753364905) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 09:48:27 (1753364907) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0873472 s, 36.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0670961 s, 46.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: e430abe0a26edd9a1ef85a5afd1f1ccb - mirror 1 checksum: dcfd7df7d79703ad9aed14d447153863 - mirror 2 checksum: e430abe0a26edd9a1ef85a5afd1f1ccb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jul 24 09:48:28 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jul 24 09:48:28 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e430abe0a26edd9a1ef85a5afd1f1ccb - mirror 1 checksum: dcfd7df7d79703ad9aed14d447153863 - mirror 2 checksum: dcfd7df7d79703ad9aed14d447153863 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jul 24 09:48:28 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jul 24 09:48:28 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:48:30 (1753364910) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135408 s, 77.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:48:33 (1753364913) Failing mds1 on oleg205-server Stopping /mnt/lustre-mds1 (opts:) on oleg205-server 09:48:41 (1753364921) shut down facet: mds1 facet_host: oleg205-server facet_failover_host: oleg205-server Failover mds1 to oleg205-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 09:48:56 (1753364936) targets are mounted 09:48:56 (1753364936) facet_failover done oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166969 s, 62.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 50A (33s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:49:06 (1753364946) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0229484 s, 892 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00912087 s, 2.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (7s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:49:13 (1753364953) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0518241 s, 395 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0149555 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:49:22 (1753364962) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00476899 s, 3.4 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:49:28 (1753364968) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0399494 s, 513 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:49:36 (1753364976) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0103016 s, 398 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0294523 s, 139 kB/s PASS 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:49:41 (1753364981) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:49:53 (1753364993) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:50:14 (1753365014) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:50:33 (1753365033) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:50:52 (1753365052) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 09:50:55 (1753365055) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55804 mirror split pid 55805 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7670: 55805 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55804 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 09:51:58 (1753365118) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5119 skipping zero length punch hole skipping zero length punch hole skipping zero size read 000005 1753365119.176254 write 0x03260a thru 0x036bb2 (0x45a9 bytes) 000007 1753365119.178353 mapread 0x01c99a thru 0x025043 (0x86aa bytes) 000008 1753365119.189557 punch from 0x00da2d to 0x00f9b2 (0x1f86 bytes) 000009 1753365119.193548 mapread 0x02e3ee thru 0x032be3 (0x47f6 bytes) 000010 1753365119.200239 mapwrite 0x03e84b thru 0x03fb42 (0x12f8 bytes) fallocating to largest ever: 0xa1d1 000011 1753365119.226755 fallocate 0x0040eb thru 0x00a1d0 (0x60e6 bytes) 000012 1753365119.229768 mapwrite 0x013c47 thru 0x016949 (0x2d03 bytes) 000013 1753365119.244229 MIRROR_EXTEND to 2 mirrors 000014 1753365119.478319 mapread 0x00b112 thru 0x018135 (0xd024 bytes) 000015 1753365119.497266 trunc from 0x00bf63 to 0x014a23 (0x8ac1 bytes) fallocating to largest ever: 0x16f25 000016 1753365119.512700 fallocate 0x00fe97 thru 0x016f24 (0x708e bytes) 000017 1753365119.518020 MIRROR_EXTEND to 3 mirrors 000018 1753365119.763717 mapwrite 0x00e8b7 thru 0x01003f (0x1789 bytes) 000019 1753365119.829592 fallocate 0x0104dc thru 0x0133a9 (0x2ece bytes) 000020 1753365119.832946 write 0x03c202 thru 0x03ffff (0x3dfe bytes) 000021 1753365119.839852 punch from 0x0031af to 0x00fec1 (0xcd13 bytes) 000022 1753365119.842896 trunc from 0x040000 to 0x000bd7 (0x3f428 bytes) 000023 1753365119.857097 read 0x000a6b thru 0x000bd7 (0x16d bytes) 000024 1753365119.859978 trunc from 0x000bc8 to 0x000bd7 (0x10 bytes) 000025 1753365119.863332 punch from 0x000337 to 0x000bd7 (0x8a1 bytes) 000026 1753365119.867457 MIRROR_RESYNC 3 mirrors 000027 1753365120.014319 write 0x028722 thru 0x03376c (0xb04b bytes) 000028 1753365120.027288 write 0x036348 thru 0x03f2f5 (0x8fae bytes) 000029 1753365120.028748 mapwrite 0x03a749 thru 0x03e96f (0x4227 bytes) 000030 1753365120.043638 mapwrite 0x03d82a thru 0x03ffff (0x27d6 bytes) 000031 1753365120.074800 write 0x00ce48 thru 0x00ea80 (0x1c39 bytes) 000032 1753365120.083017 punch from 0x005950 to 0x00dbba (0x826b bytes) 000033 1753365120.100808 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000034 1753365120.221788 mapwrite 0x03d70a thru 0x03ffff (0x28f6 bytes) 000035 1753365120.236671 read 0x00ca19 thru 0x01bce3 (0xf2cb bytes) 000036 1753365120.240692 write 0x021571 thru 0x02bc2a (0xa6ba bytes) 000037 1753365120.248858 trunc from 0x040000 to 0x011487 (0x2eb78 bytes) 000038 1753365120.256222 mapread 0x00a255 thru 0x00c49e (0x224a bytes) 000039 1753365120.260578 mapread 0x00dcbe thru 0x011487 (0x37ca bytes) 000040 1753365120.261428 trunc from 0x0064b4 to 0x009dc6 (0x3913 bytes) 000041 1753365120.264176 write 0x025ac7 thru 0x03483c (0xed76 bytes) 000042 1753365120.265704 write 0x012793 thru 0x020c28 (0xe496 bytes) 000044 1753365120.274483 MIRROR_EXTEND to 3 mirrors zero_range to largest ever: 0x24a7d 000045 1753365120.489864 trunc from 0x014f9d to 0x024a7c (0xfae0 bytes) 000046 1753365120.507062 fallocate 0x00f4d9 thru 0x01a21f (0xad47 bytes) 000047 1753365120.510182 write 0x008423 thru 0x00f7fc (0x73da bytes) 000048 1753365120.517223 mapread 0x011fae thru 0x015d64 (0x3db7 bytes) 000049 1753365120.522473 punch from 0x0097f3 to 0x00e399 (0x4ba7 bytes) 000050 1753365120.531768 mapread 0x02c4fb thru 0x02e444 (0x1f4a bytes) truncating to largest ever: 0x3b08c 000051 1753365120.537588 trunc from 0x03483d to 0x03b08b (0x684f bytes) 000052 1753365120.557837 fallocate 0x003eea thru 0x010434 (0xc54b bytes) 000053 1753365120.565543 mapwrite 0x00aa60 thru 0x018be6 (0xe187 bytes) 000054 1753365120.597292 trunc from 0x00894a to 0x0091bd (0x874 bytes) 000055 1753365120.602035 fallocate 0x003a57 thru 0x011e1a (0xe3c4 bytes) 000056 1753365120.609835 write 0x032a0d thru 0x039ef6 (0x74ea bytes) 000057 1753365120.616903 trunc from 0x01ea33 to 0x020629 (0x1bf7 bytes) 000058 1753365120.622491 mapread 0x013086 thru 0x01b162 (0x80dd bytes) 000059 1753365120.630629 trunc from 0x03b08c to 0x002bef (0x3849c bytes) 000060 1753365120.642858 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000062 1753365120.732210 punch from 0x001655 to 0x002bef (0x159b bytes) 000063 1753365120.736885 MIRROR_EXTEND to 3 mirrors fallocating to largest ever: 0x3eefc 000064 1753365120.897153 fallocate 0x0376bd thru 0x03eefb (0x783f bytes) 000065 1753365120.913015 mapwrite 0x001630 thru 0x007452 (0x5e23 bytes) 000066 1753365120.950402 fallocate 0x010fbf thru 0x016849 (0x588b bytes) 000068 1753365120.953946 write 0x00dee1 thru 0x013e7a (0x5f9a bytes) 000069 1753365120.961425 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000070 1753365121.076932 read 0x00ee81 thru 0x010f1b (0x209b bytes) 000071 1753365121.082136 trunc from 0x01684a to 0x02bf2a (0x156e1 bytes) 000073 1753365121.094195 read 0x013247 thru 0x01bcc7 (0x8a81 bytes) 000074 1753365121.101870 write 0x028ea7 thru 0x02f17b (0x62d5 bytes) 000075 1753365121.103034 trunc from 0x013f9f to 0x015e78 (0x1eda bytes) 000077 1753365121.106989 mapwrite 0x0201d2 thru 0x0220f9 (0x1f28 bytes) 000078 1753365121.118193 mapwrite 0x0226e4 thru 0x02b4aa (0x8dc7 bytes) 000079 1753365121.128322 write 0x029c2e thru 0x032b5f (0x8f32 bytes) 000080 1753365121.129540 trunc from 0x032b60 to 0x030492 (0x26cd bytes) 000081 1753365121.137203 trunc from 0x0059ad to 0x00b07e (0x56d2 bytes) 000082 1753365121.140087 fallocate 0x020be6 thru 0x028587 (0x79a2 bytes) 000083 1753365121.142762 trunc from 0x0054e5 to 0x0123dc (0xcef8 bytes) fallocating to largest ever: 0x40000 000084 1753365121.145954 fallocate 0x039432 thru 0x03ffff (0x6bce bytes) 000085 1753365121.148540 trunc from 0x01a98b to 0x01d552 (0x2bc8 bytes) 000086 1753365121.151196 write 0x02cbe3 thru 0x02d815 (0xc33 bytes) 000087 1753365121.152104 punch from 0x0235ce to 0x023e50 (0x883 bytes) 000088 1753365121.155336 mapread 0x0077dc thru 0x0161bc (0xe9e1 bytes) 000090 1753365121.189094 mapwrite 0x01be0e thru 0x0274de (0xb6d1 bytes) 000091 1753365121.215845 write 0x02f3f7 thru 0x038586 (0x9190 bytes) 000092 1753365121.217118 read 0x02eaa6 thru 0x038586 (0x9ae1 bytes) 000093 1753365121.217683 punch from 0x00e9f3 to 0x01cb6a (0xe178 bytes) 000094 1753365121.221045 fallocate 0x03213c thru 0x03402f (0x1ef4 bytes) 000095 1753365121.224371 punch from 0x02f188 to 0x03396f (0x47e8 bytes) 000096 1753365121.234567 MIRROR_EXTEND to 3 mirrors 000097 1753365121.455833 mapread 0x034edb thru 0x038586 (0x36ac bytes) 000098 1753365121.471623 trunc from 0x03686d to 0x038586 (0x1d1a bytes) 000099 1753365121.491162 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000101 1753365121.573013 trunc from 0x0093cf to 0x00d818 (0x444a bytes) 000103 1753365121.578060 MIRROR_EXTEND to 3 mirrors 000104 1753365121.791436 trunc from 0x038587 to 0x0014ca (0x370bc bytes) 000105 1753365121.830406 trunc from 0x0014cb to 0x0105a6 (0xf0dc bytes) 000106 1753365121.843365 MIRROR_RESYNC 3 mirrors 000107 1753365122.092055 mapread 0x008104 thru 0x00c2fe (0x41fb bytes) 000108 1753365122.101995 mapwrite 0x00e8bd thru 0x01068e (0x1dd2 bytes) 000111 1753365122.139792 MIRROR_RESYNC 3 mirrors 000113 1753365122.351446 mapread 0x000712 thru 0x005b41 (0x5430 bytes) 000114 1753365122.362388 write 0x00f972 thru 0x014872 (0x4f01 bytes) 000115 1753365122.380230 fallocate 0x0381ff thru 0x03d6ac (0x54ae bytes) 000116 1753365122.383264 trunc from 0x014873 to 0x03dacd (0x2925b bytes) 000117 1753365122.401767 trunc from 0x00c815 to 0x01a031 (0xd81d bytes) 000118 1753365122.410475 mapread 0x01fc33 thru 0x02f570 (0xf93e bytes) 000119 1753365122.420825 punch from 0x03acc1 to 0x03dacd (0x2e0d bytes) 000120 1753365122.424063 mapread 0x001298 thru 0x005286 (0x3fef bytes) 000121 1753365122.429307 fallocate 0x00d512 thru 0x010340 (0x2e2f bytes) 000122 1753365122.435505 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000123 1753365122.529600 read 0x02ddad thru 0x02e5ee (0x842 bytes) 000124 1753365122.533554 mapread 0x0123b5 thru 0x01e421 (0xc06d bytes) 000125 1753365122.561368 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000126 1753365122.627275 trunc from 0x03dace to 0x021984 (0x1c149 bytes) 000127 1753365122.643340 trunc from 0x009923 to 0x016f6b (0xd649 bytes) 000128 1753365122.654939 trunc from 0x01942c to 0x021984 (0x8559 bytes) 000130 1753365122.660311 trunc from 0x014991 to 0x01bf7d (0x75ed bytes) 000131 1753365122.664508 read 0x0081ab thru 0x00886d (0x6c3 bytes) 000132 1753365122.668015 trunc from 0x021985 to 0x00f776 (0x1220e bytes) 000133 1753365122.678502 fallocate 0x008690 thru 0x00a4ee (0x1e5f bytes) 000134 1753365122.683244 trunc from 0x00f777 to 0x023e4d (0x146d7 bytes) 000135 1753365122.696807 write 0x01db64 thru 0x0203a2 (0x283f bytes) 000136 1753365122.710230 read 0x00e283 thru 0x00f630 (0x13ae bytes) 000138 1753365122.720662 read 0x004081 thru 0x00630f (0x228f bytes) 000139 1753365122.721176 mapwrite 0x019a48 thru 0x01b251 (0x180a bytes) 000140 1753365122.737163 fallocate 0x009730 thru 0x018140 (0xea11 bytes) 000144 1753365122.741569 read 0x0130d1 thru 0x0174b4 (0x43e4 bytes) 000145 1753365122.742128 trunc from 0x01483a to 0x017742 (0x2f09 bytes) 000146 1753365122.745284 mapwrite 0x0113f8 thru 0x01395a (0x2563 bytes) 000147 1753365122.757112 MIRROR_EXTEND to 2 mirrors 000148 1753365123.042454 trunc from 0x023e4e to 0x017c71 (0xc1dc bytes) 000149 1753365123.084341 MIRROR_RESYNC 2 mirrors 000150 1753365123.332068 read 0x014426 thru 0x017c71 (0x384c bytes) 000151 1753365123.335499 trunc from 0x011276 to 0x017c71 (0x69fc bytes) 000152 1753365123.357052 write 0x023068 thru 0x024d21 (0x1cba bytes) 000153 1753365123.358116 mapwrite 0x038dfd thru 0x03ef3f (0x6143 bytes) 000154 1753365123.413441 MIRROR_EXTEND to 3 mirrors 000155 1753365123.777216 mapread 0x02a848 thru 0x036430 (0xbbe9 bytes) 000156 1753365123.817961 punch from 0x03c13f to 0x03dee5 (0x1da7 bytes) 000157 1753365123.838316 punch from 0x021e00 to 0x030819 (0xea1a bytes) 000158 1753365123.841623 fallocate 0x01fd58 thru 0x02218a (0x2433 bytes) 000159 1753365123.845020 read 0x035bf7 thru 0x03b641 (0x5a4b bytes) 000161 1753365123.850083 trunc from 0x019a8f to 0x02625f (0xc7d1 bytes) 000162 1753365123.853875 punch from 0x016e78 to 0x01f63d (0x87c6 bytes) 000163 1753365123.857358 mapread 0x02f171 thru 0x03ad8a (0xbc1a bytes) 000164 1753365123.863443 read 0x039b50 thru 0x03ef3f (0x53f0 bytes) 000165 1753365123.871382 fallocate 0x0162b9 thru 0x021b16 (0xb85e bytes) 000166 1753365123.875248 trunc from 0x03ef40 to 0x010d81 (0x2e1be bytes) 000167 1753365123.906640 fallocate 0x02d781 thru 0x02e907 (0x1187 bytes) 000168 1753365123.910413 mapread 0x00918a thru 0x00934e (0x1c5 bytes) 000169 1753365123.911230 mapwrite 0x02af5b thru 0x034cb4 (0x9d5a bytes) 000170 1753365123.992015 mapwrite 0x00b9ab thru 0x016209 (0xa85f bytes) 000171 1753365124.038861 trunc from 0x015b61 to 0x01f430 (0x98d0 bytes) 000172 1753365124.044160 read 0x0080e9 thru 0x01625b (0xe173 bytes) 000175 1753365124.052704 punch from 0x00721d to 0x00f064 (0x7e48 bytes) 000176 1753365124.056839 punch from 0x02131d to 0x02c5ac (0xb290 bytes) 000177 1753365124.065259 mapwrite 0x026eff thru 0x02dceb (0x6ded bytes) 000178 1753365124.123026 trunc from 0x034cb5 to 0x002921 (0x32393 bytes) 000180 1753365124.134121 mapwrite 0x0359e2 thru 0x03ffff (0xa61e bytes) 000181 1753365124.165414 read 0x026d44 thru 0x02cb04 (0x5dc1 bytes) 000182 1753365124.173283 write 0x031b49 thru 0x033571 (0x1a29 bytes) 000183 1753365124.174387 write 0x03563f thru 0x038816 (0x31d8 bytes) 000184 1753365124.175386 trunc from 0x03c2bb to 0x03ffff (0x3d45 bytes) 000185 1753365124.179585 mapwrite 0x034827 thru 0x035ed4 (0x16ae bytes) 000187 1753365124.193174 mapwrite 0x01fdf5 thru 0x027b93 (0x7d9f bytes) 000188 1753365124.206150 trunc from 0x040000 to 0x01c516 (0x23ae9 bytes) 000189 1753365124.221701 MIRROR_EXTEND to 4 mirrors 000190 1753365124.502365 fallocate 0x026ea6 thru 0x033948 (0xcaa3 bytes) 000191 1753365124.535897 read 0x003518 thru 0x01077d (0xd266 bytes) 000192 1753365124.549264 MIRROR_EXTEND to 5 mirrors 000193 1753365124.781236 write 0x01b075 thru 0x01c7d3 (0x175f bytes) 000194 1753365124.806198 read 0x017daa thru 0x01bee4 (0x413b bytes) 000195 1753365124.809621 trunc from 0x01c7d4 to 0x007645 (0x1518e bytes) 000196 1753365124.837122 trunc from 0x002806 to 0x00561f (0x2e1a bytes) 000197 1753365124.843156 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000198 1753365124.958141 fallocate 0x01d60b thru 0x02632c (0x8d22 bytes) 000199 1753365124.962261 fallocate 0x033ae6 thru 0x0393ba (0x58d5 bytes) 000200 1753365124.966103 mapread 0x00943f thru 0x00d00f (0x3bd1 bytes) 000201 1753365124.978342 mapwrite 0x024f1d thru 0x030d17 (0xbdfb bytes) 000202 1753365125.004981 read 0x02462b thru 0x028b66 (0x453c bytes) 000203 1753365125.005600 mapread 0x02b4e1 thru 0x038935 (0xd455 bytes) 000204 1753365125.011113 mapwrite 0x02a7c1 thru 0x031703 (0x6f43 bytes) 000205 1753365125.029845 MIRROR_EXTEND to 5 mirrors 000206 1753365125.285143 fallocate 0x02b5a9 thru 0x038ec2 (0xd91a bytes) 000207 1753365125.312193 MIRROR_RESYNC 5 mirrors 000208 1753365125.911038 trunc from 0x0393bb to 0x0336a6 (0x5d14 bytes) 000209 1753365125.960990 mapread 0x01537a thru 0x01c691 (0x7318 bytes) 000210 1753365125.977842 punch from 0x0211d4 to 0x027a42 (0x686f bytes) 000211 1753365125.982629 fallocate 0x03f295 thru 0x03ffff (0xd6b bytes) 000213 1753365125.986823 mapwrite 0x0225c3 thru 0x030d2a (0xe768 bytes) 000214 1753365126.027297 fallocate 0x007758 thru 0x015b2e (0xe3d7 bytes) 000215 1753365126.032436 trunc from 0x040000 to 0x011ed3 (0x2e12c bytes) 000216 1753365126.051973 trunc from 0x00ed39 to 0x011ed3 (0x319b bytes) 000217 1753365126.055465 trunc from 0x011ed4 to 0x001026 (0x10ead bytes) 000218 1753365126.067297 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000219 1753365126.208953 fallocate 0x033fee thru 0x03851a (0x452d bytes) 000220 1753365126.219765 mapwrite 0x02b0e4 thru 0x02f3b5 (0x42d2 bytes) 000221 1753365126.250177 trunc from 0x015520 to 0x023ee3 (0xe9c4 bytes) 000223 1753365126.253549 mapread 0x035f1a thru 0x03851a (0x2601 bytes) 000224 1753365126.258137 trunc from 0x020f00 to 0x024ea1 (0x3fa2 bytes) 000225 1753365126.262275 mapread 0x029486 thru 0x038116 (0xec91 bytes) 000226 1753365126.263283 mapread 0x01164b thru 0x01211f (0xad5 bytes) 000227 1753365126.269755 trunc from 0x031b88 to 0x035efb (0x4374 bytes) 000228 1753365126.273441 write 0x017e38 thru 0x01d63b (0x5804 bytes) 000229 1753365126.279505 punch from 0x008fde to 0x018076 (0xf099 bytes) 000231 1753365126.295324 mapwrite 0x037595 thru 0x03ffff (0x8a6b bytes) 000232 1753365126.338236 punch from 0x01e74c to 0x020f96 (0x284b bytes) 000233 1753365126.343317 mapwrite 0x001398 thru 0x00f1aa (0xde13 bytes) 000234 1753365126.373962 MIRROR_EXTEND to 5 mirrors 000235 1753365126.733833 fallocate 0x00b071 thru 0x018f1d (0xdead bytes) 000236 1753365126.761894 trunc from 0x040000 to 0x0157fd (0x2a802 bytes) 000237 1753365126.787781 mapread 0x011ffa thru 0x0157fd (0x3804 bytes) 000238 1753365126.800281 write 0x00a36f thru 0x00da2d (0x36bf bytes) 000239 1753365126.807738 read 0x0151f5 thru 0x0157fd (0x609 bytes) 000240 1753365126.808269 fallocate 0x032831 thru 0x0365c2 (0x3d92 bytes) 000242 1753365126.811860 read 0x0114e3 thru 0x011b3e (0x65c bytes) 000243 1753365126.815022 MIRROR_EXTEND to 6 mirrors 000244 1753365127.188453 write 0x003080 thru 0x00f83c (0xc7bd bytes) 000245 1753365127.225866 punch from 0x003e0d to 0x007b3e (0x3d32 bytes) 000246 1753365127.235156 trunc from 0x0157fe to 0x019ca2 (0x44a5 bytes) 000247 1753365127.248745 mapread 0x00631a thru 0x00dcad (0x7994 bytes) 000248 1753365127.256806 punch from 0x0087ea to 0x00b684 (0x2e9b bytes) 000249 1753365127.266301 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000250 1753365127.418775 mapwrite 0x00e1fa thru 0x01495d (0x6764 bytes) 000251 1753365127.449337 mapwrite 0x001ce8 thru 0x00e7f8 (0xcb11 bytes) 000252 1753365127.469618 trunc from 0x019ca3 to 0x00be55 (0xde4d bytes) 000253 1753365127.482586 mapread 0x009425 thru 0x00a143 (0xd1f bytes) 000254 1753365127.483432 fallocate 0x033e72 thru 0x03ffff (0xc18e bytes) 000255 1753365127.487432 write 0x019063 thru 0x01a871 (0x180f bytes) 000256 1753365127.494180 trunc from 0x008597 to 0x012afe (0xa568 bytes) 000257 1753365127.498570 mapwrite 0x00b0e5 thru 0x017797 (0xc6b3 bytes) 000259 1753365127.522390 mapread 0x02e7d4 thru 0x02ff1f (0x174c bytes) 000260 1753365127.534804 write 0x038cff thru 0x03ffff (0x7301 bytes) 000262 1753365127.555943 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000263 1753365127.694710 trunc from 0x040000 to 0x02798c (0x18673 bytes) 000264 1753365127.713850 fallocate 0x0079ec thru 0x013cfc (0xc311 bytes) 000266 1753365127.718116 trunc from 0x02798d to 0x008b24 (0x1ee68 bytes) 000267 1753365127.734559 read 0x0083b2 thru 0x008b24 (0x773 bytes) 000268 1753365127.743847 MIRROR_RESYNC 4 mirrors 000269 1753365128.058401 fallocate 0x01d9be thru 0x02d7ce (0xfe11 bytes) 000270 1753365128.084068 read 0x016747 thru 0x0168c6 (0x180 bytes) 000271 1753365128.087937 mapwrite 0x0073db thru 0x0115a0 (0xa1c6 bytes) 000272 1753365128.116099 trunc from 0x02888f to 0x02d7ce (0x4f40 bytes) 000273 1753365128.120974 write 0x00545c thru 0x00e5a7 (0x914c bytes) 000274 1753365128.124701 MIRROR_EXTEND to 5 mirrors 000275 1753365128.330184 write 0x03a115 thru 0x03ffff (0x5eeb bytes) 000276 1753365128.350494 fallocate 0x03fad0 thru 0x03ffff (0x530 bytes) 000277 1753365128.353833 mapwrite 0x01a1cf thru 0x021d2e (0x7b60 bytes) 000278 1753365128.375620 read 0x010d14 thru 0x0180c9 (0x73b6 bytes) 000279 1753365128.379716 mapwrite 0x00b188 thru 0x013973 (0x87ec bytes) 000280 1753365128.399996 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000281 1753365128.501125 fallocate 0x00365c thru 0x00b237 (0x7bdc bytes) 000282 1753365128.505118 punch from 0x03a9aa to 0x03abc2 (0x219 bytes) 000283 1753365128.508392 trunc from 0x015370 to 0x017092 (0x1d23 bytes) 000284 1753365128.512397 read 0x01067a thru 0x0154af (0x4e36 bytes) 000285 1753365128.517210 write 0x002422 thru 0x008fb7 (0x6b96 bytes) 000286 1753365128.525658 mapwrite 0x0232b9 thru 0x030487 (0xd1cf bytes) 000288 1753365128.548872 read 0x033c57 thru 0x03ffff (0xc3a9 bytes) 000289 1753365128.556723 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000291 1753365128.649368 fallocate 0x029c91 thru 0x038087 (0xe3f7 bytes) 000292 1753365128.655683 MIRROR_RESYNC 3 mirrors 000293 1753365129.047074 punch from 0x00944d to 0x00b791 (0x2345 bytes) 000294 1753365129.069602 read 0x02b475 thru 0x0310fd (0x5c89 bytes) 000295 1753365129.073991 mapwrite 0x02fd97 thru 0x032c88 (0x2ef2 bytes) 000296 1753365129.090231 fallocate 0x039f40 thru 0x03ffff (0x60c0 bytes) 000297 1753365129.094405 fallocate 0x003e62 thru 0x00fd0f (0xbeae bytes) 000298 1753365129.099549 trunc from 0x040000 to 0x0203f5 (0x1fc0a bytes) 000299 1753365129.122973 trunc from 0x0203f6 to 0x005459 (0x1af9c bytes) 000300 1753365129.130997 read 0x000240 thru 0x005459 (0x521a bytes) 000301 1753365129.137232 read 0x001bb8 thru 0x005459 (0x38a2 bytes) 000302 1753365129.137848 fallocate 0x02a37d thru 0x031960 (0x75e4 bytes) 000303 1753365129.142197 mapread 0x002a42 thru 0x005459 (0x2a18 bytes) 000304 1753365129.142968 trunc from 0x000f1b to 0x0031a8 (0x228e bytes) 000305 1753365129.147148 fallocate 0x039c99 thru 0x03c84b (0x2bb3 bytes) 000308 1753365129.151858 punch from 0x001ec9 to 0x004f85 (0x30bd bytes) 000309 1753365129.155391 write 0x03b8a2 thru 0x03ffff (0x475e bytes) 000310 1753365129.156631 trunc from 0x02bf2a to 0x034a6c (0x8b43 bytes) 000311 1753365129.162495 fallocate 0x03cc14 thru 0x03ffff (0x33ec bytes) 000312 1753365129.168865 trunc from 0x024384 to 0x02d47e (0x90fb bytes) 000316 1753365129.173597 punch from 0x03f3b4 to 0x03ffff (0xc4c bytes) 000317 1753365129.183275 trunc from 0x02ccdd to 0x03450b (0x782f bytes) 000318 1753365129.190265 punch from 0x004b54 to 0x00cb1f (0x7fcc bytes) 000319 1753365129.194471 fallocate 0x0152b4 thru 0x01967c (0x43c9 bytes) 000320 1753365129.199560 mapread 0x003b70 thru 0x00e6ee (0xab7f bytes) 000321 1753365129.211810 mapwrite 0x03c47c thru 0x03ffff (0x3b84 bytes) 000322 1753365129.223161 MIRROR_RESYNC 3 mirrors 000323 1753365129.493375 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000324 1753365129.603712 trunc from 0x040000 to 0x03e47f (0x1b80 bytes) 000325 1753365129.635357 punch from 0x03e1c7 to 0x03e47f (0x2b9 bytes) 000326 1753365129.639482 mapwrite 0x02e944 thru 0x030a25 (0x20e2 bytes) 000327 1753365129.658360 trunc from 0x03e480 to 0x0029e9 (0x3ba96 bytes) 000328 1753365129.668379 read 0x000ff4 thru 0x0029e9 (0x19f6 bytes) 000329 1753365129.671419 mapread 0x000512 thru 0x0029e9 (0x24d8 bytes) 000330 1753365129.672121 punch from 0x0005cb to 0x0029e9 (0x241f bytes) 000331 1753365129.675054 write 0x00d03d thru 0x01c2e5 (0xf2a9 bytes) 000332 1753365129.676498 punch from 0x0095b9 to 0x00c234 (0x2c7c bytes) 000334 1753365129.679296 trunc from 0x01c2e6 to 0x030404 (0x1411f bytes) 000335 1753365129.686333 mapwrite 0x007db6 thru 0x01501e (0xd269 bytes) 000336 1753365129.703655 fallocate 0x032825 thru 0x03df15 (0xb6f1 bytes) 000338 1753365129.710346 MIRROR_RESYNC 2 mirrors 000339 1753365129.881277 fallocate 0x029730 thru 0x02eb35 (0x5406 bytes) 000340 1753365129.894808 trunc from 0x03df16 to 0x028c81 (0x15294 bytes) 000341 1753365129.909823 fallocate 0x024c80 thru 0x02d305 (0x8686 bytes) 000342 1753365129.911924 write 0x030a67 thru 0x03fce7 (0xf281 bytes) 000343 1753365129.913159 punch from 0x024858 to 0x024a85 (0x22e bytes) 000344 1753365129.917321 MIRROR_RESYNC 2 mirrors 000345 1753365130.095071 trunc from 0x03fce8 to 0x0352d2 (0xaa15 bytes) 000346 1753365130.117408 fallocate 0x0265d2 thru 0x035029 (0xea58 bytes) 000347 1753365130.121170 trunc from 0x0352d3 to 0x0006e0 (0x34bf2 bytes) 000350 1753365130.137846 trunc from 0x000577 to 0x0006e0 (0x16a bytes) 000351 1753365130.145230 MIRROR_RESYNC 2 mirrors 000352 1753365130.328655 write 0x022bc2 thru 0x02651b (0x395a bytes) 000353 1753365130.345787 trunc from 0x011e3a to 0x020cb8 (0xee7f bytes) 000354 1753365130.349413 mapwrite 0x03af57 thru 0x03ffff (0x50a9 bytes) 000355 1753365130.380768 read 0x02a77e thru 0x038999 (0xe21c bytes) 000356 1753365130.385543 write 0x008399 thru 0x0102d1 (0x7f39 bytes) 000357 1753365130.397219 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000358 1753365130.475474 fallocate 0x03231f thru 0x03bddb (0x9abd bytes) 000359 1753365130.481400 write 0x00d606 thru 0x00dcb5 (0x6b0 bytes) 000361 1753365130.489022 fallocate 0x02f226 thru 0x03c0a4 (0xce7f bytes) 000362 1753365130.492309 mapwrite 0x031cce thru 0x03c7e0 (0xab13 bytes) 000363 1753365130.520274 mapwrite 0x00bdd1 thru 0x00d1a6 (0x13d6 bytes) 000364 1753365130.538924 read 0x01df37 thru 0x025010 (0x70da bytes) 000365 1753365130.545118 read 0x020b6b thru 0x024487 (0x391d bytes) 000366 1753365130.545570 punch from 0x01af1d to 0x02a3d0 (0xf4b4 bytes) 000370 1753365130.551154 mapwrite 0x0355ae thru 0x039ab0 (0x4503 bytes) 000372 1753365130.563720 punch from 0x01be4d to 0x0222b3 (0x6467 bytes) 000373 1753365130.567824 write 0x01310a thru 0x01e5a8 (0xb49f bytes) 000374 1753365130.573827 mapwrite 0x03df98 thru 0x03f4e4 (0x154d bytes) 000376 1753365130.585362 write 0x03b548 thru 0x03ffff (0x4ab8 bytes) 000377 1753365130.586359 mapwrite 0x02e417 thru 0x03202b (0x3c15 bytes) 000378 1753365130.601301 mapwrite 0x00537e thru 0x013461 (0xe0e4 bytes) 000379 1753365130.613726 read 0x0337ff thru 0x03ffff (0xc801 bytes) 000380 1753365130.614324 write 0x00cde0 thru 0x01a96f (0xdb90 bytes) 000381 1753365130.615621 trunc from 0x0323b7 to 0x03ba49 (0x9693 bytes) 000382 1753365130.619090 punch from 0x025160 to 0x02fc9e (0xab3f bytes) 000383 1753365130.622028 trunc from 0x00b4bd to 0x01a054 (0xeb98 bytes) 000384 1753365130.630544 write 0x00437e thru 0x0055f0 (0x1273 bytes) 000385 1753365130.631453 mapread 0x03673d thru 0x03ffff (0x98c3 bytes) 000386 1753365130.639014 trunc from 0x0314e1 to 0x033a21 (0x2541 bytes) 000387 1753365130.644845 MIRROR_EXTEND to 2 mirrors 000388 1753365130.917372 punch from 0x03e706 to 0x03ffff (0x18fa bytes) 000389 1753365130.937620 punch from 0x033abf to 0x03731e (0x3860 bytes) 000391 1753365130.944472 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000392 1753365131.021127 read 0x006833 thru 0x013657 (0xce25 bytes) 000393 1753365131.026942 write 0x00faeb thru 0x01aa9e (0xafb4 bytes) 000394 1753365131.035220 mapwrite 0x006e13 thru 0x008ed6 (0x20c4 bytes) 000395 1753365131.053098 mapread 0x02cc1a thru 0x03546c (0x8853 bytes) 000396 1753365131.065763 punch from 0x009b83 to 0x016326 (0xc7a4 bytes) 000397 1753365131.071072 punch from 0x02ff5a to 0x03cf2c (0xcfd3 bytes) 000398 1753365131.076873 read 0x020c6b thru 0x02ddfc (0xd192 bytes) 000399 1753365131.089172 trunc from 0x040000 to 0x03ae49 (0x51b6 bytes) 000400 1753365131.119243 trunc from 0x0062ca to 0x00dec0 (0x7bf7 bytes) 000401 1753365131.123140 punch from 0x0057a8 to 0x00eda8 (0x9601 bytes) 000402 1753365131.127229 fallocate 0x03158b thru 0x0358a6 (0x431c bytes) 000404 1753365131.131823 punch from 0x012df3 to 0x014e6c (0x207a bytes) 000405 1753365131.137365 MIRROR_EXTEND to 2 mirrors 000406 1753365131.424625 mapwrite 0x00684e thru 0x010cbf (0xa472 bytes) 000407 1753365131.481467 fallocate 0x018dbb thru 0x01f822 (0x6a68 bytes) 000408 1753365131.484326 mapwrite 0x01c6cb thru 0x02aeeb (0xe821 bytes) 000409 1753365131.503216 fallocate 0x02e9f7 thru 0x03738e (0x8998 bytes) 000410 1753365131.505898 write 0x001b13 thru 0x007fff (0x64ed bytes) 000411 1753365131.508615 trunc from 0x020756 to 0x02b8cb (0xb176 bytes) 000412 1753365131.511095 read 0x000883 thru 0x0072ee (0x6a6c bytes) 000413 1753365131.516654 MIRROR_EXTEND to 3 mirrors 000414 1753365131.743488 punch from 0x0226c7 to 0x027bce (0x5508 bytes) 000415 1753365131.762449 read 0x02f766 thru 0x031c6f (0x250a bytes) 000416 1753365131.765609 mapwrite 0x034890 thru 0x035db6 (0x1527 bytes) 000417 1753365131.782172 punch from 0x02d783 to 0x02f821 (0x209f bytes) 000418 1753365131.787260 trunc from 0x025b50 to 0x0332ce (0xd77f bytes) 000419 1753365131.790724 mapread 0x0244e7 thru 0x032ec6 (0xe9e0 bytes) 000420 1753365131.800556 fallocate 0x00ea06 thru 0x017c7f (0x927a bytes) 000422 1753365131.804361 mapwrite 0x02d225 thru 0x0392f7 (0xc0d3 bytes) 000425 1753365131.821616 write 0x031b4b thru 0x03db39 (0xbfef bytes) 000426 1753365131.825777 trunc from 0x01a816 to 0x026fae (0xc799 bytes) 000427 1753365131.830359 mapread 0x017a25 thru 0x01bbde (0x41ba bytes) 000428 1753365131.836764 mapwrite 0x01d3d8 thru 0x026176 (0x8d9f bytes) 000430 1753365131.856896 write 0x00d12c thru 0x01952f (0xc404 bytes) 000431 1753365131.865058 write 0x014bc9 thru 0x01a48c (0x58c4 bytes) 000432 1753365131.866085 fallocate 0x01ffc6 thru 0x020dec (0xe27 bytes) 000433 1753365131.869941 trunc from 0x03db3a to 0x000708 (0x3d431 bytes) 000434 1753365131.900451 MIRROR_RESYNC 3 mirrors 000435 1753365132.119565 mapwrite 0x010575 thru 0x01f828 (0xf2b4 bytes) 000437 1753365132.169169 mapread 0x01d131 thru 0x01f828 (0x26f8 bytes) 000438 1753365132.170071 punch from 0x013119 to 0x01f5bb (0xc4a3 bytes) 000439 1753365132.173308 mapwrite 0x002ee2 thru 0x00f0a5 (0xc1c4 bytes) 000440 1753365132.193308 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000441 1753365132.296440 punch from 0x00f3dd to 0x0154fb (0x611f bytes) 000443 1753365132.304260 MIRROR_EXTEND to 3 mirrors 000444 1753365132.554618 mapwrite 0x03e4d5 thru 0x03ffff (0x1b2b bytes) 000445 1753365132.592623 write 0x0037f9 thru 0x011eda (0xe6e2 bytes) 000446 1753365132.603258 punch from 0x008a79 to 0x0179e3 (0xef6b bytes) 000447 1753365132.611633 mapwrite 0x03d2c7 thru 0x03ffff (0x2d39 bytes) 000448 1753365132.626164 read 0x02b47a thru 0x0318b9 (0x6440 bytes) 000449 1753365132.631506 trunc from 0x040000 to 0x02b01d (0x14fe2 bytes) 000450 1753365132.642094 mapread 0x01ac2c thru 0x028563 (0xd938 bytes) 000451 1753365132.654024 mapwrite 0x001bc7 thru 0x0057eb (0x3c25 bytes) 000452 1753365132.669055 read 0x01100f thru 0x01a099 (0x908b bytes) 000453 1753365132.674850 fallocate 0x036f44 thru 0x03a806 (0x38c3 bytes) 000454 1753365132.680175 write 0x03a01e thru 0x03ffff (0x5fe2 bytes) 000455 1753365132.681733 trunc from 0x01026b to 0x01f148 (0xeede bytes) 000457 1753365132.686860 write 0x020389 thru 0x02d003 (0xcc7b bytes) 000458 1753365132.694980 read 0x021f14 thru 0x030a35 (0xeb22 bytes) 000459 1753365132.696031 mapwrite 0x01b25f thru 0x0212ea (0x608c bytes) 000460 1753365132.712044 read 0x01dca3 thru 0x02625d (0x85bb bytes) 000461 1753365132.712571 fallocate 0x02f5c9 thru 0x0394c8 (0x9f00 bytes) 000462 1753365132.716158 mapwrite 0x02c0a0 thru 0x02fb21 (0x3a82 bytes) 000463 1753365132.725781 mapwrite 0x03d711 thru 0x03ffff (0x28ef bytes) 000464 1753365132.733771 trunc from 0x040000 to 0x02ac3d (0x153c2 bytes) 000466 1753365132.741747 trunc from 0x002c7a to 0x011ad7 (0xee5e bytes) 000467 1753365132.745189 mapwrite 0x025887 thru 0x03416b (0xe8e5 bytes) 000468 1753365132.776745 punch from 0x0262ea to 0x0307af (0xa4c6 bytes) 000469 1753365132.780137 mapread 0x01abe8 thru 0x01d1e4 (0x25fd bytes) 000470 1753365132.781047 trunc from 0x033501 to 0x03416b (0xc6b bytes) 000471 1753365132.784144 punch from 0x01afb5 to 0x01f557 (0x45a3 bytes) 000472 1753365132.787165 read 0x027c7e thru 0x0308e0 (0x8c63 bytes) 000474 1753365132.792913 trunc from 0x03416c to 0x00ee9e (0x252cd bytes) 000475 1753365132.801035 trunc from 0x0056ef to 0x00b6ab (0x5fbd bytes) 000476 1753365132.804087 write 0x0380f5 thru 0x03ebd8 (0x6ae4 bytes) 000477 1753365132.805304 trunc from 0x03ebd9 to 0x020394 (0x1e844 bytes) 000478 1753365132.813998 read 0x00f28f thru 0x01e452 (0xf1c4 bytes) 000479 1753365132.819373 trunc from 0x020395 to 0x03ef03 (0x1eb6f bytes) 000480 1753365132.829587 MIRROR_RESYNC 3 mirrors 000481 1753365133.052580 trunc from 0x012d88 to 0x016f2d (0x41a6 bytes) 000482 1753365133.074042 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000483 1753365133.165973 write 0x00ed51 thru 0x014a84 (0x5d34 bytes) 000484 1753365133.174449 mapwrite 0x037575 thru 0x03ab5c (0x35e8 bytes) 000486 1753365133.192683 mapwrite 0x0212c6 thru 0x02cfd3 (0xbd0e bytes) 000487 1753365133.222047 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000488 1753365133.287558 punch from 0x0119f8 to 0x01a10b (0x8714 bytes) 000489 1753365133.290878 trunc from 0x03ef04 to 0x007020 (0x37ee3 bytes) 000490 1753365133.307964 punch from 0x002bad to 0x007020 (0x4474 bytes) 000491 1753365133.311796 trunc from 0x0059db to 0x007020 (0x1646 bytes) 000492 1753365133.314906 trunc from 0x0057d7 to 0x007020 (0x184a bytes) 000493 1753365133.318133 fallocate 0x0299c4 thru 0x036363 (0xc9a0 bytes) 000494 1753365133.321839 mapwrite 0x035dcb thru 0x03ffff (0xa235 bytes) 000495 1753365133.351619 punch from 0x02ca74 to 0x02ce89 (0x416 bytes) 000496 1753365133.354299 fallocate 0x004842 thru 0x0102df (0xba9e bytes) 000497 1753365133.357235 read 0x01df81 thru 0x02a8b4 (0xc934 bytes) 000500 1753365133.363304 mapwrite 0x02112b thru 0x0232e5 (0x21bb bytes) 000501 1753365133.375127 MIRROR_EXTEND to 2 mirrors 000502 1753365133.641958 MIRROR_RESYNC 2 mirrors 000503 1753365133.742864 trunc from 0x01ab4d to 0x024755 (0x9c09 bytes) 000504 1753365133.759906 trunc from 0x040000 to 0x03f057 (0xfa8 bytes) 000505 1753365133.785207 MIRROR_EXTEND to 3 mirrors 000506 1753365134.010623 punch from 0x00b814 to 0x0141e4 (0x89d1 bytes) 000507 1753365134.029356 punch from 0x000995 to 0x0066da (0x5d46 bytes) 000508 1753365134.034888 MIRROR_EXTEND to 4 mirrors 000509 1753365134.281339 trunc from 0x0332d7 to 0x03f057 (0xbd81 bytes) 000511 1753365134.304170 trunc from 0x03f058 to 0x001c86 (0x3d3d1 bytes) 000512 1753365134.319248 punch from 0x0018b1 to 0x001c86 (0x3d6 bytes) 000513 1753365134.326505 mapwrite 0x00936f thru 0x01067e (0x7310 bytes) 000515 1753365134.371061 read 0x00388c thru 0x00acb8 (0x742d bytes) 000516 1753365134.375012 punch from 0x008902 to 0x01067e (0x7d7d bytes) 000517 1753365134.378352 mapwrite 0x006df6 thru 0x015164 (0xe36f bytes) 000518 1753365134.414084 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000519 1753365134.515335 mapwrite 0x02a9a2 thru 0x033558 (0x8bb7 bytes) 000520 1753365134.550727 mapwrite 0x012afd thru 0x01b16b (0x866f bytes) 000521 1753365134.570408 write 0x021508 thru 0x02b7bd (0xa2b6 bytes) 000522 1753365134.571637 punch from 0x0063a8 to 0x014815 (0xe46e bytes) 000523 1753365134.575060 mapread 0x0068c7 thru 0x015605 (0xed3f bytes) 000524 1753365134.585395 MIRROR_RESYNC 3 mirrors 000525 1753365134.883414 mapwrite 0x013acb thru 0x01f44d (0xb983 bytes) 000526 1753365134.952477 punch from 0x00beb3 to 0x015c8f (0x9ddd bytes) 000527 1753365134.960636 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000528 1753365135.074070 read 0x00900b thru 0x0169a8 (0xd99e bytes) 000529 1753365135.081752 trunc from 0x02cbb9 to 0x031982 (0x4dca bytes) 000530 1753365135.085308 trunc from 0x033559 to 0x0276f8 (0xbe60 bytes) 000531 1753365135.106541 mapwrite 0x02a1f6 thru 0x02a5b9 (0x3c4 bytes) 000533 1753365135.134674 mapwrite 0x03c133 thru 0x03eae0 (0x29ae bytes) 000534 1753365135.174327 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000535 1753365135.236346 trunc from 0x00d530 to 0x019694 (0xc165 bytes) 000536 1753365135.238856 read 0x032b58 thru 0x03c0ca (0x9573 bytes) 000537 1753365135.243816 write 0x03938c thru 0x03ffff (0x6c74 bytes) 000538 1753365135.247924 write 0x008b1b thru 0x012189 (0x966f bytes) 000539 1753365135.254821 trunc from 0x009d8a to 0x00e3fa (0x4671 bytes) 000540 1753365135.262820 read 0x035807 thru 0x03d767 (0x7f61 bytes) 000541 1753365135.263326 trunc from 0x023256 to 0x030b15 (0xd8c0 bytes) 000542 1753365135.266147 trunc from 0x01e9bd to 0x023353 (0x4997 bytes) 000544 1753365135.270028 mapwrite 0x002eca thru 0x0123b5 (0xf4ec bytes) 000546 1753365135.289579 read 0x00b0eb thru 0x00f9a7 (0x48bd bytes) 000547 1753365135.290195 write 0x0163f9 thru 0x020e72 (0xaa7a bytes) 000548 1753365135.291377 fallocate 0x02579c thru 0x03389f (0xe104 bytes) 000550 1753365135.295631 mapwrite 0x02fb4b thru 0x0369d4 (0x6e8a bytes) 000551 1753365135.310426 trunc from 0x040000 to 0x003ae4 (0x3c51b bytes) 000552 1753365135.320747 trunc from 0x003ae5 to 0x037f52 (0x3446e bytes) 000553 1753365135.331433 write 0x01e9e0 thru 0x02451d (0x5b3e bytes) 000554 1753365135.338080 trunc from 0x0121fa to 0x012bd8 (0x9df bytes) 000555 1753365135.341326 trunc from 0x0199c0 to 0x01aeba (0x14fb bytes) 000557 1753365135.349594 MIRROR_EXTEND to 2 mirrors 000558 1753365135.563035 mapwrite 0x024ed8 thru 0x031757 (0xc880 bytes) 000559 1753365135.619435 write 0x0058ba thru 0x00751a (0x1c61 bytes) 000560 1753365135.630669 trunc from 0x016133 to 0x01de48 (0x7d16 bytes) 000561 1753365135.634593 mapwrite 0x0061bf thru 0x009525 (0x3367 bytes) 000562 1753365135.648448 mapwrite 0x022c7e thru 0x02782b (0x4bae bytes) 000563 1753365135.666445 trunc from 0x037f53 to 0x0187ab (0x1f7a7 bytes) 000564 1753365135.694163 punch from 0x002bac to 0x00651c (0x3971 bytes) 000566 1753365135.697776 fallocate 0x01bc36 thru 0x02b854 (0xfc1f bytes) 000567 1753365135.701034 write 0x002b4e thru 0x00ae22 (0x82d5 bytes) 000568 1753365135.707223 trunc from 0x02b855 to 0x03f828 (0x13fd4 bytes) 000570 1753365135.719008 mapread 0x00f55c thru 0x0120fe (0x2ba3 bytes) 000571 1753365135.732950 MIRROR_EXTEND to 3 mirrors 000572 1753365135.976669 MIRROR_EXTEND to 4 mirrors 000575 1753365136.190599 read 0x02f1b7 thru 0x036555 (0x739f bytes) 000576 1753365136.194399 mapread 0x03814e thru 0x03ef02 (0x6db5 bytes) 000577 1753365136.202917 trunc from 0x02cde1 to 0x037b5d (0xad7d bytes) 000578 1753365136.222855 MIRROR_RESYNC 4 mirrors 000579 1753365136.567154 trunc from 0x03f829 to 0x02d8ef (0x11f39 bytes) 000580 1753365136.587061 read 0x00cccf thru 0x012324 (0x5656 bytes) 000581 1753365136.591785 punch from 0x008db6 to 0x00c36d (0x35b8 bytes) 000582 1753365136.595034 write 0x02c915 thru 0x031f87 (0x5673 bytes) 000583 1753365136.598537 mapwrite 0x02641f thru 0x034081 (0xdc63 bytes) 000584 1753365136.633112 read 0x01fade thru 0x020a95 (0xfb8 bytes) 000585 1753365136.637926 trunc from 0x034082 to 0x001f9f (0x320e2 bytes) 000586 1753365136.644375 read 0x000370 thru 0x000ccf (0x960 bytes) 000587 1753365136.644956 write 0x024fc5 thru 0x029301 (0x433d bytes) 000588 1753365136.646002 fallocate 0x0149a3 thru 0x01e74d (0x9dab bytes) 000589 1753365136.648986 mapwrite 0x002285 thru 0x00a308 (0x8084 bytes) 000590 1753365136.663718 read 0x01610d thru 0x01ac4b (0x4b3f bytes) 000591 1753365136.664360 punch from 0x02796d to 0x029301 (0x1995 bytes) 000592 1753365136.667648 mapwrite 0x001c85 thru 0x0077ba (0x5b36 bytes) 000593 1753365136.679466 mapread 0x0186e9 thru 0x0187fd (0x115 bytes) 000594 1753365136.680397 fallocate 0x03d967 thru 0x03ffff (0x2699 bytes) 000595 1753365136.684671 write 0x00c0ed thru 0x010b5a (0x4a6e bytes) 000596 1753365136.685803 trunc from 0x029302 to 0x0008fb (0x28a06 bytes) 000597 1753365136.694394 write 0x00ef6c thru 0x013b05 (0x4b9a bytes) 000598 1753365136.695657 mapread 0x00f567 thru 0x010ae5 (0x157f bytes) 000599 1753365136.696401 trunc from 0x012a42 to 0x013b05 (0x10c4 bytes) 000600 1753365136.704283 write 0x016d21 thru 0x0225d6 (0xb8b6 bytes) 000601 1753365136.708105 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000602 1753365136.801024 fallocate 0x0098c6 thru 0x010159 (0x6894 bytes) 000603 1753365136.804092 fallocate 0x01032d thru 0x01222f (0x1f03 bytes) 000604 1753365136.807271 fallocate 0x01618b thru 0x01cd13 (0x6b89 bytes) 000605 1753365136.810168 mapread 0x015379 thru 0x016f30 (0x1bb8 bytes) 000606 1753365136.820343 MIRROR_EXTEND to 4 mirrors 000607 1753365137.057766 MIRROR_RESYNC 4 mirrors 000608 1753365137.302233 fallocate 0x0358a4 thru 0x03ffff (0xa75c bytes) 000609 1753365137.319752 MIRROR_RESYNC 4 mirrors 000610 1753365137.670124 punch from 0x0029c1 to 0x00693e (0x3f7e bytes) 000611 1753365137.685904 punch from 0x013c4d to 0x01608c (0x2440 bytes) 000612 1753365137.688869 read 0x03cb3d thru 0x03ec7b (0x213f bytes) 000613 1753365137.692474 mapread 0x039414 thru 0x03b155 (0x1d42 bytes) 000614 1753365137.701132 trunc from 0x040000 to 0x0083d2 (0x37c2d bytes) 000615 1753365137.712135 write 0x0212ec thru 0x028530 (0x7245 bytes) 000616 1753365137.718025 MIRROR_EXTEND to 5 mirrors 000617 1753365137.908388 fallocate 0x03ef81 thru 0x03ffff (0x107f bytes) 000618 1753365137.926388 mapwrite 0x03313d thru 0x034e42 (0x1d06 bytes) 000619 1753365137.941565 read 0x016907 thru 0x01f0cc (0x87c6 bytes) 000620 1753365137.946291 trunc from 0x031bda to 0x035487 (0x38ae bytes) 000621 1753365137.949785 fallocate 0x0277fb thru 0x02d12e (0x5934 bytes) 000622 1753365137.952690 punch from 0x016c7c to 0x024da7 (0xe12c bytes) 000623 1753365137.955606 punch from 0x02d1c2 to 0x039785 (0xc5c4 bytes) 000625 1753365137.962043 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000626 1753365138.069392 trunc from 0x03be17 to 0x03ffff (0x41e9 bytes) 000627 1753365138.072795 mapread 0x029f15 thru 0x02ffb3 (0x609f bytes) 000628 1753365138.089583 read 0x001072 thru 0x010de9 (0xfd78 bytes) 000630 1753365138.095982 trunc from 0x005f65 to 0x007ccb (0x1d67 bytes) 000631 1753365138.099861 mapread 0x027264 thru 0x028bef (0x198c bytes) 000632 1753365138.101290 trunc from 0x040000 to 0x016504 (0x29afb bytes) 000633 1753365138.124577 mapwrite 0x007e77 thru 0x013d42 (0xbecc bytes) 000634 1753365138.149263 write 0x012564 thru 0x01ba6b (0x9508 bytes) 000635 1753365138.153062 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000636 1753365138.266721 mapwrite 0x0080fc thru 0x008b9c (0xaa1 bytes) 000637 1753365138.279215 read 0x00db85 thru 0x0117bd (0x3c39 bytes) 000638 1753365138.289919 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000639 1753365138.414282 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000640 1753365138.477632 read 0x0080da thru 0x009f94 (0x1ebb bytes) 000641 1753365138.482300 write 0x03237d thru 0x037ce3 (0x5967 bytes) 000642 1753365138.489499 MIRROR_EXTEND to 2 mirrors 000643 1753365138.718971 trunc from 0x014542 to 0x01d008 (0x8ac7 bytes) 000644 1753365138.740566 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000645 1753365138.806359 write 0x01645e thru 0x019a5d (0x3600 bytes) 000647 1753365138.814221 write 0x020580 thru 0x02d649 (0xd0ca bytes) 000648 1753365138.826464 mapwrite 0x01c541 thru 0x01fb49 (0x3609 bytes) 000649 1753365138.846349 fallocate 0x033639 thru 0x037963 (0x432b bytes) 000652 1753365138.851061 mapwrite 0x01465c thru 0x014a17 (0x3bc bytes) 000654 1753365138.865854 trunc from 0x037ce4 to 0x010361 (0x27982 bytes) 000655 1753365138.885642 trunc from 0x00b28c to 0x01018c (0x4f01 bytes) 000659 1753365138.891495 read 0x00e2f9 thru 0x010361 (0x2069 bytes) 000660 1753365138.900294 MIRROR_EXTEND to 2 mirrors 000661 1753365139.157916 MIRROR_RESYNC 2 mirrors 000662 1753365139.252442 punch from 0x005a73 to 0x010361 (0xa8ef bytes) 000663 1753365139.276473 read 0x0100fb thru 0x010361 (0x267 bytes) 000664 1753365139.280093 mapwrite 0x02ca1d thru 0x033905 (0x6ee9 bytes) 000665 1753365139.338247 MIRROR_RESYNC 2 mirrors 000666 1753365139.559004 trunc from 0x033906 to 0x018c67 (0x1ac9e bytes) 000667 1753365139.584139 trunc from 0x007ba4 to 0x010e73 (0x92d0 bytes) 000668 1753365139.587334 mapwrite 0x005b18 thru 0x00f332 (0x981b bytes) 000669 1753365139.611019 fallocate 0x01e058 thru 0x02be52 (0xddfb bytes) 000670 1753365139.614417 read 0x0058c2 thru 0x00bb5b (0x629a bytes) 000671 1753365139.614991 mapwrite 0x028ba9 thru 0x035a41 (0xce99 bytes) 000673 1753365139.645394 trunc from 0x012c26 to 0x01c55c (0x9937 bytes) 000675 1753365139.649158 mapread 0x01d596 thru 0x0237c2 (0x622d bytes) 000676 1753365139.653703 mapread 0x01561b thru 0x01fc11 (0xa5f7 bytes) 000677 1753365139.658350 trunc from 0x02eebe to 0x031989 (0x2acc bytes) 000678 1753365139.661791 trunc from 0x035a42 to 0x0125ba (0x23487 bytes) 000679 1753365139.668622 mapread 0x00c7a5 thru 0x0125ba (0x5e16 bytes) 000680 1753365139.669473 write 0x035042 thru 0x037d98 (0x2d57 bytes) 000681 1753365139.670231 trunc from 0x020e07 to 0x023eda (0x30d4 bytes) 000682 1753365139.672962 fallocate 0x01042c thru 0x01c561 (0xc136 bytes) 000684 1753365139.676182 trunc from 0x037d99 to 0x00e8a4 (0x294f4 bytes) 000685 1753365139.683101 trunc from 0x00305f to 0x006304 (0x32a6 bytes) 000686 1753365139.687877 MIRROR_RESYNC 2 mirrors 000687 1753365139.817361 trunc from 0x005101 to 0x00ac0a (0x5b0a bytes) 000688 1753365139.837216 MIRROR_EXTEND to 3 mirrors 000689 1753365140.018414 fallocate 0x01e595 thru 0x024a12 (0x647e bytes) 000690 1753365140.036586 trunc from 0x00e8a5 to 0x0300c6 (0x21822 bytes) 000691 1753365140.055060 write 0x0294b3 thru 0x033814 (0xa362 bytes) 000692 1753365140.059094 trunc from 0x022a55 to 0x024458 (0x1a04 bytes) 000693 1753365140.064139 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000694 1753365140.175176 trunc from 0x033815 to 0x025972 (0xdea2 bytes) 000697 1753365140.189843 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000698 1753365140.251445 mapread 0x01a38b thru 0x025972 (0xb5e8 bytes) 000700 1753365140.266220 punch from 0x00223b to 0x00eda2 (0xcb68 bytes) 000701 1753365140.269178 fallocate 0x01e484 thru 0x01fbc4 (0x1741 bytes) 000702 1753365140.272107 write 0x0335be thru 0x03ad11 (0x7754 bytes) 000704 1753365140.276211 mapwrite 0x007a36 thru 0x01315e (0xb729 bytes) 000705 1753365140.295375 read 0x0297c7 thru 0x031161 (0x799b bytes) 000707 1753365140.300772 fallocate 0x03fe37 thru 0x03ffff (0x1c9 bytes) 000709 1753365140.304707 read 0x011345 thru 0x019dda (0x8a96 bytes) 000710 1753365140.305272 trunc from 0x026a24 to 0x0283c2 (0x199f bytes) 000711 1753365140.308219 read 0x03fdac thru 0x03ffff (0x254 bytes) 000712 1753365140.313690 write 0x03011b thru 0x03c237 (0xc11d bytes) 000714 1753365140.315621 trunc from 0x00e9c6 to 0x01b6d1 (0xcd0c bytes) 000715 1753365140.319954 write 0x02a3f4 thru 0x034bb7 (0xa7c4 bytes) 000716 1753365140.321198 write 0x03cbc5 thru 0x03ffff (0x343b bytes) 000717 1753365140.322064 mapread 0x036eea thru 0x039583 (0x269a bytes) 000718 1753365140.322811 read 0x02cc15 thru 0x02fa0b (0x2df7 bytes) 000719 1753365140.323309 fallocate 0x00c8b6 thru 0x011a5a (0x51a5 bytes) 000720 1753365140.326960 trunc from 0x040000 to 0x037758 (0x88a7 bytes) 000721 1753365140.336959 trunc from 0x0224b0 to 0x025cfd (0x384e bytes) 000722 1753365140.341389 trunc from 0x008298 to 0x014eb4 (0xcc1d bytes) 000723 1753365140.344816 trunc from 0x037759 to 0x029ec9 (0xd88f bytes) 000724 1753365140.356944 trunc from 0x029eca to 0x02d03b (0x3172 bytes) 000725 1753365140.368465 trunc from 0x02d03c to 0x03d666 (0x1062b bytes) 000726 1753365140.378083 mapread 0x03b1af thru 0x03d666 (0x24b8 bytes) 000727 1753365140.385606 mapread 0x005b22 thru 0x009ddd (0x42bc bytes) 000728 1753365140.386428 mapread 0x01c1be thru 0x023674 (0x74b7 bytes) 000729 1753365140.387247 mapread 0x02a504 thru 0x0366f3 (0xc1f0 bytes) 000731 1753365140.388974 mapread 0x0199c4 thru 0x01faaf (0x60ec bytes) 000733 1753365140.390122 fallocate 0x037714 thru 0x03ffff (0x88ec bytes) 000734 1753365140.393655 fallocate 0x00469a thru 0x0084ec (0x3e53 bytes) 000735 1753365140.397173 read 0x00cf12 thru 0x013b12 (0x6c01 bytes) 000736 1753365140.397774 trunc from 0x040000 to 0x00d9ac (0x32653 bytes) 000737 1753365140.409955 read 0x009b17 thru 0x00d9ac (0x3e96 bytes) 000738 1753365140.410782 fallocate 0x01ceff thru 0x020a5a (0x3b5c bytes) 000739 1753365140.414857 trunc from 0x01b596 to 0x01e557 (0x2fc2 bytes) 000740 1753365140.418749 read 0x01f044 thru 0x020a5a (0x1a17 bytes) 000741 1753365140.423359 write 0x003306 thru 0x004472 (0x116d bytes) 000742 1753365140.424235 write 0x014560 thru 0x01a48f (0x5f30 bytes) 000743 1753365140.425126 trunc from 0x00980e to 0x01433e (0xab31 bytes) 000744 1753365140.432007 punch from 0x012d94 to 0x0133f4 (0x661 bytes) 000747 1753365140.435711 mapread 0x01b5c3 thru 0x020a5a (0x5498 bytes) 000748 1753365140.436616 read 0x0160b8 thru 0x020a5a (0xa9a3 bytes) 000749 1753365140.437098 write 0x024f0b thru 0x02c59c (0x7692 bytes) 000750 1753365140.440912 MIRROR_EXTEND to 2 mirrors 000751 1753365140.683024 MIRROR_EXTEND to 3 mirrors 000752 1753365140.912190 mapwrite 0x0324b0 thru 0x03607b (0x3bcc bytes) 000753 1753365140.972456 punch from 0x01a850 to 0x022a2b (0x81dc bytes) 000754 1753365140.975923 write 0x032060 thru 0x033544 (0x14e5 bytes) 000755 1753365140.976983 fallocate 0x02005c thru 0x02a5ab (0xa550 bytes) 000756 1753365140.981946 trunc from 0x00fae9 to 0x014475 (0x498d bytes) 000757 1753365140.985180 fallocate 0x035fb2 thru 0x03ffff (0xa04e bytes) 000758 1753365140.989659 mapread 0x01ed77 thru 0x0223d7 (0x3661 bytes) 000759 1753365140.998794 punch from 0x038dbf to 0x03ffff (0x7241 bytes) 000760 1753365141.003432 write 0x00c806 thru 0x014917 (0x8112 bytes) 000761 1753365141.011952 fallocate 0x03f293 thru 0x03ffff (0xd6d bytes) 000762 1753365141.015864 mapread 0x033f3d thru 0x03c942 (0x8a06 bytes) 000763 1753365141.021637 trunc from 0x040000 to 0x01476c (0x2b893 bytes) 000764 1753365141.031460 mapwrite 0x02e7b7 thru 0x038eb2 (0xa6fc bytes) 000765 1753365141.055745 write 0x023bc9 thru 0x0244be (0x8f6 bytes) 000766 1753365141.060653 write 0x01f3e5 thru 0x02c4a1 (0xd0bd bytes) 000768 1753365141.062222 punch from 0x030aec to 0x031a39 (0xf4e bytes) 000769 1753365141.065172 write 0x01d4df thru 0x0240cf (0x6bf1 bytes) 000770 1753365141.066278 fallocate 0x01db06 thru 0x01e984 (0xe7f bytes) 000771 1753365141.070081 mapwrite 0x02e989 thru 0x033736 (0x4dae bytes) 000772 1753365141.082600 mapwrite 0x01b39d thru 0x01bf77 (0xbdb bytes) 000773 1753365141.095145 read 0x020db3 thru 0x026aac (0x5cfa bytes) 000774 1753365141.095781 punch from 0x00c4d1 to 0x011c17 (0x5747 bytes) 000775 1753365141.101147 MIRROR_EXTEND to 4 mirrors 000776 1753365141.323499 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000777 1753365141.451076 trunc from 0x038eb3 to 0x017193 (0x21d1f bytes) 000778 1753365141.481987 fallocate 0x012552 thru 0x013323 (0xdd2 bytes) 000781 1753365141.485406 write 0x015331 thru 0x02132c (0xbffc bytes) 000782 1753365141.492320 MIRROR_RESYNC 3 mirrors 000783 1753365141.737308 trunc from 0x0161d1 to 0x016a51 (0x881 bytes) 000784 1753365141.754748 fallocate 0x001a29 thru 0x00b489 (0x9a61 bytes) 000786 1753365141.760161 mapread 0x00a3f9 thru 0x016eff (0xcb07 bytes) 000787 1753365141.782479 trunc from 0x00b62d to 0x00f0f3 (0x3ac7 bytes) 000788 1753365141.786463 fallocate 0x02a31a thru 0x031d72 (0x7a59 bytes) 000789 1753365141.789983 write 0x00c14f thru 0x019dba (0xdc6c bytes) 000790 1753365141.794919 mapread 0x00e7b7 thru 0x014df4 (0x663e bytes) 000791 1753365141.798727 MIRROR_RESYNC 3 mirrors 000792 1753365142.018424 mapwrite 0x03fb9a thru 0x03ffff (0x466 bytes) 000793 1753365142.064959 mapwrite 0x00b932 thru 0x00e1e3 (0x28b2 bytes) 000794 1753365142.085671 read 0x00bf3b thru 0x015bd7 (0x9c9d bytes) 000795 1753365142.088945 trunc from 0x040000 to 0x007f1d (0x380e2 bytes) 000796 1753365142.103166 punch from 0x004528 to 0x007f1d (0x39f6 bytes) 000797 1753365142.109004 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000798 1753365142.209105 mapread 0x003ab9 thru 0x007f1d (0x4465 bytes) 000799 1753365142.219698 write 0x00bf06 thru 0x012682 (0x677d bytes) 000800 1753365142.223685 punch from 0x00a26c to 0x012682 (0x8417 bytes) 000801 1753365142.234979 MIRROR_EXTEND to 3 mirrors 000802 1753365142.439558 trunc from 0x00a3ab to 0x00f8c8 (0x551e bytes) 000804 1753365142.456847 punch from 0x00b787 to 0x01203e (0x68b8 bytes) 000806 1753365142.461506 write 0x01b7e8 thru 0x025eaf (0xa6c8 bytes) 000807 1753365142.465194 trunc from 0x025eb0 to 0x023e24 (0x208b bytes) 000808 1753365142.481604 trunc from 0x023e25 to 0x038934 (0x14b10 bytes) 000809 1753365142.488450 read 0x014cf7 thru 0x015cce (0xfd8 bytes) 000810 1753365142.491961 punch from 0x03416f to 0x035039 (0xecb bytes) 000813 1753365142.495478 mapwrite 0x02de8f thru 0x03746f (0x95e1 bytes) 000814 1753365142.522401 trunc from 0x0069f0 to 0x00c130 (0x5741 bytes) 000815 1753365142.527049 fallocate 0x007049 thru 0x011e1b (0xadd3 bytes) 000816 1753365142.530450 trunc from 0x038935 to 0x0276e2 (0x11252 bytes) 000818 1753365142.540449 write 0x014b4c thru 0x01bf74 (0x7429 bytes) 000819 1753365142.545154 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000820 1753365142.641245 mapwrite 0x0152d8 thru 0x01f803 (0xa52c bytes) 000821 1753365142.663299 read 0x023c43 thru 0x0276e2 (0x3aa0 bytes) 000822 1753365142.668252 trunc from 0x020e87 to 0x0276e2 (0x685c bytes) 000823 1753365142.673583 write 0x00518f thru 0x013a5c (0xe8ce bytes) 000824 1753365142.685721 punch from 0x004b3a to 0x00ddfc (0x92c3 bytes) 000825 1753365142.694335 mapwrite 0x006c0d thru 0x00ffbe (0x93b2 bytes) 000826 1753365142.710281 trunc from 0x0276e3 to 0x01fef5 (0x77ed bytes) 000827 1753365142.720718 mapwrite 0x00a62f thru 0x015579 (0xaf4b bytes) 000828 1753365142.734845 trunc from 0x01fef6 to 0x038f25 (0x19030 bytes) 000829 1753365142.743115 read 0x017422 thru 0x01fb18 (0x86f7 bytes) 000830 1753365142.743843 mapwrite 0x0102b7 thru 0x0166b9 (0x6403 bytes) 000832 1753365142.754936 trunc from 0x0014ce to 0x001fd9 (0xb0c bytes) 000833 1753365142.757754 mapread 0x02a753 thru 0x038077 (0xd925 bytes) 000834 1753365142.762686 read 0x025e04 thru 0x0301a2 (0xa39f bytes) 000835 1753365142.763271 mapwrite 0x0045b4 thru 0x00cce0 (0x872d bytes) 000836 1753365142.774978 trunc from 0x038f26 to 0x004b5f (0x343c6 bytes) 000837 1753365142.783452 trunc from 0x004b60 to 0x00abf5 (0x6096 bytes) 000838 1753365142.789197 trunc from 0x00abf6 to 0x029282 (0x1e68d bytes) 000839 1753365142.795309 fallocate 0x02939d thru 0x0321ed (0x8e51 bytes) 000840 1753365142.798280 read 0x015f03 thru 0x01792a (0x1a28 bytes) 000841 1753365142.803244 trunc from 0x029283 to 0x02f30d (0x608b bytes) 000842 1753365142.811317 trunc from 0x02f30e to 0x0105f3 (0x1ed1a bytes) 000843 1753365142.818659 trunc from 0x001b37 to 0x00d75a (0xbc24 bytes) 000844 1753365142.821999 mapread 0x00d713 thru 0x00def3 (0x7e1 bytes) 000845 1753365142.825244 trunc from 0x00fa52 to 0x0105f3 (0xba2 bytes) 000846 1753365142.830329 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000848 1753365142.893731 write 0x036b6f thru 0x03785f (0xcf1 bytes) 000849 1753365142.896778 write 0x03b628 thru 0x03ffff (0x49d8 bytes) 000850 1753365142.897690 mapwrite 0x01c0dc thru 0x028c01 (0xcb26 bytes) 000851 1753365142.918959 trunc from 0x032207 to 0x032e24 (0xc1e bytes) 000852 1753365142.921846 mapread 0x024731 thru 0x02c430 (0x7d00 bytes) 000853 1753365142.922679 mapwrite 0x0275ff thru 0x028e0f (0x1811 bytes) 000854 1753365142.930988 trunc from 0x040000 to 0x0213f1 (0x1ec0e bytes) 000855 1753365142.939351 mapwrite 0x02bb85 thru 0x036dba (0xb236 bytes) 000857 1753365142.967825 read 0x0027e6 thru 0x007dfd (0x5618 bytes) 000858 1753365142.972616 read 0x00600e thru 0x015520 (0xf513 bytes) 000860 1753365142.981690 MIRROR_EXTEND to 2 mirrors 000861 1753365143.226200 read 0x001824 thru 0x003801 (0x1fde bytes) 000862 1753365143.233676 mapwrite 0x00f434 thru 0x01c7b8 (0xd385 bytes) 000863 1753365143.296608 punch from 0x00255b to 0x010e35 (0xe8db bytes) 000864 1753365143.300022 mapread 0x008982 thru 0x00e2e9 (0x5968 bytes) 000865 1753365143.305243 trunc from 0x036dbb to 0x02b797 (0xb623 bytes) 000866 1753365143.319137 trunc from 0x02a9d4 to 0x02b797 (0xdc4 bytes) 000867 1753365143.327077 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000868 1753365143.390918 mapread 0x002530 thru 0x00c704 (0xa1d5 bytes) 000869 1753365143.399554 fallocate 0x002216 thru 0x00403c (0x1e27 bytes) 000870 1753365143.403013 mapwrite 0x0369ab thru 0x03d95d (0x6fb3 bytes) 000871 1753365143.430728 mapread 0x0214e0 thru 0x02aca3 (0x97c4 bytes) 000872 1753365143.439849 trunc from 0x001ef5 to 0x00d487 (0xb593 bytes) 000873 1753365143.444906 trunc from 0x039fa3 to 0x03d95d (0x39bb bytes) 000874 1753365143.448291 mapread 0x00fde2 thru 0x01e933 (0xeb52 bytes) 000875 1753365143.455362 trunc from 0x0244a3 to 0x028191 (0x3cef bytes) 000876 1753365143.459336 fallocate 0x01c2a7 thru 0x01dcbd (0x1a17 bytes) 000877 1753365143.462141 read 0x008413 thru 0x00f67d (0x726b bytes) 000879 1753365143.465474 punch from 0x0030a9 to 0x009a25 (0x697d bytes) 000880 1753365143.468343 fallocate 0x0109be thru 0x012de2 (0x2425 bytes) 000881 1753365143.471117 mapread 0x00fa54 thru 0x01221b (0x27c8 bytes) 000882 1753365143.471865 fallocate 0x0173b0 thru 0x026b5b (0xf7ac bytes) 000883 1753365143.475559 trunc from 0x03d95e to 0x026622 (0x1733b bytes) 000884 1753365143.485175 trunc from 0x020e6b to 0x026622 (0x57b8 bytes) 000885 1753365143.488209 punch from 0x009489 to 0x00fe35 (0x69ad bytes) 000887 1753365143.493113 fallocate 0x0218f7 thru 0x026865 (0x4f6f bytes) 000888 1753365143.496250 write 0x01e09d thru 0x02df6b (0xfecf bytes) 000889 1753365143.497656 punch from 0x02a7cf to 0x02df6b (0x379d bytes) 000890 1753365143.506884 punch from 0x0071b4 to 0x009549 (0x2396 bytes) 000891 1753365143.509599 trunc from 0x02df6c to 0x028c59 (0x5312 bytes) 000892 1753365143.516956 mapwrite 0x0222db thru 0x027785 (0x54ab bytes) 000893 1753365143.526865 read 0x001d7d thru 0x00b4fd (0x9781 bytes) 000895 1753365143.531331 read 0x00c078 thru 0x00d4a4 (0x142d bytes) 000897 1753365143.534036 MIRROR_EXTEND to 2 mirrors 000899 1753365143.781017 mapread 0x02578b thru 0x026512 (0xd88 bytes) 000900 1753365143.802473 mapwrite 0x022f9f thru 0x02add4 (0x7e36 bytes) 000901 1753365143.877898 MIRROR_RESYNC 2 mirrors 000902 1753365144.025722 trunc from 0x0269fc to 0x02add4 (0x43d9 bytes) 000904 1753365144.045620 mapwrite 0x0236e1 thru 0x02c110 (0x8a30 bytes) 000905 1753365144.086183 write 0x037921 thru 0x03ffff (0x86df bytes) 000907 1753365144.088059 punch from 0x001594 to 0x001c34 (0x6a1 bytes) 000908 1753365144.092473 punch from 0x004452 to 0x00cb88 (0x8737 bytes) 000909 1753365144.095489 trunc from 0x040000 to 0x029bcb (0x16434 bytes) 000910 1753365144.108949 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000912 1753365144.181995 fallocate 0x032689 thru 0x035a6e (0x33e6 bytes) 000913 1753365144.185431 read 0x02745d thru 0x032ed8 (0xba7c bytes) 000914 1753365144.189340 read 0x020e58 thru 0x0277dc (0x6985 bytes) 000915 1753365144.194281 trunc from 0x035a6f to 0x02f752 (0x631c bytes) 000916 1753365144.203658 mapread 0x00d93b thru 0x01270b (0x4dd1 bytes) 000917 1753365144.204311 mapread 0x0277c3 thru 0x027da6 (0x5e4 bytes) 000918 1753365144.204885 mapwrite 0x03f579 thru 0x03ffff (0xa87 bytes) 000920 1753365144.227096 trunc from 0x040000 to 0x001840 (0x3e7bf bytes) 000921 1753365144.235365 punch from 0x001219 to 0x001840 (0x628 bytes) 000923 1753365144.243171 MIRROR_EXTEND to 2 mirrors 000925 1753365144.456783 trunc from 0x000417 to 0x001840 (0x142a bytes) 000926 1753365144.472162 write 0x02d8bf thru 0x02f5a9 (0x1ceb bytes) 000929 1753365144.479987 fallocate 0x02a72d thru 0x03a493 (0xfd67 bytes) 000930 1753365144.483638 read 0x02c840 thru 0x02e0e2 (0x18a3 bytes) 000931 1753365144.487150 fallocate 0x01e0e0 thru 0x022afe (0x4a1f bytes) 000933 1753365144.491423 punch from 0x01284f to 0x019b40 (0x72f2 bytes) 000934 1753365144.498110 MIRROR_RESYNC 2 mirrors 000935 1753365144.652959 mapread 0x01ccbe thru 0x01f7ba (0x2afd bytes) 000938 1753365144.663820 write 0x02e45a thru 0x035ac7 (0x766e bytes) 000939 1753365144.683339 mapwrite 0x03e5cc thru 0x03ffff (0x1a34 bytes) 000940 1753365144.718480 punch from 0x034910 to 0x03ffff (0xb6f0 bytes) 000941 1753365144.721955 mapwrite 0x038801 thru 0x03ffff (0x77ff bytes) 000942 1753365144.735535 trunc from 0x040000 to 0x034bfc (0xb403 bytes) 000943 1753365144.743572 mapread 0x033571 thru 0x033673 (0x103 bytes) 000944 1753365144.744383 write 0x006cc6 thru 0x00c98e (0x5cc9 bytes) 000945 1753365144.749422 trunc from 0x0051db to 0x00fbfd (0xaa23 bytes) 000946 1753365144.755573 trunc from 0x034bfd to 0x03c530 (0x7934 bytes) 000947 1753365144.762761 trunc from 0x03c531 to 0x01a3a3 (0x2218d bytes) 000948 1753365144.772011 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000949 1753365144.832312 trunc from 0x01a3a4 to 0x027fba (0xdc17 bytes) 000950 1753365144.841570 write 0x0344c7 thru 0x03a9d6 (0x6510 bytes) 000951 1753365144.842554 trunc from 0x03a9d7 to 0x01760f (0x233c7 bytes) 000952 1753365144.853263 MIRROR_EXTEND to 2 mirrors 000953 1753365145.019658 read 0x0117d9 thru 0x012748 (0xf70 bytes) 000954 1753365145.026987 trunc from 0x017610 to 0x007e83 (0xf78c bytes) 000955 1753365145.061689 MIRROR_RESYNC 2 mirrors 000956 1753365145.194425 trunc from 0x007e84 to 0x02bfee (0x2416b bytes) 000957 1753365145.212874 trunc from 0x01a302 to 0x01aad2 (0x7d1 bytes) 000958 1753365145.218470 MIRROR_EXTEND to 3 mirrors 000959 1753365145.388343 trunc from 0x012b9d to 0x01dfad (0xb411 bytes) 000960 1753365145.402285 fallocate 0x02bc3d thru 0x037cc9 (0xc08d bytes) 000961 1753365145.405126 punch from 0x01bf3f to 0x025d5a (0x9e1c bytes) 000962 1753365145.407943 punch from 0x0112c5 to 0x01f6c2 (0xe3fe bytes) 000963 1753365145.410858 trunc from 0x02bfef to 0x00fd83 (0x1c26b bytes) 000964 1753365145.421335 fallocate 0x029c87 thru 0x037ca8 (0xe022 bytes) 000965 1753365145.426456 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000966 1753365145.514053 mapwrite 0x000fe4 thru 0x008b38 (0x7b55 bytes) 000967 1753365145.538796 mapread 0x0239f3 thru 0x02b31c (0x792a bytes) 000968 1753365145.543850 punch from 0x0100be to 0x01dc08 (0xdb4b bytes) 000969 1753365145.549071 read 0x011238 thru 0x01260c (0x13d5 bytes) 000970 1753365145.552471 trunc from 0x00fd57 to 0x015074 (0x531e bytes) 000971 1753365145.554998 mapread 0x00cbb9 thru 0x011de7 (0x522f bytes) 000972 1753365145.559398 punch from 0x01859b to 0x02436c (0xbdd2 bytes) 000973 1753365145.562672 write 0x0346c3 thru 0x03ba2a (0x7368 bytes) 000974 1753365145.569665 mapwrite 0x016fd5 thru 0x021fe1 (0xb00d bytes) 000975 1753365145.587201 mapwrite 0x02f872 thru 0x030eb5 (0x1644 bytes) 000976 1753365145.596178 write 0x00c56b thru 0x011998 (0x542e bytes) 000977 1753365145.597049 write 0x00055a thru 0x00292f (0x23d6 bytes) 000979 1753365145.598115 trunc from 0x010e13 to 0x012100 (0x12ee bytes) 000980 1753365145.608561 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000981 1753365145.662637 fallocate 0x01bf03 thru 0x02b175 (0xf273 bytes) 000982 1753365145.665446 trunc from 0x03ba2b to 0x02bfb5 (0xfa75 bytes) 000983 1753365145.675655 fallocate 0x02e528 thru 0x03aecc (0xc9a5 bytes) 000987 1753365145.682941 MIRROR_EXTEND to 2 mirrors 000988 1753365145.872932 trunc from 0x00663d to 0x009cba (0x367e bytes) 000989 1753365145.889008 mapwrite 0x01bb79 thru 0x01e862 (0x2cea bytes) 000990 1753365145.909242 read 0x01091d thru 0x01361c (0x2d00 bytes) 000991 1753365145.915571 MIRROR_EXTEND to 3 mirrors 000992 1753365146.164130 MIRROR_RESYNC 3 mirrors 000993 1753365146.398257 trunc from 0x03a944 to 0x03aecc (0x589 bytes) 000994 1753365146.415449 punch from 0x00b1b6 to 0x01893d (0xd788 bytes) 000995 1753365146.419490 read 0x00207c thru 0x009ac8 (0x7a4d bytes) 000996 1753365146.424301 fallocate 0x03ee48 thru 0x03ffff (0x11b8 bytes) 000997 1753365146.430319 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x372:0x0]: Invalid argument (22) 000998 1753365146.505812 write 0x02150e thru 0x022612 (0x1105 bytes) 000999 1753365146.511330 trunc from 0x040000 to 0x01715c (0x28ea3 bytes) 001000 1753365146.517747 mapread 0x012156 thru 0x01715c (0x5007 bytes) All operations completed A-OK! PASS 70a (30s) == sanity-flr test 71: check mirror extend parent fid ==== 09:52:28 (1753365148) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x53:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 200a: stress test ===================== 09:52:33 (1753365153) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3517792 .. Extending file size to 3871392 .. failed Extending file size to 4537760 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5053408 .. Extending file size to 7967616 .. Extending file size to 6771520 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7338464 .. Extending file size to 8182720 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8268928 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8369504 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8472480 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8737376 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9358752 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9012160 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60503 60504 60505 60508 60509 done 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:53:39 (1753365219) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3456864 .. Extending file size to 4411456 .. failed Extending file size to 7232352 .. Extending file size to 5885888 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7837536 .. Extending file size to 8273664 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8630528 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8810688 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8668416 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8971680 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8925088 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9008352 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9124448 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 64163 64164 64166 64168 64170 done 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:54:44 (1753365284) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2540 1285148 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21580 3585440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15728 3591292 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37308 7176732 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2628 1285060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2540 1285148 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21580 3585440 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15728 3591292 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37308 7176732 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0464045 s, 45.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0866784 s, 24.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:54:51 (1753365291) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.039974 s, 52.5 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 09:54:54 (1753365294) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:54:57 (1753365297) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0356496 s, 58.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x516:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x514:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x515:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:55:00 (1753365300) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0389999 s, 53.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:55:03 (1753365303) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0360026 s, 58.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:55:28 (1753365328) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016874 s, 62.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x517:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x517:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x516:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x517:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x517:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x516:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x518:0x0] } PASS 204d (40s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:56:08 (1753365368) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0344844 s, 60.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:56:43 (1753365403) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg205-server: Pool lustre.test_204f created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0309643 s, 67.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg205-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:57:32 (1753365452) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:57:35 (1753365455) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:57:38 (1753365458) oleg205-server: Pool lustre.test_206 created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg205-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:57:51 (1753365471) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307309 s, 34.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132025 s, 79.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x9118c571 Mirror 3: 0x9118c571 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:57:55 (1753365475) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:57:56 (1753365476) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:57:57 (1753365477) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.014701 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1500000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1500000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1500000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2150000 Read this (4 bytes): BBBB 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:58:15 (1753365495) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00474528 s, 1.7 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:58:18 (1753365498) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176892 s, 59.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-24 09:58:18.000000000 -0400 Modify: 2025-07-24 09:58:18.000000000 -0400 Change: 2025-07-24 09:58:19.000000000 -0400 Birth: 2025-07-24 09:58:19.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:58:21 (1753365501) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148817 s, 70.5 MB/s oleg205-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg205-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg205-server: O/280000401/d26/ba: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:58:37 (1753365517) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0546242 s, 750 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0106761 s, 384 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 24 09:58 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x85:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 24 09:58 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 932 sec ============ 09:58:39 (1753365519) === sanity-flr: start cleanup 09:58:40 (1753365520) === === sanity-flr: finish cleanup 09:58:43 (1753365523) ===