-----============= acceptance-small: sanity-flr ============----- Wed May 28 17:49:40 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg322-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 17:49:45 (1748468985) === oleg322-client.virtnet: executing check_config_client /mnt/lustre oleg322-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg322-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5cc7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5cc7800.idle_timeout=debug disable quota as required oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 17:49:53 (1748468993) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7279] == sanity-flr test 0a: lfs mirror create with -N option == 17:49:53 (1748468993) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:49:57 (1748468997) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:49:57 (1748468997) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:49:58 (1748468998) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:50:03 (1748469003) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:50:04 (1748469004) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 17:50:05 (1748469005) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5cc7800.stats=clear osc.lustre-OST0001-osc-ffff8800b5cc7800.stats=clear PASS 0g (1s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:50:06 (1748469006) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:50:08 (1748469008) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:50:10 (1748469010) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:50:11 (1748469011) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 17:50:13 (1748469013) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:50:14 (1748469014) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:50:15 (1748469015) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:50:17 (1748469017) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:50:19 (1748469019) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:50:21 (1748469021) 14+0 records in 14+0 records out 14680064 bytes (15 MB) copied, 0.333648 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0383846 s, 27.3 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:50:23 (1748469023) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0222595 s, 141 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:50:25 (1748469025) Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5cc7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5cc7800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5cc7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5cc7800.ost_server_uuid in DISCONN state after 3 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (23s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:50:48 (1748469048) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0782344 s, 67.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5cc7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5cc7800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:50:58 (1748469058) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5cc7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5cc7800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5cc7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5cc7800.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:51:19 (1748469079) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.800709 s, 65.5 MB/s -rw-r--r-- 1 root root 52428800 May 28 17:51 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.210s user 0m0.000s sys 0m0.114s Stopping /mnt/lustre-ost1 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5cc7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5cc7800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.222s user 0m0.000s sys 0m0.116s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg322-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg322-server oleg322-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5cc7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5cc7800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.230s user 0m0.001s sys 0m0.117s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg322-server: oleg322-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg322-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5cc7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5cc7800.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:51:40 (1748469100) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:51:41 (1748469101) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:51:42 (1748469102) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 17:51:43 (1748469103) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 17:51:44 (1748469104) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.104108 s, 121 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:51:47 (1748469107) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00717119 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00657874 s, 159 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:51:48 (1748469108) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:51:50 (1748469110) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000751925 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000647059 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000661323 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000715401 s, 1.4 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 17:51:52 (1748469112) create test files with size 15733234 .. 8019b4703c0e263cebb2b5e3aca89091 - 070d10653d4b87b89ad63e142d6e9652 - d2b4b59378b3988bdb70257a759e9e3b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 17:51:57 (1748469117) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:52:00 (1748469120) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:52:01 (1748469121) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237743 s, 44.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00890534 s, 118 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 17:52:03 (1748469123) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0211855 s, 99.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0810192 s, 51.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0329789 s, 63.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 17:52:06 (1748469126) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 17:52:06 (1748469126) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:52:07 (1748469127) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00781092 s, 134 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 17:52:09 (1748469129) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (0s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:52:09 (1748469129) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:52:11 (1748469131) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0507508 s, 207 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (25s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:52:36 (1748469156) llite.lustre-ffff88012d672800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0675209 s, 155 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg322-client.virtnet /mnt/lustre (opts:) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a1149800.xattr_cache=1 PASS 44e (2s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:52:38 (1748469158) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (7s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:52:45 (1748469165) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:52:52 (1748469172) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 17:52:52 (1748469172) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0629327 s, 50.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0484245 s, 65.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 412cafa54e21ef9153575006321a225a - mirror 1 checksum: 959cece5877d543e00ed844512be4fc7 - mirror 2 checksum: 412cafa54e21ef9153575006321a225a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 28 17:52:54 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 28 17:52:54 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 412cafa54e21ef9153575006321a225a - mirror 1 checksum: 959cece5877d543e00ed844512be4fc7 - mirror 2 checksum: 959cece5877d543e00ed844512be4fc7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 28 17:52:54 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 28 17:52:54 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:52:55 (1748469175) Failing mds1 on oleg322-server Stopping /mnt/lustre-mds1 (opts:) on oleg322-server 17:52:57 (1748469177) shut down facet: mds1 facet_host: oleg322-server facet_failover_host: oleg322-server Failover mds1 to oleg322-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Started lustre-MDT0000 17:53:09 (1748469189) targets are mounted 17:53:09 (1748469189) facet_failover done oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00966388 s, 109 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:53:14 (1748469194) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:53:15 (1748469195) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (0s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:53:15 (1748469195) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00226391 s, 7.2 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:53:17 (1748469197) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:53:18 (1748469198) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (0s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:53:19 (1748469199) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:53:22 (1748469202) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:53:40 (1748469220) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:53:57 (1748469237) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:54:14 (1748469254) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 17:54:15 (1748469255) mirror create pid 22870 mirror split pid 22871 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22870 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22871 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 17:55:17 (1748469317) Seed set to 9317 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x32b5 000011 1748469317.512496 trunc from 00000000 to 0x0032b4 (0x32b5 bytes) truncating to largest ever: 0x25992 000012 1748469317.516676 trunc from 0x0032b5 to 0x025991 (0x226dd bytes) 000013 1748469317.520336 read 0x0048b5 thru 0x00d721 (0x8e6d bytes) 000014 1748469317.522872 read 0x009130 thru 0x01159f (0x8470 bytes) truncating to largest ever: 0x25e3f 000015 1748469317.525752 trunc from 0x025992 to 0x025e3e (0x4ad bytes) 000016 1748469317.529503 write 0x0319ff thru 0x03ad96 (0x9398 bytes) 000017 1748469317.530197 mapwrite 0x01e125 thru 0x027f2e (0x9e0a bytes) 000018 1748469317.560915 mapread 0x032538 thru 0x03ad96 (0x885f bytes) 000019 1748469317.561635 mapread 0x017e77 thru 0x021e0c (0x9f96 bytes) 000021 1748469317.562299 write 0x035622 thru 0x0357e6 (0x1c5 bytes) 000022 1748469317.562928 write 0x025f1f thru 0x029b79 (0x3c5b bytes) 000024 1748469317.563968 trunc from 0x03ad97 to 0x012e85 (0x27f11 bytes) 000025 1748469317.569580 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x319a6 000026 1748469317.656657 trunc from 0x012e86 to 0x0319a5 (0x1eb20 bytes) 000027 1748469317.673677 trunc from 0x0319a6 to 0x0248b1 (0xd0f4 bytes) 000028 1748469317.678125 mapwrite 0x003bc9 thru 0x011710 (0xdb48 bytes) 000029 1748469317.699010 mapread 0x01c4ab thru 0x0248b1 (0x8407 bytes) 000032 1748469317.703481 mapread 0x017528 thru 0x01aab2 (0x358b bytes) 000033 1748469317.705274 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000034 1748469317.734317 MIRROR_EXTEND to 2 mirrors 000038 1748469317.828014 write 0x00fdf2 thru 0x018125 (0x8334 bytes) 000040 1748469317.844408 MIRROR_EXTEND to 3 mirrors 000043 1748469317.936305 mapread 0x01ee23 thru 0x0248b1 (0x5a8f bytes) 000044 1748469317.942029 write 0x0065ab thru 0x0116d3 (0xb129 bytes) 000045 1748469317.957599 mapwrite 0x01fbdb thru 0x01fcc6 (0xec bytes) 000046 1748469317.973429 write 0x0179cb thru 0x01cbf7 (0x522d bytes) 000047 1748469317.976035 mapread 0x01a822 thru 0x0248b1 (0xa090 bytes) 000048 1748469317.976622 mapread 0x0082f5 thru 0x013bf4 (0xb900 bytes) 000050 1748469317.977398 write 0x01c65c thru 0x02165b (0x5000 bytes) 000051 1748469317.978074 write 0x02c966 thru 0x03b21d (0xe8b8 bytes) 000052 1748469317.979057 trunc from 0x03b21e to 0x000270 (0x3afad bytes) 000053 1748469317.988542 MIRROR_EXTEND to 4 mirrors 000055 1748469318.083210 mapwrite 0x022136 thru 0x02249f (0x36a bytes) 000056 1748469318.113043 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000058 1748469318.161615 trunc from 0x0224a0 to 0x01bff5 (0x64aa bytes) 000059 1748469318.165072 write 0x00ff70 thru 0x013f01 (0x3f92 bytes) 000060 1748469318.168687 MIRROR_EXTEND to 4 mirrors 000062 1748469318.274024 mapwrite 0x002d9a thru 0x00687a (0x3ae1 bytes) 000063 1748469318.319888 mapwrite 0x003483 thru 0x00671d (0x329b bytes) 000066 1748469318.334006 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000069 1748469318.380766 read 0x015f2d thru 0x01b445 (0x5519 bytes) 000071 1748469318.383667 mapread 0x017693 thru 0x01bff5 (0x4963 bytes) 000072 1748469318.384184 trunc from 0x01bff6 to 0x00b8f5 (0x10700 bytes) 000073 1748469318.387531 write 0x0161c3 thru 0x0243e0 (0xe21e bytes) 000075 1748469318.388577 mapread 0x00b524 thru 0x00c886 (0x1363 bytes) 000077 1748469318.391457 mapread 0x021c31 thru 0x0243e0 (0x27b0 bytes) 000078 1748469318.391831 mapread 0x0169c7 thru 0x023044 (0xc67e bytes) 000080 1748469318.392751 write 0x0004df thru 0x00d872 (0xd394 bytes) 000081 1748469318.395557 mapread 0x011e98 thru 0x018f67 (0x70d0 bytes) 000085 1748469318.400026 MIRROR_EXTEND to 4 mirrors 000086 1748469318.502647 read 0x00e143 thru 0x014334 (0x61f2 bytes) 000087 1748469318.506694 write 0x029cbc thru 0x034206 (0xa54b bytes) 000088 1748469318.520707 mapread 0x01d50b thru 0x02897e (0xb474 bytes) 000089 1748469318.526316 write 0x032670 thru 0x03d289 (0xac1a bytes) 000090 1748469318.528373 MIRROR_RESYNC 4 mirrors 000091 1748469318.660772 read 0x02bb80 thru 0x036a38 (0xaeb9 bytes) 000095 1748469318.663968 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000098 1748469318.718554 read 0x03c346 thru 0x03d289 (0xf44 bytes) 000099 1748469318.720041 mapread 0x036c9b thru 0x039df0 (0x3156 bytes) 000103 1748469318.724626 write 0x017abc thru 0x023bde (0xc123 bytes) 000107 1748469318.740874 write 0x030f65 thru 0x03c5a2 (0xb63e bytes) 000108 1748469318.744511 mapread 0x031570 thru 0x038168 (0x6bf9 bytes) 000109 1748469318.744996 mapread 0x00ef80 thru 0x01c0eb (0xd16c bytes) 000110 1748469318.747197 mapwrite 0x038d3f thru 0x03ffff (0x72c1 bytes) 000111 1748469318.770654 trunc from 0x040000 to 0x02de9e (0x12161 bytes) 000112 1748469318.774284 read 0x008a94 thru 0x00a108 (0x1675 bytes) 000113 1748469318.774619 write 0x03c3b7 thru 0x03ffff (0x3c49 bytes) 000114 1748469318.775210 mapread 0x002676 thru 0x0055d7 (0x2f62 bytes) 000120 1748469318.780168 write 0x01ff0c thru 0x02770e (0x7803 bytes) 000121 1748469318.781062 mapwrite 0x030bff thru 0x03fce1 (0xf0e3 bytes) 000122 1748469318.796650 mapwrite 0x007312 thru 0x014c9c (0xd98b bytes) 000123 1748469318.810216 mapwrite 0x008242 thru 0x009ed2 (0x1c91 bytes) 000126 1748469318.828036 mapwrite 0x03e0fe thru 0x03ffff (0x1f02 bytes) 000127 1748469318.839082 mapwrite 0x03d255 thru 0x03ffff (0x2dab bytes) 000129 1748469318.850958 trunc from 0x040000 to 0x02f402 (0x10bfd bytes) truncating to largest ever: 0x3c9d0 000130 1748469318.854566 trunc from 0x02f403 to 0x03c9cf (0xd5cd bytes) 000131 1748469318.857432 mapread 0x007d29 thru 0x00ba33 (0x3d0b bytes) 000132 1748469318.857840 mapread 0x027c7a thru 0x02cf9d (0x5324 bytes) 000134 1748469318.858483 read 0x00c6fc thru 0x00e401 (0x1d06 bytes) 000135 1748469318.859717 MIRROR_EXTEND to 4 mirrors truncating to largest ever: 0x3e16f 000136 1748469318.946772 trunc from 0x03c9d0 to 0x03e16e (0x179f bytes) 000139 1748469318.967072 write 0x0338d1 thru 0x03de9f (0xa5cf bytes) 000140 1748469318.969580 write 0x013ad4 thru 0x019c3d (0x616a bytes) 000142 1748469318.973119 MIRROR_RESYNC 4 mirrors 000144 1748469319.105582 trunc from 0x03e16f to 0x030ca0 (0xd4ce bytes) 000148 1748469319.121248 write 0x014de4 thru 0x023b85 (0xeda2 bytes) 000151 1748469319.124402 read 0x003d33 thru 0x00f21d (0xb4eb bytes) 000152 1748469319.127788 MIRROR_EXTEND to 5 mirrors 000153 1748469319.228151 read 0x012b41 thru 0x01961c (0x6adc bytes) 000156 1748469319.233203 write 0x03cf50 thru 0x03ffff (0x30b0 bytes) 000157 1748469319.248652 mapwrite 0x03b860 thru 0x03ffff (0x47a0 bytes) 000158 1748469319.264455 MIRROR_EXTEND to 6 mirrors 000159 1748469319.359630 mapwrite 0x00dcf6 thru 0x016173 (0x847e bytes) 000161 1748469319.405373 MIRROR_RESYNC 6 mirrors 000163 1748469319.563622 write 0x010956 thru 0x01549a (0x4b45 bytes) 000164 1748469319.576954 write 0x02e961 thru 0x037205 (0x88a5 bytes) 000166 1748469319.580976 read 0x007428 thru 0x00fafc (0x86d5 bytes) 000168 1748469319.583517 write 0x02cbbb thru 0x03aef0 (0xe336 bytes) truncating to largest ever: 0x3e77e 000172 1748469319.586431 trunc from 0x040000 to 0x03e77d (0x1882 bytes) 000173 1748469319.595445 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000175 1748469319.642196 mapwrite 0x017054 thru 0x01a2db (0x3288 bytes) 000176 1748469319.657283 mapread 0x00d8e6 thru 0x00f714 (0x1e2f bytes) 000177 1748469319.659500 trunc from 0x03e77e to 0x01d5fc (0x21181 bytes) 000178 1748469319.663100 mapwrite 0x02fb06 thru 0x03c366 (0xc861 bytes) 000181 1748469319.684073 mapwrite 0x032bea thru 0x03d22c (0xa643 bytes) 000182 1748469319.704016 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000183 1748469319.746741 trunc from 0x03d22d to 0x00e8ae (0x2e97e bytes) 000184 1748469319.750147 read 0x001819 thru 0x006872 (0x505a bytes) 000187 1748469319.752925 mapwrite 0x03aefa thru 0x03ffff (0x5106 bytes) 000188 1748469319.774813 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000189 1748469319.816437 mapwrite 0x0214b3 thru 0x02cbad (0xb6fb bytes) 000191 1748469319.835938 write 0x008c9a thru 0x01713c (0xe4a3 bytes) 000192 1748469319.844119 MIRROR_RESYNC 3 mirrors 000193 1748469319.952055 trunc from 0x040000 to 0x03c015 (0x3fea bytes) 000194 1748469319.968444 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000195 1748469320.011549 trunc from 0x03c016 to 0x037635 (0x49e0 bytes) 000197 1748469320.014968 write 0x022f61 thru 0x024f1d (0x1fbd bytes) 000200 1748469320.017845 mapwrite 0x02dd8b thru 0x02f0d4 (0x134a bytes) 000201 1748469320.034468 write 0x011901 thru 0x01c5ff (0xacff bytes) 000202 1748469320.038909 write 0x015a9d thru 0x019559 (0x3abd bytes) 000205 1748469320.041122 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000207 1748469320.076070 mapread 0x00dadc thru 0x00faa6 (0x1fcb bytes) 000208 1748469320.080648 mapwrite 0x002413 thru 0x00a5fa (0x81e8 bytes) 000212 1748469320.095405 read 0x0278d7 thru 0x0284b9 (0xbe3 bytes) 000213 1748469320.096873 mapwrite 0x03491b thru 0x03ffff (0xb6e5 bytes) 000216 1748469320.119775 trunc from 0x040000 to 0x02aed7 (0x15128 bytes) 000217 1748469320.125327 MIRROR_EXTEND to 2 mirrors 000219 1748469320.220931 trunc from 0x02aed8 to 0x03a802 (0xf92b bytes) 000221 1748469320.240945 trunc from 0x03a803 to 0x006898 (0x33f6a bytes) 000222 1748469320.245808 read 0x005272 thru 0x006898 (0x1627 bytes) 000224 1748469320.247769 read 0x004272 thru 0x006898 (0x2627 bytes) 000225 1748469320.249745 mapread 0x005036 thru 0x006898 (0x1863 bytes) 000227 1748469320.250495 read 0x000469 thru 0x004f3c (0x4ad4 bytes) 000229 1748469320.251140 write 0x011b04 thru 0x01f7e9 (0xdce6 bytes) 000231 1748469320.252245 mapwrite 0x039744 thru 0x03ffff (0x68bc bytes) 000232 1748469320.273207 mapread 0x00a373 thru 0x0129d1 (0x865f bytes) 000233 1748469320.273736 mapwrite 0x022c08 thru 0x0294ce (0x68c7 bytes) 000234 1748469320.285809 mapwrite 0x0268be thru 0x02be8e (0x55d1 bytes) 000237 1748469320.298628 mapread 0x00d94a thru 0x010628 (0x2cdf bytes) 000238 1748469320.299075 mapwrite 0x00b283 thru 0x00d5f1 (0x236f bytes) 000241 1748469320.310766 read 0x02423f thru 0x02ba0c (0x77ce bytes) 000242 1748469320.312454 MIRROR_EXTEND to 3 mirrors 000244 1748469320.410685 MIRROR_EXTEND to 4 mirrors 000245 1748469320.523475 mapwrite 0x02c385 thru 0x031323 (0x4f9f bytes) 000248 1748469320.568601 mapread 0x026a41 thru 0x02f652 (0x8c12 bytes) 000249 1748469320.570669 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000250 1748469320.616790 trunc from 0x040000 to 0x00e25e (0x31da1 bytes) 000252 1748469320.625569 mapread 0x004796 thru 0x00e25e (0x9ac9 bytes) 000253 1748469320.633143 write 0x0269d2 thru 0x02c76a (0x5d99 bytes) 000256 1748469320.634631 read 0x004526 thru 0x00b11e (0x6bf9 bytes) 000257 1748469320.634930 mapread 0x006ff1 thru 0x008e9f (0x1eaf bytes) 000258 1748469320.635384 read 0x01b440 thru 0x026d0c (0xb8cd bytes) 000259 1748469320.638304 trunc from 0x02c76b to 0x01e0a1 (0xe6c9 bytes) 000260 1748469320.642756 mapwrite 0x0088b8 thru 0x013829 (0xaf72 bytes) 000262 1748469320.657478 write 0x0342e3 thru 0x039eac (0x5bca bytes) 000264 1748469320.658369 mapread 0x031c7c thru 0x039dd6 (0x815b bytes) 000266 1748469320.661499 write 0x01e550 thru 0x02e2ae (0xfd5f bytes) 000267 1748469320.662478 mapread 0x0335e1 thru 0x038671 (0x5091 bytes) 000268 1748469320.664481 MIRROR_RESYNC 3 mirrors 000275 1748469320.789852 mapread 0x0241c3 thru 0x033f21 (0xfd5f bytes) 000279 1748469320.796453 read 0x027275 thru 0x02b138 (0x3ec4 bytes) 000280 1748469320.796626 mapread 0x0143cb thru 0x0222f9 (0xdf2f bytes) 000283 1748469320.801251 mapread 0x023274 thru 0x02bf84 (0x8d11 bytes) 000285 1748469320.801614 mapread 0x010ac0 thru 0x01e411 (0xd952 bytes) 000286 1748469320.804801 MIRROR_RESYNC 3 mirrors 000287 1748469320.846774 mapread 0x0381c9 thru 0x039eac (0x1ce4 bytes) 000290 1748469320.847516 read 0x021cbf thru 0x027fd0 (0x6312 bytes) 000292 1748469320.847953 write 0x01a5fe thru 0x020fc0 (0x69c3 bytes) 000294 1748469320.863594 trunc from 0x039ead to 0x01a33c (0x1fb70 bytes) 000295 1748469320.872618 mapread 0x00b40b thru 0x0118a6 (0x649c bytes) 000296 1748469320.875422 mapread 0x014245 thru 0x01a063 (0x5e1f bytes) 000299 1748469320.876340 mapread 0x00aa39 thru 0x01297a (0x7f42 bytes) 000303 1748469320.878798 MIRROR_EXTEND to 4 mirrors 000306 1748469320.974471 read 0x00f01f thru 0x014822 (0x5804 bytes) 000307 1748469320.977430 write 0x03c418 thru 0x03fa00 (0x35e9 bytes) 000309 1748469320.993116 MIRROR_RESYNC 4 mirrors 000310 1748469321.128383 read 0x032515 thru 0x0375f2 (0x50de bytes) 000311 1748469321.130712 write 0x032bf5 thru 0x03ffff (0xd40b bytes) 000313 1748469321.148128 trunc from 0x040000 to 0x00ea7f (0x31580 bytes) 000315 1748469321.152885 write 0x022cab thru 0x02af6f (0x82c5 bytes) 000319 1748469321.155614 MIRROR_EXTEND to 5 mirrors 000320 1748469321.279508 write 0x00f961 thru 0x01712d (0x77cd bytes) 000321 1748469321.296700 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000324 1748469321.350049 write 0x0325ac thru 0x03ee72 (0xc8c7 bytes) 000325 1748469321.351031 trunc from 0x03ee73 to 0x03924e (0x5c24 bytes) 000326 1748469321.356161 MIRROR_RESYNC 4 mirrors 000329 1748469321.507152 mapread 0x0300ca thru 0x03924e (0x9185 bytes) 000331 1748469321.511848 trunc from 0x03924f to 0x0210eb (0x18163 bytes) 000333 1748469321.527795 write 0x03d41d thru 0x03ffff (0x2be3 bytes) 000335 1748469321.530091 MIRROR_RESYNC 4 mirrors 000337 1748469321.672912 read 0x01bd1f thru 0x01dd0f (0x1ff1 bytes) 000338 1748469321.674523 write 0x0093be thru 0x00e536 (0x5179 bytes) 000340 1748469321.693195 MIRROR_EXTEND to 5 mirrors 000342 1748469321.811397 MIRROR_EXTEND to 6 mirrors 000343 1748469321.915707 mapread 0x033e44 thru 0x035612 (0x17cf bytes) 000344 1748469321.920198 read 0x027ecf thru 0x02d7cf (0x5901 bytes) 000346 1748469321.923152 MIRROR_RESYNC 6 mirrors 000348 1748469322.065526 write 0x01056d thru 0x012c41 (0x26d5 bytes) 000351 1748469322.082362 mapread 0x021a79 thru 0x0318d7 (0xfe5f bytes) 000353 1748469322.087610 trunc from 0x040000 to 0x00bfd7 (0x34028 bytes) 000357 1748469322.094561 mapread 0x00810a thru 0x00bfd7 (0x3ece bytes) 000358 1748469322.097022 mapread 0x008027 thru 0x00bfd7 (0x3fb1 bytes) 000360 1748469322.097671 write 0x038a79 thru 0x03d4fe (0x4a86 bytes) 000363 1748469322.099114 mapwrite 0x00773a thru 0x00c474 (0x4d3b bytes) 000365 1748469322.115980 mapwrite 0x02e84f thru 0x03c8b5 (0xe067 bytes) 000366 1748469322.133780 read 0x030676 thru 0x03d4fe (0xce89 bytes) 000367 1748469322.134218 read 0x017cb7 thru 0x018aa1 (0xdeb bytes) 000368 1748469322.137469 trunc from 0x03d4ff to 0x02fe0c (0xd6f2 bytes) 000370 1748469322.141770 trunc from 0x02fe0d to 0x00ad20 (0x250ec bytes) 000371 1748469322.146131 MIRROR_RESYNC 6 mirrors 000372 1748469322.301857 write 0x0004cf thru 0x00913b (0x8c6d bytes) 000374 1748469322.318665 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000375 1748469322.367884 mapwrite 0x02d235 thru 0x038f87 (0xbd53 bytes) 000376 1748469322.397035 read 0x00b1af thru 0x017b41 (0xc993 bytes) 000380 1748469322.400800 mapwrite 0x02872a thru 0x033f1b (0xb7f2 bytes) 000381 1748469322.415030 mapwrite 0x039b8b thru 0x03ffff (0x6475 bytes) 000382 1748469322.435883 mapread 0x002ad6 thru 0x00a019 (0x7544 bytes) 000384 1748469322.440257 mapwrite 0x024467 thru 0x02fac0 (0xb65a bytes) 000385 1748469322.457817 MIRROR_EXTEND to 6 mirrors 000386 1748469322.566103 mapread 0x02314e thru 0x023488 (0x33b bytes) 000387 1748469322.569846 mapwrite 0x02308c thru 0x02ff6c (0xcee1 bytes) 000388 1748469322.616141 write 0x037d89 thru 0x037e90 (0x108 bytes) 000389 1748469322.618776 trunc from 0x040000 to 0x026fdc (0x19023 bytes) 000392 1748469322.627869 write 0x03db53 thru 0x03ffff (0x24ad bytes) 000395 1748469322.629187 read 0x01d1e6 thru 0x023018 (0x5e33 bytes) 000396 1748469322.629590 trunc from 0x040000 to 0x032848 (0xd7b7 bytes) 000400 1748469322.633793 write 0x030231 thru 0x03866c (0x843c bytes) 000402 1748469322.636890 read 0x0002bc thru 0x004c8e (0x49d3 bytes) 000403 1748469322.639804 mapread 0x00b852 thru 0x01b265 (0xfa14 bytes) 000404 1748469322.641812 MIRROR_RESYNC 6 mirrors 000405 1748469322.824792 mapwrite 0x00acde thru 0x00c646 (0x1969 bytes) 000406 1748469322.870210 mapread 0x02b6d1 thru 0x0368c7 (0xb1f7 bytes) 000408 1748469322.875018 mapwrite 0x00674c thru 0x00d1a8 (0x6a5d bytes) 000409 1748469322.887779 mapwrite 0x03aeea thru 0x03ffff (0x5116 bytes) 000410 1748469322.912561 read 0x021da1 thru 0x030a98 (0xecf8 bytes) 000411 1748469322.916143 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000412 1748469322.964680 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000416 1748469323.012697 read 0x03f8d3 thru 0x03ffff (0x72d bytes) 000422 1748469323.015779 mapwrite 0x0312d2 thru 0x03b2fe (0xa02d bytes) 000423 1748469323.034682 read 0x01b551 thru 0x021a31 (0x64e1 bytes) 000424 1748469323.038744 mapread 0x01cc0f thru 0x02337f (0x6771 bytes) 000425 1748469323.040669 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000426 1748469323.086879 write 0x0079fb thru 0x00b299 (0x389f bytes) 000427 1748469323.089852 trunc from 0x040000 to 0x0091ca (0x36e35 bytes) 000428 1748469323.094618 read 0x006917 thru 0x0091ca (0x28b4 bytes) 000430 1748469323.096914 mapread 0x0038d6 thru 0x0056fa (0x1e25 bytes) 000431 1748469323.099673 mapread 0x0046c0 thru 0x005455 (0xd96 bytes) 000432 1748469323.101403 MIRROR_EXTEND to 4 mirrors 000433 1748469323.193142 mapread 0x0029fc thru 0x0036cc (0xcd1 bytes) 000434 1748469323.197820 write 0x0152ed thru 0x01e8ba (0x95ce bytes) 000436 1748469323.213526 MIRROR_RESYNC 4 mirrors 000437 1748469323.335524 trunc from 0x01e8bb to 0x007b1b (0x16d9f bytes) 000438 1748469323.352642 read 0x007156 thru 0x007b1b (0x9c6 bytes) 000439 1748469323.354165 write 0x00c5a8 thru 0x01275d (0x61b6 bytes) 000440 1748469323.354736 trunc from 0x01275e to 0x03b889 (0x2912c bytes) 000442 1748469323.359579 mapwrite 0x0341f9 thru 0x03cb5e (0x8966 bytes) 000443 1748469323.380330 trunc from 0x03cb5f to 0x03c4e0 (0x67e bytes) 000444 1748469323.383855 read 0x00fb80 thru 0x011336 (0x17b7 bytes) 000446 1748469323.384554 write 0x027e74 thru 0x0301cf (0x835c bytes) 000448 1748469323.388355 read 0x015411 thru 0x01fee5 (0xaad5 bytes) 000450 1748469323.389013 mapwrite 0x034d31 thru 0x03c961 (0x7c31 bytes) 000451 1748469323.405511 read 0x02fcef thru 0x03b9fa (0xbd0c bytes) 000453 1748469323.406053 trunc from 0x03c962 to 0x00688f (0x360d2 bytes) 000454 1748469323.409902 mapwrite 0x021927 thru 0x0276be (0x5d98 bytes) 000455 1748469323.430203 mapread 0x01e78a thru 0x026fdd (0x8854 bytes) 000457 1748469323.430956 write 0x02d275 thru 0x03a438 (0xd1c4 bytes) 000458 1748469323.431947 read 0x033a6f thru 0x03a438 (0x69ca bytes) 000459 1748469323.432326 read 0x00c04a thru 0x00db96 (0x1b4d bytes) 000460 1748469323.433948 MIRROR_RESYNC 4 mirrors 000463 1748469323.563627 trunc from 0x03a439 to 0x01be91 (0x1e5a7 bytes) 000467 1748469323.580327 mapwrite 0x03a6c1 thru 0x03ffff (0x593f bytes) 000468 1748469323.602130 mapread 0x03da29 thru 0x03ffff (0x25d7 bytes) 000469 1748469323.602609 write 0x022341 thru 0x0316f6 (0xf3b6 bytes) 000470 1748469323.606999 mapwrite 0x036a31 thru 0x03ffff (0x95cf bytes) 000472 1748469323.622674 MIRROR_EXTEND to 5 mirrors 000473 1748469323.723328 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000474 1748469323.791051 MIRROR_RESYNC 4 mirrors 000475 1748469323.908194 read 0x03b7ab thru 0x03f5b8 (0x3e0e bytes) 000479 1748469323.910525 read 0x03bf57 thru 0x03ffff (0x40a9 bytes) 000483 1748469323.910974 trunc from 0x040000 to 0x02c224 (0x13ddb bytes) 000485 1748469323.932040 MIRROR_RESYNC 4 mirrors 000486 1748469324.065276 read 0x027607 thru 0x02c224 (0x4c1e bytes) 000487 1748469324.068015 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000488 1748469324.125944 trunc from 0x02c225 to 0x001ffe (0x2a226 bytes) 000490 1748469324.142162 MIRROR_EXTEND to 4 mirrors 000491 1748469324.233593 write 0x0139d0 thru 0x017c92 (0x42c3 bytes) 000493 1748469324.248359 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000497 1748469324.303584 MIRROR_RESYNC 3 mirrors 000498 1748469324.416019 mapread 0x013a85 thru 0x017c92 (0x420e bytes) 000499 1748469324.421506 MIRROR_EXTEND to 4 mirrors 000501 1748469324.527324 trunc from 0x017c93 to 0x015467 (0x282b bytes) 000503 1748469324.542642 write 0x02e3cf thru 0x02e442 (0x74 bytes) 000504 1748469324.543324 trunc from 0x02e443 to 0x03b151 (0xcd0f bytes) 000505 1748469324.548260 write 0x014a6e thru 0x01c03a (0x75cd bytes) 000506 1748469324.551095 read 0x02b619 thru 0x039ed1 (0xe8b9 bytes) 000509 1748469324.556116 MIRROR_EXTEND to 5 mirrors 000511 1748469324.665960 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000512 1748469324.723395 write 0x017b48 thru 0x025232 (0xd6eb bytes) 000513 1748469324.738920 mapwrite 0x039617 thru 0x0399a6 (0x390 bytes) 000514 1748469324.753192 trunc from 0x03b152 to 0x004451 (0x36d00 bytes) 000517 1748469324.761861 write 0x02c04d thru 0x0392b7 (0xd26b bytes) 000521 1748469324.763450 read 0x00647b thru 0x00844a (0x1fd0 bytes) 000523 1748469324.766350 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000524 1748469324.812592 mapwrite 0x00ca63 thru 0x017735 (0xacd3 bytes) 000525 1748469324.832751 trunc from 0x0392b8 to 0x00b223 (0x2e094 bytes) 000526 1748469324.836719 mapwrite 0x02d217 thru 0x03ba56 (0xe840 bytes) 000528 1748469324.859846 MIRROR_RESYNC 3 mirrors 000529 1748469324.967580 trunc from 0x03ba57 to 0x03a851 (0x1205 bytes) 000530 1748469324.983775 read 0x0315fd thru 0x035b72 (0x4576 bytes) 000531 1748469324.985944 trunc from 0x03a852 to 0x038ef9 (0x1958 bytes) 000532 1748469324.990153 mapwrite 0x0151d7 thru 0x024af3 (0xf91d bytes) 000534 1748469325.011515 trunc from 0x038efa to 0x0041dc (0x34d1d bytes) 000535 1748469325.015467 read 0x00322a thru 0x0041dc (0xfb3 bytes) 000536 1748469325.017620 write 0x021b5c thru 0x031578 (0xfa1d bytes) 000538 1748469325.018773 write 0x027600 thru 0x03095b (0x935c bytes) 000539 1748469325.019510 read 0x00baef thru 0x01a50e (0xea20 bytes) 000540 1748469325.023102 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3fcc7 000541 1748469325.125767 trunc from 0x031579 to 0x03fcc6 (0xe74e bytes) 000542 1748469325.141786 mapwrite 0x0201d5 thru 0x026511 (0x633d bytes) 000543 1748469325.161038 MIRROR_EXTEND to 4 mirrors 000545 1748469325.262876 read 0x02fb0f thru 0x032f79 (0x346b bytes) 000546 1748469325.266690 MIRROR_RESYNC 4 mirrors 000548 1748469325.374726 trunc from 0x03fcc7 to 0x038e2c (0x6e9a bytes) 000549 1748469325.393184 read 0x01aff2 thru 0x01d05b (0x206a bytes) 000550 1748469325.395361 mapread 0x02998c thru 0x033907 (0x9f7c bytes) 000551 1748469325.401607 trunc from 0x038e2d to 0x022640 (0x167ec bytes) 000552 1748469325.406422 read 0x01692a thru 0x022640 (0xbd17 bytes) 000554 1748469325.409202 mapwrite 0x034f88 thru 0x03ca39 (0x7ab2 bytes) 000555 1748469325.432166 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000558 1748469325.484670 mapread 0x017c54 thru 0x01a352 (0x26ff bytes) 000561 1748469325.490838 MIRROR_EXTEND to 4 mirrors 000562 1748469325.588094 write 0x00487c thru 0x006499 (0x1c1e bytes) 000563 1748469325.604279 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000564 1748469325.651851 mapread 0x02d0b7 thru 0x02d619 (0x563 bytes) 000566 1748469325.653689 write 0x023201 thru 0x03017a (0xcf7a bytes) 000567 1748469325.657552 trunc from 0x03ca3a to 0x00c23f (0x307fa bytes) 000569 1748469325.664787 write 0x023b24 thru 0x02820f (0x46ec bytes) 000570 1748469325.665628 mapread 0x000d11 thru 0x0074d1 (0x67c1 bytes) 000571 1748469325.669761 read 0x027cfb thru 0x02820f (0x515 bytes) 000572 1748469325.670203 write 0x0084b1 thru 0x00ce6e (0x49be bytes) 000573 1748469325.670672 trunc from 0x028210 to 0x000ef3 (0x2731c bytes) 000574 1748469325.673956 write 0x01f8f3 thru 0x02ba1c (0xc12a bytes) 000575 1748469325.674774 write 0x02c9d2 thru 0x034e8d (0x84bc bytes) 000576 1748469325.675385 mapread 0x00631f thru 0x00b259 (0x4f3b bytes) 000577 1748469325.677764 mapread 0x01b36a thru 0x01dc84 (0x291b bytes) 000578 1748469325.680137 read 0x00e239 thru 0x0172a2 (0x906a bytes) 000580 1748469325.682578 mapwrite 0x031194 thru 0x03dcc0 (0xcb2d bytes) 000581 1748469325.701280 read 0x03770f thru 0x03dcc0 (0x65b2 bytes) 000582 1748469325.701583 mapwrite 0x027c76 thru 0x035f7d (0xe308 bytes) 000583 1748469325.714397 mapread 0x030c8c thru 0x03a535 (0x98aa bytes) 000584 1748469325.715101 mapwrite 0x037cf1 thru 0x03ffff (0x830f bytes) 000585 1748469325.735670 MIRROR_EXTEND to 4 mirrors 000586 1748469325.840306 mapwrite 0x00dcd7 thru 0x01c546 (0xe870 bytes) 000587 1748469325.888419 write 0x029356 thru 0x035090 (0xbd3b bytes) 000588 1748469325.893258 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000589 1748469325.945266 write 0x021532 thru 0x02bfa4 (0xaa73 bytes) 000592 1748469325.948898 mapread 0x015549 thru 0x01b981 (0x6439 bytes) 000593 1748469325.951483 mapread 0x02b3cc thru 0x02f7d9 (0x440e bytes) 000594 1748469325.953718 trunc from 0x040000 to 0x032bb4 (0xd44b bytes) 000596 1748469325.961805 trunc from 0x032bb5 to 0x01c6bf (0x164f5 bytes) 000598 1748469325.965297 write 0x007ad8 thru 0x007ec6 (0x3ef bytes) 000599 1748469325.967478 read 0x011539 thru 0x0172d8 (0x5da0 bytes) 000600 1748469325.968852 read 0x000409 thru 0x0100d3 (0xfccb bytes) 000602 1748469325.970378 trunc from 0x01c6c0 to 0x0003be (0x1c301 bytes) 000603 1748469325.974398 read 0x000285 thru 0x0003be (0x13a bytes) 000607 1748469325.975884 read 0x0003bd thru 0x0003be (0x2 bytes) 000609 1748469325.976479 mapread 0x000007 thru 0x0003be (0x3b8 bytes) 000610 1748469325.977103 read 0x0001af thru 0x0003be (0x210 bytes) 000613 1748469325.977816 mapread 0x0002b2 thru 0x0003be (0x10d bytes) 000615 1748469325.979933 MIRROR_RESYNC 3 mirrors 000617 1748469326.080019 mapread 0x0001e0 thru 0x0003be (0x1df bytes) 000622 1748469326.080897 mapread 0x00009d thru 0x0003be (0x322 bytes) 000624 1748469326.082187 MIRROR_EXTEND to 4 mirrors 000628 1748469326.177390 read 0x000180 thru 0x0003be (0x23f bytes) 000629 1748469326.180377 mapwrite 0x027149 thru 0x032d0c (0xbbc4 bytes) 000632 1748469326.216492 mapread 0x030755 thru 0x0327ca (0x2076 bytes) 000633 1748469326.217087 mapread 0x02fa92 thru 0x032d0c (0x327b bytes) 000634 1748469326.217528 write 0x021cc8 thru 0x029e0b (0x8144 bytes) 000635 1748469326.220127 write 0x039e8f thru 0x03ffff (0x6171 bytes) 000636 1748469326.220903 write 0x022856 thru 0x0289cd (0x6178 bytes) 000638 1748469326.221797 mapwrite 0x0255a9 thru 0x0269f8 (0x1450 bytes) 000640 1748469326.232881 mapread 0x03a119 thru 0x03ffff (0x5ee7 bytes) 000641 1748469326.233337 trunc from 0x040000 to 0x004c11 (0x3b3ee bytes) 000643 1748469326.237743 trunc from 0x004c12 to 0x019d20 (0x1510f bytes) 000647 1748469326.241481 mapwrite 0x00ffc2 thru 0x019674 (0x96b3 bytes) 000648 1748469326.257598 mapwrite 0x034465 thru 0x034e9e (0xa3a bytes) 000649 1748469326.275464 MIRROR_EXTEND to 5 mirrors 000650 1748469326.366547 write 0x00ef4b thru 0x0129dd (0x3a93 bytes) 000651 1748469326.381918 mapread 0x00a9a8 thru 0x016c34 (0xc28d bytes) 000653 1748469326.386314 mapread 0x02f276 thru 0x0328c5 (0x3650 bytes) 000657 1748469326.389286 read 0x029fa0 thru 0x031c00 (0x7c61 bytes) 000658 1748469326.389703 write 0x02be26 thru 0x02c3dd (0x5b8 bytes) 000660 1748469326.390360 write 0x02d739 thru 0x035ac1 (0x8389 bytes) 000661 1748469326.390952 write 0x0040b0 thru 0x012aea (0xea3b bytes) 000662 1748469326.393410 mapwrite 0x03b40a thru 0x03bcf1 (0x8e8 bytes) 000663 1748469326.417467 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000666 1748469326.461938 mapwrite 0x02e674 thru 0x02f51a (0xea7 bytes) 000667 1748469326.474940 write 0x00bbc3 thru 0x00f94b (0x3d89 bytes) 000668 1748469326.478844 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000669 1748469326.524466 trunc from 0x03bcf2 to 0x039f27 (0x1dca bytes) 000671 1748469326.528090 mapread 0x02c649 thru 0x039f27 (0xd8df bytes) 000674 1748469326.536631 MIRROR_RESYNC 3 mirrors 000675 1748469326.640114 read 0x008c6a thru 0x00ab06 (0x1e9d bytes) 000679 1748469326.641915 mapwrite 0x039cc0 thru 0x03d1b8 (0x34f9 bytes) 000681 1748469326.674194 read 0x039e22 thru 0x03d048 (0x3227 bytes) 000682 1748469326.674515 mapwrite 0x01acf6 thru 0x01bb3d (0xe48 bytes) 000684 1748469326.688367 write 0x03f448 thru 0x03fbe2 (0x79b bytes) 000686 1748469326.689027 write 0x0167b3 thru 0x0228b6 (0xc104 bytes) 000687 1748469326.689638 mapwrite 0x034459 thru 0x03f012 (0xabba bytes) 000689 1748469326.704034 trunc from 0x03fbe3 to 0x01b0e6 (0x24afc bytes) 000690 1748469326.708543 read 0x0054be thru 0x00d67d (0x81c0 bytes) 000691 1748469326.708863 trunc from 0x01b0e7 to 0x03e642 (0x2355c bytes) 000693 1748469326.713386 MIRROR_EXTEND to 4 mirrors 000695 1748469326.803130 MIRROR_RESYNC 4 mirrors 000700 1748469326.905793 trunc from 0x03e643 to 0x0030ee (0x3b554 bytes) 000702 1748469326.926928 mapread 0x0006df thru 0x0030ee (0x2a10 bytes) 000705 1748469326.933493 trunc from 0x0030ef to 0x00a153 (0x7065 bytes) 000706 1748469326.938517 write 0x01c525 thru 0x025809 (0x92e5 bytes) 000708 1748469326.939984 mapwrite 0x010903 thru 0x01c4a0 (0xbb9e bytes) 000709 1748469326.956460 mapread 0x006498 thru 0x01044f (0x9fb8 bytes) 000711 1748469326.959300 mapread 0x017d36 thru 0x01e573 (0x683e bytes) 000712 1748469326.959749 write 0x00c4e6 thru 0x00e930 (0x244b bytes) 000714 1748469326.960528 mapwrite 0x015386 thru 0x022255 (0xced0 bytes) 000715 1748469326.975209 read 0x004eaf thru 0x007e29 (0x2f7b bytes) 000716 1748469326.975784 read 0x00709b thru 0x00ab76 (0x3adc bytes) 000718 1748469326.976386 read 0x01820a thru 0x020fd8 (0x8dcf bytes) 000720 1748469326.976948 mapread 0x011343 thru 0x018564 (0x7222 bytes) 000721 1748469326.977420 write 0x02de24 thru 0x0302bb (0x2498 bytes) 000723 1748469326.978404 read 0x02d65d thru 0x0302bb (0x2c5f bytes) 000725 1748469326.979014 write 0x00bbed thru 0x011a52 (0x5e66 bytes) 000727 1748469326.979941 mapread 0x0005f1 thru 0x006eaf (0x68bf bytes) 000730 1748469326.982720 MIRROR_RESYNC 4 mirrors 000733 1748469327.117367 read 0x001ed0 thru 0x00851d (0x664e bytes) 000734 1748469327.119416 write 0x039318 thru 0x03ffff (0x6ce8 bytes) 000735 1748469327.133019 write 0x003286 thru 0x011d30 (0xeaab bytes) 000736 1748469327.135627 mapwrite 0x00bee5 thru 0x011b9f (0x5cbb bytes) 000737 1748469327.148576 trunc from 0x040000 to 0x02512c (0x1aed3 bytes) 000738 1748469327.153305 mapwrite 0x02ebc5 thru 0x02f2f0 (0x72c bytes) 000741 1748469327.171001 MIRROR_EXTEND to 5 mirrors 000742 1748469327.259961 write 0x003bd0 thru 0x00ada1 (0x71d2 bytes) 000743 1748469327.275683 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000744 1748469327.324996 write 0x00aa09 thru 0x00f63d (0x4c35 bytes) 000745 1748469327.327516 read 0x01c4a6 thru 0x027c3f (0xb79a bytes) 000746 1748469327.329638 mapread 0x0178b9 thru 0x01a1bf (0x2907 bytes) 000747 1748469327.331891 read 0x029844 thru 0x02f2f0 (0x5aad bytes) 000748 1748469327.334499 trunc from 0x02f2f1 to 0x00d6f1 (0x21bff bytes) 000749 1748469327.342276 read 0x00d3b8 thru 0x00d6f1 (0x33a bytes) 000750 1748469327.342564 mapread 0x003b72 thru 0x00d6f1 (0x9b80 bytes) 000752 1748469327.343491 trunc from 0x00d6f2 to 0x026f47 (0x19856 bytes) 000753 1748469327.348272 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000754 1748469327.393353 trunc from 0x026f48 to 0x0027c6 (0x24781 bytes) 000755 1748469327.396959 mapread 0x0015da thru 0x0027c6 (0x11ed bytes) 000758 1748469327.400345 mapread 0x0017d9 thru 0x0027c6 (0xfee bytes) 000759 1748469327.400748 write 0x00a012 thru 0x00b49e (0x148d bytes) 000761 1748469327.401458 read 0x001c67 thru 0x00b49e (0x9838 bytes) 000762 1748469327.403875 mapread 0x00b3f5 thru 0x00b49e (0xaa bytes) 000764 1748469327.404643 trunc from 0x00b49f to 0x028b13 (0x1d675 bytes) 000765 1748469327.408028 write 0x0359e1 thru 0x03b852 (0x5e72 bytes) 000767 1748469327.409280 mapread 0x01197c thru 0x018dbe (0x7443 bytes) 000768 1748469327.411749 mapwrite 0x028665 thru 0x02d2ed (0x4c89 bytes) 000769 1748469327.426199 read 0x00c2b3 thru 0x0188c3 (0xc611 bytes) 000770 1748469327.427846 MIRROR_EXTEND to 4 mirrors 000771 1748469327.518441 write 0x01d45a thru 0x02219a (0x4d41 bytes) 000772 1748469327.533519 mapwrite 0x027148 thru 0x02af56 (0x3e0f bytes) 000774 1748469327.549050 read 0x0366b4 thru 0x03b852 (0x519f bytes) 000777 1748469327.552567 mapwrite 0x015456 thru 0x0235cc (0xe177 bytes) 000778 1748469327.568694 mapread 0x02387f thru 0x027421 (0x3ba3 bytes) 000780 1748469327.569521 trunc from 0x03b853 to 0x034450 (0x7402 bytes) 000781 1748469327.577524 read 0x007bb0 thru 0x01106b (0x94bc bytes) 000782 1748469327.580344 mapwrite 0x00c4f1 thru 0x0193c8 (0xced8 bytes) 000783 1748469327.595265 read 0x00406c thru 0x00f2a1 (0xb236 bytes) 000787 1748469327.596581 write 0x017a8d thru 0x020ee4 (0x9458 bytes) 000788 1748469327.597209 trunc from 0x034451 to 0x0337db (0xc75 bytes) 000793 1748469327.602717 write 0x02f0ad thru 0x038430 (0x9384 bytes) 000796 1748469327.604140 trunc from 0x038431 to 0x00d783 (0x2acad bytes) 000797 1748469327.607841 write 0x00bb4f thru 0x015e28 (0xa2da bytes) 000799 1748469327.609969 MIRROR_RESYNC 4 mirrors 000800 1748469327.733788 mapread 0x00d5fc thru 0x015e28 (0x882d bytes) truncating to largest ever: 0x3fe3b 000802 1748469327.738004 trunc from 0x015e29 to 0x03fe3a (0x2a012 bytes) 000804 1748469327.755119 mapread 0x011512 thru 0x012e2d (0x191c bytes) 000806 1748469327.758146 trunc from 0x03fe3b to 0x012ca4 (0x2d196 bytes) 000807 1748469327.762445 mapread 0x00a44f thru 0x01252c (0x80de bytes) 000808 1748469327.764888 trunc from 0x012ca5 to 0x01ea77 (0xbdd3 bytes) 000812 1748469327.769325 write 0x000382 thru 0x0083e7 (0x8066 bytes) 000816 1748469327.771974 trunc from 0x01ea78 to 0x016d17 (0x7d60 bytes) 000817 1748469327.775526 trunc from 0x016d18 to 0x0358e6 (0x1ebcf bytes) 000819 1748469327.779148 write 0x0060f1 thru 0x006839 (0x749 bytes) 000822 1748469327.779959 read 0x002030 thru 0x005aae (0x3a7f bytes) 000825 1748469327.780817 mapwrite 0x011196 thru 0x01f6db (0xe546 bytes) 000826 1748469327.799367 mapread 0x012cfe thru 0x0186b0 (0x59b3 bytes) 000827 1748469327.801262 MIRROR_RESYNC 4 mirrors 000828 1748469327.924171 trunc from 0x0358e7 to 0x00b288 (0x2a65e bytes) 000832 1748469327.943416 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000835 1748469327.989018 mapread 0x00a6aa thru 0x00b288 (0xbdf bytes) 000836 1748469327.991731 write 0x03b04a thru 0x03ffff (0x4fb6 bytes) 000842 1748469327.993568 write 0x033a05 thru 0x03ec16 (0xb212 bytes) 000843 1748469327.996411 trunc from 0x040000 to 0x03f2b2 (0xd4d bytes) 000845 1748469328.000541 trunc from 0x03f2b3 to 0x00e158 (0x3115a bytes) 000847 1748469328.004618 write 0x006cd1 thru 0x00c101 (0x5431 bytes) 000848 1748469328.007345 write 0x0318da thru 0x03f51f (0xdc46 bytes) 000849 1748469328.008118 read 0x00599f thru 0x00b041 (0x56a3 bytes) 000850 1748469328.008446 read 0x03e8fe thru 0x03f51f (0xc22 bytes) 000854 1748469328.009349 mapwrite 0x0142bb thru 0x0156ac (0x13f2 bytes) 000855 1748469328.025286 mapwrite 0x02bafd thru 0x034a12 (0x8f16 bytes) 000857 1748469328.038799 MIRROR_EXTEND to 4 mirrors 000858 1748469328.134553 MIRROR_RESYNC 4 mirrors 000859 1748469328.244615 read 0x03dcdf thru 0x03f51f (0x1841 bytes) 000862 1748469328.246623 read 0x00198a thru 0x010f85 (0xf5fc bytes) 000863 1748469328.249382 read 0x02d822 thru 0x03a6c3 (0xcea2 bytes) 000865 1748469328.249830 trunc from 0x03f520 to 0x03802d (0x74f2 bytes) 000866 1748469328.268178 write 0x0158ba thru 0x0165ba (0xd01 bytes) 000869 1748469328.273007 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000870 1748469328.320317 read 0x0145c9 thru 0x01a222 (0x5c5a bytes) 000871 1748469328.322942 write 0x03eb36 thru 0x03ffff (0x14ca bytes) 000872 1748469328.323482 write 0x021e81 thru 0x022c49 (0xdc9 bytes) 000873 1748469328.326140 mapwrite 0x008582 thru 0x0117bd (0x923c bytes) 000876 1748469328.343996 mapread 0x034455 thru 0x03ffff (0xbbab bytes) 000879 1748469328.349610 read 0x006669 thru 0x00d19b (0x6b33 bytes) 000882 1748469328.350468 write 0x0285f1 thru 0x033717 (0xb127 bytes) 000884 1748469328.353863 mapread 0x03f652 thru 0x03ffff (0x9ae bytes) 000886 1748469328.354664 trunc from 0x040000 to 0x01094f (0x2f6b0 bytes) 000887 1748469328.358675 mapread 0x004ff4 thru 0x00dbb0 (0x8bbd bytes) 000888 1748469328.359295 read 0x00e8a7 thru 0x01094f (0x20a9 bytes) 000890 1748469328.360083 write 0x03568d thru 0x0389ec (0x3360 bytes) 000892 1748469328.361187 read 0x029abe thru 0x035b62 (0xc0a5 bytes) 000894 1748469328.364405 mapwrite 0x0360f6 thru 0x03664e (0x559 bytes) 000896 1748469328.376575 write 0x019fd5 thru 0x027832 (0xd85e bytes) 000897 1748469328.377469 read 0x0287ac thru 0x0366dc (0xdf31 bytes) 000900 1748469328.378414 read 0x023303 thru 0x0318d0 (0xe5ce bytes) 000902 1748469328.379014 mapwrite 0x03d9b3 thru 0x03ffff (0x264d bytes) 000903 1748469328.397738 mapread 0x02629c thru 0x02b358 (0x50bd bytes) 000908 1748469328.399202 mapread 0x00b06a thru 0x01852c (0xd4c3 bytes) 000910 1748469328.399907 read 0x03fd38 thru 0x03ffff (0x2c8 bytes) 000911 1748469328.400243 trunc from 0x040000 to 0x000e4e (0x3f1b1 bytes) 000914 1748469328.404682 read 0x000aef thru 0x000e4e (0x360 bytes) 000916 1748469328.405199 mapread 0x000903 thru 0x000e4e (0x54c bytes) 000917 1748469328.405588 read 0x000657 thru 0x000e4e (0x7f8 bytes) 000918 1748469328.406002 trunc from 0x000e4f to 0x02e901 (0x2dab3 bytes) 000919 1748469328.409683 trunc from 0x02e902 to 0x0094e4 (0x2541d bytes) 000920 1748469328.412951 mapwrite 0x026608 thru 0x027d6b (0x1764 bytes) 000921 1748469328.430286 trunc from 0x027d6c to 0x02b646 (0x38db bytes) 000922 1748469328.433541 read 0x01ab13 thru 0x029cf3 (0xf1e1 bytes) 000923 1748469328.435386 write 0x02d615 thru 0x02fb8f (0x257b bytes) 000926 1748469328.436631 trunc from 0x02fb90 to 0x03af66 (0xb3d7 bytes) 000927 1748469328.440572 mapwrite 0x0064a1 thru 0x012070 (0xbbd0 bytes) 000928 1748469328.454718 trunc from 0x03af67 to 0x0372ba (0x3cac bytes) 000931 1748469328.459137 mapwrite 0x0191d9 thru 0x028cca (0xfaf2 bytes) 000933 1748469328.474252 read 0x01755f thru 0x024542 (0xcfe4 bytes) 000934 1748469328.477336 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000936 1748469328.521641 mapwrite 0x01cd65 thru 0x01f5cc (0x2868 bytes) 000937 1748469328.537765 write 0x00b5c3 thru 0x0118f0 (0x632e bytes) 000938 1748469328.540836 write 0x038cfd thru 0x03f054 (0x6358 bytes) 000943 1748469328.542480 read 0x015682 thru 0x01bb7e (0x64fd bytes) 000944 1748469328.543939 mapread 0x039abc thru 0x03f054 (0x5599 bytes) 000946 1748469328.545980 MIRROR_RESYNC 2 mirrors 000948 1748469328.639383 trunc from 0x03f055 to 0x01276d (0x2c8e7 bytes) 000951 1748469328.657718 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9d7:0x0]: Invalid argument (22) 000952 1748469328.687401 read 0x00161f thru 0x00a19c (0x8b7e bytes) 000953 1748469328.689961 mapread 0x0116c0 thru 0x01276d (0x10ae bytes) 000955 1748469328.693702 read 0x01261f thru 0x01276d (0x14f bytes) 000963 1748469328.696009 read 0x00be52 thru 0x01276d (0x691c bytes) 000964 1748469328.699205 write 0x024698 thru 0x030e0a (0xc773 bytes) 000965 1748469328.700738 mapread 0x002d11 thru 0x00d788 (0xaa78 bytes) 000966 1748469328.701429 read 0x01d814 thru 0x02ba59 (0xe246 bytes) 000967 1748469328.704334 trunc from 0x030e0b to 0x009418 (0x279f2 bytes) 000968 1748469328.708689 read 0x00241f thru 0x009418 (0x6ffa bytes) 000969 1748469328.709057 read 0x005337 thru 0x007957 (0x2621 bytes) 000972 1748469328.709951 mapread 0x009167 thru 0x009418 (0x2b2 bytes) 000973 1748469328.710419 read 0x0002e4 thru 0x009418 (0x9135 bytes) 000977 1748469328.711440 read 0x0028b7 thru 0x009418 (0x6b62 bytes) 000978 1748469328.711793 write 0x0216e5 thru 0x0288e9 (0x7205 bytes) 000980 1748469328.712744 read 0x00e75e thru 0x01c289 (0xdb2c bytes) 000982 1748469328.716930 MIRROR_EXTEND to 2 mirrors 000983 1748469328.815498 write 0x025c8d thru 0x02866d (0x29e1 bytes) 000984 1748469328.831593 trunc from 0x0288ea to 0x02f159 (0x6870 bytes) 000987 1748469328.839830 read 0x00e053 thru 0x01cbed (0xeb9b bytes) 000993 1748469328.843979 mapwrite 0x03cefe thru 0x03d987 (0xa8a bytes) 000995 1748469328.863044 mapread 0x02d812 thru 0x031cdd (0x44cc bytes) 000997 1748469328.867362 MIRROR_EXTEND to 3 mirrors 000998 1748469328.962066 read 0x015345 thru 0x016c50 (0x190c bytes) 000999 1748469328.965538 MIRROR_RESYNC 3 mirrors 001000 1748469329.062287 trunc from 0x03d988 to 0x01d75b (0x2022c bytes) All operations completed A-OK! PASS 70a (13s) == sanity-flr test 71: check mirror extend parent fid ==== 17:55:30 (1748469330) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (0s) == sanity-flr test 200a: stress test ===================== 17:55:30 (1748469330) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3822912 .. Extending file size to 6247328 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8244128 .. Extending file size to 4426944 .. Extending file size to 5035360 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6966496 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7977376 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8247968 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8540000 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8826560 .. Extending file size to 8712672 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8730272 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8869088 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9322336 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 30847 30848 30850 30851 30854 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) 192.168.203.122@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:56:34 (1748469394) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4120192 .. Extending file size to 1331040 .. failed Extending file size to 1917600 .. Extending file size to 8630784 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 4488480 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7971680 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8615744 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9125792 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9316704 .. Extending file size to 8810464 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8819296 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9136608 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 2607 2608 2610 2612 2614 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) 192.168.203.122@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:57:37 (1748469457) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5376 2202752 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5376 2202752 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0343887 s, 61.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0810893 s, 25.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:57:42 (1748469462) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0249674 s, 84.0 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 17:57:43 (1748469463) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa25:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd53:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd53:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd54:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xd56:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:57:45 (1748469465) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0264437 s, 79.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd57:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd58:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:57:46 (1748469466) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0254138 s, 82.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd57:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd59:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd57:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd59:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5a:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:57:48 (1748469468) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0243935 s, 86.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd58:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd58:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5c:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:58:11 (1748469491) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200106 s, 52.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd59:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd59:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5a:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:58:54 (1748469534) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0250052 s, 83.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:59:40 (1748469580) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg322-server mds-ost sync done. Creating new pool test_204f oleg322-server: Pool lustre.test_204f created Adding targets to pool oleg322-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg322-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0219504 s, 95.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd60:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd5d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd60:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg322-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg322-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg322-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (57s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:00:37 (1748469637) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:00:38 (1748469638) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:00:41 (1748469641) oleg322-server: Pool lustre.test_206 created Adding targets to pool oleg322-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg322-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg322-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg322-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg322-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (12s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:00:53 (1748469653) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333178 s, 31.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113496 s, 92.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x66cae330 Mirror 3: 0x66cae330 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:00:56 (1748469656) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:00:56 (1748469656) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:00:57 (1748469657) Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.015481 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x17b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x17b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x17b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1680000 Read this (4 bytes): BBBB 192.168.203.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg322-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:01:11 (1748469671) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00237761 s, 3.4 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:01:12 (1748469672) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125399 s, 83.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa4a lmm_fid: [0x200000404:0xa4a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa4a lmm_fid: [0x200000404:0xa4a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6a:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836554 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:01:12.000000000 -0400 Modify: 2025-05-28 18:01:13.000000000 -0400 Change: 2025-05-28 18:01:13.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:01:14 (1748469674) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:01:15 (1748469675) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.109123 s, 375 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0201732 s, 203 kB/s size after second write -rw-r--r-- 1 root root 4096 May 28 18:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa4c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 28 18:01 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 696 sec ============ 18:01:17 (1748469677) === sanity-flr: start cleanup 18:01:17 (1748469677) === === sanity-flr: finish cleanup 18:01:19 (1748469679) ===