-----============= acceptance-small: sanity-flr ============----- Mon Jun 9 10:12:40 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg308-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:12:44 (1749478364) === oleg308-client.virtnet: executing check_config_client /mnt/lustre oleg308-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg308-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b60ab000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b60ab000.idle_timeout=debug disable quota as required oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:12:53 (1749478373) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7275] == sanity-flr test 0a: lfs mirror create with -N option == 10:12:53 (1749478373) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:12:56 (1749478376) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:12:57 (1749478377) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:12:58 (1749478378) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:13:02 (1749478382) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:13:03 (1749478383) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 10:13:03 (1749478383) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b60ab000.stats=clear osc.lustre-OST0001-osc-ffff8800b60ab000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:13:05 (1749478385) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:13:07 (1749478387) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:13:09 (1749478389) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:13:10 (1749478390) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 10:13:11 (1749478391) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:13:13 (1749478393) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:13:14 (1749478394) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:13:15 (1749478395) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:13:18 (1749478398) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:13:19 (1749478399) 13+0 records in 13+0 records out 13631488 bytes (14 MB) copied, 0.320578 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0336737 s, 31.1 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:13:21 (1749478401) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.0658854 s, 175 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:13:23 (1749478403) Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b60ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b60ab000.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b60ab000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b60ab000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:13:45 (1749478425) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.077391 s, 54.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b60ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b60ab000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (8s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:13:53 (1749478433) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b60ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b60ab000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b60ab000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b60ab000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (16s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:14:09 (1749478449) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.78488 s, 66.8 MB/s -rw-r--r-- 1 root root 52428800 Jun 9 10:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.255s user 0m0.000s sys 0m0.138s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b60ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b60ab000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.200s user 0m0.000s sys 0m0.113s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b60ab000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b60ab000.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.211s user 0m0.000s sys 0m0.114s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg308-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b60ab000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b60ab000.ost_server_uuid in FULL state after 0 sec PASS 33b (20s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:14:29 (1749478469) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:14:29 (1749478469) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:14:30 (1749478470) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 10:14:31 (1749478471) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 10:14:32 (1749478472) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0989173 s, 127 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:14:35 (1749478475) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00693507 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0085825 s, 122 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:14:36 (1749478476) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:14:38 (1749478478) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000571073 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00047115 s, 2.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000492575 s, 2.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000399138 s, 2.6 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 10:14:40 (1749478480) create test files with size 15757537 .. b321c12df2a04d755c6e3317264a4f76 - 332c26d725245c6acc0eb8798a260304 - 9e9ec12f4b1304e130f0b34c39721922 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 10:14:44 (1749478484) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:14:47 (1749478487) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:14:48 (1749478488) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198233 s, 52.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00672304 s, 156 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 10:14:50 (1749478490) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0232371 s, 90.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0671422 s, 62.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0325011 s, 64.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 10:14:52 (1749478492) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 10:14:53 (1749478493) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (0s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:14:53 (1749478493) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00750487 s, 140 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 10:14:55 (1749478495) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:14:56 (1749478496) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:14:58 (1749478498) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0615786 s, 170 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (24s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:15:22 (1749478522) llite.lustre-ffff88012a630000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0667824 s, 157 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff88012a632800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:15:25 (1749478525) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:15:31 (1749478531) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:15:37 (1749478537) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 10:15:38 (1749478538) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0580534 s, 54.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0512366 s, 61.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: a31dc18c1c80979892908f76e782240d - mirror 1 checksum: b0a2f5a0e2843c82519479f2e026e88f - mirror 2 checksum: a31dc18c1c80979892908f76e782240d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 9 10:15:37 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 9 10:15:37 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a31dc18c1c80979892908f76e782240d - mirror 1 checksum: b0a2f5a0e2843c82519479f2e026e88f - mirror 2 checksum: b0a2f5a0e2843c82519479f2e026e88f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 9 10:15:37 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 9 10:15:37 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:15:40 (1749478540) Failing mds1 on oleg308-server Stopping /mnt/lustre-mds1 (opts:) on oleg308-server 10:15:41 (1749478541) shut down facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server Failover mds1 to oleg308-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 10:15:54 (1749478554) targets are mounted 10:15:54 (1749478554) facet_failover done oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00740826 s, 142 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (19s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:15:59 (1749478559) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:15:59 (1749478559) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:16:00 (1749478560) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00236309 s, 6.9 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:16:01 (1749478561) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:16:02 (1749478562) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:16:03 (1749478563) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:16:07 (1749478567) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:16:25 (1749478585) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (16s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:16:41 (1749478601) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:16:58 (1749478618) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 10:17:00 (1749478620) mirror create pid 22793 mirror split pid 22795 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22793 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 22795 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 10:18:01 (1749478681) Seed set to 8681 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1749478681.913295 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x2535 000004 1749478681.969965 trunc from 00000000 to 0x002534 (0x2535 bytes) 000005 1749478681.988027 read 0x001ec0 thru 0x002534 (0x675 bytes) 000006 1749478681.989543 write 0x001456 thru 0x00fa44 (0xe5ef bytes) 000007 1749478681.990404 mapwrite 0x0287a4 thru 0x02b7c5 (0x3022 bytes) 000008 1749478682.015302 MIRROR_EXTEND to 3 mirrors 000011 1749478682.114328 write 0x02b64c thru 0x03748e (0xbe43 bytes) 000012 1749478682.129797 mapwrite 0x035064 thru 0x03ffff (0xaf9c bytes) 000013 1749478682.152869 MIRROR_EXTEND to 4 mirrors 000015 1749478682.243476 write 0x00ae4b thru 0x01a14d (0xf303 bytes) 000016 1749478682.260201 mapread 0x03c0d4 thru 0x03ff18 (0x3e45 bytes) 000019 1749478682.263314 write 0x007cae thru 0x00ae06 (0x3159 bytes) 000020 1749478682.265506 mapwrite 0x0012dc thru 0x00ab6b (0x9890 bytes) 000021 1749478682.280918 write 0x02790a thru 0x02b5f8 (0x3cef bytes) truncating to largest ever: 0xbdeb 000024 1749478682.283921 trunc from 0x040000 to 0x00bdea (0x34215 bytes) 000025 1749478682.292580 MIRROR_EXTEND to 5 mirrors 000027 1749478682.375958 write 0x034c05 thru 0x03ee5c (0xa258 bytes) 000031 1749478682.391879 read 0x03848a thru 0x03dd75 (0x58ec bytes) 000032 1749478682.393564 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000035 1749478682.439688 write 0x032eb5 thru 0x038acd (0x5c19 bytes) 000036 1749478682.442796 read 0x03525c thru 0x03a874 (0x5619 bytes) 000037 1749478682.444426 mapwrite 0x002cd5 thru 0x00cda3 (0xa0cf bytes) 000039 1749478682.467630 mapread 0x01c996 thru 0x020c6e (0x42d9 bytes) truncating to largest ever: 0x3e633 000040 1749478682.470836 trunc from 0x03ee5d to 0x03e632 (0x82a bytes) 000041 1749478682.479678 MIRROR_RESYNC 4 mirrors 000042 1749478682.621592 write 0x026558 thru 0x02c3e5 (0x5e8e bytes) 000046 1749478682.639355 mapwrite 0x03f830 thru 0x03ffff (0x7d0 bytes) 000047 1749478682.665457 mapread 0x03f249 thru 0x03ffff (0xdb7 bytes) 000048 1749478682.666168 mapread 0x0246f1 thru 0x03017c (0xba8c bytes) 000050 1749478682.672577 MIRROR_RESYNC 4 mirrors 000051 1749478682.810629 mapwrite 0x02677a thru 0x035fa8 (0xf82f bytes) 000053 1749478682.856686 mapwrite 0x02423f thru 0x0277f6 (0x35b8 bytes) 000054 1749478682.872417 read 0x017a9a thru 0x020fe7 (0x954e bytes) 000056 1749478682.875033 read 0x00079d thru 0x002e26 (0x268a bytes) 000060 1749478682.879529 MIRROR_RESYNC 4 mirrors 000061 1749478683.016201 MIRROR_EXTEND to 5 mirrors 000063 1749478683.118828 mapwrite 0x023dbd thru 0x02b60a (0x784e bytes) 000065 1749478683.161932 mapwrite 0x01d35b thru 0x028698 (0xb33e bytes) 000066 1749478683.175454 mapwrite 0x018bc9 thru 0x01bc8c (0x30c4 bytes) 000069 1749478683.189172 mapwrite 0x033c14 thru 0x03ffff (0xc3ec bytes) 000070 1749478683.204675 trunc from 0x040000 to 0x011d12 (0x2e2ed bytes) 000074 1749478683.210140 mapread 0x006d70 thru 0x011d12 (0xafa3 bytes) 000075 1749478683.213864 write 0x0250b2 thru 0x026001 (0xf50 bytes) 000078 1749478683.214792 mapread 0x01f27a thru 0x026001 (0x6d88 bytes) 000079 1749478683.216995 read 0x020fc9 thru 0x026001 (0x5039 bytes) 000081 1749478683.217515 write 0x00656c thru 0x01308c (0xcb21 bytes) 000082 1749478683.219238 read 0x009623 thru 0x015f84 (0xc962 bytes) 000084 1749478683.222323 MIRROR_EXTEND to 6 mirrors 000086 1749478683.320126 MIRROR_EXTEND to 7 mirrors 000087 1749478683.412786 read 0x00e4e5 thru 0x01de83 (0xf99f bytes) 000089 1749478683.418028 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000090 1749478683.470737 trunc from 0x026002 to 0x00c38e (0x19c73 bytes) 000091 1749478683.489986 mapwrite 0x003154 thru 0x00f179 (0xc026 bytes) 000092 1749478683.517498 read 0x00ebe3 thru 0x00f179 (0x597 bytes) 000095 1749478683.518502 write 0x03ba0a thru 0x03ffff (0x45f6 bytes) 000096 1749478683.519199 read 0x019af2 thru 0x01ebd8 (0x50e7 bytes) 000097 1749478683.522257 mapwrite 0x00dd79 thru 0x00fcb4 (0x1f3c bytes) 000098 1749478683.534504 mapread 0x037457 thru 0x03a35e (0x2f08 bytes) 000102 1749478683.536359 mapwrite 0x01942d thru 0x021710 (0x82e4 bytes) 000103 1749478683.550243 trunc from 0x040000 to 0x00c618 (0x339e7 bytes) 000104 1749478683.554013 read 0x00aa65 thru 0x00c618 (0x1bb4 bytes) 000106 1749478683.554575 mapread 0x0021f5 thru 0x007ea2 (0x5cae bytes) 000107 1749478683.556130 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000108 1749478683.602298 trunc from 0x00c619 to 0x006bd0 (0x5a48 bytes) 000109 1749478683.606937 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000110 1749478683.652309 mapwrite 0x032977 thru 0x03ffff (0xd689 bytes) truncating to largest ever: 0x3f4b9 000111 1749478683.675643 trunc from 0x040000 to 0x03f4b8 (0xb47 bytes) 000113 1749478683.679320 trunc from 0x03f4b9 to 0x039234 (0x6284 bytes) 000116 1749478683.683035 trunc from 0x039235 to 0x01d84f (0x1b9e5 bytes) 000117 1749478683.687713 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000118 1749478683.731743 read 0x008f75 thru 0x016946 (0xd9d2 bytes) 000122 1749478683.735509 write 0x035177 thru 0x03ac3e (0x5ac8 bytes) 000123 1749478683.736313 trunc from 0x03ac3f to 0x036038 (0x4c06 bytes) 000125 1749478683.741760 MIRROR_RESYNC 3 mirrors 000128 1749478683.857896 read 0x00e781 thru 0x0192cc (0xab4c bytes) 000129 1749478683.860488 mapread 0x00abf2 thru 0x014fec (0xa3fb bytes) 000133 1749478683.865544 read 0x010e43 thru 0x01f671 (0xe82f bytes) 000134 1749478683.868433 mapwrite 0x03afd6 thru 0x03eb86 (0x3bb1 bytes) 000135 1749478683.901496 trunc from 0x03eb87 to 0x005823 (0x39363 bytes) 000137 1749478683.906481 mapread 0x00545f thru 0x005823 (0x3c5 bytes) 000138 1749478683.909269 MIRROR_RESYNC 3 mirrors 000139 1749478684.005953 mapread 0x003286 thru 0x005823 (0x259e bytes) 000140 1749478684.008573 read 0x001dd9 thru 0x0029e5 (0xc0d bytes) 000141 1749478684.010175 mapwrite 0x01c77e thru 0x01dc21 (0x14a4 bytes) 000144 1749478684.041814 trunc from 0x01dc22 to 0x02189c (0x3c7b bytes) 000145 1749478684.046244 mapwrite 0x02b744 thru 0x033237 (0x7af4 bytes) 000146 1749478684.063741 read 0x010e97 thru 0x01571e (0x4888 bytes) 000150 1749478684.067709 read 0x001a8d thru 0x004792 (0x2d06 bytes) 000152 1749478684.068494 read 0x00384b thru 0x00964d (0x5e03 bytes) 000154 1749478684.069288 write 0x023ec7 thru 0x02bd6d (0x7ea7 bytes) 000155 1749478684.070127 trunc from 0x033238 to 0x014959 (0x1e8de bytes) 000157 1749478684.074584 trunc from 0x01495a to 0x02f810 (0x1aeb7 bytes) 000159 1749478684.078476 read 0x008443 thru 0x00be36 (0x39f4 bytes) 000160 1749478684.078891 mapread 0x02379b thru 0x02f810 (0xc076 bytes) 000162 1749478684.082498 read 0x013d34 thru 0x016e3a (0x3107 bytes) 000164 1749478684.083211 mapwrite 0x01f99c thru 0x026774 (0x6dd9 bytes) 000166 1749478684.097475 mapread 0x025471 thru 0x02b33f (0x5ecf bytes) 000167 1749478684.098187 mapread 0x00932b thru 0x00e389 (0x505f bytes) 000169 1749478684.099172 read 0x0109b2 thru 0x0124d3 (0x1b22 bytes) 000170 1749478684.099547 read 0x00ecf4 thru 0x01e810 (0xfb1d bytes) 000171 1749478684.099881 mapread 0x00e892 thru 0x010594 (0x1d03 bytes) 000172 1749478684.100265 mapwrite 0x02151f thru 0x02b0eb (0x9bcd bytes) 000173 1749478684.114541 mapwrite 0x005fe4 thru 0x015b78 (0xfb95 bytes) 000175 1749478684.129090 read 0x02f6bc thru 0x02f810 (0x155 bytes) 000177 1749478684.129743 mapread 0x00289f thru 0x00a5a3 (0x7d05 bytes) 000179 1749478684.130381 mapwrite 0x0189a5 thru 0x01d44d (0x4aa9 bytes) 000182 1749478684.142941 write 0x0324cb thru 0x039ee1 (0x7a17 bytes) 000183 1749478684.143761 write 0x0008a3 thru 0x002fe8 (0x2746 bytes) 000184 1749478684.144384 trunc from 0x039ee2 to 0x012b01 (0x273e0 bytes) 000185 1749478684.148498 read 0x00a448 thru 0x00b46f (0x1028 bytes) 000187 1749478684.150249 MIRROR_EXTEND to 4 mirrors 000188 1749478684.232002 write 0x013f98 thru 0x021382 (0xd3eb bytes) 000192 1749478684.248620 read 0x01ff79 thru 0x021382 (0x140a bytes) 000194 1749478684.249393 mapwrite 0x01e66f thru 0x02e2fd (0xfc8f bytes) 000195 1749478684.280532 read 0x0180b3 thru 0x01cf77 (0x4ec5 bytes) 000197 1749478684.281109 trunc from 0x02e2fe to 0x00d03d (0x212c0 bytes) 000199 1749478684.285318 trunc from 0x00d03e to 0x03a898 (0x2d85b bytes) 000203 1749478684.290358 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000205 1749478684.332271 mapread 0x0178da thru 0x01dfd0 (0x66f7 bytes) 000207 1749478684.337324 mapread 0x0178b0 thru 0x01e5a0 (0x6cf1 bytes) 000208 1749478684.337788 trunc from 0x03a899 to 0x00e252 (0x2c646 bytes) 000209 1749478684.342063 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000211 1749478684.385328 MIRROR_RESYNC 2 mirrors 000212 1749478684.465911 read 0x00199f thru 0x004de4 (0x3446 bytes) 000214 1749478684.467785 trunc from 0x00e253 to 0x016a8a (0x8838 bytes) 000215 1749478684.484764 mapwrite 0x01b50d thru 0x01de39 (0x292d bytes) 000216 1749478684.505685 write 0x02b44a thru 0x030449 (0x5000 bytes) 000218 1749478684.506778 mapread 0x00749d thru 0x00f9f6 (0x855a bytes) 000219 1749478684.511590 trunc from 0x03044a to 0x01a797 (0x15cb2 bytes) 000220 1749478684.515400 trunc from 0x01a798 to 0x03525f (0x1aac8 bytes) 000221 1749478684.520286 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000222 1749478684.548715 write 0x01838c thru 0x023ae4 (0xb759 bytes) 000223 1749478684.551387 mapread 0x01df23 thru 0x0269ab (0x8a89 bytes) 000227 1749478684.555581 mapread 0x0111c8 thru 0x016b01 (0x593a bytes) 000228 1749478684.558024 mapread 0x01b55a thru 0x0238ad (0x8354 bytes) 000229 1749478684.558550 trunc from 0x035260 to 0x036800 (0x15a1 bytes) 000231 1749478684.562310 write 0x035700 thru 0x03ffff (0xa900 bytes) 000240 1749478684.566874 write 0x03182b thru 0x03e5eb (0xcdc1 bytes) 000241 1749478684.568805 MIRROR_EXTEND to 2 mirrors 000242 1749478684.658784 mapwrite 0x00f178 thru 0x01b23b (0xc0c4 bytes) 000243 1749478684.704469 MIRROR_RESYNC 2 mirrors 000244 1749478684.787332 mapread 0x03a792 thru 0x03c424 (0x1c93 bytes) 000245 1749478684.792035 trunc from 0x040000 to 0x0234c3 (0x1cb3c bytes) 000246 1749478684.808912 mapwrite 0x02bc81 thru 0x02f455 (0x37d5 bytes) 000247 1749478684.828943 write 0x01159e thru 0x012518 (0xf7b bytes) 000249 1749478684.832352 mapwrite 0x0152ce thru 0x01cfa8 (0x7cdb bytes) 000250 1749478684.847538 read 0x00ef00 thru 0x01c46e (0xd56f bytes) 000251 1749478684.848080 mapread 0x013342 thru 0x0144ab (0x116a bytes) 000252 1749478684.848688 trunc from 0x02f456 to 0x012bab (0x1c8aa bytes) 000253 1749478684.853869 MIRROR_EXTEND to 3 mirrors 000254 1749478684.930270 write 0x020f4e thru 0x0213dc (0x48f bytes) 000255 1749478684.943548 read 0x005eb5 thru 0x009505 (0x3651 bytes) 000256 1749478684.945701 mapwrite 0x005c77 thru 0x0128f2 (0xcc7c bytes) 000258 1749478684.964275 mapread 0x0138de thru 0x01ea07 (0xb12a bytes) 000260 1749478684.967233 mapwrite 0x00e008 thru 0x018eb0 (0xaea9 bytes) 000261 1749478684.980423 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000265 1749478685.030572 mapwrite 0x01677f thru 0x017343 (0xbc5 bytes) 000267 1749478685.044186 trunc from 0x0213dd to 0x01d06e (0x436e bytes) 000270 1749478685.052050 read 0x012a70 thru 0x01d06e (0xa5ff bytes) 000271 1749478685.054469 write 0x0085af thru 0x016da0 (0xe7f2 bytes) 000272 1749478685.057035 trunc from 0x01d06f to 0x03b4ed (0x1e47f bytes) 000273 1749478685.061745 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000274 1749478685.091507 read 0x007686 thru 0x0157be (0xe139 bytes) 000275 1749478685.093902 write 0x01fcf8 thru 0x02d1c9 (0xd4d2 bytes) 000276 1749478685.097788 MIRROR_EXTEND to 2 mirrors 000277 1749478685.190931 read 0x038615 thru 0x03b4ed (0x2ed9 bytes) 000279 1749478685.193251 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000281 1749478685.219594 read 0x02cffa thru 0x030276 (0x327d bytes) 000282 1749478685.221771 read 0x0197ea thru 0x01ddba (0x45d1 bytes) 000284 1749478685.225007 mapwrite 0x00a061 thru 0x013c20 (0x9bc0 bytes) 000288 1749478685.263080 trunc from 0x03b4ee to 0x017648 (0x23ea5 bytes) 000289 1749478685.269807 mapread 0x00ed4b thru 0x017648 (0x88fe bytes) 000290 1749478685.274583 mapread 0x012842 thru 0x017648 (0x4e07 bytes) 000292 1749478685.275216 mapread 0x009636 thru 0x00a336 (0xd01 bytes) 000297 1749478685.277483 read 0x00671d thru 0x0161b0 (0xfa94 bytes) 000298 1749478685.279696 trunc from 0x017649 to 0x02badd (0x14495 bytes) 000299 1749478685.283033 write 0x02a527 thru 0x02e581 (0x405b bytes) 000301 1749478685.285813 mapwrite 0x00b579 thru 0x00bc35 (0x6bd bytes) 000303 1749478685.297999 mapwrite 0x03351a thru 0x03ffff (0xcae6 bytes) 000304 1749478685.315353 write 0x00d0ce thru 0x013e2b (0x6d5e bytes) 000308 1749478685.316457 trunc from 0x040000 to 0x011cbd (0x2e342 bytes) 000310 1749478685.320472 mapread 0x00de47 thru 0x010f4e (0x3108 bytes) 000314 1749478685.321636 read 0x007366 thru 0x008236 (0xed1 bytes) 000315 1749478685.321885 mapread 0x002fd4 thru 0x00efa8 (0xbfd5 bytes) 000316 1749478685.322341 mapread 0x0059b5 thru 0x00d9ab (0x7ff7 bytes) 000317 1749478685.322724 trunc from 0x011cbe to 0x018eca (0x720d bytes) 000318 1749478685.326140 mapwrite 0x02e001 thru 0x03590b (0x790b bytes) 000325 1749478685.345432 mapread 0x022a79 thru 0x0264b2 (0x3a3a bytes) 000326 1749478685.345842 mapread 0x03493b thru 0x03590b (0xfd1 bytes) 000327 1749478685.346184 trunc from 0x03590c to 0x004f7d (0x3098e bytes) 000328 1749478685.350064 mapwrite 0x01b3cf thru 0x02875e (0xd390 bytes) 000329 1749478685.381652 write 0x01dd0c thru 0x025ed4 (0x81c9 bytes) 000330 1749478685.382398 mapwrite 0x002ac8 thru 0x0061b4 (0x36ed bytes) 000333 1749478685.397912 read 0x018133 thru 0x020ae9 (0x89b7 bytes) 000335 1749478685.400968 write 0x00f771 thru 0x01c41e (0xccae bytes) 000338 1749478685.402436 mapread 0x0159a3 thru 0x022da3 (0xd401 bytes) 000339 1749478685.402871 read 0x0087e4 thru 0x00faa7 (0x72c4 bytes) 000342 1749478685.404851 MIRROR_EXTEND to 2 mirrors 000343 1749478685.497931 read 0x0235e2 thru 0x02875e (0x517d bytes) 000346 1749478685.501028 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000349 1749478685.527159 trunc from 0x02875f to 0x00d641 (0x1b11d bytes) 000351 1749478685.535121 mapread 0x001338 thru 0x00d641 (0xc30a bytes) 000355 1749478685.541993 mapwrite 0x005a73 thru 0x012ec6 (0xd454 bytes) 000356 1749478685.562156 mapwrite 0x013958 thru 0x01be7a (0x8523 bytes) 000357 1749478685.582531 read 0x002fb7 thru 0x00abae (0x7bf8 bytes) 000361 1749478685.583943 write 0x03a04c thru 0x03ffff (0x5fb4 bytes) 000362 1749478685.584770 read 0x0248d0 thru 0x0254ac (0xbdd bytes) 000365 1749478685.588841 trunc from 0x040000 to 0x014d9e (0x2b261 bytes) 000366 1749478685.592795 mapwrite 0x019bd1 thru 0x022780 (0x8bb0 bytes) 000372 1749478685.621070 read 0x009fb4 thru 0x00f731 (0x577e bytes) 000373 1749478685.621395 mapwrite 0x00e07b thru 0x018147 (0xa0cd bytes) 000375 1749478685.634353 read 0x003811 thru 0x0040a1 (0x891 bytes) 000376 1749478685.634633 read 0x0069d6 thru 0x00c376 (0x59a1 bytes) 000377 1749478685.634883 mapread 0x009d6d thru 0x00a2d6 (0x56a bytes) 000378 1749478685.635312 mapwrite 0x030ff5 thru 0x03b7d4 (0xa7e0 bytes) 000380 1749478685.653568 mapread 0x034cbe thru 0x03aacc (0x5e0f bytes) 000384 1749478685.656350 MIRROR_EXTEND to 2 mirrors 000385 1749478685.743539 mapwrite 0x03d624 thru 0x03ffff (0x29dc bytes) 000386 1749478685.775699 trunc from 0x040000 to 0x01b8e9 (0x24716 bytes) 000387 1749478685.781371 MIRROR_EXTEND to 3 mirrors 000388 1749478685.861757 write 0x00c0c8 thru 0x00f94c (0x3885 bytes) 000389 1749478685.876549 read 0x00d0a5 thru 0x01b8e9 (0xe845 bytes) 000390 1749478685.878905 read 0x008317 thru 0x00c1b6 (0x3ea0 bytes) 000392 1749478685.881239 mapread 0x01118c thru 0x01b8e9 (0xa75e bytes) 000393 1749478685.882762 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000394 1749478685.924931 trunc from 0x01b8ea to 0x02d8d9 (0x11ff0 bytes) 000397 1749478685.931618 read 0x02a52d thru 0x02d8d9 (0x33ad bytes) 000398 1749478685.933292 mapwrite 0x01b126 thru 0x02a4dc (0xf3b7 bytes) 000400 1749478685.954297 write 0x00d252 thru 0x01b85a (0xe609 bytes) 000404 1749478685.958241 read 0x009a5f thru 0x00e053 (0x45f5 bytes) 000405 1749478685.958537 write 0x03cb9c thru 0x03ffff (0x3464 bytes) 000406 1749478685.959067 write 0x03fa5c thru 0x03ffff (0x5a4 bytes) 000409 1749478685.960953 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000411 1749478685.991955 write 0x02f251 thru 0x0389b3 (0x9763 bytes) 000412 1749478685.994556 write 0x004626 thru 0x0062ac (0x1c87 bytes) 000413 1749478685.996796 trunc from 0x040000 to 0x00a6f0 (0x3590f bytes) 000415 1749478686.000994 mapread 0x009193 thru 0x00a6f0 (0x155e bytes) 000421 1749478686.004724 MIRROR_EXTEND to 2 mirrors 000422 1749478686.096540 mapwrite 0x0285f4 thru 0x02ad17 (0x2724 bytes) 000423 1749478686.126337 write 0x008bbd thru 0x00f546 (0x698a bytes) 000425 1749478686.129177 read 0x009fbd thru 0x0180fd (0xe141 bytes) 000426 1749478686.131818 write 0x01733a thru 0x0238c0 (0xc587 bytes) 000427 1749478686.133915 MIRROR_RESYNC 2 mirrors 000428 1749478686.217378 write 0x03c836 thru 0x03ffff (0x37ca bytes) 000429 1749478686.230770 trunc from 0x040000 to 0x008904 (0x376fb bytes) 000431 1749478686.238834 mapread 0x007d96 thru 0x008904 (0xb6f bytes) 000432 1749478686.241919 trunc from 0x008905 to 0x00a147 (0x1843 bytes) 000433 1749478686.245278 write 0x010c28 thru 0x01d8a5 (0xcc7e bytes) 000437 1749478686.247068 trunc from 0x01d8a6 to 0x01344f (0xa456 bytes) 000439 1749478686.251621 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000442 1749478686.283008 mapwrite 0x019022 thru 0x027a98 (0xea77 bytes) 000443 1749478686.306299 mapread 0x01b530 thru 0x0248e7 (0x93b8 bytes) 000444 1749478686.306882 read 0x015012 thru 0x02369b (0xe68a bytes) 000446 1749478686.307433 mapread 0x00eee6 thru 0x01d475 (0xe590 bytes) 000447 1749478686.309890 mapwrite 0x0075c4 thru 0x014cf2 (0xd72f bytes) 000451 1749478686.326114 mapread 0x004f60 thru 0x00c7f3 (0x7894 bytes) 000452 1749478686.326635 trunc from 0x027a99 to 0x00b8b6 (0x1c1e2 bytes) 000453 1749478686.330497 read 0x004b48 thru 0x00b8b6 (0x6d6f bytes) 000454 1749478686.330931 mapread 0x00b559 thru 0x00b8b6 (0x35e bytes) 000455 1749478686.331409 mapwrite 0x0308f9 thru 0x034617 (0x3d1f bytes) 000457 1749478686.351791 MIRROR_EXTEND to 2 mirrors 000458 1749478686.447137 trunc from 0x034618 to 0x018c13 (0x1ba04 bytes) 000459 1749478686.465686 write 0x027c6d thru 0x036587 (0xe91b bytes) 000460 1749478686.466655 mapwrite 0x02dafb thru 0x032fbe (0x54c4 bytes) 000461 1749478686.480018 mapread 0x02b019 thru 0x036587 (0xb56f bytes) 000462 1749478686.480728 mapwrite 0x02bd70 thru 0x033c3a (0x7ecb bytes) 000463 1749478686.493553 mapread 0x00c1f4 thru 0x015e25 (0x9c32 bytes) 000465 1749478686.499411 write 0x0290a4 thru 0x0322ce (0x922b bytes) 000467 1749478686.500328 trunc from 0x036588 to 0x039468 (0x2ee1 bytes) 000468 1749478686.505367 write 0x0147d6 thru 0x01b821 (0x704c bytes) 000469 1749478686.507234 MIRROR_EXTEND to 3 mirrors 000472 1749478686.607800 mapread 0x014cf5 thru 0x017947 (0x2c53 bytes) 000475 1749478686.613490 mapwrite 0x01be62 thru 0x026343 (0xa4e2 bytes) 000478 1749478686.657561 mapwrite 0x00fbd2 thru 0x012364 (0x2793 bytes) 000479 1749478686.672133 write 0x0189b8 thru 0x0274ad (0xeaf6 bytes) 000481 1749478686.674051 write 0x03b4eb thru 0x03ffff (0x4b15 bytes) 000483 1749478686.675473 write 0x03a065 thru 0x03ffff (0x5f9b bytes) 000484 1749478686.678133 mapread 0x01fb20 thru 0x020568 (0xa49 bytes) 000486 1749478686.678889 mapread 0x02c914 thru 0x033d6a (0x7457 bytes) 000487 1749478686.680824 write 0x03c1e9 thru 0x03ffff (0x3e17 bytes) 000489 1749478686.681679 mapread 0x01cfef thru 0x0226d4 (0x56e6 bytes) 000490 1749478686.682087 write 0x00307f thru 0x0127b7 (0xf739 bytes) 000492 1749478686.685024 read 0x01649d thru 0x017503 (0x1067 bytes) 000493 1749478686.685346 read 0x00e895 thru 0x013bc0 (0x532c bytes) 000495 1749478686.685838 trunc from 0x040000 to 0x007e34 (0x381cb bytes) 000497 1749478686.694626 mapread 0x00747f thru 0x007e34 (0x9b6 bytes) 000499 1749478686.695312 mapread 0x00335e thru 0x007e34 (0x4ad7 bytes) 000500 1749478686.695834 trunc from 0x007e35 to 0x024b3d (0x1cd09 bytes) 000502 1749478686.699307 mapread 0x003491 thru 0x00e471 (0xafe1 bytes) 000504 1749478686.702715 trunc from 0x024b3e to 0x025f8b (0x144e bytes) 000508 1749478686.706593 read 0x011c66 thru 0x016512 (0x48ad bytes) 000510 1749478686.707110 write 0x01084d thru 0x01cd55 (0xc509 bytes) 000511 1749478686.707695 write 0x0298e7 thru 0x038786 (0xeea0 bytes) 000512 1749478686.708418 trunc from 0x038787 to 0x022c94 (0x15af2 bytes) 000516 1749478686.712735 mapread 0x0209a0 thru 0x022c94 (0x22f5 bytes) 000517 1749478686.713250 mapwrite 0x0036d2 thru 0x00415b (0xa8a bytes) 000518 1749478686.728168 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000520 1749478686.769577 trunc from 0x022c95 to 0x01e049 (0x4c4b bytes) 000521 1749478686.772803 trunc from 0x01e04a to 0x02b29f (0xd256 bytes) 000522 1749478686.776042 write 0x00e3c4 thru 0x011a99 (0x36d6 bytes) 000523 1749478686.778528 read 0x00396c thru 0x0083ae (0x4a43 bytes) 000524 1749478686.780531 read 0x01a783 thru 0x02118c (0x6a0a bytes) 000525 1749478686.784174 MIRROR_EXTEND to 3 mirrors 000527 1749478686.871384 mapwrite 0x02e4b3 thru 0x03110f (0x2c5d bytes) 000528 1749478686.904132 mapwrite 0x01f2e5 thru 0x029839 (0xa555 bytes) 000530 1749478686.919881 trunc from 0x031110 to 0x01eb22 (0x125ed bytes) 000531 1749478686.924148 write 0x00ff1c thru 0x00ff39 (0x1e bytes) 000532 1749478686.926495 mapwrite 0x02acc7 thru 0x034f79 (0xa2b3 bytes) 000534 1749478686.947360 write 0x003781 thru 0x008be2 (0x5462 bytes) 000536 1749478686.950172 write 0x02fdbd thru 0x03e22a (0xe46e bytes) 000538 1749478686.951415 mapwrite 0x00bf16 thru 0x01b123 (0xf20e bytes) 000539 1749478686.968375 write 0x00c354 thru 0x0169c2 (0xa66f bytes) 000540 1749478686.970885 MIRROR_RESYNC 3 mirrors 000541 1749478687.086422 write 0x03fd14 thru 0x03ffff (0x2ec bytes) 000544 1749478687.099291 trunc from 0x040000 to 0x03f15f (0xea0 bytes) 000545 1749478687.104526 MIRROR_EXTEND to 4 mirrors 000546 1749478687.205444 write 0x01eaf2 thru 0x022e6a (0x4379 bytes) 000548 1749478687.221591 mapwrite 0x02efd0 thru 0x0358d6 (0x6907 bytes) 000549 1749478687.237505 write 0x03d9e9 thru 0x03ffff (0x2617 bytes) 000552 1749478687.240111 trunc from 0x040000 to 0x021b63 (0x1e49c bytes) 000554 1749478687.244845 mapread 0x010a22 thru 0x012e98 (0x2477 bytes) 000555 1749478687.247337 mapread 0x01088a thru 0x013a99 (0x3210 bytes) 000556 1749478687.248823 MIRROR_RESYNC 4 mirrors 000557 1749478687.368899 mapread 0x0214bb thru 0x021b63 (0x6a9 bytes) 000559 1749478687.369459 read 0x01204d thru 0x017e73 (0x5e27 bytes) 000563 1749478687.372144 read 0x00d37a thru 0x012bba (0x5841 bytes) 000564 1749478687.374580 write 0x0161ab thru 0x01a80c (0x4662 bytes) 000568 1749478687.391382 write 0x0181a2 thru 0x01e7db (0x663a bytes) 000570 1749478687.393442 read 0x01fb13 thru 0x021b63 (0x2051 bytes) 000575 1749478687.396784 read 0x000139 thru 0x00734a (0x7212 bytes) 000576 1749478687.398309 MIRROR_EXTEND to 5 mirrors 000577 1749478687.505371 MIRROR_RESYNC 5 mirrors 000581 1749478687.623552 mapread 0x0018f5 thru 0x002ba0 (0x12ac bytes) 000582 1749478687.626229 read 0x013fb7 thru 0x0188f5 (0x493f bytes) 000584 1749478687.628260 read 0x003889 thru 0x005eda (0x2652 bytes) 000585 1749478687.630759 read 0x015443 thru 0x021b63 (0xc721 bytes) 000586 1749478687.630981 trunc from 0x021b64 to 0x01f624 (0x253f bytes) 000589 1749478687.648721 trunc from 0x01f625 to 0x02407b (0x4a57 bytes) 000590 1749478687.652876 write 0x03cc5a thru 0x03f89b (0x2c42 bytes) 000593 1749478687.654238 mapread 0x039e24 thru 0x03c2b0 (0x248d bytes) 000596 1749478687.658572 trunc from 0x03f89c to 0x004659 (0x3b242 bytes) 000599 1749478687.662544 mapread 0x000a70 thru 0x0037c4 (0x2d55 bytes) 000602 1749478687.664340 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000603 1749478687.712303 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000608 1749478687.757657 mapwrite 0x009222 thru 0x01658c (0xd36b bytes) 000611 1749478687.782550 read 0x008254 thru 0x01304b (0xadf8 bytes) 000612 1749478687.784032 MIRROR_RESYNC 3 mirrors 000614 1749478687.885470 mapread 0x002517 thru 0x00a724 (0x820e bytes) 000615 1749478687.890887 MIRROR_EXTEND to 4 mirrors 000616 1749478687.981417 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000617 1749478688.033309 MIRROR_RESYNC 3 mirrors 000618 1749478688.070132 trunc from 0x01658d to 0x002f71 (0x1361b bytes) 000621 1749478688.088589 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000622 1749478688.134702 MIRROR_RESYNC 2 mirrors 000623 1749478688.219045 read 0x002283 thru 0x002f71 (0xcef bytes) 000626 1749478688.219623 read 0x0011cd thru 0x002f71 (0x1da5 bytes) 000628 1749478688.221405 write 0x00ecf3 thru 0x011831 (0x2b3f bytes) 000629 1749478688.233592 write 0x0200d8 thru 0x02d5c1 (0xd4ea bytes) 000631 1749478688.234765 mapread 0x01a062 thru 0x01e96a (0x4909 bytes) 000634 1749478688.237881 trunc from 0x02d5c2 to 0x00bca9 (0x21918 bytes) 000635 1749478688.242439 write 0x003edf thru 0x00958d (0x56af bytes) 000637 1749478688.244342 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000639 1749478688.276474 write 0x029b4c thru 0x02e7b0 (0x4c65 bytes) 000640 1749478688.277291 mapwrite 0x01e58c thru 0x027556 (0x8fcb bytes) 000641 1749478688.296067 mapread 0x006503 thru 0x00b56d (0x506b bytes) 000642 1749478688.298893 read 0x01e14d thru 0x02a269 (0xc11d bytes) 000643 1749478688.299271 trunc from 0x02e7b1 to 0x02b242 (0x356e bytes) 000644 1749478688.303098 write 0x0121df thru 0x01fac3 (0xd8e5 bytes) 000647 1749478688.304512 write 0x00b49f thru 0x00e165 (0x2cc7 bytes) 000648 1749478688.305002 read 0x01ac98 thru 0x021e37 (0x71a0 bytes) 000650 1749478688.305661 write 0x03a71e thru 0x03ffff (0x58e2 bytes) 000652 1749478688.306713 mapwrite 0x002fc3 thru 0x011c87 (0xecc5 bytes) 000653 1749478688.325283 trunc from 0x040000 to 0x0397fa (0x6805 bytes) 000654 1749478688.329093 mapread 0x016ff3 thru 0x021815 (0xa823 bytes) 000657 1749478688.330453 read 0x01706b thru 0x0189b5 (0x194b bytes) 000660 1749478688.331490 mapread 0x017283 thru 0x01a860 (0x35de bytes) 000667 1749478688.334281 mapread 0x00d996 thru 0x01d3bb (0xfa26 bytes) 000669 1749478688.335224 mapwrite 0x024b2d thru 0x031a99 (0xcf6d bytes) 000670 1749478688.350770 write 0x012cb8 thru 0x01f32f (0xc678 bytes) 000674 1749478688.352846 write 0x030b15 thru 0x03bdf5 (0xb2e1 bytes) 000675 1749478688.353632 read 0x00b2b9 thru 0x01348b (0x81d3 bytes) 000677 1749478688.354212 write 0x03c8df thru 0x03ffff (0x3721 bytes) 000678 1749478688.354841 read 0x0108c6 thru 0x0183f5 (0x7b30 bytes) 000682 1749478688.356326 mapwrite 0x02242f thru 0x028145 (0x5d17 bytes) 000683 1749478688.370813 read 0x02ce79 thru 0x03acc2 (0xde4a bytes) 000689 1749478688.372913 write 0x0371ea thru 0x03bc33 (0x4a4a bytes) 000690 1749478688.373617 mapread 0x03619d thru 0x03dcdd (0x7b41 bytes) 000692 1749478688.374540 trunc from 0x040000 to 0x03b8a6 (0x4759 bytes) 000693 1749478688.378452 mapread 0x01b849 thru 0x02b6ee (0xfea6 bytes) 000694 1749478688.379038 trunc from 0x03b8a7 to 0x0245bb (0x172eb bytes) 000697 1749478688.382966 mapread 0x00fa9c thru 0x01c0b3 (0xc618 bytes) 000698 1749478688.384499 MIRROR_EXTEND to 2 mirrors 000699 1749478688.469316 trunc from 0x0245bc to 0x03ae24 (0x16869 bytes) 000700 1749478688.487005 read 0x016160 thru 0x0245a8 (0xe449 bytes) 000704 1749478688.490436 trunc from 0x03ae25 to 0x00b46f (0x2f9b5 bytes) 000705 1749478688.496646 MIRROR_EXTEND to 3 mirrors 000707 1749478688.581809 trunc from 0x00b470 to 0x002acd (0x89a2 bytes) 000708 1749478688.601537 write 0x0159d0 thru 0x017c68 (0x2299 bytes) 000709 1749478688.602310 mapread 0x006e03 thru 0x00dcef (0x6eed bytes) 000711 1749478688.607142 write 0x02066c thru 0x02be6e (0xb803 bytes) 000712 1749478688.608124 mapread 0x01113c thru 0x015f4a (0x4e0f bytes) 000713 1749478688.610535 trunc from 0x02be6f to 0x0091b1 (0x22cbd bytes) 000716 1749478688.615925 write 0x00ce43 thru 0x012261 (0x541f bytes) 000718 1749478688.617915 MIRROR_EXTEND to 4 mirrors 000720 1749478688.712931 read 0x00b976 thru 0x00f572 (0x3bfd bytes) 000721 1749478688.715279 write 0x009404 thru 0x017d72 (0xe96f bytes) 000722 1749478688.732677 mapwrite 0x000bdc thru 0x0057fc (0x4c21 bytes) 000724 1749478688.750553 mapread 0x000dfc thru 0x0032b7 (0x24bc bytes) 000725 1749478688.751114 trunc from 0x017d73 to 0x02457c (0xc80a bytes) 000727 1749478688.759275 mapwrite 0x000f33 thru 0x001917 (0x9e5 bytes) 000730 1749478688.772243 write 0x018c61 thru 0x01b8d2 (0x2c72 bytes) 000731 1749478688.774936 mapread 0x01ecef thru 0x02457c (0x588e bytes) 000732 1749478688.777965 MIRROR_EXTEND to 5 mirrors 000735 1749478688.876862 read 0x002343 thru 0x00d8f5 (0xb5b3 bytes) 000738 1749478688.881893 mapread 0x00e314 thru 0x0189cf (0xa6bc bytes) 000739 1749478688.889197 read 0x006603 thru 0x00b61f (0x501d bytes) 000741 1749478688.891385 write 0x03c822 thru 0x03ffff (0x37de bytes) 000742 1749478688.907180 write 0x006e64 thru 0x00e863 (0x7a00 bytes) 000743 1749478688.910736 mapread 0x031bc8 thru 0x03540c (0x3845 bytes) 000745 1749478688.911706 write 0x02c367 thru 0x03934e (0xcfe8 bytes) 000746 1749478688.912596 trunc from 0x040000 to 0x01a50f (0x25af0 bytes) 000747 1749478688.921124 read 0x00e3fa thru 0x014214 (0x5e1b bytes) 000750 1749478688.921871 write 0x01ff5e thru 0x023d46 (0x3de9 bytes) 000751 1749478688.923495 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000752 1749478688.973650 write 0x02d062 thru 0x02ed91 (0x1d30 bytes) 000753 1749478688.974281 write 0x0280c6 thru 0x0286e4 (0x61f bytes) 000757 1749478688.976517 mapwrite 0x002728 thru 0x0122dd (0xfbb6 bytes) 000759 1749478688.998840 trunc from 0x02ed92 to 0x03ba20 (0xcc8f bytes) 000760 1749478689.002721 write 0x037f12 thru 0x03bb0b (0x3bfa bytes) 000761 1749478689.005375 trunc from 0x03bb0c to 0x00f48b (0x2c680 bytes) 000763 1749478689.009274 mapread 0x008495 thru 0x00bad6 (0x3642 bytes) 000767 1749478689.010398 read 0x003ca1 thru 0x00f48b (0xb7eb bytes) 000768 1749478689.010754 mapwrite 0x02a3e5 thru 0x03312e (0x8d4a bytes) 000771 1749478689.032020 mapread 0x01bfae thru 0x01f2c9 (0x331c bytes) 000772 1749478689.034934 mapread 0x00a1a8 thru 0x011d79 (0x7bd2 bytes) 000774 1749478689.039246 MIRROR_RESYNC 4 mirrors 000775 1749478689.167945 trunc from 0x03312f to 0x0157f8 (0x1d936 bytes) 000776 1749478689.184147 write 0x0220a2 thru 0x0294c1 (0x7420 bytes) 000778 1749478689.185137 trunc from 0x0294c2 to 0x014712 (0x14daf bytes) 000779 1749478689.189553 mapread 0x0043c4 thru 0x004898 (0x4d5 bytes) 000780 1749478689.191444 mapwrite 0x017976 thru 0x023d0f (0xc39a bytes) 000781 1749478689.218644 MIRROR_EXTEND to 5 mirrors 000785 1749478689.313554 mapwrite 0x02e023 thru 0x02fe9f (0x1e7d bytes) 000788 1749478689.351334 trunc from 0x02fea0 to 0x01c6c2 (0x137dd bytes) 000789 1749478689.356674 trunc from 0x01c6c3 to 0x00fb65 (0xcb5d bytes) 000793 1749478689.362496 write 0x01600d thru 0x016aef (0xae3 bytes) 000794 1749478689.363238 mapread 0x013054 thru 0x0141b1 (0x115e bytes) 000796 1749478689.367818 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000797 1749478689.416236 trunc from 0x016af0 to 0x03488c (0x1dd9d bytes) 000798 1749478689.420008 trunc from 0x03488d to 0x002efe (0x3198e bytes) 000799 1749478689.423872 trunc from 0x002eff to 0x02d767 (0x2a869 bytes) 000800 1749478689.427735 trunc from 0x02d768 to 0x0351ba (0x7a53 bytes) 000802 1749478689.431546 mapread 0x01d6c5 thru 0x02145e (0x3d9a bytes) 000804 1749478689.436393 mapwrite 0x025916 thru 0x033d1c (0xe407 bytes) 000805 1749478689.459429 write 0x0016c3 thru 0x00822a (0x6b68 bytes) 000806 1749478689.463007 trunc from 0x0351bb to 0x01943f (0x1bd7b bytes) 000809 1749478689.468990 mapwrite 0x028a9b thru 0x02c0b1 (0x3617 bytes) 000810 1749478689.491490 mapwrite 0x020083 thru 0x021aa6 (0x1a24 bytes) 000812 1749478689.509118 trunc from 0x02c0b2 to 0x01cd0c (0xf3a5 bytes) 000813 1749478689.514076 mapread 0x00feda thru 0x01853a (0x8661 bytes) 000814 1749478689.516853 mapread 0x0092cc thru 0x0146f1 (0xb426 bytes) 000815 1749478689.517486 mapwrite 0x02a1e1 thru 0x0390b8 (0xeed8 bytes) 000817 1749478689.541197 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000819 1749478689.585530 read 0x00b4d5 thru 0x015054 (0x9b80 bytes) 000821 1749478689.588470 mapread 0x02de08 thru 0x038d5b (0xaf54 bytes) 000822 1749478689.596064 mapwrite 0x03de84 thru 0x03ffff (0x217c bytes) 000824 1749478689.615332 mapread 0x02ede4 thru 0x035307 (0x6524 bytes) 000825 1749478689.615994 trunc from 0x040000 to 0x00e5fe (0x31a01 bytes) 000826 1749478689.619595 trunc from 0x00e5ff to 0x032c63 (0x24665 bytes) 000827 1749478689.622848 mapwrite 0x01bc6c thru 0x0288ee (0xcc83 bytes) 000828 1749478689.643978 read 0x011e33 thru 0x018dec (0x6fba bytes) 000829 1749478689.647247 trunc from 0x032c64 to 0x002649 (0x3061a bytes) 000831 1749478689.651232 trunc from 0x00264a to 0x023046 (0x209fd bytes) 000833 1749478689.654698 trunc from 0x023047 to 0x002302 (0x20d44 bytes) 000834 1749478689.658025 mapwrite 0x03cce5 thru 0x03e60a (0x1926 bytes) 000835 1749478689.675351 write 0x03ce40 thru 0x03ffff (0x31c0 bytes) 000837 1749478689.676180 read 0x018f5b thru 0x022111 (0x91b7 bytes) 000838 1749478689.679471 read 0x006745 thru 0x013a2d (0xd2e9 bytes) 000841 1749478689.680321 mapwrite 0x00e753 thru 0x012d75 (0x4623 bytes) 000844 1749478689.693694 mapwrite 0x03a42b thru 0x03ffff (0x5bd5 bytes) 000846 1749478689.707350 mapread 0x00b66f thru 0x0155ce (0x9f60 bytes) 000847 1749478689.707876 mapwrite 0x002ddf thru 0x0122e4 (0xf506 bytes) 000848 1749478689.722967 trunc from 0x040000 to 0x038adf (0x7520 bytes) 000849 1749478689.728572 MIRROR_RESYNC 3 mirrors 000852 1749478689.836371 mapwrite 0x029940 thru 0x037ed9 (0xe59a bytes) 000853 1749478689.881254 mapread 0x00fa9f thru 0x018c43 (0x91a5 bytes) 000854 1749478689.885542 write 0x03c136 thru 0x03ef75 (0x2e40 bytes) 000855 1749478689.886104 read 0x0186ad thru 0x01f4b8 (0x6e0c bytes) 000857 1749478689.886660 mapwrite 0x03dbdb thru 0x03ffff (0x2425 bytes) 000858 1749478689.908530 mapread 0x03106f thru 0x03381b (0x27ad bytes) 000860 1749478689.910331 MIRROR_RESYNC 3 mirrors 000861 1749478690.018082 write 0x033238 thru 0x03b653 (0x841c bytes) 000862 1749478690.032960 write 0x01b806 thru 0x01f02f (0x382a bytes) 000863 1749478690.035471 write 0x0306f4 thru 0x03c685 (0xbf92 bytes) 000865 1749478690.040855 MIRROR_EXTEND to 4 mirrors 000866 1749478690.139021 write 0x02082a thru 0x02190a (0x10e1 bytes) 000867 1749478690.155119 write 0x02bafc thru 0x02f3e5 (0x38ea bytes) 000868 1749478690.157396 read 0x0314b2 thru 0x03a2ed (0x8e3c bytes) 000874 1749478690.160632 trunc from 0x040000 to 0x0174a3 (0x28b5c bytes) 000875 1749478690.166133 MIRROR_RESYNC 4 mirrors 000878 1749478690.289621 MIRROR_EXTEND to 5 mirrors 000879 1749478690.375956 mapread 0x0062ca thru 0x00da30 (0x7767 bytes) 000881 1749478690.380289 write 0x017b4f thru 0x01914e (0x1600 bytes) 000884 1749478690.396206 read 0x004886 thru 0x0075e0 (0x2d5b bytes) 000889 1749478690.398655 trunc from 0x01914f to 0x019cbb (0xb6d bytes) 000891 1749478690.406341 write 0x023a01 thru 0x027894 (0x3e94 bytes) 000892 1749478690.407033 trunc from 0x027895 to 0x03e46a (0x16bd6 bytes) 000895 1749478690.411942 MIRROR_RESYNC 5 mirrors 000896 1749478690.561017 MIRROR_RESYNC 5 mirrors 000897 1749478690.601516 mapwrite 0x0293ae thru 0x033d34 (0xa987 bytes) 000900 1749478690.646892 mapread 0x0244c0 thru 0x033161 (0xeca2 bytes) 000901 1749478690.647595 read 0x00743e thru 0x011424 (0x9fe7 bytes) 000902 1749478690.649840 mapread 0x00acab thru 0x00e225 (0x357b bytes) 000903 1749478690.650264 mapread 0x0195ee thru 0x01b115 (0x1b28 bytes) 000907 1749478690.653789 read 0x016ca0 thru 0x01f6a7 (0x8a08 bytes) 000908 1749478690.654183 mapwrite 0x02bb48 thru 0x031ed6 (0x638f bytes) 000909 1749478690.666007 read 0x01e7f3 thru 0x023da4 (0x55b2 bytes) 000910 1749478690.666356 read 0x00c8f9 thru 0x01196f (0x5077 bytes) 000911 1749478690.666769 trunc from 0x03e46b to 0x001445 (0x3d025 bytes) 000913 1749478690.675856 mapread 0x000b0b thru 0x001445 (0x93b bytes) 000914 1749478690.677727 trunc from 0x001446 to 0x016854 (0x1540f bytes) 000917 1749478690.681735 trunc from 0x016855 to 0x034aec (0x1e298 bytes) 000919 1749478690.685396 trunc from 0x034aed to 0x013baa (0x20f42 bytes) 000920 1749478690.689178 mapwrite 0x008e12 thru 0x00928d (0x47c bytes) 000923 1749478690.705691 mapwrite 0x017505 thru 0x023ed9 (0xc9d5 bytes) 000924 1749478690.726523 mapread 0x002ba1 thru 0x00a755 (0x7bb5 bytes) 000925 1749478690.728582 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000926 1749478690.771850 mapread 0x0102fd thru 0x013877 (0x357b bytes) 000927 1749478690.776338 mapwrite 0x032e7f thru 0x039b61 (0x6ce3 bytes) 000930 1749478690.793958 read 0x027344 thru 0x02cd4e (0x5a0b bytes) 000931 1749478690.796010 mapwrite 0x035113 thru 0x03ffff (0xaeed bytes) 000932 1749478690.813648 mapwrite 0x00e706 thru 0x01d5fb (0xeef6 bytes) 000933 1749478690.827760 trunc from 0x040000 to 0x02fadd (0x10522 bytes) 000934 1749478690.831484 write 0x02a8f0 thru 0x032fbb (0x86cc bytes) 000936 1749478690.832519 mapwrite 0x01ed9d thru 0x021824 (0x2a88 bytes) 000937 1749478690.844762 write 0x010737 thru 0x013077 (0x2941 bytes) 000939 1749478690.845442 mapwrite 0x01c96e thru 0x029edd (0xd570 bytes) 000940 1749478690.861862 MIRROR_EXTEND to 5 mirrors 000941 1749478690.954785 write 0x032ec0 thru 0x03a4bc (0x75fd bytes) 000943 1749478690.971487 mapwrite 0x011c8d thru 0x020ee9 (0xf25d bytes) 000944 1749478690.992400 write 0x030539 thru 0x039ae4 (0x95ac bytes) 000947 1749478690.995464 write 0x0226c4 thru 0x026d6e (0x46ab bytes) 000949 1749478690.997567 mapwrite 0x00f1cb thru 0x01e78a (0xf5c0 bytes) 000952 1749478691.013544 write 0x03a00e thru 0x03ffff (0x5ff2 bytes) 000953 1749478691.015809 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000956 1749478691.061708 trunc from 0x040000 to 0x009516 (0x36ae9 bytes) 000957 1749478691.069032 trunc from 0x009517 to 0x02db9e (0x24688 bytes) 000958 1749478691.073395 MIRROR_RESYNC 4 mirrors 000959 1749478691.202944 mapread 0x01c1e1 thru 0x028508 (0xc328 bytes) 000960 1749478691.209036 mapwrite 0x007e81 thru 0x00bc96 (0x3e16 bytes) 000963 1749478691.249662 trunc from 0x02db9f to 0x027a5a (0x6144 bytes) 000964 1749478691.255141 MIRROR_EXTEND to 5 mirrors 000967 1749478691.357108 mapwrite 0x0072a3 thru 0x01056b (0x92c9 bytes) 000968 1749478691.402823 read 0x012eac thru 0x01f328 (0xc47d bytes) 000969 1749478691.405377 mapwrite 0x00f2ba thru 0x00fa92 (0x7d9 bytes) 000970 1749478691.415122 mapread 0x01012a thru 0x01a494 (0xa36b bytes) 000971 1749478691.416517 mapwrite 0x02d994 thru 0x0314f1 (0x3b5e bytes) 000973 1749478691.434301 mapwrite 0x018a0e thru 0x01a0be (0x16b1 bytes) 000974 1749478691.444657 MIRROR_RESYNC 5 mirrors 000976 1749478691.584042 trunc from 0x0314f2 to 0x0291b4 (0x833d bytes) 000978 1749478691.603250 write 0x011f26 thru 0x01f258 (0xd333 bytes) 000981 1749478691.606350 write 0x01f17b thru 0x025820 (0x66a6 bytes) 000982 1749478691.607858 trunc from 0x0291b5 to 0x02d59d (0x43e9 bytes) 000983 1749478691.612028 read 0x01933b thru 0x024bee (0xb8b4 bytes) 000987 1749478691.614303 MIRROR_EXTEND to 6 mirrors 000988 1749478691.703542 read 0x0212d2 thru 0x02a95c (0x968b bytes) 000989 1749478691.705896 write 0x00101d thru 0x007f07 (0x6eeb bytes) 000990 1749478691.723354 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9b1:0x0]: Invalid argument (22) 000991 1749478691.772434 trunc from 0x02d59e to 0x02c291 (0x130c bytes) 000992 1749478691.780422 MIRROR_EXTEND to 6 mirrors 000993 1749478691.888736 mapwrite 0x03f10b thru 0x03ffff (0xef5 bytes) 000996 1749478691.919373 mapread 0x03c623 thru 0x03f213 (0x2bf1 bytes) 000999 1749478691.923366 trunc from 0x040000 to 0x014b9d (0x2b462 bytes) All operations completed A-OK! PASS 70a (11s) == sanity-flr test 71: check mirror extend parent fid ==== 10:18:13 (1749478693) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 10:18:13 (1749478693) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7704480 .. Extending file size to 1842144 .. done Extending file size to 7842720 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8562880 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8517600 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8555744 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9090880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8972064 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9005760 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9206752 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9377984 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 31311 31312 31314 31315 31318 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:19:17 (1749478757) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1520704 .. Extending file size to 3599072 .. Extending file size to 7556032 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4534912 .. failed Extending file size to 8231264 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7855104 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8685120 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8701696 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9287584 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8828128 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9173056 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 3279 3280 3282 3283 3286 failed 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:20:21 (1749478821) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5376 2202752 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 19456 3749888 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5376 2202752 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 19456 3749888 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.030351 s, 69.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0817516 s, 25.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:20:25 (1749478825) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0237912 s, 88.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 10:20:27 (1749478827) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x9fa:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe23:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe23:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe24:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xe28:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:20:28 (1749478828) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0245202 s, 85.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe25:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe29:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:20:30 (1749478830) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0313516 s, 66.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe26:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe27:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe26:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe27:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2c:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:20:32 (1749478832) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0211469 s, 99.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe28:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe28:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2e:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:20:54 (1749478854) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154871 s, 67.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe29:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe29:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2a:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:21:39 (1749478899) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0318654 s, 65.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe31:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe31:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (41s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:22:20 (1749478940) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. Creating new pool test_204f oleg308-server: Pool lustre.test_204f created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0202961 s, 103 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe32:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe32:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg308-server: Pool lustre.test_204f destroyed PASS 204f (55s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:23:15 (1749478995) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:23:16 (1749478996) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:23:18 (1749478998) oleg308-server: Pool lustre.test_206 created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg308-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:23:31 (1749479011) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183703 s, 57.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00706649 s, 148 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xd225a034 Mirror 3: 0xd225a034 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:23:33 (1749479013) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:23:33 (1749479013) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:23:34 (1749479014) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0136297 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf10000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x660000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x660000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x660000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x6b0000 Read this (4 bytes): BBBB 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 209a (14s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:23:48 (1749479028) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00244316 s, 3.4 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:23:49 (1749479029) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00909041 s, 115 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa1f lmm_fid: [0x200000404:0xa1f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa1f lmm_fid: [0x200000404:0xa1f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3c:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836511 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-09 10:23:50.000000000 -0400 Modify: 2025-06-09 10:23:50.000000000 -0400 Change: 2025-06-09 10:23:50.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:23:51 (1749479031) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:23:52 (1749479032) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.119963 s, 341 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0145551 s, 281 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 9 10:23 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa21:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 9 10:23 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 674 sec ============ 10:23:54 (1749479034) === sanity-flr: start cleanup 10:23:54 (1749479034) === === sanity-flr: finish cleanup 10:23:55 (1749479035) ===