-----============= acceptance-small: sanity-flr ============----- Wed May 7 09:45:39 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg349-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:45:44 (1746625544) === oleg349-client.virtnet: executing check_config_client /mnt/lustre oleg349-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg349-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6180800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6180800.idle_timeout=debug disable quota as required oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:45:51 (1746625551) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7024] == sanity-flr test 0a: lfs mirror create with -N option == 09:45:51 (1746625551) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:45:55 (1746625555) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:45:55 (1746625555) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:45:56 (1746625556) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:46:00 (1746625560) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:46:01 (1746625561) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:46:02 (1746625562) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b6180800.stats=clear osc.lustre-OST0001-osc-ffff8800b6180800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:46:04 (1746625564) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:46:06 (1746625566) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:46:08 (1746625568) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (0s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:46:08 (1746625568) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 09:46:10 (1746625570) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:46:12 (1746625572) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:46:13 (1746625573) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:46:14 (1746625574) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:46:16 (1746625576) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:46:18 (1746625578) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.421023 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0447669 s, 23.4 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:46:20 (1746625580) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.114882 s, 164 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:46:23 (1746625583) Stopping /mnt/lustre-ost1 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6180800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6180800.ost_server_uuid in DISCONN state after 1 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg349-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6180800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6180800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg349-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (19s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:46:42 (1746625602) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.118476 s, 62.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6180800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6180800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg349-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:46:51 (1746625611) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6180800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6180800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg349-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6180800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6180800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg349-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:47:11 (1746625631) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.825741 s, 63.5 MB/s -rw-r--r-- 1 root root 52428800 May 7 09:47 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.269s user 0m0.000s sys 0m0.118s Stopping /mnt/lustre-ost1 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6180800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6180800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.246s user 0m0.000s sys 0m0.110s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg349-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6180800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6180800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost1)... real 0m0.222s user 0m0.000s sys 0m0.095s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg349-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6180800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6180800.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:47:33 (1746625653) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:47:34 (1746625654) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:47:34 (1746625654) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 09:47:35 (1746625655) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 09:47:37 (1746625657) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.0872235 s, 144 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:47:39 (1746625659) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0079961 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00752632 s, 139 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:47:41 (1746625661) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:47:43 (1746625663) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000627401 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000541609 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000621585 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00051681 s, 2.0 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 09:47:45 (1746625665) create test files with size 15749926 .. 17b448248632cab7ccd5d87a13b80b7b - 924bf36ef5c18d51b28e97dd5973b1ac - edf7bd5fc62189a58138030964827788 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg349-client.virtnet /mnt/lustre (opts:) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 09:47:51 (1746625671) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:47:58 (1746625678) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:48:01 (1746625681) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399607 s, 26.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195175 s, 53.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 09:48:04 (1746625684) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0644123 s, 32.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.178581 s, 23.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.111872 s, 18.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 09:48:09 (1746625689) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:48:10 (1746625690) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:48:12 (1746625692) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139729 s, 75.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 09:48:14 (1746625694) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:48:15 (1746625695) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:48:17 (1746625697) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0718566 s, 146 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg349-client.virtnet /mnt/lustre (opts:) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (21s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:48:38 (1746625718) llite.lustre-ffff880136464000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0818667 s, 128 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg349-client.virtnet /mnt/lustre (opts:) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a57c6800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:48:42 (1746625722) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:48:54 (1746625734) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:49:04 (1746625744) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 09:49:05 (1746625745) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0664464 s, 47.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0522562 s, 60.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 42d1173b71dd653988e8716de47b6568 - mirror 1 checksum: 6e07b529626c10d776ad04dec79311ab - mirror 2 checksum: 42d1173b71dd653988e8716de47b6568 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 7 09:49:06 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 7 09:49:06 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 42d1173b71dd653988e8716de47b6568 - mirror 1 checksum: 6e07b529626c10d776ad04dec79311ab - mirror 2 checksum: 6e07b529626c10d776ad04dec79311ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 7 09:49:06 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 7 09:49:06 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:49:09 (1746625749) Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 09:49:10 (1746625750) shut down facet: mds1 facet_host: oleg349-server facet_failover_host: oleg349-server Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 09:49:23 (1746625763) targets are mounted 09:49:23 (1746625763) facet_failover done oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107731 s, 97.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (20s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:49:29 (1746625769) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:49:30 (1746625770) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:49:30 (1746625770) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00485312 s, 3.4 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:49:33 (1746625773) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (0s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:49:33 (1746625773) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:49:34 (1746625774) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:49:38 (1746625778) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:49:57 (1746625797) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:50:14 (1746625814) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:50:32 (1746625832) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 09:50:34 (1746625834) mirror create pid 22492 mirror split pid 22493 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22492 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 22493 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 09:51:36 (1746625896) Seed set to 5896 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1746625896.729292 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0xbaf3 000002 1746625896.838610 trunc from 00000000 to 0x00baf2 (0xbaf3 bytes) 000004 1746625896.875437 mapread 0x003f61 thru 0x00520d (0x12ad bytes) 000007 1746625896.886989 write 0x01c084 thru 0x02aac7 (0xea44 bytes) 000009 1746625896.888727 mapread 0x022a49 thru 0x02aac7 (0x807f bytes) truncating to largest ever: 0x245c9 000010 1746625896.889362 trunc from 0x02aac8 to 0x0245c8 (0x64ff bytes) 000011 1746625896.902336 read 0x00b7e4 thru 0x00e208 (0x2a25 bytes) 000012 1746625896.904763 write 0x01edac thru 0x0209f5 (0x1c4a bytes) 000013 1746625896.905431 mapwrite 0x01a896 thru 0x01b437 (0xba2 bytes) 000014 1746625896.942100 mapwrite 0x0267ea thru 0x02d1e6 (0x69fd bytes) 000015 1746625896.977553 trunc from 0x02d1e7 to 0x01e361 (0xee85 bytes) 000018 1746625896.984484 mapread 0x0120b7 thru 0x01bf35 (0x9e7f bytes) 000019 1746625896.988156 read 0x016596 thru 0x01e361 (0x7dcc bytes) 000021 1746625896.991280 MIRROR_RESYNC 2 mirrors 000027 1746625897.165996 trunc from 0x01e362 to 0x019632 (0x4d2f bytes) 000028 1746625897.191900 mapwrite 0x028bc4 thru 0x028ed1 (0x30e bytes) truncating to largest ever: 0x3c8b5 000029 1746625897.218457 trunc from 0x028ed2 to 0x03c8b4 (0x139e3 bytes) 000030 1746625897.223068 mapwrite 0x03b2fc thru 0x03ffff (0x4d04 bytes) 000031 1746625897.251949 mapwrite 0x0360fa thru 0x038a73 (0x297a bytes) 000035 1746625897.270402 write 0x03314d thru 0x03ffff (0xceb3 bytes) 000036 1746625897.275570 mapread 0x014609 thru 0x021e0a (0xd802 bytes) 000037 1746625897.283773 trunc from 0x040000 to 0x0241a3 (0x1be5c bytes) 000038 1746625897.288940 read 0x01b733 thru 0x0241a3 (0x8a71 bytes) 000039 1746625897.289473 mapread 0x00abe0 thru 0x00b646 (0xa67 bytes) 000043 1746625897.294189 read 0x022944 thru 0x0241a3 (0x1860 bytes) 000046 1746625897.295253 mapwrite 0x03e812 thru 0x03ffff (0x17ee bytes) 000047 1746625897.326055 mapread 0x00683d thru 0x0073fb (0xbbf bytes) 000050 1746625897.327533 trunc from 0x040000 to 0x028f99 (0x17066 bytes) 000051 1746625897.333191 mapwrite 0x032f4e thru 0x039543 (0x65f6 bytes) 000053 1746625897.370013 trunc from 0x039544 to 0x033363 (0x61e0 bytes) 000055 1746625897.377068 mapwrite 0x021eca thru 0x02f48d (0xd5c4 bytes) 000059 1746625897.421876 MIRROR_EXTEND to 3 mirrors 000061 1746625897.566259 mapwrite 0x00a716 thru 0x012ac9 (0x83b4 bytes) 000062 1746625897.635301 mapwrite 0x001645 thru 0x003873 (0x222f bytes) 000063 1746625897.657256 write 0x014697 thru 0x022428 (0xdd92 bytes) 000064 1746625897.662316 read 0x029c5f thru 0x032dc5 (0x9167 bytes) 000066 1746625897.666665 mapread 0x0131c4 thru 0x01ffd7 (0xce14 bytes) 000071 1746625897.670479 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000072 1746625897.737305 mapwrite 0x00dd6f thru 0x0137fa (0x5a8c bytes) 000073 1746625897.765125 mapwrite 0x0081ca thru 0x00ff85 (0x7dbc bytes) 000074 1746625897.788829 MIRROR_EXTEND to 3 mirrors 000077 1746625897.954039 trunc from 0x033364 to 0x033bec (0x889 bytes) 000078 1746625897.995489 MIRROR_RESYNC 3 mirrors 000079 1746625898.177682 trunc from 0x033bed to 0x021394 (0x12858 bytes) 000080 1746625898.211410 read 0x00ce9a thru 0x014810 (0x7977 bytes) 000082 1746625898.216426 mapread 0x00e413 thru 0x01ab17 (0xc705 bytes) 000083 1746625898.225020 write 0x014b57 thru 0x01bd87 (0x7231 bytes) 000084 1746625898.225781 read 0x000c45 thru 0x007248 (0x6604 bytes) 000088 1746625898.232525 read 0x010ca2 thru 0x011e6f (0x11ce bytes) 000089 1746625898.233125 write 0x031036 thru 0x03afa1 (0x9f6c bytes) 000090 1746625898.233994 write 0x026ebb thru 0x027ddf (0xf25 bytes) 000092 1746625898.243694 MIRROR_EXTEND to 4 mirrors 000093 1746625898.417079 write 0x0123b8 thru 0x01bdeb (0x9a34 bytes) 000096 1746625898.443083 read 0x00436d thru 0x00bdf4 (0x7a88 bytes) 000104 1746625898.448711 mapread 0x01f710 thru 0x025be7 (0x64d8 bytes) 000105 1746625898.453259 mapread 0x02a799 thru 0x039910 (0xf178 bytes) 000106 1746625898.459503 mapwrite 0x039a09 thru 0x03ffff (0x65f7 bytes) 000108 1746625898.508608 write 0x00d7a4 thru 0x011c90 (0x44ed bytes) 000112 1746625898.513307 mapread 0x012441 thru 0x01c07c (0x9c3c bytes) 000114 1746625898.517119 mapwrite 0x031147 thru 0x03d392 (0xc24c bytes) 000115 1746625898.538041 read 0x0222f7 thru 0x02718d (0x4e97 bytes) 000116 1746625898.538496 write 0x03903e thru 0x03ffff (0x6fc2 bytes) 000117 1746625898.539224 mapwrite 0x013486 thru 0x01c976 (0x94f1 bytes) 000118 1746625898.558878 mapread 0x01f7ae thru 0x028e02 (0x9655 bytes) 000119 1746625898.559550 mapread 0x036b8f thru 0x039882 (0x2cf4 bytes) 000121 1746625898.562751 MIRROR_RESYNC 4 mirrors 000122 1746625898.787340 trunc from 0x040000 to 0x030b79 (0xf486 bytes) 000123 1746625898.822949 MIRROR_EXTEND to 5 mirrors 000124 1746625898.999754 mapread 0x00fb50 thru 0x0119dc (0x1e8d bytes) 000128 1746625899.017163 write 0x0239c5 thru 0x02b081 (0x76bd bytes) 000129 1746625899.039560 write 0x01fd3a thru 0x02e1f0 (0xe4b7 bytes) 000131 1746625899.043333 trunc from 0x030b7a to 0x02716f (0x9a0a bytes) 000134 1746625899.059091 MIRROR_EXTEND to 6 mirrors 000135 1746625899.260451 mapread 0x003d8f thru 0x0109d8 (0xcc4a bytes) 000136 1746625899.278862 read 0x012844 thru 0x013f1d (0x16da bytes) 000139 1746625899.291055 trunc from 0x027170 to 0x03c468 (0x152f9 bytes) 000140 1746625899.318998 mapread 0x023a0f thru 0x02dff0 (0xa5e2 bytes) 000141 1746625899.329385 mapread 0x0135f1 thru 0x0155ba (0x1fca bytes) 000142 1746625899.334609 MIRROR_RESYNC 6 mirrors 000143 1746625899.655572 read 0x03965d thru 0x03c468 (0x2e0c bytes) 000144 1746625899.658903 mapwrite 0x02d231 thru 0x031b6c (0x493c bytes) 000145 1746625899.733022 write 0x00ae4f thru 0x0153e8 (0xa59a bytes) 000147 1746625899.741663 mapread 0x00323c thru 0x0115df (0xe3a4 bytes) 000148 1746625899.742581 mapwrite 0x003d1b thru 0x00e445 (0xa72b bytes) 000149 1746625899.774565 trunc from 0x03c469 to 0x030d02 (0xb766 bytes) 000150 1746625899.793010 trunc from 0x030d03 to 0x02890f (0x83f3 bytes) 000152 1746625899.801029 mapwrite 0x030f94 thru 0x03e03c (0xd0a9 bytes) 000153 1746625899.857313 write 0x02b30f thru 0x031a5b (0x674d bytes) 000156 1746625899.861845 MIRROR_RESYNC 6 mirrors 000158 1746625900.244962 mapwrite 0x02cff1 thru 0x03ad5b (0xdd6b bytes) 000159 1746625900.339007 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000160 1746625900.424567 mapwrite 0x025eb8 thru 0x02cf44 (0x708d bytes) 000161 1746625900.473311 read 0x01de7b thru 0x0257d1 (0x7957 bytes) 000164 1746625900.479443 mapwrite 0x011aab thru 0x016b43 (0x5099 bytes) 000169 1746625900.520915 mapwrite 0x0147cc thru 0x01ae4c (0x6681 bytes) 000170 1746625900.547431 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000171 1746625900.628144 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000173 1746625900.718868 mapread 0x0064ae thru 0x0095f9 (0x314c bytes) 000175 1746625900.731405 trunc from 0x03e03d to 0x027f3a (0x16102 bytes) 000176 1746625900.736780 read 0x023507 thru 0x027f3a (0x4a34 bytes) 000178 1746625900.742398 write 0x0250ac thru 0x02a708 (0x565d bytes) 000179 1746625900.743247 read 0x010bdc thru 0x01e2b0 (0xd6d5 bytes) 000181 1746625900.749904 write 0x036f61 thru 0x03be2c (0x4ecc bytes) 000182 1746625900.750737 mapread 0x02f88d thru 0x035699 (0x5e0d bytes) 000184 1746625900.754745 write 0x03a5b5 thru 0x03ffff (0x5a4b bytes) 000186 1746625900.755898 write 0x02e736 thru 0x03bd5d (0xd628 bytes) 000187 1746625900.756810 write 0x0040be thru 0x0101f6 (0xc139 bytes) 000188 1746625900.757736 mapread 0x03d095 thru 0x03ffff (0x2f6b bytes) 000193 1746625900.759648 read 0x00cff8 thru 0x01b754 (0xe75d bytes) 000196 1746625900.760690 write 0x01560a thru 0x01ab9b (0x5592 bytes) 000198 1746625900.761653 read 0x006e14 thru 0x00b612 (0x47ff bytes) 000199 1746625900.761958 write 0x006a63 thru 0x00c997 (0x5f35 bytes) 000200 1746625900.762661 write 0x038e3f thru 0x03a6cf (0x1891 bytes) 000202 1746625900.763697 read 0x00bd0a thru 0x015587 (0x987e bytes) 000204 1746625900.771297 mapread 0x0313a1 thru 0x032091 (0xcf1 bytes) 000208 1746625900.773467 write 0x038b6b thru 0x039cfa (0x1190 bytes) 000209 1746625900.774065 mapread 0x01c8f2 thru 0x0292d7 (0xc9e6 bytes) 000210 1746625900.774675 read 0x023df0 thru 0x02e250 (0xa461 bytes) 000213 1746625900.775661 mapwrite 0x01c61e thru 0x0219d6 (0x53b9 bytes) 000214 1746625900.807117 MIRROR_RESYNC 3 mirrors 000216 1746625901.047693 read 0x02018d thru 0x027772 (0x75e6 bytes) 000217 1746625901.053649 MIRROR_RESYNC 3 mirrors 000219 1746625901.111751 read 0x00a8c1 thru 0x013e74 (0x95b4 bytes) 000220 1746625901.118808 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000221 1746625901.202083 write 0x021e95 thru 0x023f0a (0x2076 bytes) 000223 1746625901.223468 read 0x00672d thru 0x00a80a (0x40de bytes) 000224 1746625901.228801 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000225 1746625901.287985 mapread 0x021744 thru 0x0244f6 (0x2db3 bytes) 000227 1746625901.298652 mapwrite 0x015fb5 thru 0x018bba (0x2c06 bytes) 000231 1746625901.322781 read 0x03ccc3 thru 0x03ffff (0x333d bytes) 000234 1746625901.326465 trunc from 0x040000 to 0x0042d5 (0x3bd2a bytes) 000236 1746625901.340663 trunc from 0x0042d6 to 0x01a1a1 (0x15ecc bytes) 000238 1746625901.348383 read 0x00c113 thru 0x018617 (0xc505 bytes) 000241 1746625901.353616 write 0x002070 thru 0x00808f (0x6020 bytes) 000244 1746625901.354940 trunc from 0x01a1a2 to 0x02c4dc (0x1233b bytes) 000247 1746625901.361740 mapread 0x01b75f thru 0x025b34 (0xa3d6 bytes) 000248 1746625901.367231 write 0x018911 thru 0x01c48c (0x3b7c bytes) 000249 1746625901.368040 mapread 0x012aa8 thru 0x01579e (0x2cf7 bytes) 000250 1746625901.368623 read 0x024aa9 thru 0x02c4dc (0x7a34 bytes) 000251 1746625901.369058 trunc from 0x02c4dd to 0x02f0f2 (0x2c16 bytes) 000252 1746625901.375194 mapwrite 0x00d92c thru 0x01a4cb (0xcba0 bytes) 000253 1746625901.396465 write 0x018577 thru 0x01dc00 (0x568a bytes) 000254 1746625901.397194 write 0x03278d thru 0x033f96 (0x180a bytes) 000255 1746625901.397768 trunc from 0x033f97 to 0x035c1d (0x1c87 bytes) 000256 1746625901.404660 read 0x00781c thru 0x010a8f (0x9274 bytes) 000260 1746625901.408325 MIRROR_EXTEND to 2 mirrors 000261 1746625901.571750 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000265 1746625901.617244 read 0x0161a1 thru 0x02573a (0xf59a bytes) 000267 1746625901.622532 mapread 0x00f3b9 thru 0x01899f (0x95e7 bytes) 000269 1746625901.629443 mapwrite 0x01de4f thru 0x02bed8 (0xe08a bytes) 000272 1746625901.672558 trunc from 0x035c1e to 0x0379be (0x1da1 bytes) 000274 1746625901.683116 read 0x0319a9 thru 0x0379be (0x6016 bytes) 000276 1746625901.688740 mapwrite 0x03f66a thru 0x03fa60 (0x3f7 bytes) 000278 1746625901.723350 mapread 0x01ab3b thru 0x01b0d7 (0x59d bytes) 000281 1746625901.725296 write 0x01d37f thru 0x022fbd (0x5c3f bytes) 000282 1746625901.725950 read 0x01930f thru 0x024f69 (0xbc5b bytes) 000283 1746625901.726420 read 0x034039 thru 0x03db33 (0x9afb bytes) 000284 1746625901.726839 write 0x022e17 thru 0x027cb9 (0x4ea3 bytes) 000286 1746625901.727834 read 0x02e4fb thru 0x036a38 (0x853e bytes) 000289 1746625901.728810 mapread 0x03ee4c thru 0x03fa60 (0xc15 bytes) 000290 1746625901.729528 trunc from 0x03fa61 to 0x01727f (0x287e1 bytes) 000292 1746625901.736180 read 0x0138cb thru 0x01727f (0x39b5 bytes) 000295 1746625901.737191 mapread 0x014a7f thru 0x01727f (0x2801 bytes) 000297 1746625901.737993 read 0x011891 thru 0x01727f (0x59ef bytes) 000298 1746625901.740297 MIRROR_EXTEND to 2 mirrors 000299 1746625901.908257 trunc from 0x017280 to 0x009223 (0xe05c bytes) 000300 1746625901.948126 mapread 0x00888e thru 0x009223 (0x996 bytes) 000307 1746625901.955941 mapwrite 0x01957b thru 0x02104d (0x7ad3 bytes) 000308 1746625901.992325 trunc from 0x02104e to 0x0006f6 (0x20957 bytes) 000311 1746625901.998654 mapwrite 0x013bc3 thru 0x0235bb (0xf9f9 bytes) 000312 1746625902.032505 write 0x03df08 thru 0x03ffff (0x20f8 bytes) 000313 1746625902.033343 trunc from 0x040000 to 0x02d35f (0x12ca0 bytes) 000314 1746625902.038858 write 0x01a20e thru 0x027e11 (0xdc04 bytes) 000315 1746625902.042608 mapwrite 0x01657f thru 0x016f06 (0x988 bytes) 000317 1746625902.062309 mapwrite 0x00ad36 thru 0x0156e1 (0xa9ac bytes) 000321 1746625902.082289 mapwrite 0x005ca1 thru 0x01083c (0xab9c bytes) 000323 1746625902.107649 MIRROR_RESYNC 2 mirrors 000324 1746625902.246195 mapread 0x01852c thru 0x01a5c1 (0x2096 bytes) 000326 1746625902.257160 read 0x013ca4 thru 0x015e3d (0x219a bytes) 000330 1746625902.258298 read 0x00aa2d thru 0x01a0df (0xf6b3 bytes) 000331 1746625902.265527 trunc from 0x02d360 to 0x003bbb (0x297a4 bytes) 000332 1746625902.300182 mapwrite 0x02fd82 thru 0x0345c2 (0x4841 bytes) 000335 1746625902.338134 MIRROR_RESYNC 2 mirrors 000336 1746625902.490474 write 0x036965 thru 0x03eb64 (0x8200 bytes) 000337 1746625902.512259 write 0x010049 thru 0x019c09 (0x9bc1 bytes) 000340 1746625902.524528 trunc from 0x03eb65 to 0x0007a6 (0x3e3be bytes) 000341 1746625902.540245 read 0x000065 thru 0x0007a6 (0x742 bytes) 000342 1746625902.543388 mapread 0x000589 thru 0x0007a6 (0x21e bytes) 000344 1746625902.545074 mapwrite 0x027499 thru 0x0310d5 (0x9c3d bytes) 000346 1746625902.590002 trunc from 0x0310d6 to 0x011789 (0x1f94c bytes) 000349 1746625902.596362 mapwrite 0x001fbf thru 0x00e2e5 (0xc327 bytes) 000350 1746625902.625260 MIRROR_RESYNC 2 mirrors 000356 1746625902.770646 write 0x00c20f thru 0x00c99c (0x78e bytes) 000358 1746625902.800879 MIRROR_EXTEND to 3 mirrors 000359 1746625903.002033 MIRROR_EXTEND to 4 mirrors 000362 1746625903.172295 read 0x0087a4 thru 0x00bf61 (0x37be bytes) 000363 1746625903.174758 mapwrite 0x007e44 thru 0x00a46f (0x262c bytes) 000364 1746625903.242841 mapread 0x0098ec thru 0x00e8e5 (0x4ffa bytes) 000366 1746625903.243728 trunc from 0x01178a to 0x004227 (0xd562 bytes) 000367 1746625903.255988 mapwrite 0x018260 thru 0x01e7ba (0x655b bytes) 000370 1746625903.288555 mapwrite 0x030a22 thru 0x03cf2b (0xc50a bytes) 000371 1746625903.324861 trunc from 0x03cf2c to 0x0275fd (0x1592e bytes) 000372 1746625903.330624 write 0x026524 thru 0x0284ff (0x1fdc bytes) 000374 1746625903.339784 MIRROR_EXTEND to 5 mirrors 000375 1746625903.515121 write 0x0387ce thru 0x03cec2 (0x46f5 bytes) 000377 1746625903.536902 read 0x033436 thru 0x03cec2 (0x9a8d bytes) 000379 1746625903.543732 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000381 1746625903.615109 write 0x010aff thru 0x01d08d (0xc58f bytes) 000382 1746625903.621349 write 0x03ea0a thru 0x03ffff (0x15f6 bytes) 000383 1746625903.622028 mapwrite 0x01e449 thru 0x02805e (0x9c16 bytes) 000386 1746625903.665555 mapread 0x01fa92 thru 0x027f61 (0x84d0 bytes) 000387 1746625903.668441 MIRROR_RESYNC 4 mirrors 000390 1746625903.881183 MIRROR_EXTEND to 5 mirrors 000392 1746625904.037483 mapwrite 0x025327 thru 0x0260c5 (0xd9f bytes) 000393 1746625904.103613 write 0x032bb0 thru 0x033a45 (0xe96 bytes) 000399 1746625904.109658 mapwrite 0x00e3ca thru 0x0100a6 (0x1cdd bytes) 000401 1746625904.134698 read 0x0223d3 thru 0x023fae (0x1bdc bytes) 000402 1746625904.135145 read 0x009a97 thru 0x01148f (0x79f9 bytes) 000403 1746625904.135576 mapwrite 0x02d8ee thru 0x03832c (0xaa3f bytes) 000405 1746625904.156842 write 0x025eb7 thru 0x032030 (0xc17a bytes) 000406 1746625904.161305 MIRROR_EXTEND to 6 mirrors 000407 1746625904.318353 mapwrite 0x026f96 thru 0x028c90 (0x1cfb bytes) 000408 1746625904.385247 mapread 0x03ec71 thru 0x03ffff (0x138f bytes) 000409 1746625904.391502 MIRROR_EXTEND to 7 mirrors 000410 1746625904.550262 mapread 0x02225d thru 0x023b8d (0x1931 bytes) 000411 1746625904.559927 trunc from 0x040000 to 0x0278fc (0x18703 bytes) 000414 1746625904.594951 read 0x004c35 thru 0x0120c9 (0xd495 bytes) 000415 1746625904.599380 mapread 0x001e4f thru 0x00b949 (0x9afb bytes) 000416 1746625904.601907 write 0x035967 thru 0x03cdaa (0x7444 bytes) 000417 1746625904.602779 trunc from 0x03cdab to 0x037c20 (0x518a bytes) 000419 1746625904.612397 write 0x03af46 thru 0x03ffff (0x50ba bytes) 000422 1746625904.615995 MIRROR_EXTEND to 8 mirrors 000423 1746625904.776788 read 0x015881 thru 0x021891 (0xc011 bytes) 000424 1746625904.782593 mapwrite 0x0196c8 thru 0x02181f (0x8158 bytes) 000425 1746625904.847627 trunc from 0x040000 to 0x018a6e (0x27591 bytes) 000426 1746625904.859561 trunc from 0x018a6f to 0x009164 (0xf90a bytes) 000428 1746625904.867493 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000429 1746625904.960932 write 0x037bf5 thru 0x03af7b (0x3387 bytes) 000431 1746625904.962250 read 0x025692 thru 0x02db7b (0x84ea bytes) 000432 1746625904.971627 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000433 1746625905.045009 MIRROR_EXTEND to 7 mirrors 000436 1746625905.244604 read 0x023189 thru 0x02623f (0x30b7 bytes) 000437 1746625905.250033 mapwrite 0x029f47 thru 0x039422 (0xf4dc bytes) 000438 1746625905.357566 mapwrite 0x03e7d0 thru 0x03ffff (0x1830 bytes) 000441 1746625905.397554 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000442 1746625905.504558 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000444 1746625905.595717 trunc from 0x040000 to 0x02fa24 (0x105db bytes) 000445 1746625905.601978 mapwrite 0x0037d4 thru 0x006b1f (0x334c bytes) truncating to largest ever: 0x3de18 000446 1746625905.649320 trunc from 0x02fa25 to 0x03de17 (0xe3f3 bytes) 000447 1746625905.659954 MIRROR_EXTEND to 6 mirrors 000448 1746625905.891530 trunc from 0x03de18 to 0x016cfe (0x27119 bytes) 000449 1746625905.936801 mapwrite 0x0194ee thru 0x027acb (0xe5de bytes) 000451 1746625906.000413 read 0x00fee9 thru 0x01b5f5 (0xb70d bytes) 000452 1746625906.006723 trunc from 0x027acc to 0x038693 (0x10bc8 bytes) 000453 1746625906.018587 MIRROR_EXTEND to 7 mirrors 000454 1746625906.198467 read 0x028a5a thru 0x02c1c2 (0x3769 bytes) 000455 1746625906.204624 read 0x01f093 thru 0x021e74 (0x2de2 bytes) 000460 1746625906.222915 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000461 1746625906.342121 mapread 0x02e1b1 thru 0x02f717 (0x1567 bytes) 000464 1746625906.365307 mapwrite 0x026a73 thru 0x032dfe (0xc38c bytes) 000467 1746625906.454759 read 0x008d83 thru 0x00e480 (0x56fe bytes) 000468 1746625906.458010 trunc from 0x038694 to 0x030c4e (0x7a45 bytes) 000470 1746625906.473046 read 0x017132 thru 0x025a79 (0xe948 bytes) 000472 1746625906.479477 mapread 0x01cc0c thru 0x02016a (0x355f bytes) 000473 1746625906.480034 read 0x01b69d thru 0x02706f (0xb9d3 bytes) 000481 1746625906.483601 read 0x01c627 thru 0x01df18 (0x18f2 bytes) 000484 1746625906.484600 mapwrite 0x026ec3 thru 0x033218 (0xc356 bytes) 000486 1746625906.523103 write 0x02aa89 thru 0x02b58f (0xb07 bytes) 000487 1746625906.523882 read 0x016e22 thru 0x01e2c0 (0x749f bytes) 000488 1746625906.524390 trunc from 0x033219 to 0x0316b7 (0x1b61 bytes) 000490 1746625906.530903 mapwrite 0x00b703 thru 0x016652 (0xaf50 bytes) 000491 1746625906.558443 MIRROR_RESYNC 6 mirrors 000492 1746625906.875245 trunc from 0x0316b8 to 0x034361 (0x2caa bytes) 000493 1746625906.914724 mapread 0x02af92 thru 0x034361 (0x93d0 bytes) 000497 1746625906.938630 trunc from 0x034362 to 0x0115be (0x22da3 bytes) 000501 1746625906.947940 mapread 0x000ffd thru 0x00ad18 (0x9d1c bytes) 000502 1746625906.962536 MIRROR_EXTEND to 7 mirrors 000505 1746625907.145083 mapwrite 0x005bf2 thru 0x00e6b7 (0x8ac6 bytes) 000508 1746625907.212177 write 0x023275 thru 0x028e5b (0x5be7 bytes) 000509 1746625907.213110 write 0x0056d7 thru 0x011eb4 (0xc7de bytes) 000510 1746625907.216376 read 0x0242bf thru 0x028e5b (0x4b9d bytes) 000511 1746625907.216826 trunc from 0x028e5c to 0x017f4f (0x10f0c bytes) 000512 1746625907.223531 mapwrite 0x02c1b9 thru 0x038026 (0xbe6e bytes) 000513 1746625907.260519 mapwrite 0x02a3e9 thru 0x02dc38 (0x3850 bytes) 000514 1746625907.280926 mapwrite 0x0301e9 thru 0x03dc0c (0xda24 bytes) 000515 1746625907.327438 mapread 0x01cb61 thru 0x029ebc (0xd35c bytes) 000519 1746625907.338435 read 0x00d05a thru 0x012ff0 (0x5f97 bytes) 000520 1746625907.341557 write 0x02e9b3 thru 0x035ae1 (0x712f bytes) 000521 1746625907.342365 trunc from 0x03dc0d to 0x00439f (0x3986d bytes) 000522 1746625907.349168 mapread 0x000893 thru 0x00439f (0x3b0d bytes) 000525 1746625907.350374 read 0x00339f thru 0x00439f (0x1001 bytes) 000526 1746625907.350786 read 0x00300d thru 0x00439f (0x1393 bytes) 000528 1746625907.351402 read 0x000593 thru 0x00439f (0x3e0d bytes) 000530 1746625907.352004 write 0x02764f thru 0x036f66 (0xf918 bytes) 000532 1746625907.353182 mapwrite 0x01ba8f thru 0x01d46e (0x19e0 bytes) 000533 1746625907.378273 mapwrite 0x01564c thru 0x020817 (0xb1cc bytes) 000535 1746625907.403356 read 0x019888 thru 0x025c0c (0xc385 bytes) 000536 1746625907.405855 MIRROR_EXTEND to 8 mirrors 000537 1746625907.561967 read 0x017ced thru 0x020b27 (0x8e3b bytes) 000538 1746625907.567560 read 0x00fe8f thru 0x0130c3 (0x3235 bytes) 000540 1746625907.577255 read 0x031ea9 thru 0x036f66 (0x50be bytes) 000541 1746625907.579042 read 0x00a49f thru 0x00fbb7 (0x5719 bytes) 000546 1746625907.589363 read 0x01a646 thru 0x024dac (0xa767 bytes) 000551 1746625907.599608 read 0x01b8a7 thru 0x021241 (0x599b bytes) 000553 1746625907.604418 trunc from 0x036f67 to 0x006c25 (0x30341 bytes) 000554 1746625907.648898 read 0x00610d thru 0x006c25 (0xb19 bytes) 000555 1746625907.652388 read 0x004edb thru 0x006c25 (0x1d4b bytes) 000558 1746625907.653532 read 0x003412 thru 0x006c25 (0x3814 bytes) 000560 1746625907.654226 read 0x006113 thru 0x006c25 (0xb13 bytes) 000561 1746625907.654601 write 0x005463 thru 0x012117 (0xccb5 bytes) 000562 1746625907.657435 MIRROR_RESYNC 8 mirrors 000563 1746625908.035452 MIRROR_EXTEND to 9 mirrors 000565 1746625908.237673 mapread 0x00f3b1 thru 0x012117 (0x2d67 bytes) 000566 1746625908.244005 mapwrite 0x01650b thru 0x01ab4f (0x4645 bytes) 000567 1746625908.308542 write 0x00fc53 thru 0x015bae (0x5f5c bytes) 000569 1746625908.311368 trunc from 0x01ab50 to 0x024d46 (0xa1f7 bytes) 000570 1746625908.319070 mapwrite 0x01d8ef thru 0x02b585 (0xdc97 bytes) 000571 1746625908.358615 trunc from 0x02b586 to 0x02112d (0xa458 bytes) 000573 1746625908.365660 mapwrite 0x0207a8 thru 0x029777 (0x8fd0 bytes) 000575 1746625908.404578 mapwrite 0x021008 thru 0x02c04b (0xb044 bytes) 000576 1746625908.446006 MIRROR_RESYNC 9 mirrors 000579 1746625908.912518 mapwrite 0x00dbf0 thru 0x011860 (0x3c71 bytes) 000580 1746625908.998881 read 0x0141c3 thru 0x0239aa (0xf7e8 bytes) 000581 1746625909.002396 read 0x01525e thru 0x017d84 (0x2b27 bytes) 000583 1746625909.003098 mapwrite 0x02f9eb thru 0x0360c9 (0x66df bytes) 000584 1746625909.043979 trunc from 0x0360ca to 0x0294e1 (0xcbe8 bytes) 000586 1746625909.048983 write 0x00f929 thru 0x0155d7 (0x5caf bytes) 000588 1746625909.050400 write 0x00810a thru 0x00faf4 (0x79eb bytes) 000589 1746625909.051133 write 0x0168cd thru 0x01e5a5 (0x7cd9 bytes) 000590 1746625909.051751 mapread 0x028ff7 thru 0x0294e1 (0x4eb bytes) 000591 1746625909.052287 trunc from 0x0294e2 to 0x024e28 (0x46b9 bytes) 000592 1746625909.058274 mapread 0x0055b2 thru 0x013655 (0xe0a4 bytes) 000593 1746625909.058858 read 0x01769e thru 0x01998e (0x22f1 bytes) 000595 1746625909.061467 MIRROR_RESYNC 9 mirrors 000598 1746625909.492479 mapwrite 0x0327a2 thru 0x03bae1 (0x9340 bytes) 000599 1746625909.559944 mapread 0x01713e thru 0x0242b6 (0xd179 bytes) 000600 1746625909.568001 write 0x01179b thru 0x017f3f (0x67a5 bytes) 000601 1746625909.575368 read 0x0068d2 thru 0x014c8c (0xe3bb bytes) 000605 1746625909.578970 trunc from 0x03bae2 to 0x001a93 (0x3a04e bytes) 000609 1746625909.591564 write 0x032d10 thru 0x039dfa (0x70eb bytes) 000611 1746625909.599115 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000614 1746625909.690132 MIRROR_EXTEND to 9 mirrors 000616 1746625909.896319 write 0x0179ff thru 0x01959a (0x1b9c bytes) 000617 1746625909.925354 write 0x030118 thru 0x03635d (0x6246 bytes) 000618 1746625909.932113 mapread 0x030b0a thru 0x03484e (0x3d45 bytes) 000621 1746625909.933743 mapwrite 0x02a195 thru 0x0384b4 (0xe320 bytes) 000623 1746625909.970768 trunc from 0x039dfb to 0x015f94 (0x23e66 bytes) 000625 1746625909.991353 write 0x020d87 thru 0x028373 (0x75ed bytes) 000626 1746625909.992325 trunc from 0x028374 to 0x003815 (0x24b5e bytes) 000627 1746625909.999404 mapread 0x000a0f thru 0x003815 (0x2e07 bytes) 000629 1746625910.005596 mapwrite 0x01f556 thru 0x024667 (0x5112 bytes) 000631 1746625910.054241 mapread 0x01fbc8 thru 0x024667 (0x4aa0 bytes) 000632 1746625910.055196 trunc from 0x024668 to 0x02210c (0x255b bytes) 000634 1746625910.066438 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000636 1746625910.159024 trunc from 0x02210d to 0x000630 (0x21adc bytes) 000637 1746625910.165611 write 0x025fbe thru 0x02b98e (0x59d1 bytes) 000640 1746625910.167911 mapwrite 0x011cf4 thru 0x01e017 (0xc324 bytes) 000641 1746625910.213531 mapread 0x02716a thru 0x029d24 (0x2bbb bytes) 000643 1746625910.214964 write 0x03b47a thru 0x03ffff (0x4b86 bytes) 000645 1746625910.216103 write 0x03ce4c thru 0x03ffff (0x31b4 bytes) 000646 1746625910.216977 trunc from 0x040000 to 0x003844 (0x3c7bb bytes) 000647 1746625910.231346 mapread 0x001add thru 0x003844 (0x1d68 bytes) 000649 1746625910.238700 mapread 0x000ce0 thru 0x003844 (0x2b65 bytes) 000651 1746625910.242866 write 0x03c05d thru 0x03db08 (0x1aac bytes) 000652 1746625910.244698 read 0x022e53 thru 0x02ec94 (0xbe42 bytes) 000655 1746625910.250737 write 0x02ed4c thru 0x03b0bb (0xc370 bytes) 000656 1746625910.254241 mapwrite 0x000755 thru 0x00603b (0x58e7 bytes) 000657 1746625910.295087 read 0x01958a thru 0x01e0c8 (0x4b3f bytes) 000660 1746625910.302694 trunc from 0x03db09 to 0x030516 (0xd5f2 bytes) 000661 1746625910.312802 write 0x004b82 thru 0x005744 (0xbc3 bytes) 000662 1746625910.313395 mapwrite 0x02ae4f thru 0x02cb8e (0x1d40 bytes) 000663 1746625910.345809 mapread 0x026775 thru 0x030516 (0x9da2 bytes) 000666 1746625910.347118 mapwrite 0x00f5cd thru 0x0167a4 (0x71d8 bytes) 000670 1746625910.374715 mapread 0x01c192 thru 0x026bcd (0xaa3c bytes) 000672 1746625910.378054 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000674 1746625910.489709 mapread 0x01d24c thru 0x02507d (0x7e32 bytes) 000676 1746625910.502941 write 0x010024 thru 0x018399 (0x8376 bytes) 000679 1746625910.507842 trunc from 0x030517 to 0x007e91 (0x28685 bytes) 000680 1746625910.513635 mapread 0x002f43 thru 0x007e91 (0x4f4f bytes) 000682 1746625910.518665 mapread 0x002ed7 thru 0x004220 (0x134a bytes) 000683 1746625910.519272 write 0x01dcd0 thru 0x024206 (0x6537 bytes) 000686 1746625910.520841 mapread 0x01b70d thru 0x023309 (0x7bfd bytes) 000687 1746625910.524593 mapread 0x010283 thru 0x01d32b (0xd0a9 bytes) 000688 1746625910.531527 trunc from 0x024207 to 0x00ff5a (0x142ac bytes) 000690 1746625910.538708 trunc from 0x00ff5b to 0x028bee (0x18c94 bytes) 000691 1746625910.543794 write 0x0328ac thru 0x0367f3 (0x3f48 bytes) 000695 1746625910.545529 read 0x025fed thru 0x033c6a (0xdc7e bytes) 000696 1746625910.549877 mapread 0x0240c6 thru 0x02cb3d (0x8a78 bytes) 000699 1746625910.553989 mapread 0x0202fa thru 0x028c95 (0x899c bytes) 000702 1746625910.556041 trunc from 0x0367f4 to 0x032ba1 (0x3c52 bytes) 000703 1746625910.562783 trunc from 0x032ba2 to 0x008154 (0x2aa4d bytes) 000711 1746625910.574524 write 0x038c6c thru 0x03ffff (0x7394 bytes) 000712 1746625910.580056 MIRROR_RESYNC 7 mirrors 000713 1746625911.014771 mapwrite 0x024ac6 thru 0x034433 (0xf96e bytes) 000714 1746625911.109941 mapwrite 0x039ddf thru 0x03a816 (0xa38 bytes) 000715 1746625911.130154 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000717 1746625911.198801 read 0x01a915 thru 0x024fd6 (0xa6c2 bytes) 000721 1746625911.204155 read 0x0139c4 thru 0x022062 (0xe69f bytes) 000725 1746625911.211285 mapwrite 0x00cf8a thru 0x0165bb (0x9632 bytes) 000726 1746625911.240514 trunc from 0x040000 to 0x01f3ee (0x20c11 bytes) 000728 1746625911.258198 MIRROR_EXTEND to 7 mirrors 000729 1746625911.409255 write 0x02359f thru 0x02f554 (0xbfb6 bytes) 000730 1746625911.442562 MIRROR_EXTEND to 8 mirrors 000731 1746625911.600374 mapwrite 0x00da06 thru 0x015402 (0x79fd bytes) 000733 1746625911.658737 read 0x018dd5 thru 0x01e128 (0x5354 bytes) 000735 1746625911.662424 trunc from 0x02f555 to 0x02fb52 (0x5fe bytes) 000736 1746625911.674323 trunc from 0x02fb53 to 0x03889b (0x8d49 bytes) 000737 1746625911.679675 read 0x019c25 thru 0x02978f (0xfb6b bytes) 000738 1746625911.686520 MIRROR_RESYNC 8 mirrors 000739 1746625911.977406 read 0x002835 thru 0x00a713 (0x7edf bytes) 000740 1746625911.980949 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000741 1746625912.073652 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000743 1746625912.166888 trunc from 0x03889c to 0x030ddc (0x7abf bytes) 000744 1746625912.194729 read 0x01410f thru 0x01d3b8 (0x92aa bytes) 000746 1746625912.197887 read 0x02c7ff thru 0x030ddc (0x45de bytes) 000748 1746625912.202191 read 0x00608e thru 0x0070c8 (0x103b bytes) 000749 1746625912.202607 mapread 0x000f10 thru 0x005a47 (0x4b38 bytes) 000751 1746625912.208602 MIRROR_EXTEND to 7 mirrors 000752 1746625912.324654 mapread 0x022a04 thru 0x025652 (0x2c4f bytes) 000755 1746625912.332036 write 0x038ac3 thru 0x03ffff (0x753d bytes) 000762 1746625912.350871 trunc from 0x040000 to 0x00e479 (0x31b86 bytes) 000763 1746625912.362808 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000764 1746625912.422154 write 0x00da39 thru 0x00f24e (0x1816 bytes) 000765 1746625912.424199 mapwrite 0x01b17e thru 0x02a33a (0xf1bd bytes) 000766 1746625912.455423 write 0x00cca2 thru 0x01cc4a (0xffa9 bytes) 000768 1746625912.459052 read 0x01b7e4 thru 0x01ca85 (0x12a2 bytes) 000769 1746625912.459462 write 0x0107d0 thru 0x0180aa (0x78db bytes) 000771 1746625912.460373 write 0x01df20 thru 0x02813c (0xa21d bytes) 000775 1746625912.462096 mapwrite 0x01cc76 thru 0x022c84 (0x600f bytes) 000776 1746625912.485990 write 0x00f806 thru 0x01ece5 (0xf4e0 bytes) 000778 1746625912.488937 MIRROR_RESYNC 6 mirrors 000780 1746625912.704693 MIRROR_EXTEND to 7 mirrors 000781 1746625912.848463 mapwrite 0x0186f9 thru 0x02063d (0x7f45 bytes) 000783 1746625912.903703 trunc from 0x02a33b to 0x02e534 (0x41fa bytes) 000784 1746625912.912712 mapread 0x0158f4 thru 0x024ccc (0xf3d9 bytes) 000790 1746625912.917144 write 0x030093 thru 0x03440f (0x437d bytes) 000791 1746625912.917944 trunc from 0x034410 to 0x0275f1 (0xce1e bytes) 000793 1746625912.923520 write 0x01464e thru 0x0223e0 (0xdd93 bytes) 000794 1746625912.924319 trunc from 0x0275f2 to 0x01c2a8 (0xb349 bytes) 000796 1746625912.929186 read 0x0194d6 thru 0x01c2a8 (0x2dd3 bytes) 000797 1746625912.929709 read 0x002e01 thru 0x009a54 (0x6c54 bytes) 000798 1746625912.934002 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000799 1746625912.996588 MIRROR_RESYNC 6 mirrors 000803 1746625913.209177 trunc from 0x01c2a9 to 0x02456d (0x82c5 bytes) 000805 1746625913.226753 write 0x03e6fb thru 0x03ffff (0x1905 bytes) 000806 1746625913.227486 trunc from 0x040000 to 0x0239f2 (0x1c60d bytes) 000807 1746625913.234136 MIRROR_RESYNC 6 mirrors 000808 1746625913.436402 write 0x002c2c thru 0x010c25 (0xdffa bytes) 000809 1746625913.459485 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000810 1746625913.513696 read 0x0061bf thru 0x00e250 (0x8092 bytes) 000812 1746625913.518007 MIRROR_EXTEND to 6 mirrors 000813 1746625913.640930 trunc from 0x0239f3 to 0x0323af (0xe9bd bytes) 000816 1746625913.670479 trunc from 0x0323b0 to 0x0263ce (0xbfe1 bytes) 000817 1746625913.675850 read 0x01fb28 thru 0x022c51 (0x312a bytes) 000819 1746625913.679525 MIRROR_RESYNC 6 mirrors 000820 1746625913.940057 mapread 0x016365 thru 0x02242d (0xc0c9 bytes) 000821 1746625913.951733 write 0x00e9f5 thru 0x0105b0 (0x1bbc bytes) 000822 1746625913.980899 mapwrite 0x035957 thru 0x03c01a (0x66c4 bytes) 000823 1746625914.034131 trunc from 0x03c01b to 0x012ed0 (0x2914a bytes) 000825 1746625914.046123 MIRROR_RESYNC 6 mirrors 000827 1746625914.327041 mapread 0x005b73 thru 0x00ccce (0x715c bytes) 000829 1746625914.335077 mapwrite 0x01c46b thru 0x021d95 (0x592b bytes) 000831 1746625914.388454 write 0x01d41e thru 0x02aae8 (0xd6cb bytes) 000832 1746625914.389461 mapread 0x01a052 thru 0x01a3ee (0x39d bytes) 000833 1746625914.391931 MIRROR_RESYNC 6 mirrors 000834 1746625914.693165 write 0x032c9b thru 0x035034 (0x239a bytes) 000835 1746625914.714783 write 0x009f05 thru 0x0172e3 (0xd3df bytes) 000838 1746625914.721028 mapread 0x0046f4 thru 0x011709 (0xd016 bytes) 000841 1746625914.727902 trunc from 0x035035 to 0x017ff6 (0x1d03e bytes) 000843 1746625914.742880 mapwrite 0x01737c thru 0x020f1a (0x9b9f bytes) 000845 1746625914.783059 read 0x018636 thru 0x020f1a (0x88e5 bytes) 000847 1746625914.783621 write 0x01ab81 thru 0x01eb8a (0x400a bytes) 000848 1746625914.784309 mapread 0x00164b thru 0x009d58 (0x870e bytes) 000850 1746625914.785104 read 0x00b02e thru 0x00c536 (0x1509 bytes) 000851 1746625914.785497 read 0x0035b2 thru 0x0125de (0xf02d bytes) 000854 1746625914.786621 trunc from 0x020f1b to 0x023423 (0x2509 bytes) 000855 1746625914.792521 write 0x0063a4 thru 0x0145fc (0xe259 bytes) 000861 1746625914.795014 mapwrite 0x007f0c thru 0x0172bc (0xf3b1 bytes) 000864 1746625914.824714 mapwrite 0x00074c thru 0x001f70 (0x1825 bytes) 000866 1746625914.856367 MIRROR_RESYNC 6 mirrors 000867 1746625915.188251 mapwrite 0x017c34 thru 0x02685b (0xec28 bytes) 000868 1746625915.263805 trunc from 0x02685c to 0x00dcfb (0x18b60 bytes) 000869 1746625915.278739 mapread 0x00262a thru 0x0043c4 (0x1d9b bytes) 000870 1746625915.288878 MIRROR_RESYNC 6 mirrors 000871 1746625915.645387 mapwrite 0x038312 thru 0x03e41e (0x610d bytes) 000873 1746625915.741105 read 0x022561 thru 0x0301f7 (0xdc97 bytes) 000874 1746625915.749338 MIRROR_EXTEND to 7 mirrors 000880 1746625916.006543 trunc from 0x03e41f to 0x0010b1 (0x3d36d bytes) 000881 1746625916.059594 write 0x020309 thru 0x02bdd2 (0xbaca bytes) 000885 1746625916.061705 read 0x005094 thru 0x00ed11 (0x9c7e bytes) 000886 1746625916.067611 trunc from 0x02bdd3 to 0x02da38 (0x1c66 bytes) 000889 1746625916.082264 read 0x026793 thru 0x0290d7 (0x2945 bytes) 000890 1746625916.085269 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000891 1746625916.178318 read 0x01cbf8 thru 0x02c487 (0xf890 bytes) 000896 1746625916.184855 write 0x012d44 thru 0x017031 (0x42ee bytes) 000897 1746625916.194895 mapwrite 0x0074cd thru 0x0079e4 (0x518 bytes) 000898 1746625916.222028 mapread 0x026bfd thru 0x028cc3 (0x20c7 bytes) 000901 1746625916.225122 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000903 1746625916.305172 mapread 0x018a54 thru 0x01e407 (0x59b4 bytes) 000904 1746625916.312707 trunc from 0x02da39 to 0x02322b (0xa80d bytes) 000906 1746625916.318561 mapwrite 0x009d87 thru 0x013474 (0x96ee bytes) 000907 1746625916.348427 read 0x01071f thru 0x015b55 (0x5437 bytes) 000908 1746625916.348844 write 0x003a87 thru 0x010b10 (0xd08a bytes) 000909 1746625916.349800 write 0x031cde thru 0x03da39 (0xbd5c bytes) 000910 1746625916.350836 read 0x02e6d0 thru 0x0328b1 (0x41e2 bytes) 000911 1746625916.354924 read 0x023b3e thru 0x028d64 (0x5227 bytes) 000912 1746625916.357805 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000914 1746625916.437929 read 0x0262a0 thru 0x0273a2 (0x1103 bytes) 000916 1746625916.442958 mapread 0x0042fd thru 0x00880a (0x450e bytes) 000917 1746625916.454730 MIRROR_EXTEND to 5 mirrors 000918 1746625916.626941 mapread 0x02d461 thru 0x0331f3 (0x5d93 bytes) 000920 1746625916.642464 MIRROR_RESYNC 5 mirrors 000923 1746625916.847444 write 0x00da75 thru 0x01d4f1 (0xfa7d bytes) 000924 1746625916.872549 MIRROR_EXTEND to 6 mirrors 000925 1746625917.043056 write 0x0022f9 thru 0x00dc90 (0xb998 bytes) 000927 1746625917.069417 MIRROR_EXTEND to 7 mirrors 000928 1746625917.255955 read 0x0176aa thru 0x01ff07 (0x885e bytes) 000932 1746625917.273352 mapread 0x00f3bb thru 0x00f695 (0x2db bytes) 000933 1746625917.280231 mapwrite 0x0249b2 thru 0x02524b (0x89a bytes) 000934 1746625917.345969 mapread 0x02c030 thru 0x038848 (0xc819 bytes) 000935 1746625917.351926 mapread 0x00871e thru 0x01283a (0xa11d bytes) 000936 1746625917.357508 read 0x014ed8 thru 0x020fde (0xc107 bytes) 000937 1746625917.360326 trunc from 0x03da3a to 0x032790 (0xb2a9 bytes) 000939 1746625917.366882 mapwrite 0x0317c3 thru 0x03b349 (0x9b87 bytes) 000940 1746625917.410948 mapread 0x0301c5 thru 0x03b349 (0xb185 bytes) 000941 1746625917.411701 mapwrite 0x03c243 thru 0x03c477 (0x235 bytes) 000943 1746625917.447769 MIRROR_RESYNC 7 mirrors 000944 1746625917.725877 write 0x0376bb thru 0x038b3c (0x1482 bytes) 000948 1746625917.746667 trunc from 0x03c478 to 0x002456 (0x3a021 bytes) 000949 1746625917.758867 read 0x00001b thru 0x002456 (0x243c bytes) 000950 1746625917.760951 trunc from 0x002457 to 0x013abe (0x11668 bytes) 000951 1746625917.765593 trunc from 0x013abf to 0x0018e5 (0x121d9 bytes) 000952 1746625917.771846 mapread 0x00156d thru 0x0018e5 (0x379 bytes) 000954 1746625917.775223 MIRROR_EXTEND to 8 mirrors 000955 1746625917.915730 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000956 1746625917.986385 read 0x000f82 thru 0x0018e5 (0x964 bytes) 000960 1746625917.992254 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) 000961 1746625918.062959 read 0x000b0b thru 0x0018e5 (0xddb bytes) 000964 1746625918.066368 mapread 0x000552 thru 0x0018e5 (0x1394 bytes) 000965 1746625918.066960 read 0x000b97 thru 0x0018e5 (0xd4f bytes) 000967 1746625918.067708 read 0x000960 thru 0x0018e5 (0xf86 bytes) 000969 1746625918.068254 mapread 0x001896 thru 0x0018e5 (0x50 bytes) 000970 1746625918.068702 mapwrite 0x02fc4e thru 0x03994d (0x9d00 bytes) 000973 1746625918.124818 read 0x023799 thru 0x032dbe (0xf626 bytes) 000980 1746625918.127135 write 0x029b97 thru 0x02d45f (0x38c9 bytes) 000981 1746625918.127804 write 0x02ad2d thru 0x02d402 (0x26d6 bytes) 000983 1746625918.128587 write 0x016aa1 thru 0x01ed50 (0x82b0 bytes) 000987 1746625918.130232 write 0x0163d3 thru 0x024acd (0xe6fb bytes) 000989 1746625918.131439 mapread 0x014680 thru 0x0159e2 (0x1363 bytes) 000990 1746625918.132086 mapwrite 0x0071cf thru 0x010c7c (0x9aae bytes) 000992 1746625918.157792 read 0x020cc8 thru 0x02ecfe (0xe037 bytes) 000993 1746625918.158302 mapwrite 0x00b39d thru 0x017fc2 (0xcc26 bytes) 000994 1746625918.186736 trunc from 0x03994e to 0x01967c (0x202d1 bytes) 000996 1746625918.194275 trunc from 0x01967d to 0x0036cf (0x15fad bytes) 000998 1746625918.201671 mapread 0x001287 thru 0x0036cf (0x2449 bytes) 000999 1746625918.204995 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5de:0x0]: Invalid argument (22) All operations completed A-OK! PASS 70a (23s) == sanity-flr test 71: check mirror extend parent fid ==== 09:51:59 (1746625919) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 09:52:00 (1746625920) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1251520 .. Extending file size to 6602624 .. failed Extending file size to 5006752 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6854880 .. failed Extending file size to 5896704 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7394720 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8625408 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8992224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9021344 .. failed Extending file size to 9106688 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9329632 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9226112 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9358592 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 27761 27762 27764 27765 27768 done 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) 192.168.203.149@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg349-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:53:05 (1746625985) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3827648 .. Extending file size to 3073248 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7405184 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3324160 .. Extending file size to 7668288 .. Extending file size to 9126272 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8699008 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8726624 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9202656 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 31410 31411 31414 31415 31416 done 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) 192.168.203.149@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg349-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:54:09 (1746626049) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4992 2203392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4992 2203392 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0360049 s, 58.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0944865 s, 22.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:54:14 (1746626054) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0271842 s, 77.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 09:54:16 (1746626056) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x628:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7cc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7cc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7cd:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x7c7:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:54:17 (1746626057) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0315506 s, 66.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:54:19 (1746626059) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0342259 s, 61.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ca:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ca:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cb:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:54:21 (1746626061) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0420713 s, 49.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cd:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:54:44 (1746626084) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg349-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0258109 s, 40.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d3:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:55:28 (1746626128) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg349-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0378694 s, 55.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:56:14 (1746626174) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg349-server mds-ost sync done. Creating new pool test_204f oleg349-server: Pool lustre.test_204f created Adding targets to pool oleg349-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg349-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.040629 s, 51.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x7d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg349-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg349-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg349-server: Pool lustre.test_204f destroyed PASS 204f (56s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:57:10 (1746626230) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:57:12 (1746626232) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:57:14 (1746626234) oleg349-server: Pool lustre.test_206 created Adding targets to pool oleg349-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg349-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg349-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg349-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg349-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:57:28 (1746626248) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0212207 s, 49.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00976196 s, 107 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x1a58f493 Mirror 3: 0x1a58f493 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:57:30 (1746626250) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:57:31 (1746626251) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:57:32 (1746626252) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0281158 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1990000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2030000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2030000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2030000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf50000 Read this (4 bytes): BBBB 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:57:47 (1746626267) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00788261 s, 1.0 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:57:49 (1746626269) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111484 s, 94.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x64d lmm_fid: [0x200000404:0x64d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x64d lmm_fid: [0x200000404:0x64d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x7df:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835533 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-07 09:57:49.000000000 -0400 Modify: 2025-05-07 09:57:49.000000000 -0400 Change: 2025-05-07 09:57:50.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:57:51 (1746626271) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:57:52 (1746626272) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.201281 s, 203 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0246414 s, 166 kB/s size after second write -rw-r--r-- 1 root root 4096 May 7 09:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x64f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 7 09:57 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 735 sec ============ 09:57:54 (1746626274) === sanity-flr: start cleanup 09:57:55 (1746626275) === === sanity-flr: finish cleanup 09:57:57 (1746626277) ===