-----============= acceptance-small: sanity-flr ============----- Tue Dec 24 23:04:07 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg219-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:04:15 (1735099455) === oleg219-client.virtnet: executing check_config_client /mnt/lustre oleg219-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg219-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5909000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5909000.idle_timeout=debug disable quota as required oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:04:26 (1735099466) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f6992] == sanity-flr test 0a: lfs mirror create with -N option == 23:04:27 (1735099467) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:04:33 (1735099473) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:04:34 (1735099474) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:04:35 (1735099475) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:04:44 (1735099484) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:04:45 (1735099485) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 23:04:46 (1735099486) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5909000.stats=clear osc.lustre-OST0001-osc-ffff8800b5909000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:04:48 (1735099488) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:04:52 (1735099492) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:04:55 (1735099495) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:04:56 (1735099496) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 23:05:00 (1735099500) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:05:03 (1735099503) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:05:04 (1735099504) lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:05:07 (1735099507) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:05:11 (1735099511) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:05:13 (1735099513) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.365404 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10537 s, 10.0 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:05:18 (1735099518) 17+0 records in 17+0 records out 17825792 bytes (18 MB) copied, 0.177593 s, 100 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:05:22 (1735099522) Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5909000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5909000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5909000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5909000.ost_server_uuid in DISCONN state after 1 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:05:51 (1735099551) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.132348 s, 47.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5909000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5909000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:06:07 (1735099567) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5909000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5909000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5909000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5909000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:06:31 (1735099591) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.00776 s, 52.0 MB/s -rw-r--r-- 1 root root 52428800 Dec 24 23:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.269s user 0m0.000s sys 0m0.124s Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5909000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5909000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.240s user 0m0.000s sys 0m0.123s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5909000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5909000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.307s user 0m0.000s sys 0m0.127s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5909000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5909000.ost_server_uuid in FULL state after 0 sec PASS 33b (24s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:06:55 (1735099615) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:06:56 (1735099616) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:06:57 (1735099617) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 23:06:58 (1735099618) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 23:07:00 (1735099620) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.148837 s, 84.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:07:03 (1735099623) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108949 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0097038 s, 108 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:07:05 (1735099625) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:07:08 (1735099628) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000857301 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000816488 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000833143 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000674915 s, 1.5 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 23:07:10 (1735099630) create test files with size 15734567 .. b50957f8505c69a03adfd28a947b28f4 - 958733442acb491c5baeb21b35b346f1 - f8c3038457e91f3e8b0225aa8a1a57fa - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 23:07:16 (1735099636) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:07:21 (1735099641) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:07:23 (1735099643) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0307924 s, 34.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109689 s, 95.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 23:07:25 (1735099645) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.03671 s, 57.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0990214 s, 42.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0508053 s, 41.3 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 23:07:28 (1735099648) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 23:07:29 (1735099649) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:07:30 (1735099650) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00837827 s, 125 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 23:07:32 (1735099652) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:07:33 (1735099653) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x13:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:07:36 (1735099656) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0703083 s, 149 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:07:58 (1735099678) llite.lustre-ffff8800a4fbe000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0708643 s, 148 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a2a57000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:08:01 (1735099681) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:08:09 (1735099689) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (9s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:08:18 (1735099698) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 23:08:19 (1735099699) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0853596 s, 36.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0614509 s, 51.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 603f6613cdc792bb491d0849901cda1d - mirror 1 checksum: 6f4401d3c902ec30b0aabc56ea64cead - mirror 2 checksum: 603f6613cdc792bb491d0849901cda1d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 24 23:08:19 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 24 23:08:19 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 603f6613cdc792bb491d0849901cda1d - mirror 1 checksum: 6f4401d3c902ec30b0aabc56ea64cead - mirror 2 checksum: 6f4401d3c902ec30b0aabc56ea64cead - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 24 23:08:19 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 24 23:08:19 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:08:23 (1735099703) Failing mds1 on oleg219-server Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 23:08:25 (1735099705) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server Failover mds1 to oleg219-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 23:08:38 (1735099718) targets are mounted 23:08:38 (1735099718) facet_failover done oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100785 s, 104 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (20s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:08:43 (1735099723) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (0s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:08:43 (1735099723) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:08:44 (1735099724) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00232929 s, 7.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:08:46 (1735099726) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:08:47 (1735099727) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:08:48 (1735099728) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:08:53 (1735099733) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:09:11 (1735099751) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:09:28 (1735099768) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:09:45 (1735099785) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 23:09:47 (1735099787) mirror create pid 22447 mirror split pid 22448 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6729: 22448 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6729: 22448 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 22447 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 22448 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 23:10:49 (1735099849) Seed set to 9850 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1735099850.291947 MIRROR_EXTEND to 2 mirrors skipping zero size read truncating to largest ever: 0x3e661 000009 1735099850.406527 trunc from 00000000 to 0x03e660 (0x3e661 bytes) 000011 1735099850.442542 mapread 0x027383 thru 0x0292ac (0x1f2a bytes) 000012 1735099850.450119 read 0x028b71 thru 0x02bfa9 (0x3439 bytes) 000013 1735099850.450497 write 0x0340f0 thru 0x03f509 (0xb41a bytes) 000015 1735099850.455237 mapread 0x0377cb thru 0x03f509 (0x7d3f bytes) 000016 1735099850.455752 read 0x0218f5 thru 0x02e9eb (0xd0f7 bytes) 000018 1735099850.458416 MIRROR_RESYNC 2 mirrors 000020 1735099850.619074 trunc from 0x03f50a to 0x02d321 (0x121e8 bytes) 000021 1735099850.656901 read 0x01bcf6 thru 0x01da85 (0x1d90 bytes) 000022 1735099850.660440 mapread 0x013917 thru 0x017168 (0x3852 bytes) 000024 1735099850.669558 write 0x00ebba thru 0x01907b (0xa4c2 bytes) 000025 1735099850.670465 trunc from 0x02d322 to 0x021d0a (0xb617 bytes) 000027 1735099850.683587 mapread 0x012215 thru 0x01c212 (0x9ffe bytes) 000028 1735099850.686540 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000029 1735099850.750304 MIRROR_EXTEND to 2 mirrors 000030 1735099850.918172 mapread 0x01d2e7 thru 0x01f428 (0x2142 bytes) 000031 1735099850.924969 write 0x02f61f thru 0x03efe8 (0xf9ca bytes) 000032 1735099850.945518 read 0x01a1a9 thru 0x023c1c (0x9a74 bytes) 000033 1735099850.949378 mapwrite 0x01cf30 thru 0x02baaf (0xeb80 bytes) 000036 1735099850.978431 write 0x003c85 thru 0x00f6d2 (0xba4e bytes) 000037 1735099850.983758 read 0x00e197 thru 0x0186ce (0xa538 bytes) 000039 1735099850.988706 write 0x002383 thru 0x003bc4 (0x1842 bytes) 000040 1735099850.989500 write 0x00f93e thru 0x01efdb (0xf69e bytes) 000041 1735099850.994084 MIRROR_RESYNC 2 mirrors 000042 1735099851.172156 write 0x02de37 thru 0x03186e (0x3a38 bytes) 000045 1735099851.198673 read 0x00acba thru 0x01157e (0x68c5 bytes) 000046 1735099851.202117 trunc from 0x03efe9 to 0x03949f (0x5b49 bytes) 000047 1735099851.211485 write 0x0150de thru 0x01bec0 (0x6de3 bytes) 000048 1735099851.215502 mapread 0x00ee85 thru 0x01dae6 (0xec62 bytes) 000049 1735099851.216254 mapwrite 0x03f7a2 thru 0x03ffff (0x85e bytes) 000050 1735099851.251582 read 0x03778a thru 0x03efe4 (0x785b bytes) 000053 1735099851.256580 read 0x000087 thru 0x00a8b4 (0xa82e bytes) 000054 1735099851.257026 mapwrite 0x01b763 thru 0x022494 (0x6d32 bytes) 000059 1735099851.285727 mapread 0x017059 thru 0x0195cf (0x2577 bytes) 000061 1735099851.286834 mapread 0x0123af thru 0x018797 (0x63e9 bytes) 000063 1735099851.287729 write 0x01d5c1 thru 0x0264df (0x8f1f bytes) 000064 1735099851.288557 mapwrite 0x00d2b2 thru 0x00f268 (0x1fb7 bytes) 000065 1735099851.320561 write 0x03b722 thru 0x03e394 (0x2c73 bytes) 000066 1735099851.321180 mapwrite 0x01c713 thru 0x01e527 (0x1e15 bytes) 000068 1735099851.345518 mapwrite 0x02bc78 thru 0x0319b7 (0x5d40 bytes) 000069 1735099851.374402 read 0x03f5b1 thru 0x03ffff (0xa4f bytes) 000071 1735099851.375852 read 0x035f27 thru 0x03e535 (0x860f bytes) 000072 1735099851.376299 write 0x032d3f thru 0x033653 (0x915 bytes) 000073 1735099851.378751 trunc from 0x040000 to 0x0118ae (0x2e751 bytes) 000074 1735099851.386620 mapwrite 0x02e8ee thru 0x038b7a (0xa28d bytes) 000075 1735099851.454375 read 0x00377f thru 0x00c435 (0x8cb7 bytes) 000078 1735099851.455923 mapread 0x019656 thru 0x021c9c (0x8647 bytes) 000079 1735099851.456591 mapwrite 0x031797 thru 0x03e91b (0xd185 bytes) 000082 1735099851.512177 read 0x0003f0 thru 0x005720 (0x5331 bytes) 000083 1735099851.515076 trunc from 0x03e91c to 0x039da7 (0x4b74 bytes) 000084 1735099851.523444 mapwrite 0x00c4a9 thru 0x014b26 (0x867e bytes) 000087 1735099851.558764 write 0x03b6f6 thru 0x03ffff (0x490a bytes) 000088 1735099851.559476 read 0x014ecc thru 0x02399c (0xead1 bytes) 000089 1735099851.561704 MIRROR_RESYNC 2 mirrors 000090 1735099851.711337 write 0x01b056 thru 0x0263d1 (0xb37c bytes) 000091 1735099851.757774 read 0x0225a1 thru 0x03044f (0xdeaf bytes) 000092 1735099851.762340 trunc from 0x040000 to 0x019ba1 (0x2645e bytes) 000093 1735099851.778380 mapwrite 0x02e9b3 thru 0x037be5 (0x9233 bytes) 000095 1735099851.825243 trunc from 0x037be6 to 0x01d1ba (0x1aa2b bytes) 000096 1735099851.831875 write 0x01ead6 thru 0x0239c7 (0x4ef2 bytes) 000101 1735099851.834130 write 0x01014b thru 0x0149d4 (0x488a bytes) 000102 1735099851.840914 MIRROR_EXTEND to 3 mirrors 000103 1735099852.045352 MIRROR_RESYNC 3 mirrors 000104 1735099852.239187 trunc from 0x0239c8 to 0x006dbc (0x1cc0b bytes) 000105 1735099852.284555 trunc from 0x006dbd to 0x036562 (0x2f7a6 bytes) 000107 1735099852.297679 write 0x005c61 thru 0x0114c1 (0xb861 bytes) 000108 1735099852.303591 trunc from 0x036563 to 0x01de2e (0x18734 bytes) 000109 1735099852.311060 write 0x026d63 thru 0x035104 (0xe3a2 bytes) 000110 1735099852.312251 mapwrite 0x027408 thru 0x028d4f (0x1948 bytes) 000112 1735099852.352490 MIRROR_RESYNC 3 mirrors 000113 1735099852.568496 mapread 0x0134a8 thru 0x021337 (0xde90 bytes) 000115 1735099852.587615 trunc from 0x035105 to 0x02a58f (0xab75 bytes) 000116 1735099852.630723 read 0x020200 thru 0x02a58f (0xa390 bytes) 000119 1735099852.636769 MIRROR_EXTEND to 4 mirrors 000122 1735099852.843373 mapread 0x024aac thru 0x027f80 (0x34d5 bytes) 000123 1735099852.853751 trunc from 0x02a590 to 0x03cf87 (0x129f8 bytes) 000124 1735099852.891757 mapread 0x038d8f thru 0x03cf87 (0x41f9 bytes) 000127 1735099852.899183 MIRROR_EXTEND to 5 mirrors 000133 1735099853.172443 mapread 0x02df9a thru 0x02e376 (0x3dd bytes) truncating to largest ever: 0x3fdcb 000134 1735099853.181985 trunc from 0x03cf88 to 0x03fdca (0x2e43 bytes) 000136 1735099853.211165 write 0x01bbfc thru 0x024717 (0x8b1c bytes) 000141 1735099853.218228 mapread 0x025bfb thru 0x0299fc (0x3e02 bytes) 000143 1735099853.224024 mapwrite 0x023063 thru 0x02b416 (0x83b4 bytes) 000145 1735099853.264701 mapwrite 0x01cf3f thru 0x02a26a (0xd32c bytes) 000146 1735099853.300886 read 0x01ab45 thru 0x021b95 (0x7051 bytes) 000147 1735099853.301502 trunc from 0x03fdcb to 0x035660 (0xa76a bytes) 000148 1735099853.309785 trunc from 0x035661 to 0x00a237 (0x2b429 bytes) 000149 1735099853.315675 mapwrite 0x032b12 thru 0x035426 (0x2915 bytes) 000150 1735099853.352839 mapwrite 0x03e043 thru 0x03ffff (0x1fbd bytes) 000151 1735099853.389869 mapread 0x032f48 thru 0x03c99f (0x9a58 bytes) 000152 1735099853.393638 mapread 0x02bc98 thru 0x03b1e7 (0xf550 bytes) 000153 1735099853.394312 write 0x014b86 thru 0x019d43 (0x51be bytes) 000154 1735099853.398933 mapwrite 0x00b381 thru 0x01a94b (0xf5cb bytes) 000155 1735099853.438362 read 0x00ffd8 thru 0x01288a (0x28b3 bytes) 000159 1735099853.439523 mapread 0x030675 thru 0x039bd0 (0x955c bytes) 000162 1735099853.440751 mapread 0x024f0c thru 0x02d423 (0x8518 bytes) 000163 1735099853.445646 mapread 0x003d65 thru 0x003f76 (0x212 bytes) 000164 1735099853.446247 read 0x0161ab thru 0x019767 (0x35bd bytes) 000166 1735099853.446967 trunc from 0x040000 to 0x00b850 (0x347af bytes) 000167 1735099853.455146 read 0x001f5f thru 0x005a60 (0x3b02 bytes) 000170 1735099853.456365 mapread 0x004562 thru 0x0051e6 (0xc85 bytes) 000171 1735099853.457164 mapread 0x00a420 thru 0x00b850 (0x1431 bytes) 000172 1735099853.457882 write 0x01e984 thru 0x0251fa (0x6877 bytes) 000173 1735099853.459118 write 0x02c542 thru 0x0368c5 (0xa384 bytes) 000174 1735099853.462379 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000176 1735099853.544770 trunc from 0x0368c6 to 0x03ce79 (0x65b4 bytes) 000177 1735099853.555932 MIRROR_RESYNC 4 mirrors 000179 1735099853.862025 trunc from 0x03ce7a to 0x00d304 (0x2fb75 bytes) 000181 1735099853.906747 mapread 0x008b94 thru 0x00d304 (0x4771 bytes) 000182 1735099853.920383 write 0x030d1f thru 0x035e38 (0x511a bytes) 000183 1735099853.921284 write 0x02dc89 thru 0x03d3ac (0xf724 bytes) 000184 1735099853.926229 write 0x032f9c thru 0x03e267 (0xb2cc bytes) 000185 1735099853.927151 write 0x02c1d0 thru 0x032651 (0x6482 bytes) 000187 1735099853.936999 MIRROR_RESYNC 4 mirrors 000189 1735099854.284964 mapread 0x032c12 thru 0x03e267 (0xb656 bytes) 000192 1735099854.299037 mapwrite 0x0096bd thru 0x010949 (0x728d bytes) 000193 1735099854.393639 read 0x00b7b9 thru 0x017649 (0xbe91 bytes) 000195 1735099854.399588 mapread 0x038afe thru 0x0396d9 (0xbdc bytes) 000196 1735099854.405889 write 0x0164d2 thru 0x024b8d (0xe6bc bytes) 000197 1735099854.412767 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000198 1735099854.507988 write 0x008346 thru 0x017748 (0xf403 bytes) 000200 1735099854.512941 trunc from 0x03e268 to 0x001e0b (0x3c45c bytes) 000201 1735099854.521594 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000202 1735099854.624096 mapread 0x001beb thru 0x001e0b (0x221 bytes) 000205 1735099854.630493 mapwrite 0x0247e3 thru 0x033b02 (0xf320 bytes) 000207 1735099854.700472 read 0x023879 thru 0x0324c5 (0xec4d bytes) 000208 1735099854.702149 read 0x0213cf thru 0x026660 (0x5292 bytes) 000209 1735099854.702775 write 0x0146c4 thru 0x01bf32 (0x786f bytes) 000210 1735099854.711998 write 0x007101 thru 0x00dde8 (0x6ce8 bytes) 000211 1735099854.712797 trunc from 0x033b03 to 0x01ed09 (0x14df9 bytes) 000213 1735099854.721387 trunc from 0x01ed0a to 0x00c7ef (0x1251a bytes) 000215 1735099854.731622 write 0x03b447 thru 0x03ffff (0x4bb9 bytes) 000217 1735099854.732550 mapwrite 0x020a4c thru 0x0237ac (0x2d61 bytes) 000219 1735099854.784686 read 0x02e650 thru 0x037dc0 (0x9771 bytes) 000222 1735099854.797646 MIRROR_RESYNC 2 mirrors 000223 1735099855.002647 trunc from 0x040000 to 0x03af8d (0x5072 bytes) 000225 1735099855.052941 mapread 0x022d89 thru 0x02be66 (0x90de bytes) 000227 1735099855.063501 read 0x031404 thru 0x0394f3 (0x80f0 bytes) 000230 1735099855.069703 trunc from 0x03af8e to 0x017f44 (0x23049 bytes) 000232 1735099855.080979 mapread 0x0160b1 thru 0x017f44 (0x1e94 bytes) 000234 1735099855.092234 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000237 1735099855.167703 MIRROR_EXTEND to 2 mirrors 000238 1735099855.375677 write 0x03415d thru 0x03ffff (0xbea3 bytes) 000239 1735099855.407323 trunc from 0x040000 to 0x028839 (0x177c6 bytes) 000241 1735099855.427431 trunc from 0x02883a to 0x033389 (0xab50 bytes) 000242 1735099855.435039 mapread 0x02545c thru 0x027250 (0x1df5 bytes) 000243 1735099855.448025 mapread 0x02165e thru 0x02d831 (0xc1d4 bytes) 000244 1735099855.448614 trunc from 0x03338a to 0x025dc8 (0xd5c1 bytes) 000245 1735099855.457368 read 0x022637 thru 0x025dc8 (0x3792 bytes) 000249 1735099855.458181 write 0x007067 thru 0x00e139 (0x70d3 bytes) 000250 1735099855.462455 mapread 0x00bbb6 thru 0x01a93e (0xed89 bytes) 000251 1735099855.476065 mapread 0x003798 thru 0x0103bc (0xcc25 bytes) 000253 1735099855.480636 MIRROR_RESYNC 2 mirrors 000255 1735099855.668079 mapwrite 0x026022 thru 0x0270be (0x109d bytes) 000256 1735099855.736524 mapwrite 0x032454 thru 0x03740c (0x4fb9 bytes) 000258 1735099855.788136 trunc from 0x03740d to 0x039cc8 (0x28bc bytes) 000259 1735099855.798122 mapread 0x006804 thru 0x00abc7 (0x43c4 bytes) 000260 1735099855.803268 write 0x01231c thru 0x015631 (0x3316 bytes) 000261 1735099855.806925 mapwrite 0x025625 thru 0x02681b (0x11f7 bytes) 000266 1735099855.836716 write 0x010995 thru 0x018dab (0x8417 bytes) 000268 1735099855.840345 read 0x02225e thru 0x02b263 (0x9006 bytes) 000269 1735099855.840802 write 0x0364b5 thru 0x03ffff (0x9b4b bytes) 000271 1735099855.841955 trunc from 0x040000 to 0x0129c3 (0x2d63c bytes) 000272 1735099855.849473 write 0x00f6af thru 0x01dc82 (0xe5d4 bytes) 000274 1735099855.850881 write 0x0015e9 thru 0x005fc5 (0x49dd bytes) 000275 1735099855.854084 mapwrite 0x02364e thru 0x02bf61 (0x8914 bytes) 000277 1735099855.895780 mapwrite 0x01a9d8 thru 0x01c652 (0x1c7b bytes) 000281 1735099855.920013 mapwrite 0x02c7be thru 0x02e72e (0x1f71 bytes) 000284 1735099855.970895 write 0x01492c thru 0x021e1c (0xd4f1 bytes) 000285 1735099855.972071 trunc from 0x02e72f to 0x023170 (0xb5be bytes) 000286 1735099855.982149 mapread 0x0124a3 thru 0x01273f (0x29d bytes) 000287 1735099855.982944 mapread 0x00f016 thru 0x01cb27 (0xdb12 bytes) 000288 1735099855.983544 mapwrite 0x038167 thru 0x03ffff (0x7e99 bytes) 000292 1735099856.034418 mapread 0x03d0a2 thru 0x03ffff (0x2f5e bytes) 000293 1735099856.035030 read 0x012793 thru 0x01a03f (0x78ad bytes) 000294 1735099856.037801 MIRROR_RESYNC 2 mirrors 000296 1735099856.244510 mapwrite 0x015aeb thru 0x0200f2 (0xa608 bytes) 000298 1735099856.316400 write 0x018c38 thru 0x028852 (0xfc1b bytes) 000299 1735099856.319909 trunc from 0x040000 to 0x00c2aa (0x33d55 bytes) 000300 1735099856.328194 trunc from 0x00c2ab to 0x034880 (0x285d6 bytes) 000302 1735099856.334744 read 0x017ab9 thru 0x01de20 (0x6368 bytes) 000309 1735099856.339467 mapread 0x01ef5c thru 0x02c570 (0xd615 bytes) 000310 1735099856.350627 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000311 1735099856.402195 write 0x00c1cf thru 0x014513 (0x8345 bytes) 000313 1735099856.408455 mapwrite 0x01d99e thru 0x02b2b1 (0xd914 bytes) 000314 1735099856.452327 read 0x01ea67 thru 0x020717 (0x1cb1 bytes) 000315 1735099856.452767 write 0x03ccb6 thru 0x03ffff (0x334a bytes) 000316 1735099856.453810 read 0x005e94 thru 0x00d7cb (0x7938 bytes) 000317 1735099856.460010 trunc from 0x040000 to 0x00f27d (0x30d82 bytes) 000323 1735099856.475561 MIRROR_EXTEND to 2 mirrors 000327 1735099856.692645 read 0x00e632 thru 0x00f27d (0xc4c bytes) 000329 1735099856.695630 mapread 0x008597 thru 0x00c0b5 (0x3b1f bytes) 000330 1735099856.711816 MIRROR_RESYNC 2 mirrors 000331 1735099856.772293 mapwrite 0x015616 thru 0x0236c4 (0xe0af bytes) 000333 1735099856.861010 trunc from 0x0236c5 to 0x006ddc (0x1c8e8 bytes) 000334 1735099856.870795 mapwrite 0x025855 thru 0x034bc5 (0xf371 bytes) 000335 1735099856.933801 mapread 0x0005a1 thru 0x00683a (0x629a bytes) 000336 1735099856.944024 read 0x00349d thru 0x00fdd7 (0xc93b bytes) 000340 1735099856.945832 mapread 0x025760 thru 0x02908e (0x392f bytes) 000342 1735099856.947113 write 0x014a37 thru 0x01f49d (0xaa67 bytes) 000343 1735099856.953071 read 0x030972 thru 0x034bc5 (0x4254 bytes) 000344 1735099856.953574 mapwrite 0x0087e3 thru 0x00a115 (0x1933 bytes) 000346 1735099856.988683 mapwrite 0x001a4c thru 0x00d94a (0xbeff bytes) 000347 1735099857.023452 mapread 0x028e5a thru 0x034bc5 (0xbd6c bytes) 000348 1735099857.024564 trunc from 0x034bc6 to 0x005ad3 (0x2f0f2 bytes) 000349 1735099857.035575 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000350 1735099857.095041 write 0x037258 thru 0x03ffff (0x8da8 bytes) 000352 1735099857.096441 write 0x00df8d thru 0x00f65b (0x16cf bytes) 000353 1735099857.101732 read 0x0002d5 thru 0x001c7e (0x19aa bytes) 000354 1735099857.104936 mapwrite 0x02a378 thru 0x035e76 (0xbaff bytes) 000355 1735099857.147424 write 0x018a0b thru 0x02773a (0xed30 bytes) 000356 1735099857.153265 read 0x01abb3 thru 0x01f05f (0x44ad bytes) 000358 1735099857.153992 mapwrite 0x03566f thru 0x03ffff (0xa991 bytes) 000359 1735099857.183116 mapread 0x0068d9 thru 0x010416 (0x9b3e bytes) 000360 1735099857.190899 mapread 0x0109fc thru 0x01387e (0x2e83 bytes) 000362 1735099857.191700 trunc from 0x040000 to 0x009173 (0x36e8c bytes) 000366 1735099857.203360 MIRROR_EXTEND to 2 mirrors 000367 1735099857.382016 mapwrite 0x00dcf5 thru 0x00fdc4 (0x20d0 bytes) 000368 1735099857.471968 mapwrite 0x0047aa thru 0x00abf5 (0x644c bytes) 000369 1735099857.508434 read 0x000741 thru 0x00bcbb (0xb57b bytes) 000370 1735099857.510769 mapread 0x002ac0 thru 0x004010 (0x1551 bytes) 000371 1735099857.511460 read 0x0069e9 thru 0x00a0d0 (0x36e8 bytes) 000374 1735099857.512490 write 0x01e5a4 thru 0x02971c (0xb179 bytes) 000375 1735099857.513441 read 0x01f31e thru 0x02971c (0xa3ff bytes) 000377 1735099857.514237 write 0x004c0b thru 0x008c96 (0x408c bytes) 000379 1735099857.515378 mapwrite 0x02fef6 thru 0x0389e6 (0x8af1 bytes) 000380 1735099857.574802 read 0x03299b thru 0x0389e6 (0x604c bytes) 000382 1735099857.576824 read 0x00e29e thru 0x015166 (0x6ec9 bytes) 000384 1735099857.577575 trunc from 0x0389e7 to 0x00553c (0x334aa bytes) 000385 1735099857.589026 MIRROR_RESYNC 2 mirrors 000387 1735099857.735108 trunc from 0x00553d to 0x028590 (0x23054 bytes) 000388 1735099857.783968 mapread 0x017c2a thru 0x025782 (0xdb59 bytes) 000390 1735099857.801160 mapwrite 0x00f391 thru 0x01a5d2 (0xb242 bytes) 000392 1735099857.850338 write 0x02097e thru 0x02836a (0x79ed bytes) 000393 1735099857.851146 trunc from 0x028591 to 0x021759 (0x6e37 bytes) 000394 1735099857.862888 read 0x0181f1 thru 0x021759 (0x9569 bytes) 000395 1735099857.867381 MIRROR_EXTEND to 3 mirrors 000396 1735099858.102444 read 0x00292e thru 0x0073c5 (0x4a98 bytes) 000402 1735099858.109539 read 0x01e742 thru 0x021759 (0x3018 bytes) 000404 1735099858.115078 trunc from 0x02175a to 0x017c08 (0x9b51 bytes) 000405 1735099858.155543 read 0x00a806 thru 0x017c08 (0xd403 bytes) 000406 1735099858.163077 write 0x00576a thru 0x009e49 (0x46e0 bytes) 000409 1735099858.165435 write 0x03e73b thru 0x03ffff (0x18c5 bytes) 000410 1735099858.168076 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000413 1735099858.290542 mapread 0x00b43e thru 0x00f071 (0x3c34 bytes) 000415 1735099858.298614 mapwrite 0x037220 thru 0x03ffff (0x8de0 bytes) 000416 1735099858.332189 trunc from 0x040000 to 0x016f87 (0x29078 bytes) 000417 1735099858.342286 write 0x032661 thru 0x03ffff (0xd99f bytes) 000418 1735099858.343421 mapwrite 0x03e9c0 thru 0x03ffff (0x1640 bytes) 000419 1735099858.371607 trunc from 0x040000 to 0x03197d (0xe682 bytes) 000420 1735099858.378723 read 0x00b3a8 thru 0x00f3ac (0x4005 bytes) 000421 1735099858.379240 mapread 0x007742 thru 0x008122 (0x9e1 bytes) 000422 1735099858.383717 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000426 1735099858.445457 mapread 0x02f863 thru 0x03197d (0x211b bytes) 000427 1735099858.454642 write 0x02fc07 thru 0x03c489 (0xc883 bytes) 000428 1735099858.455874 trunc from 0x03c48a to 0x000d70 (0x3b719 bytes) 000429 1735099858.463247 mapread 0x000315 thru 0x000d70 (0xa5c bytes) 000432 1735099858.467168 mapwrite 0x03a9aa thru 0x03c849 (0x1ea0 bytes) 000437 1735099858.518337 write 0x016b7c thru 0x019013 (0x2498 bytes) 000443 1735099858.523896 trunc from 0x03c84a to 0x010fdb (0x2b86e bytes) 000446 1735099858.530979 trunc from 0x010fdc to 0x000f65 (0x10076 bytes) 000451 1735099858.539322 mapread 0x000684 thru 0x000f65 (0x8e2 bytes) 000457 1735099858.543453 MIRROR_EXTEND to 2 mirrors 000459 1735099858.710753 trunc from 0x000f66 to 0x015d9b (0x14e36 bytes) 000460 1735099858.756089 read 0x004b96 thru 0x00549a (0x905 bytes) 000463 1735099858.762272 mapwrite 0x01083b thru 0x0167a4 (0x5f6a bytes) 000464 1735099858.828919 trunc from 0x0167a5 to 0x03b984 (0x251e0 bytes) 000466 1735099858.841028 mapwrite 0x0300ae thru 0x03a4d6 (0xa429 bytes) 000468 1735099858.883671 trunc from 0x03b985 to 0x0219fd (0x19f87 bytes) 000470 1735099858.892143 mapwrite 0x033d38 thru 0x03e8c7 (0xab90 bytes) 000471 1735099858.933692 mapwrite 0x00a283 thru 0x01867a (0xe3f8 bytes) 000474 1735099858.972279 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000475 1735099859.013354 mapread 0x0227d7 thru 0x027223 (0x4a4d bytes) 000479 1735099859.025060 mapwrite 0x034401 thru 0x03ffff (0xbbff bytes) 000480 1735099859.066906 read 0x0367b4 thru 0x03f858 (0x90a5 bytes) 000481 1735099859.067399 mapwrite 0x013898 thru 0x01d06f (0x97d8 bytes) 000482 1735099859.097224 read 0x00f8ef thru 0x011cff (0x2411 bytes) 000484 1735099859.097921 read 0x036e67 thru 0x038fec (0x2186 bytes) 000485 1735099859.098321 mapread 0x03f1f7 thru 0x03ffff (0xe09 bytes) 000486 1735099859.098844 mapwrite 0x02cdc1 thru 0x03366c (0x68ac bytes) 000488 1735099859.128323 read 0x028c76 thru 0x035b20 (0xceab bytes) 000489 1735099859.128752 mapwrite 0x021cdd thru 0x02dec9 (0xc1ed bytes) 000492 1735099859.155952 read 0x034749 thru 0x03a11d (0x59d5 bytes) 000493 1735099859.156401 write 0x00b0f0 thru 0x0163b3 (0xb2c4 bytes) 000496 1735099859.161626 trunc from 0x040000 to 0x00f6b7 (0x30948 bytes) 000497 1735099859.168586 mapread 0x00206c thru 0x00bdea (0x9d7f bytes) 000498 1735099859.169370 mapread 0x00a3d4 thru 0x00f6b7 (0x52e4 bytes) 000499 1735099859.169880 read 0x000861 thru 0x004386 (0x3b26 bytes) 000500 1735099859.172491 MIRROR_EXTEND to 2 mirrors 000501 1735099859.319989 MIRROR_EXTEND to 3 mirrors 000502 1735099859.462888 read 0x00c12e thru 0x00f6b7 (0x358a bytes) 000503 1735099859.467618 trunc from 0x00f6b8 to 0x0093f1 (0x62c6 bytes) 000504 1735099859.499370 mapwrite 0x00563d thru 0x015536 (0xfefa bytes) 000505 1735099859.539014 mapwrite 0x03ed1a thru 0x03ffff (0x12e6 bytes) 000509 1735099859.583324 trunc from 0x040000 to 0x003e16 (0x3c1e9 bytes) 000510 1735099859.589943 trunc from 0x003e17 to 0x022a95 (0x1ec7f bytes) 000511 1735099859.598662 mapwrite 0x03020a thru 0x037cfd (0x7af4 bytes) 000512 1735099859.630246 mapwrite 0x001503 thru 0x010a78 (0xf576 bytes) 000517 1735099859.660173 mapwrite 0x00f17f thru 0x017046 (0x7ec8 bytes) 000518 1735099859.690633 write 0x0037f5 thru 0x008075 (0x4881 bytes) 000519 1735099859.691427 mapread 0x00037f thru 0x00c5c3 (0xc245 bytes) 000521 1735099859.696767 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000522 1735099859.774326 MIRROR_EXTEND to 3 mirrors 000524 1735099859.922969 read 0x01e5af thru 0x02c64f (0xe0a1 bytes) 000525 1735099859.927403 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000527 1735099859.995412 read 0x030aee thru 0x037cfd (0x7210 bytes) 000528 1735099860.000378 mapread 0x01c2b4 thru 0x02a402 (0xe14f bytes) 000529 1735099860.011020 trunc from 0x037cfe to 0x00fe55 (0x27ea8 bytes) 000530 1735099860.027787 write 0x01723f thru 0x023884 (0xc646 bytes) 000531 1735099860.028919 read 0x00a04e thru 0x011f5a (0x7f0d bytes) 000532 1735099860.036004 MIRROR_EXTEND to 3 mirrors 000534 1735099860.241408 write 0x00494c thru 0x00e050 (0x9705 bytes) 000535 1735099860.270971 trunc from 0x023885 to 0x02c6f4 (0x8e70 bytes) 000537 1735099860.287242 mapwrite 0x0164ef thru 0x02261e (0xc130 bytes) 000538 1735099860.342083 mapwrite 0x01520d thru 0x0196f4 (0x44e8 bytes) 000542 1735099860.370763 mapwrite 0x009a10 thru 0x0178d5 (0xdec6 bytes) 000545 1735099860.404869 write 0x003a86 thru 0x004c54 (0x11cf bytes) 000546 1735099860.408438 trunc from 0x02c6f5 to 0x02d83a (0x1146 bytes) 000547 1735099860.415736 mapwrite 0x00a3da thru 0x00f83e (0x5465 bytes) 000548 1735099860.451586 read 0x00b970 thru 0x017fbc (0xc64d bytes) 000549 1735099860.452291 trunc from 0x02d83b to 0x0382ef (0xaab5 bytes) 000551 1735099860.462084 write 0x02cd68 thru 0x034fb2 (0x824b bytes) 000554 1735099860.469412 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000555 1735099860.546960 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000556 1735099860.605649 write 0x01c78f thru 0x0274cb (0xad3d bytes) 000557 1735099860.609679 read 0x019811 thru 0x01b778 (0x1f68 bytes) 000559 1735099860.612445 write 0x002e83 thru 0x006e20 (0x3f9e bytes) 000561 1735099860.617048 write 0x001a3e thru 0x003720 (0x1ce3 bytes) 000562 1735099860.617607 mapread 0x030271 thru 0x0382ef (0x807f bytes) 000563 1735099860.621632 mapread 0x00167f thru 0x0054da (0x3e5c bytes) 000566 1735099860.622559 mapwrite 0x01a21e thru 0x01c897 (0x267a bytes) 000568 1735099860.653757 mapwrite 0x023c34 thru 0x0258b5 (0x1c82 bytes) 000569 1735099860.677486 mapwrite 0x02e643 thru 0x0343b1 (0x5d6f bytes) 000573 1735099860.700671 mapread 0x0053d6 thru 0x0113ea (0xc015 bytes) 000576 1735099860.705185 mapread 0x028640 thru 0x036592 (0xdf53 bytes) 000578 1735099860.708260 trunc from 0x0382f0 to 0x02704a (0x112a5 bytes) 000580 1735099860.715997 mapwrite 0x013329 thru 0x016c8e (0x3966 bytes) 000582 1735099860.745938 write 0x009852 thru 0x011e7e (0x862d bytes) 000583 1735099860.746841 mapwrite 0x0390e4 thru 0x03ffff (0x6f1c bytes) 000587 1735099860.782868 mapread 0x03ee9f thru 0x03ffff (0x1161 bytes) 000588 1735099860.783485 write 0x0206fb thru 0x02ae7a (0xa780 bytes) 000589 1735099860.786374 mapread 0x038f57 thru 0x03ffff (0x70a9 bytes) 000591 1735099860.787565 read 0x031b89 thru 0x038f3f (0x73b7 bytes) 000592 1735099860.791901 read 0x008e69 thru 0x009f05 (0x109d bytes) 000597 1735099860.793545 mapread 0x039307 thru 0x03d401 (0x40fb bytes) 000598 1735099860.794116 read 0x0319d0 thru 0x03ae35 (0x9466 bytes) 000599 1735099860.794474 write 0x03cb8e thru 0x03ffff (0x3472 bytes) 000602 1735099860.795630 write 0x01d715 thru 0x02bb8e (0xe47a bytes) 000603 1735099860.796527 mapread 0x0178c5 thru 0x020fe0 (0x971c bytes) 000605 1735099860.797375 read 0x01e76d thru 0x026585 (0x7e19 bytes) 000606 1735099860.797717 mapread 0x03a778 thru 0x03ffff (0x5888 bytes) 000607 1735099860.798239 read 0x0024c2 thru 0x00d861 (0xb3a0 bytes) 000608 1735099860.798592 read 0x0350a4 thru 0x03be5e (0x6dbb bytes) 000609 1735099860.798934 trunc from 0x040000 to 0x0074eb (0x38b14 bytes) 000613 1735099860.806215 trunc from 0x0074ec to 0x03f6d3 (0x381e8 bytes) 000614 1735099860.811490 read 0x01e7b2 thru 0x028de5 (0xa634 bytes) 000618 1735099860.816987 read 0x02debe thru 0x03521a (0x735d bytes) 000622 1735099860.818464 mapwrite 0x0305f4 thru 0x03ed9e (0xe7ab bytes) 000625 1735099860.845112 write 0x03bdb1 thru 0x03ffff (0x424f bytes) 000626 1735099860.846085 trunc from 0x040000 to 0x03f0e6 (0xf19 bytes) 000628 1735099860.851811 trunc from 0x03f0e7 to 0x01d832 (0x218b4 bytes) 000630 1735099860.861822 trunc from 0x01d833 to 0x01d763 (0xcf bytes) 000632 1735099860.870784 mapread 0x01a22d thru 0x01aa14 (0x7e8 bytes) 000633 1735099860.871557 mapread 0x007638 thru 0x00ad46 (0x370f bytes) 000635 1735099860.872538 trunc from 0x01d764 to 0x004a14 (0x18d4f bytes) 000636 1735099860.879494 trunc from 0x004a15 to 0x036583 (0x31b6f bytes) 000637 1735099860.886591 read 0x019fb5 thru 0x020a7b (0x6ac7 bytes) 000638 1735099860.891994 mapread 0x00733d thru 0x00d025 (0x5ce9 bytes) 000639 1735099860.893346 mapwrite 0x0161ed thru 0x01f00f (0x8e23 bytes) 000641 1735099860.918171 trunc from 0x036584 to 0x03ef00 (0x897d bytes) 000642 1735099860.923803 mapread 0x032dea thru 0x036494 (0x36ab bytes) 000643 1735099860.924445 mapread 0x01e3a0 thru 0x029f92 (0xbbf3 bytes) 000644 1735099860.926559 MIRROR_EXTEND to 2 mirrors 000645 1735099861.106334 mapread 0x015620 thru 0x01cdc0 (0x77a1 bytes) 000646 1735099861.115138 mapread 0x02f0a1 thru 0x0350f2 (0x6052 bytes) 000648 1735099861.122832 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000649 1735099861.181683 read 0x0279e7 thru 0x032164 (0xa77e bytes) 000650 1735099861.186866 read 0x00e8b3 thru 0x01615c (0x78aa bytes) 000652 1735099861.198891 MIRROR_EXTEND to 2 mirrors 000657 1735099861.374764 mapwrite 0x011e8b thru 0x014fbb (0x3131 bytes) 000660 1735099861.445245 write 0x02250b thru 0x0253e7 (0x2edd bytes) 000661 1735099861.453051 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000664 1735099861.502346 trunc from 0x03ef01 to 0x02c5ed (0x12913 bytes) 000665 1735099861.508551 mapwrite 0x00dbdc thru 0x01448b (0x68b0 bytes) 000667 1735099861.547991 write 0x00d50a thru 0x01974a (0xc241 bytes) 000668 1735099861.552572 mapwrite 0x03594a thru 0x037471 (0x1b28 bytes) 000669 1735099861.596932 trunc from 0x037472 to 0x03f674 (0x8203 bytes) 000670 1735099861.603787 mapwrite 0x013ed6 thru 0x01a32f (0x645a bytes) 000672 1735099861.643538 mapwrite 0x00eb38 thru 0x0140fb (0x55c4 bytes) 000673 1735099861.663676 trunc from 0x03f675 to 0x002ff4 (0x3c680 bytes) 000677 1735099861.670956 read 0x0015b9 thru 0x002ff4 (0x1a3c bytes) 000678 1735099861.673833 mapread 0x000588 thru 0x002ff4 (0x2a6d bytes) 000681 1735099861.676896 read 0x0004bf thru 0x002ff4 (0x2b36 bytes) 000683 1735099861.677759 read 0x0018c9 thru 0x002ff4 (0x172c bytes) 000686 1735099861.678884 read 0x00202c thru 0x002ff4 (0xfc9 bytes) 000687 1735099861.679286 write 0x0154b5 thru 0x01b46c (0x5fb8 bytes) 000690 1735099861.680643 mapread 0x00b4d6 thru 0x01487e (0x93a9 bytes) 000695 1735099861.687138 trunc from 0x01b46d to 0x011781 (0x9ceb bytes) 000697 1735099861.695069 mapwrite 0x02848e thru 0x0290ff (0xc72 bytes) 000699 1735099861.727120 trunc from 0x029100 to 0x03209c (0x8f9d bytes) 000700 1735099861.733782 trunc from 0x03209d to 0x02e2d9 (0x3dc3 bytes) 000702 1735099861.741581 trunc from 0x02e2da to 0x0285ec (0x5ced bytes) 000703 1735099861.752147 read 0x001da9 thru 0x010830 (0xea88 bytes) 000704 1735099861.752721 read 0x025fc0 thru 0x0285ec (0x262d bytes) 000705 1735099861.753137 write 0x00eed9 thru 0x014df7 (0x5f1f bytes) 000706 1735099861.757476 trunc from 0x0285ed to 0x021800 (0x6dec bytes) 000707 1735099861.765571 mapread 0x00e27f thru 0x012e82 (0x4c04 bytes) 000708 1735099861.766177 mapwrite 0x028de1 thru 0x031a84 (0x8ca4 bytes) 000709 1735099861.811253 trunc from 0x031a85 to 0x006b5b (0x2af29 bytes) 000711 1735099861.818374 mapwrite 0x0200c3 thru 0x025aac (0x59ea bytes) 000716 1735099861.881332 trunc from 0x025aad to 0x025083 (0xa29 bytes) 000717 1735099861.889487 read 0x021f15 thru 0x025083 (0x316f bytes) 000720 1735099861.890444 mapwrite 0x00a59d thru 0x0161af (0xbc13 bytes) 000724 1735099861.929317 mapread 0x013f8a thru 0x01f469 (0xb4e0 bytes) 000733 1735099861.932700 read 0x01992b thru 0x0219c2 (0x8098 bytes) 000734 1735099861.933115 trunc from 0x025084 to 0x00fcc8 (0x153bb bytes) 000738 1735099861.941677 write 0x028a51 thru 0x02915c (0x70c bytes) 000740 1735099861.942764 mapread 0x00a438 thru 0x013e1e (0x99e7 bytes) 000741 1735099861.948307 mapread 0x01c9e9 thru 0x02318e (0x67a6 bytes) 000742 1735099861.949756 mapread 0x01b71f thru 0x01f4a5 (0x3d87 bytes) 000744 1735099861.950925 trunc from 0x02915d to 0x035afd (0xc9a1 bytes) 000745 1735099861.957818 mapwrite 0x01f00c thru 0x023709 (0x46fe bytes) 000746 1735099861.987385 mapwrite 0x030246 thru 0x03913a (0x8ef5 bytes) 000749 1735099862.022190 mapread 0x012183 thru 0x020806 (0xe684 bytes) 000750 1735099862.022939 write 0x02a31d thru 0x0321a7 (0x7e8b bytes) 000752 1735099862.023916 mapread 0x02091a thru 0x02e2a7 (0xd98e bytes) 000759 1735099862.025968 read 0x021bfd thru 0x03099c (0xeda0 bytes) 000760 1735099862.026370 read 0x034c79 thru 0x03913a (0x44c2 bytes) 000761 1735099862.026703 write 0x00ec5c thru 0x01788c (0x8c31 bytes) 000762 1735099862.029407 MIRROR_EXTEND to 2 mirrors 000763 1735099862.187236 mapwrite 0x0227a2 thru 0x029030 (0x688f bytes) 000764 1735099862.257177 mapread 0x02be12 thru 0x035419 (0x9608 bytes) 000767 1735099862.264297 mapwrite 0x0198b5 thru 0x027721 (0xde6d bytes) 000768 1735099862.288429 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6f1:0x0]: Invalid argument (22) 000770 1735099862.321904 write 0x028e10 thru 0x02c58b (0x377c bytes) 000772 1735099862.325778 mapread 0x02c5ac thru 0x030f2e (0x4983 bytes) 000777 1735099862.331133 write 0x0254d7 thru 0x028e9a (0x39c4 bytes) 000778 1735099862.334718 mapwrite 0x02e8b6 thru 0x039367 (0xaab2 bytes) 000779 1735099862.371817 mapread 0x01b9c0 thru 0x027eb7 (0xc4f8 bytes) 000780 1735099862.376300 mapwrite 0x02877a thru 0x02a6ea (0x1f71 bytes) 000781 1735099862.393735 trunc from 0x039368 to 0x03703b (0x232c bytes) 000782 1735099862.399963 trunc from 0x03703c to 0x03d6fb (0x66c0 bytes) 000785 1735099862.407263 mapread 0x020a06 thru 0x0265e3 (0x5bde bytes) 000786 1735099862.407870 mapwrite 0x030aa8 thru 0x03b2e6 (0xa83f bytes) 000787 1735099862.433912 mapwrite 0x019b0c thru 0x027a52 (0xdf47 bytes) 000788 1735099862.455764 mapwrite 0x01a2ad thru 0x025978 (0xb6cc bytes) 000789 1735099862.478559 MIRROR_EXTEND to 2 mirrors 000790 1735099862.610860 MIRROR_EXTEND to 3 mirrors 000791 1735099862.741664 trunc from 0x03d6fc to 0x033aee (0x9c0d bytes) 000793 1735099862.766656 trunc from 0x033aef to 0x03d041 (0x9553 bytes) 000795 1735099862.771565 mapread 0x035ba6 thru 0x03d041 (0x749c bytes) 000799 1735099862.777913 mapread 0x01229d thru 0x01a727 (0x848b bytes) 000802 1735099862.783287 read 0x038a51 thru 0x03d041 (0x45f1 bytes) 000804 1735099862.783919 write 0x00e221 thru 0x01885e (0xa63e bytes) 000806 1735099862.785102 mapwrite 0x034151 thru 0x0365d6 (0x2486 bytes) 000808 1735099862.803059 trunc from 0x03d042 to 0x029d24 (0x1331d bytes) 000810 1735099862.808838 trunc from 0x029d25 to 0x02d107 (0x33e3 bytes) 000811 1735099862.815874 MIRROR_RESYNC 3 mirrors 000812 1735099862.970041 trunc from 0x02d108 to 0x01c485 (0x10c82 bytes) 000814 1735099863.001819 read 0x008254 thru 0x00bfd7 (0x3d84 bytes) 000816 1735099863.005580 mapwrite 0x01a848 thru 0x022889 (0x8042 bytes) 000817 1735099863.038783 mapwrite 0x007eea thru 0x00eed3 (0x6fea bytes) 000819 1735099863.057708 write 0x02ad83 thru 0x030344 (0x55c2 bytes) 000820 1735099863.058447 read 0x01a05a thru 0x01c552 (0x24f9 bytes) 000821 1735099863.058794 mapwrite 0x00feea thru 0x01428f (0x43a6 bytes) 000827 1735099863.079152 mapread 0x001b7a thru 0x0088fe (0x6d85 bytes) 000829 1735099863.081947 MIRROR_EXTEND to 4 mirrors 000831 1735099863.261466 mapwrite 0x00091a thru 0x00d2c8 (0xc9af bytes) 000832 1735099863.325341 trunc from 0x030345 to 0x02717b (0x91c9 bytes) 000834 1735099863.336609 write 0x001579 thru 0x003b86 (0x260e bytes) 000835 1735099863.337253 trunc from 0x02717c to 0x0111c3 (0x15fb8 bytes) 000836 1735099863.342458 mapwrite 0x011a3f thru 0x01b6e7 (0x9ca9 bytes) 000837 1735099863.375483 mapread 0x009396 thru 0x00f405 (0x6070 bytes) 000839 1735099863.376488 read 0x01a3bd thru 0x01b6e7 (0x132b bytes) 000841 1735099863.377229 read 0x0085ef thru 0x00956c (0xf7e bytes) 000842 1735099863.377610 read 0x00439d thru 0x0094b0 (0x5114 bytes) 000843 1735099863.378010 mapwrite 0x005cb5 thru 0x013438 (0xd784 bytes) 000844 1735099863.404562 MIRROR_EXTEND to 5 mirrors 000846 1735099863.550292 write 0x038c72 thru 0x03da95 (0x4e24 bytes) 000848 1735099863.576720 mapwrite 0x005831 thru 0x006199 (0x969 bytes) 000850 1735099863.618433 mapread 0x0076fe thru 0x0115ad (0x9eb0 bytes) 000851 1735099863.627422 mapwrite 0x031f70 thru 0x0347c6 (0x2857 bytes) 000854 1735099863.649480 write 0x007274 thru 0x00f558 (0x82e5 bytes) 000856 1735099863.650571 mapwrite 0x0399da thru 0x03ffff (0x6626 bytes) 000858 1735099863.687328 write 0x01c64b thru 0x01d809 (0x11bf bytes) 000859 1735099863.691676 mapread 0x00c6d5 thru 0x01476d (0x8099 bytes) 000862 1735099863.692997 trunc from 0x040000 to 0x00e7a1 (0x3185e bytes) 000865 1735099863.701079 mapwrite 0x022b92 thru 0x02cdf4 (0xa263 bytes) 000867 1735099863.742560 mapwrite 0x00f61d thru 0x01316f (0x3b53 bytes) 000868 1735099863.766895 read 0x014e04 thru 0x01ac95 (0x5e92 bytes) 000869 1735099863.770059 trunc from 0x02cdf5 to 0x033b1f (0x6d2b bytes) 000870 1735099863.779303 mapread 0x02e600 thru 0x033b1f (0x5520 bytes) 000871 1735099863.783442 write 0x0300e7 thru 0x03ed3c (0xec56 bytes) 000872 1735099863.784687 trunc from 0x03ed3d to 0x0219ae (0x1d38e bytes) 000873 1735099863.791985 write 0x002556 thru 0x00594d (0x33f8 bytes) 000875 1735099863.793268 trunc from 0x0219af to 0x012391 (0xf61d bytes) 000876 1735099863.800662 trunc from 0x012392 to 0x011f13 (0x47e bytes) 000877 1735099863.808074 mapwrite 0x020900 thru 0x029d37 (0x9438 bytes) 000878 1735099863.850894 mapread 0x01e74a thru 0x023b49 (0x5400 bytes) 000879 1735099863.851689 mapwrite 0x0133fc thru 0x0161c7 (0x2dcc bytes) 000880 1735099863.883595 MIRROR_EXTEND to 6 mirrors 000881 1735099864.105963 trunc from 0x029d38 to 0x0131e7 (0x16b50 bytes) 000882 1735099864.133433 read 0x009901 thru 0x0131e7 (0x98e7 bytes) 000883 1735099864.136813 read 0x0002bf thru 0x006e6c (0x6bae bytes) 000884 1735099864.142126 write 0x0257a7 thru 0x030a4b (0xb2a5 bytes) 000885 1735099864.142935 read 0x0297c5 thru 0x030a4b (0x7287 bytes) 000886 1735099864.143319 mapwrite 0x014b42 thru 0x01bbdd (0x709c bytes) 000888 1735099864.182401 mapwrite 0x039cc8 thru 0x03ffff (0x6338 bytes) 000889 1735099864.225595 MIRROR_EXTEND to 7 mirrors 000890 1735099864.401788 trunc from 0x040000 to 0x01df98 (0x22067 bytes) 000892 1735099864.440179 mapread 0x0126a4 thru 0x016df0 (0x474d bytes) 000893 1735099864.448124 trunc from 0x01df99 to 0x009e4e (0x1414a bytes) 000894 1735099864.457241 trunc from 0x009e4f to 0x003f1f (0x5f2f bytes) 000897 1735099864.465811 write 0x00a320 thru 0x00b5d2 (0x12b3 bytes) 000898 1735099864.466544 mapread 0x004cdb thru 0x00b5d2 (0x68f8 bytes) 000899 1735099864.471571 mapread 0x000543 thru 0x00b5d2 (0xb090 bytes) 000900 1735099864.477321 write 0x03e56f thru 0x03ffff (0x1a91 bytes) 000901 1735099864.478151 trunc from 0x040000 to 0x023517 (0x1cae8 bytes) 000903 1735099864.486788 write 0x03fb03 thru 0x03ffff (0x4fd bytes) 000907 1735099864.488524 trunc from 0x040000 to 0x018637 (0x279c8 bytes) 000908 1735099864.497287 write 0x0265f9 thru 0x02d5da (0x6fe2 bytes) 000912 1735099864.499093 write 0x003132 thru 0x00c397 (0x9266 bytes) 000913 1735099864.503767 write 0x037e46 thru 0x03ffff (0x81ba bytes) 000914 1735099864.506839 MIRROR_RESYNC 7 mirrors 000917 1735099864.875922 read 0x01b7b6 thru 0x026fbe (0xb809 bytes) 000918 1735099864.880924 trunc from 0x040000 to 0x02ce7e (0x13181 bytes) 000919 1735099864.933748 trunc from 0x02ce7f to 0x03afe1 (0xe163 bytes) 000920 1735099864.943670 trunc from 0x03afe2 to 0x0042b2 (0x36d2f bytes) 000922 1735099864.950140 mapwrite 0x00c05f thru 0x015538 (0x94da bytes) 000924 1735099864.994164 mapwrite 0x009ea8 thru 0x00a61b (0x774 bytes) 000925 1735099865.022367 trunc from 0x015539 to 0x03f1e4 (0x29cac bytes) 000927 1735099865.030160 read 0x031186 thru 0x036314 (0x518f bytes) 000929 1735099865.036383 trunc from 0x03f1e5 to 0x01e4ad (0x20d37 bytes) 000931 1735099865.043366 trunc from 0x01e4ae to 0x024c52 (0x67a5 bytes) 000932 1735099865.053861 MIRROR_RESYNC 7 mirrors 000933 1735099865.394535 mapread 0x0039dc thru 0x0085ad (0x4bd2 bytes) 000934 1735099865.403640 mapread 0x003b5e thru 0x011fe4 (0xe487 bytes) 000935 1735099865.408357 mapread 0x0165ee thru 0x01ddf0 (0x7803 bytes) 000936 1735099865.412450 trunc from 0x024c53 to 0x0303a1 (0xb74f bytes) 000937 1735099865.447350 mapwrite 0x015437 thru 0x025080 (0xfc4a bytes) 000938 1735099865.479685 trunc from 0x0303a2 to 0x038359 (0x7fb8 bytes) 000940 1735099865.488567 read 0x01a874 thru 0x021e2a (0x75b7 bytes) 000941 1735099865.488998 write 0x033860 thru 0x03ffff (0xc7a0 bytes) 000942 1735099865.493951 mapwrite 0x0081ca thru 0x009ddc (0x1c13 bytes) 000943 1735099865.521330 write 0x0227a1 thru 0x024cab (0x250b bytes) 000944 1735099865.522061 mapwrite 0x030b29 thru 0x03ffff (0xf4d7 bytes) 000945 1735099865.552700 MIRROR_RESYNC 7 mirrors 000946 1735099865.899767 read 0x00576e thru 0x00c812 (0x70a5 bytes) 000953 1735099865.905230 trunc from 0x040000 to 0x00a47e (0x35b81 bytes) 000954 1735099865.936827 trunc from 0x00a47f to 0x034a07 (0x2a589 bytes) 000956 1735099865.943711 read 0x01da2e thru 0x02baf8 (0xe0cb bytes) 000959 1735099865.949071 read 0x00e6fc thru 0x015dd1 (0x76d6 bytes) 000961 1735099865.949831 mapread 0x01f326 thru 0x0260d8 (0x6db3 bytes) 000962 1735099865.950492 mapread 0x028f72 thru 0x02e524 (0x55b3 bytes) 000963 1735099865.951034 trunc from 0x034a08 to 0x009887 (0x2b180 bytes) 000964 1735099865.956882 write 0x011f0d thru 0x018e01 (0x6ef5 bytes) 000965 1735099865.957760 read 0x004942 thru 0x00c144 (0x7803 bytes) 000968 1735099865.961824 mapread 0x00c11a thru 0x010d48 (0x4c2f bytes) 000969 1735099865.962475 write 0x006eff thru 0x0119bf (0xaac1 bytes) 000971 1735099865.963665 read 0x00796a thru 0x016377 (0xea0e bytes) 000972 1735099865.964154 mapwrite 0x006ada thru 0x00f05e (0x8585 bytes) 000973 1735099865.993950 mapwrite 0x006b0e thru 0x016052 (0xf545 bytes) 000974 1735099866.023942 write 0x016d8b thru 0x020085 (0x92fb bytes) 000975 1735099866.025458 trunc from 0x020086 to 0x0207bf (0x73a bytes) 000976 1735099866.034714 trunc from 0x0207c0 to 0x01deaa (0x2915 bytes) 000977 1735099866.042910 mapread 0x014739 thru 0x01deaa (0x9772 bytes) 000979 1735099866.043955 mapread 0x00c847 thru 0x0116ac (0x4e66 bytes) 000980 1735099866.044817 mapread 0x00bcd2 thru 0x01bac3 (0xfdf2 bytes) 000981 1735099866.045445 write 0x028335 thru 0x0347a2 (0xc46e bytes) 000984 1735099866.047233 trunc from 0x0347a3 to 0x02c2ed (0x84b5 bytes) 000985 1735099866.054576 mapread 0x016e74 thru 0x01f1ca (0x8357 bytes) 000986 1735099866.057877 write 0x00fcbc thru 0x01cdbd (0xd102 bytes) 000987 1735099866.058854 read 0x02b5e4 thru 0x02c2ed (0xd0a bytes) 000988 1735099866.059283 write 0x019efe thru 0x01fa94 (0x5b97 bytes) 000989 1735099866.061869 MIRROR_EXTEND to 8 mirrors 000991 1735099866.243720 write 0x021198 thru 0x029734 (0x859d bytes) 000992 1735099866.278821 MIRROR_RESYNC 8 mirrors 000993 1735099866.639888 write 0x004cea thru 0x00c2e1 (0x75f8 bytes) 000995 1735099866.670368 mapwrite 0x035cde thru 0x03c775 (0x6a98 bytes) 000997 1735099866.717083 read 0x00eb6e thru 0x0146c6 (0x5b59 bytes) 000999 1735099866.721231 write 0x029e82 thru 0x02ea7a (0x4bf9 bytes) 001000 1735099866.726951 mapwrite 0x0302c2 thru 0x032775 (0x24b4 bytes) All operations completed A-OK! PASS 70a (18s) == sanity-flr test 200a: stress test ===================== 23:11:07 (1735099867) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5023424 .. Extending file size to 7562592 .. done Extending file size to 5880416 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7932480 .. Extending file size to 8369984 .. Extending file size to 8926432 .. Extending file size to 8252768 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8688224 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9013856 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9074176 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 30952 30953 30954 30955 30956 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) 192.168.202.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg219-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:12:12 (1735099932) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5117888 .. Extending file size to 2227808 .. failed Extending file size to 5142080 .. Extending file size to 8489728 .. Extending file size to 3073440 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9123008 .. Extending file size to 6595232 .. Extending file size to 7297984 .. Extending file size to 8377760 .. Extending file size to 8733440 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8895264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8909952 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9205120 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9121760 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9230816 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 1204 1205 1206 1209 1211 failed 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) 192.168.202.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg219-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:13:21 (1735100001) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0706393 s, 29.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.30407 s, 6.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:13:33 (1735100013) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0794938 s, 26.4 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 23:13:38 (1735100018) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x728:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ca:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ca:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8cb:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x8cd:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:13:41 (1735100021) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0496872 s, 42.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:13:45 (1735100025) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0522818 s, 40.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d1:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:13:48 (1735100028) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0526461 s, 39.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d3:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:14:13 (1735100053) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0388296 s, 27.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d1:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:14:59 (1735100099) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0232039 s, 90.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:15:42 (1735100142) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. Creating new pool test_204f oleg219-server: Pool lustre.test_204f created Adding targets to pool oleg219-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg219-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0500657 s, 41.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg219-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg219-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg219-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (54s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:16:36 (1735100196) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:16:38 (1735100198) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:16:40 (1735100200) oleg219-server: Pool lustre.test_206 created Adding targets to pool oleg219-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg219-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg219-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg219-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg219-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:16:53 (1735100213) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198945 s, 52.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115946 s, 90.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xf6660098 Mirror 3: 0xf6660098 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:16:56 (1735100216) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:16:57 (1735100217) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:16:58 (1735100218) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0276477 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x810000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x810000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x810000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17f0000 Read this (4 bytes): BBBB 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:17:13 (1735100233) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0041148 s, 2.0 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:17:15 (1735100235) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114046 s, 91.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x74d lmm_fid: [0x200000404:0x74d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x74d lmm_fid: [0x200000404:0x74d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8e1:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835789 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-24 23:17:15.000000000 -0500 Modify: 2024-12-24 23:17:15.000000000 -0500 Change: 2024-12-24 23:17:15.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:17:17 (1735100237) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:17:18 (1735100238) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.196897 s, 208 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0251852 s, 163 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 24 23:17 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x74f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 24 23:17 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 792 sec ============ 23:17:20 (1735100240) === sanity-flr: start cleanup 23:17:21 (1735100241) === === sanity-flr: finish cleanup 23:17:22 (1735100242) ===