-----============= acceptance-small: sanity-flr ============----- Tue Apr 8 15:28:12 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg251-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:28:17 (1744140497) === oleg251-client.virtnet: executing check_config_client /mnt/lustre oleg251-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg251-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b640f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b640f000.idle_timeout=debug disable quota as required oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:28:23 (1744140503) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7262] == sanity-flr test 0a: lfs mirror create with -N option == 15:28:23 (1744140503) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:28:28 (1744140508) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:28:29 (1744140509) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:28:30 (1744140510) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:28:37 (1744140517) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:28:38 (1744140518) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 15:28:39 (1744140519) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b640f000.stats=clear osc.lustre-OST0001-osc-ffff8800b640f000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:28:41 (1744140521) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:28:44 (1744140524) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:28:46 (1744140526) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:28:47 (1744140527) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 15:28:50 (1744140530) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:28:52 (1744140532) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:28:53 (1744140533) lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:28:55 (1744140535) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:28:58 (1744140538) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:28:59 (1744140539) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 0.534823 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0521246 s, 20.1 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:29:03 (1744140543) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0214336 s, 97.8 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:29:05 (1744140545) Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b640f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b640f000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b640f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b640f000.ost_server_uuid in DISCONN state after 3 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:29:30 (1744140570) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0592997 s, 53.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b640f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b640f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:29:41 (1744140581) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b640f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b640f000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b640f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b640f000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (23s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:30:04 (1744140604) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.7879 s, 66.5 MB/s -rw-r--r-- 1 root root 52428800 Apr 8 15:30 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.263s user 0m0.000s sys 0m0.127s Stopping /mnt/lustre-ost1 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b640f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b640f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.287s user 0m0.000s sys 0m0.127s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg251-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg251-server oleg251-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b640f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b640f000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.264s user 0m0.000s sys 0m0.115s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg251-server: oleg251-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg251-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b640f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b640f000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:30:26 (1744140626) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (0s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:30:26 (1744140626) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:30:27 (1744140627) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 15:30:28 (1744140628) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 15:30:30 (1744140630) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.126523 s, 99.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:30:33 (1744140633) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00927057 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00888344 s, 118 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:30:35 (1744140635) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:30:37 (1744140637) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000707482 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000737411 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000655189 s, 1.6 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0006341 s, 1.6 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 15:30:40 (1744140640) create test files with size 15739902 .. eb6ff4d274815630b93f7eda9e7e4469 - 1d9d12994c49cb5a079f557457d17266 - b13ebcd6946ee12e456e773af5c08fff - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 15:30:45 (1744140645) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:30:48 (1744140648) PASS 39 (1s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:30:49 (1744140649) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0283172 s, 37.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102263 s, 103 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 15:30:52 (1744140652) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0389377 s, 53.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0927636 s, 45.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0501305 s, 41.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 15:30:54 (1744140654) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 15:30:55 (1744140655) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:30:56 (1744140656) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00955015 s, 110 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 15:30:58 (1744140658) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (0s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:30:59 (1744140659) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:31:01 (1744140661) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0591372 s, 177 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:31:24 (1744140684) llite.lustre-ffff88012b5e7800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0567075 s, 185 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg251-client.virtnet /mnt/lustre (opts:) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800b6cab800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:31:27 (1744140687) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:31:36 (1744140696) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (9s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:31:44 (1744140704) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 15:31:45 (1744140705) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0753322 s, 41.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0491796 s, 64.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 46389fda195ecbd312c343dda7b12e53 - mirror 1 checksum: 6fabc9e5d44906da2896d5ae0acfba8c - mirror 2 checksum: 46389fda195ecbd312c343dda7b12e53 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 8 15:31:44 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 8 15:31:44 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 46389fda195ecbd312c343dda7b12e53 - mirror 1 checksum: 6fabc9e5d44906da2896d5ae0acfba8c - mirror 2 checksum: 6fabc9e5d44906da2896d5ae0acfba8c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 8 15:31:44 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 8 15:31:44 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:31:48 (1744140708) Failing mds1 on oleg251-server Stopping /mnt/lustre-mds1 (opts:) on oleg251-server 15:31:50 (1744140710) shut down facet: mds1 facet_host: oleg251-server facet_failover_host: oleg251-server Failover mds1 to oleg251-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 15:32:03 (1744140723) targets are mounted 15:32:03 (1744140723) facet_failover done oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116569 s, 90.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (20s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:32:08 (1744140728) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:32:09 (1744140729) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:32:10 (1744140730) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00368899 s, 4.4 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:32:12 (1744140732) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:32:13 (1744140733) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:32:14 (1744140734) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:32:20 (1744140740) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:32:38 (1744140758) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:32:55 (1744140775) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:33:12 (1744140792) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 15:33:14 (1744140794) mirror create pid 22851 mirror split pid 22852 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4930: 22851 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4930: 22852 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 15:34:16 (1744140856) Seed set to 856 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x2394f 000006 1744140856.600846 trunc from 00000000 to 0x02394e (0x2394f bytes) truncating to largest ever: 0x3bdf3 000008 1744140856.605137 trunc from 0x02394f to 0x03bdf2 (0x184a4 bytes) 000010 1744140856.609698 read 0x01cd93 thru 0x028968 (0xbbd6 bytes) 000011 1744140856.613306 mapread 0x01bbc2 thru 0x02a427 (0xe866 bytes) 000012 1744140856.619037 read 0x034e08 thru 0x0353dc (0x5d5 bytes) 000015 1744140856.623626 mapwrite 0x00f083 thru 0x017ba6 (0x8b24 bytes) 000017 1744140856.647473 trunc from 0x03bdf3 to 0x01e588 (0x1d86a bytes) 000018 1744140856.652909 mapread 0x012c6b thru 0x01e588 (0xb91e bytes) 000019 1744140856.653657 mapread 0x0177e3 thru 0x01e588 (0x6da6 bytes) 000020 1744140856.654183 trunc from 0x01e589 to 0x02f05a (0x10ad2 bytes) 000021 1744140856.658527 trunc from 0x02f05b to 0x0083ea (0x26c70 bytes) 000023 1744140856.662802 mapread 0x0059d1 thru 0x006a09 (0x1039 bytes) 000024 1744140856.663289 mapwrite 0x018c8e thru 0x01f898 (0x6c0b bytes) 000025 1744140856.698555 mapread 0x018392 thru 0x01aa18 (0x2687 bytes) 000028 1744140856.699662 write 0x0253f6 thru 0x02716f (0x1d7a bytes) 000030 1744140856.700857 read 0x02283f thru 0x024307 (0x1ac9 bytes) 000032 1744140856.704726 trunc from 0x027170 to 0x01cc8d (0xa4e2 bytes) 000033 1744140856.708862 trunc from 0x01cc8e to 0x008e43 (0x13e4a bytes) 000036 1744140856.716057 mapwrite 0x001332 thru 0x008c8e (0x795d bytes) 000038 1744140856.731871 MIRROR_EXTEND to 2 mirrors 000040 1744140856.817496 trunc from 0x008e44 to 0x0038f4 (0x554f bytes) 000041 1744140856.836311 read 0x002cbd thru 0x00310e (0x452 bytes) 000042 1744140856.839427 MIRROR_EXTEND to 3 mirrors 000044 1744140856.925794 MIRROR_EXTEND to 4 mirrors 000045 1744140857.010306 read 0x002e65 thru 0x0038f4 (0xa90 bytes) 000046 1744140857.012558 mapread 0x0015b1 thru 0x003251 (0x1ca1 bytes) 000047 1744140857.015389 write 0x03c9d7 thru 0x03ffff (0x3629 bytes) 000048 1744140857.027816 read 0x0314e5 thru 0x03ffff (0xeb1b bytes) 000050 1744140857.032342 read 0x022ade thru 0x027617 (0x4b3a bytes) 000051 1744140857.032611 mapwrite 0x0076b1 thru 0x01589e (0xe1ee bytes) 000052 1744140857.049311 MIRROR_RESYNC 4 mirrors 000055 1744140857.190698 write 0x0339bb thru 0x034254 (0x89a bytes) 000056 1744140857.204821 read 0x022eec thru 0x030b3e (0xdc53 bytes) 000057 1744140857.207024 trunc from 0x040000 to 0x03082c (0xf7d3 bytes) 000058 1744140857.214657 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000059 1744140857.255987 mapwrite 0x033896 thru 0x03536d (0x1ad8 bytes) 000068 1744140857.277309 read 0x014e23 thru 0x01f9ba (0xab98 bytes) 000069 1744140857.279692 write 0x03eed8 thru 0x03ffff (0x1128 bytes) 000072 1744140857.280724 mapwrite 0x01f30c thru 0x029159 (0x9e4e bytes) 000073 1744140857.296477 mapread 0x025881 thru 0x029069 (0x37e9 bytes) 000074 1744140857.296943 mapwrite 0x0157ce thru 0x01f039 (0x986c bytes) 000075 1744140857.309874 trunc from 0x040000 to 0x02da2c (0x125d3 bytes) 000076 1744140857.313449 write 0x003a06 thru 0x00572a (0x1d25 bytes) 000077 1744140857.316079 write 0x02c05b thru 0x03100d (0x4fb3 bytes) 000080 1744140857.317322 mapwrite 0x029500 thru 0x035231 (0xbd32 bytes) 000081 1744140857.335936 read 0x001962 thru 0x00dbb7 (0xc256 bytes) 000083 1744140857.338037 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000084 1744140857.384400 read 0x01eef2 thru 0x024d0e (0x5e1d bytes) 000086 1744140857.387292 read 0x0064c8 thru 0x008079 (0x1bb2 bytes) 000088 1744140857.390574 trunc from 0x035232 to 0x036856 (0x1625 bytes) 000091 1744140857.396160 write 0x0188c4 thru 0x01ea30 (0x616d bytes) 000094 1744140857.397214 mapread 0x02216b thru 0x022771 (0x607 bytes) 000095 1744140857.397796 read 0x020c15 thru 0x02f796 (0xeb82 bytes) 000098 1744140857.398870 write 0x013b35 thru 0x0143b6 (0x882 bytes) 000099 1744140857.399269 mapwrite 0x017bff thru 0x01ae77 (0x3279 bytes) 000101 1744140857.413140 mapread 0x013f64 thru 0x017312 (0x33af bytes) 000103 1744140857.413737 mapread 0x00723f thru 0x011007 (0x9dc9 bytes) 000107 1744140857.414650 mapread 0x03600c thru 0x036856 (0x84b bytes) 000108 1744140857.416586 mapwrite 0x00a31b thru 0x00c6f0 (0x23d6 bytes) 000110 1744140857.426670 read 0x01b69d thru 0x026568 (0xaecc bytes) 000111 1744140857.428098 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000112 1744140857.456544 mapread 0x02a187 thru 0x02e43f (0x42b9 bytes) 000116 1744140857.462781 mapwrite 0x03d778 thru 0x03ffff (0x2888 bytes) 000117 1744140857.482570 mapread 0x02d50b thru 0x0375d3 (0xa0c9 bytes) 000121 1744140857.485712 mapread 0x027440 thru 0x02abf0 (0x37b1 bytes) 000123 1744140857.486340 read 0x00b40f thru 0x016d95 (0xb987 bytes) 000125 1744140857.489582 mapread 0x02b709 thru 0x02fc03 (0x44fb bytes) 000126 1744140857.490086 write 0x000062 thru 0x00aae0 (0xaa7f bytes) 000127 1744140857.493137 trunc from 0x040000 to 0x036d7a (0x9285 bytes) 000132 1744140857.498140 read 0x01560f thru 0x01d084 (0x7a76 bytes) 000133 1744140857.500611 write 0x007da2 thru 0x0088c6 (0xb25 bytes) 000134 1744140857.501152 trunc from 0x036d7b to 0x00d0fb (0x29c7f bytes) 000136 1744140857.505493 write 0x0162d7 thru 0x025096 (0xedc0 bytes) 000138 1744140857.506566 mapwrite 0x009e6f thru 0x016b28 (0xccba bytes) 000139 1744140857.529115 read 0x00c77d thru 0x0141e5 (0x7a69 bytes) 000145 1744140857.530818 mapwrite 0x005008 thru 0x007b70 (0x2b69 bytes) 000146 1744140857.543361 mapread 0x019ef9 thru 0x0242a7 (0xa3af bytes) 000147 1744140857.543981 mapread 0x009704 thru 0x00b3ff (0x1cfc bytes) 000149 1744140857.544649 mapread 0x0126ac thru 0x01bad1 (0x9426 bytes) 000150 1744140857.545078 mapread 0x0240a0 thru 0x025096 (0xff7 bytes) 000151 1744140857.545367 read 0x023899 thru 0x025096 (0x17fe bytes) 000152 1744140857.545699 trunc from 0x025097 to 0x02e5cd (0x9537 bytes) 000156 1744140857.550251 read 0x0280ab thru 0x02e5cd (0x6523 bytes) 000159 1744140857.554188 write 0x03414b thru 0x03f3f9 (0xb2af bytes) 000160 1744140857.554782 trunc from 0x03f3fa to 0x0042bf (0x3b13a bytes) 000165 1744140857.559512 read 0x0004f5 thru 0x000893 (0x39f bytes) 000166 1744140857.559785 mapread 0x000f51 thru 0x0042bf (0x336f bytes) 000168 1744140857.560452 write 0x00cd29 thru 0x0197ed (0xcac5 bytes) 000173 1744140857.561981 read 0x00702c thru 0x00a840 (0x3815 bytes) 000175 1744140857.564434 mapwrite 0x031a74 thru 0x031b66 (0xf3 bytes) 000176 1744140857.585700 read 0x017cef thru 0x019901 (0x1c13 bytes) 000177 1744140857.586136 mapwrite 0x00d269 thru 0x019610 (0xc3a8 bytes) 000184 1744140857.604776 write 0x021bb1 thru 0x030b16 (0xef66 bytes) 000187 1744140857.605883 mapwrite 0x0251a2 thru 0x02a606 (0x5465 bytes) 000188 1744140857.619061 mapwrite 0x03d1d8 thru 0x03ffff (0x2e28 bytes) 000190 1744140857.639369 mapwrite 0x01126e thru 0x011648 (0x3db bytes) 000191 1744140857.651494 trunc from 0x040000 to 0x019648 (0x269b7 bytes) 000192 1744140857.655571 mapread 0x005070 thru 0x009e5e (0x4def bytes) 000193 1744140857.656181 trunc from 0x019649 to 0x024cc0 (0xb678 bytes) 000197 1744140857.660223 trunc from 0x024cc1 to 0x00640c (0x1e8b4 bytes) 000198 1744140857.663840 mapwrite 0x0046b6 thru 0x013919 (0xf264 bytes) 000200 1744140857.708090 trunc from 0x01391a to 0x0005c1 (0x13358 bytes) 000201 1744140857.714633 write 0x0118a7 thru 0x018c4a (0x73a4 bytes) 000202 1744140857.715590 mapread 0x00cc13 thru 0x018c4a (0xc038 bytes) 000203 1744140857.723857 mapwrite 0x037d09 thru 0x03d217 (0x550f bytes) 000205 1744140857.753087 write 0x00133a thru 0x00c6c6 (0xb38d bytes) 000206 1744140857.756085 MIRROR_EXTEND to 2 mirrors 000207 1744140857.876441 write 0x02352a thru 0x0236f1 (0x1c8 bytes) 000209 1744140857.891715 trunc from 0x03d218 to 0x002e1f (0x3a3f8 bytes) 000211 1744140857.900407 mapread 0x002a9a thru 0x002e1f (0x386 bytes) 000212 1744140857.902443 write 0x03a531 thru 0x03ffff (0x5acf bytes) 000213 1744140857.903134 trunc from 0x040000 to 0x02f9c7 (0x10638 bytes) 000217 1744140857.911209 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000218 1744140857.947886 read 0x00e3e2 thru 0x0117a5 (0x33c4 bytes) 000224 1744140857.951673 mapwrite 0x03d406 thru 0x03ffff (0x2bfa bytes) 000225 1744140857.973282 write 0x03cb7a thru 0x03ffff (0x3486 bytes) truncating to largest ever: 0x3d187 000226 1744140857.973827 trunc from 0x040000 to 0x03d186 (0x2e79 bytes) 000227 1744140857.977263 write 0x007e8e thru 0x0127b2 (0xa925 bytes) 000230 1744140857.981882 mapread 0x03b3bf thru 0x03d186 (0x1dc8 bytes) 000231 1744140857.984449 MIRROR_EXTEND to 2 mirrors 000233 1744140858.110767 write 0x025cfc thru 0x027d2b (0x2030 bytes) 000235 1744140858.127157 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000238 1744140858.159426 read 0x02a62b thru 0x0314be (0x6e94 bytes) 000239 1744140858.161700 mapwrite 0x015d07 thru 0x01757a (0x1874 bytes) 000241 1744140858.179164 trunc from 0x03d187 to 0x02ad31 (0x12455 bytes) 000242 1744140858.183185 read 0x016049 thru 0x0221b4 (0xc16c bytes) 000243 1744140858.185831 mapread 0x00b9f5 thru 0x014f27 (0x9533 bytes) 000244 1744140858.186299 mapread 0x007c18 thru 0x00b48a (0x3873 bytes) 000245 1744140858.186633 mapwrite 0x00c58a thru 0x00fc3a (0x36b1 bytes) 000246 1744140858.200325 mapwrite 0x0301a2 thru 0x031844 (0x16a3 bytes) 000247 1744140858.218305 mapread 0x0107c3 thru 0x01c459 (0xbc97 bytes) 000248 1744140858.218801 trunc from 0x031845 to 0x01b832 (0x16012 bytes) 000250 1744140858.222887 trunc from 0x01b833 to 0x039908 (0x1e0d6 bytes) 000255 1744140858.227736 write 0x0150ce thru 0x01a586 (0x54b9 bytes) 000256 1744140858.228343 read 0x00853f thru 0x017af3 (0xf5b5 bytes) 000257 1744140858.228766 read 0x025848 thru 0x030c35 (0xb3ee bytes) 000258 1744140858.232005 read 0x01a3e1 thru 0x01f90e (0x552e bytes) 000259 1744140858.232333 trunc from 0x039909 to 0x0303a0 (0x9568 bytes) 000263 1744140858.236818 read 0x01c4df thru 0x023cc0 (0x77e2 bytes) 000265 1744140858.238871 trunc from 0x0303a1 to 0x001f83 (0x2e41d bytes) 000270 1744140858.244223 mapread 0x00079f thru 0x001f83 (0x17e5 bytes) truncating to largest ever: 0x3dda1 000271 1744140858.244853 trunc from 0x001f84 to 0x03dda0 (0x3be1d bytes) 000273 1744140858.248714 mapread 0x007324 thru 0x00d8c9 (0x65a6 bytes) 000276 1744140858.252901 write 0x01c06a thru 0x02134e (0x52e5 bytes) 000278 1744140858.254872 MIRROR_EXTEND to 2 mirrors 000280 1744140858.371198 trunc from 0x03dda1 to 0x006743 (0x3765d bytes) 000281 1744140858.388551 mapread 0x001fa4 thru 0x006743 (0x47a0 bytes) 000283 1744140858.392595 read 0x004f1f thru 0x006743 (0x1825 bytes) 000285 1744140858.394448 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000287 1744140858.424566 mapread 0x000c35 thru 0x001d50 (0x111c bytes) 000289 1744140858.429313 read 0x0042f7 thru 0x006743 (0x244d bytes) 000291 1744140858.431555 write 0x0044f0 thru 0x00cd14 (0x8825 bytes) 000292 1744140858.432380 mapread 0x002589 thru 0x004bf5 (0x266d bytes) 000293 1744140858.434360 read 0x004320 thru 0x00cd14 (0x89f5 bytes) 000298 1744140858.435628 trunc from 0x00cd15 to 0x00351e (0x97f6 bytes) 000299 1744140858.440065 mapwrite 0x00f2c7 thru 0x01e2ea (0xf024 bytes) 000300 1744140858.467983 trunc from 0x01e2eb to 0x01e1dd (0x10d bytes) 000301 1744140858.472385 write 0x02f085 thru 0x033ea5 (0x4e21 bytes) 000308 1744140858.474799 mapread 0x008368 thru 0x01712c (0xedc5 bytes) 000309 1744140858.475380 trunc from 0x033ea6 to 0x0068b3 (0x2d5f2 bytes) 000311 1744140858.480483 mapwrite 0x00a085 thru 0x00a7e9 (0x765 bytes) 000313 1744140858.500644 write 0x035009 thru 0x03ffff (0xaff7 bytes) 000314 1744140858.501477 mapwrite 0x01fce9 thru 0x02f668 (0xf980 bytes) 000317 1744140858.520323 write 0x004fa9 thru 0x00ecca (0x9d22 bytes) 000319 1744140858.522826 write 0x030218 thru 0x03b965 (0xb74e bytes) 000322 1744140858.524246 mapread 0x01c4f7 thru 0x0220e8 (0x5bf2 bytes) 000323 1744140858.527752 write 0x02f5fb thru 0x03ecbf (0xf6c5 bytes) 000325 1744140858.528670 mapwrite 0x0260b9 thru 0x02c91e (0x6866 bytes) 000326 1744140858.542461 write 0x02db1a thru 0x0331f6 (0x56dd bytes) 000328 1744140858.543166 read 0x030366 thru 0x030bff (0x89a bytes) 000330 1744140858.543532 read 0x01398e thru 0x017a03 (0x4076 bytes) 000331 1744140858.546924 mapwrite 0x002017 thru 0x00678a (0x4774 bytes) 000340 1744140858.562933 mapread 0x025e3b thru 0x0275e8 (0x17ae bytes) 000343 1744140858.564208 mapread 0x02a892 thru 0x035124 (0xa893 bytes) 000344 1744140858.564760 read 0x0125c8 thru 0x01a208 (0x7c41 bytes) 000345 1744140858.565140 mapwrite 0x010118 thru 0x018f86 (0x8e6f bytes) 000348 1744140858.583460 mapread 0x03f062 thru 0x03ffff (0xf9e bytes) 000349 1744140858.583915 read 0x01bfca thru 0x027a22 (0xba59 bytes) 000351 1744140858.584565 write 0x004070 thru 0x00705e (0x2fef bytes) 000353 1744140858.585719 read 0x03c47f thru 0x03ffff (0x3b81 bytes) 000355 1744140858.586391 mapread 0x021d64 thru 0x027c2f (0x5ecc bytes) 000358 1744140858.587530 read 0x039da6 thru 0x03ffff (0x625a bytes) 000359 1744140858.587916 write 0x00414c thru 0x00a737 (0x65ec bytes) 000364 1744140858.589647 mapwrite 0x018f74 thru 0x01f0c0 (0x614d bytes) 000365 1744140858.605515 mapread 0x00a197 thru 0x011c67 (0x7ad1 bytes) 000367 1744140858.606314 trunc from 0x040000 to 0x00eec6 (0x31139 bytes) 000371 1744140858.613052 MIRROR_EXTEND to 2 mirrors 000372 1744140858.717250 mapwrite 0x0200ef thru 0x027f34 (0x7e46 bytes) 000374 1744140858.761426 read 0x0001d5 thru 0x001a1d (0x1849 bytes) 000376 1744140858.764205 read 0x016144 thru 0x01c228 (0x60e5 bytes) 000377 1744140858.767324 trunc from 0x027f35 to 0x01a614 (0xd920 bytes) 000378 1744140858.772979 mapwrite 0x02b59b thru 0x02c182 (0xbe8 bytes) 000379 1744140858.793975 mapwrite 0x007672 thru 0x015830 (0xe1bf bytes) 000380 1744140858.807958 read 0x0015a6 thru 0x00267d (0x10d8 bytes) 000386 1744140858.809775 mapwrite 0x00f25a thru 0x01e645 (0xf3ec bytes) 000389 1744140858.826031 mapread 0x025a7f thru 0x02c182 (0x6704 bytes) 000390 1744140858.826450 mapread 0x0255f7 thru 0x02c182 (0x6b8c bytes) 000391 1744140858.826858 read 0x00a6c5 thru 0x012694 (0x7fd0 bytes) 000392 1744140858.827153 mapread 0x02a857 thru 0x02c182 (0x192c bytes) 000397 1744140858.828294 mapwrite 0x025a99 thru 0x031a8e (0xbff6 bytes) 000398 1744140858.847338 read 0x016c4b thru 0x01ab9d (0x3f53 bytes) 000399 1744140858.847630 mapread 0x01883e thru 0x021519 (0x8cdc bytes) 000400 1744140858.848059 read 0x0078f3 thru 0x00dd24 (0x6432 bytes) 000402 1744140858.849776 MIRROR_EXTEND to 3 mirrors 000403 1744140858.943064 MIRROR_EXTEND to 4 mirrors 000404 1744140859.050647 mapread 0x030ca2 thru 0x0315c4 (0x923 bytes) 000406 1744140859.052185 write 0x022ade thru 0x02d007 (0xa52a bytes) 000407 1744140859.069410 mapwrite 0x02c313 thru 0x03069c (0x438a bytes) 000408 1744140859.083685 read 0x022f7c thru 0x02dc40 (0xacc5 bytes) 000409 1744140859.084015 read 0x019323 thru 0x01e19b (0x4e79 bytes) 000410 1744140859.086720 write 0x014810 thru 0x017447 (0x2c38 bytes) 000412 1744140859.087428 read 0x00f68d thru 0x015b1e (0x6492 bytes) 000413 1744140859.087827 write 0x022c8f thru 0x02c5d8 (0x994a bytes) 000414 1744140859.088431 trunc from 0x031a8f to 0x03bfbd (0xa52f bytes) 000415 1744140859.096274 MIRROR_EXTEND to 5 mirrors 000419 1744140859.206631 MIRROR_RESYNC 5 mirrors 000420 1744140859.345236 mapwrite 0x01485d thru 0x01ab3f (0x62e3 bytes) 000423 1744140859.385652 read 0x02b36d thru 0x03302f (0x7cc3 bytes) 000427 1744140859.389571 MIRROR_EXTEND to 6 mirrors 000429 1744140859.478119 read 0x02a864 thru 0x035950 (0xb0ed bytes) 000431 1744140859.481311 write 0x009369 thru 0x00f1ce (0x5e66 bytes) 000433 1744140859.496123 mapwrite 0x015782 thru 0x01727b (0x1afa bytes) 000434 1744140859.510016 mapwrite 0x002abd thru 0x00e180 (0xb6c4 bytes) 000435 1744140859.524351 write 0x0362a8 thru 0x0389f4 (0x274d bytes) 000436 1744140859.527823 MIRROR_RESYNC 6 mirrors 000437 1744140859.694514 mapread 0x01677b thru 0x01ef29 (0x87af bytes) 000439 1744140859.699375 read 0x007322 thru 0x00d234 (0x5f13 bytes) 000440 1744140859.701595 read 0x024fc3 thru 0x03177f (0xc7bd bytes) 000442 1744140859.705000 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000443 1744140859.763061 mapwrite 0x009109 thru 0x016593 (0xd48b bytes) 000444 1744140859.805379 mapread 0x016296 thru 0x018c77 (0x29e2 bytes) 000447 1744140859.806272 mapwrite 0x033e67 thru 0x03ce37 (0x8fd1 bytes) 000450 1744140859.831720 read 0x01fc2e thru 0x021c96 (0x2069 bytes) 000451 1744140859.834341 MIRROR_EXTEND to 6 mirrors 000454 1744140859.922700 mapread 0x006d46 thru 0x00da94 (0x6d4f bytes) 000455 1744140859.928368 write 0x0161b9 thru 0x01b6bc (0x5504 bytes) 000456 1744140859.946746 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000457 1744140859.992635 MIRROR_RESYNC 5 mirrors 000458 1744140860.137636 read 0x02169b thru 0x024e10 (0x3776 bytes) 000461 1744140860.139536 write 0x02b136 thru 0x037937 (0xc802 bytes) 000462 1744140860.155138 read 0x036252 thru 0x03ce37 (0x6be6 bytes) 000463 1744140860.157953 mapread 0x01495f thru 0x01bcf2 (0x7394 bytes) 000472 1744140860.160311 read 0x02a8c1 thru 0x02abe1 (0x321 bytes) 000474 1744140860.160853 write 0x02efcd thru 0x0372d5 (0x8309 bytes) 000476 1744140860.161557 write 0x02f950 thru 0x032888 (0x2f39 bytes) 000477 1744140860.161989 read 0x02616d thru 0x032039 (0xbecd bytes) 000478 1744140860.162244 read 0x01b524 thru 0x029b0c (0xe5e9 bytes) 000480 1744140860.162946 mapread 0x00c3c3 thru 0x0123b1 (0x5fef bytes) 000481 1744140860.163327 trunc from 0x03ce38 to 0x01ba92 (0x213a5 bytes) 000482 1744140860.168849 write 0x034566 thru 0x036fd1 (0x2a6c bytes) 000484 1744140860.169641 mapread 0x00fe0c thru 0x013fe2 (0x41d7 bytes) 000485 1744140860.170063 write 0x00f4e1 thru 0x01f4c0 (0xffe0 bytes) 000486 1744140860.173373 MIRROR_EXTEND to 6 mirrors 000487 1744140860.265319 mapread 0x01aef1 thru 0x01e041 (0x3151 bytes) 000491 1744140860.273019 mapread 0x01685f thru 0x017d06 (0x14a8 bytes) 000492 1744140860.275745 read 0x032175 thru 0x036fd1 (0x4e5d bytes) 000494 1744140860.278943 mapwrite 0x001e5a thru 0x011cee (0xfe95 bytes) 000495 1744140860.326581 write 0x02125a thru 0x02c079 (0xae20 bytes) 000496 1744140860.329782 mapread 0x01419e thru 0x01966c (0x54cf bytes) 000501 1744140860.331872 write 0x007001 thru 0x0089e3 (0x19e3 bytes) 000502 1744140860.332239 mapwrite 0x01b4c2 thru 0x01d2bf (0x1dfe bytes) 000503 1744140860.343508 mapread 0x02781a thru 0x0306d2 (0x8eb9 bytes) 000505 1744140860.346436 read 0x02ee87 thru 0x036fd1 (0x814b bytes) 000506 1744140860.346824 read 0x0150c9 thru 0x01628d (0x11c5 bytes) 000508 1744140860.347300 mapread 0x0238d9 thru 0x02a322 (0x6a4a bytes) 000509 1744140860.347821 read 0x0299b4 thru 0x036c30 (0xd27d bytes) 000514 1744140860.349237 trunc from 0x036fd2 to 0x00bfcd (0x2b004 bytes) 000517 1744140860.359334 MIRROR_EXTEND to 7 mirrors 000518 1744140860.444194 MIRROR_EXTEND to 8 mirrors 000519 1744140860.528519 read 0x00391c thru 0x00bfcd (0x86b2 bytes) 000520 1744140860.531167 trunc from 0x00bfce to 0x028b82 (0x1cbb5 bytes) 000522 1744140860.551195 mapread 0x026985 thru 0x028690 (0x1d0c bytes) 000523 1744140860.555375 mapread 0x000485 thru 0x00b594 (0xb110 bytes) 000526 1744140860.564183 MIRROR_RESYNC 8 mirrors 000527 1744140860.784002 mapwrite 0x03a041 thru 0x03ffff (0x5fbf bytes) 000529 1744140860.822583 write 0x01dd92 thru 0x02313b (0x53aa bytes) 000530 1744140860.825026 mapwrite 0x03decc thru 0x03f14a (0x127f bytes) 000533 1744140860.836908 mapread 0x00f3ee thru 0x0125f8 (0x320b bytes) 000535 1744140860.839625 mapwrite 0x002a42 thru 0x008cd9 (0x6298 bytes) 000537 1744140860.853292 mapread 0x021ec9 thru 0x025392 (0x34ca bytes) 000538 1744140860.853789 mapread 0x032925 thru 0x032f21 (0x5fd bytes) 000539 1744140860.856154 mapwrite 0x0257b4 thru 0x028968 (0x31b5 bytes) 000541 1744140860.869287 write 0x0198e6 thru 0x01f950 (0x606b bytes) 000542 1744140860.869969 mapwrite 0x035902 thru 0x039299 (0x3998 bytes) 000543 1744140860.885060 trunc from 0x040000 to 0x023f18 (0x1c0e7 bytes) 000545 1744140860.890702 mapwrite 0x017e98 thru 0x019bf7 (0x1d60 bytes) 000546 1744140860.906157 trunc from 0x023f19 to 0x00d887 (0x16691 bytes) 000547 1744140860.911252 MIRROR_EXTEND to 9 mirrors 000549 1744140860.995648 read 0x002714 thru 0x00d887 (0xb174 bytes) 000550 1744140860.998479 read 0x00d644 thru 0x00d887 (0x244 bytes) 000552 1744140861.000391 read 0x001c95 thru 0x00c4eb (0xa857 bytes) 000554 1744140861.003966 mapwrite 0x00b7f1 thru 0x00b996 (0x1a6 bytes) 000556 1744140861.052255 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000557 1744140861.100226 write 0x0372e5 thru 0x03d1ec (0x5f08 bytes) 000558 1744140861.101076 trunc from 0x03d1ed to 0x009dbb (0x33431 bytes) 000560 1744140861.112054 mapread 0x005410 thru 0x009dbb (0x49ac bytes) 000561 1744140861.118101 read 0x007c1d thru 0x009dbb (0x219f bytes) 000562 1744140861.118513 mapread 0x008582 thru 0x009dbb (0x183a bytes) 000568 1744140861.120316 write 0x0330a0 thru 0x0350d5 (0x2036 bytes) 000570 1744140861.121351 mapread 0x0252f2 thru 0x02a676 (0x5385 bytes) 000571 1744140861.126850 MIRROR_RESYNC 8 mirrors 000572 1744140861.359564 mapwrite 0x000770 thru 0x003ebe (0x374f bytes) 000573 1744140861.409263 mapwrite 0x03ac5d thru 0x03f265 (0x4609 bytes) 000575 1744140861.436736 read 0x016f4c thru 0x016f58 (0xd bytes) 000581 1744140861.439627 mapread 0x01346e thru 0x01b284 (0x7e17 bytes) 000583 1744140861.445951 MIRROR_EXTEND to 9 mirrors 000585 1744140861.558104 write 0x0280c3 thru 0x0337db (0xb719 bytes) 000586 1744140861.577059 trunc from 0x03f266 to 0x022b45 (0x1c720 bytes) 000588 1744140861.589426 MIRROR_EXTEND to 10 mirrors 000589 1744140861.703850 write 0x0007dd thru 0x00571b (0x4f3f bytes) 000590 1744140861.725801 read 0x0191bd thru 0x01da8e (0x48d2 bytes) 000592 1744140861.729795 read 0x00794a thru 0x00bef8 (0x45af bytes) 000593 1744140861.734023 read 0x013a19 thru 0x01c1f9 (0x87e1 bytes) 000594 1744140861.734519 read 0x00e83a thru 0x00e8b6 (0x7d bytes) 000596 1744140861.735156 read 0x00f944 thru 0x0102a7 (0x964 bytes) 000598 1744140861.736034 mapread 0x005a37 thru 0x009002 (0x35cc bytes) 000600 1744140861.737052 trunc from 0x022b46 to 0x02453a (0x19f5 bytes) 000601 1744140861.749579 write 0x02b9f9 thru 0x033379 (0x7981 bytes) 000605 1744140861.751128 mapwrite 0x03c3d8 thru 0x03ffff (0x3c28 bytes) 000607 1744140861.781005 mapread 0x01be00 thru 0x01ff69 (0x416a bytes) 000608 1744140861.781561 mapwrite 0x0103c5 thru 0x01a045 (0x9c81 bytes) 000609 1744140861.804779 mapwrite 0x03833a thru 0x03bec9 (0x3b90 bytes) 000611 1744140861.822963 mapread 0x009277 thru 0x0155fb (0xc385 bytes) truncating to largest ever: 0x3df79 000613 1744140861.823723 trunc from 0x040000 to 0x03df78 (0x2087 bytes) 000614 1744140861.827842 write 0x0398e7 thru 0x03ffff (0x6719 bytes) 000615 1744140861.828627 read 0x00611c thru 0x012374 (0xc259 bytes) 000617 1744140861.829315 mapwrite 0x01d143 thru 0x01fb2c (0x29ea bytes) 000620 1744140861.850382 read 0x00f923 thru 0x01890a (0x8fe8 bytes) 000622 1744140861.851204 mapwrite 0x02b158 thru 0x0326e6 (0x758f bytes) 000623 1744140861.874577 MIRROR_EXTEND to 11 mirrors 000625 1744140861.991840 trunc from 0x040000 to 0x02ef0d (0x110f2 bytes) 000627 1744140862.015241 trunc from 0x02ef0e to 0x016138 (0x18dd5 bytes) 000628 1744140862.020286 mapread 0x00ce8d thru 0x016138 (0x92ac bytes) 000631 1744140862.028183 read 0x00104b thru 0x00be23 (0xadd9 bytes) 000632 1744140862.031229 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000633 1744140862.080556 read 0x00b410 thru 0x014a15 (0x9606 bytes) 000634 1744140862.083129 write 0x02e5d7 thru 0x0303b7 (0x1de1 bytes) 000635 1744140862.085101 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000642 1744140862.140750 read 0x0214b4 thru 0x023703 (0x2250 bytes) 000645 1744140862.145230 MIRROR_EXTEND to 10 mirrors 000646 1744140862.256181 trunc from 0x0303b8 to 0x015e06 (0x1a5b1 bytes) 000647 1744140862.277043 mapwrite 0x022c03 thru 0x02489a (0x1c98 bytes) 000649 1744140862.300969 read 0x00d9da thru 0x00d9ed (0x14 bytes) 000650 1744140862.304389 MIRROR_EXTEND to 11 mirrors 000652 1744140862.404992 write 0x011db3 thru 0x01500a (0x3258 bytes) 000653 1744140862.424069 read 0x01d490 thru 0x02489a (0x740b bytes) 000654 1744140862.426561 mapwrite 0x00eb4e thru 0x013609 (0x4abc bytes) 000656 1744140862.446721 read 0x006f15 thru 0x007087 (0x173 bytes) 000657 1744140862.450008 read 0x00e3e2 thru 0x01e333 (0xff52 bytes) 000658 1744140862.450507 write 0x030199 thru 0x0332e0 (0x3148 bytes) 000659 1744140862.451192 read 0x00a7c6 thru 0x012046 (0x7881 bytes) 000662 1744140862.452198 mapwrite 0x02d829 thru 0x03c86f (0xf047 bytes) 000663 1744140862.480596 mapread 0x022e34 thru 0x02fe36 (0xd003 bytes) 000664 1744140862.482500 MIRROR_RESYNC 11 mirrors 000665 1744140862.787878 trunc from 0x03c870 to 0x03c48d (0x3e2 bytes) 000666 1744140862.805071 MIRROR_EXTEND to 12 mirrors 000668 1744140862.901449 mapwrite 0x002b5a thru 0x011049 (0xe4f0 bytes) 000670 1744140862.953626 trunc from 0x03c48e to 0x00fb62 (0x2c92b bytes) 000674 1744140862.962695 mapread 0x006281 thru 0x00fb62 (0x98e2 bytes) 000675 1744140862.963188 mapread 0x00f525 thru 0x00fb62 (0x63e bytes) 000676 1744140862.963566 write 0x01ef1d thru 0x027bb1 (0x8c95 bytes) 000677 1744140862.964211 trunc from 0x027bb2 to 0x00eba8 (0x19009 bytes) 000678 1744140862.971247 MIRROR_EXTEND to 13 mirrors 000680 1744140863.075506 mapwrite 0x01e8c1 thru 0x0253c5 (0x6b05 bytes) 000681 1744140863.120680 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000682 1744140863.173181 write 0x027f46 thru 0x02b158 (0x3213 bytes) 000683 1744140863.173865 trunc from 0x02b159 to 0x000758 (0x2aa00 bytes) 000686 1744140863.178230 write 0x001315 thru 0x00885a (0x7546 bytes) 000687 1744140863.178824 trunc from 0x00885b to 0x011976 (0x911c bytes) 000688 1744140863.183654 read 0x00b456 thru 0x00f1fd (0x3da8 bytes) 000689 1744140863.186399 write 0x02069a thru 0x02a354 (0x9cbb bytes) 000690 1744140863.187301 mapwrite 0x01e2cd thru 0x02774b (0x947f bytes) 000691 1744140863.208671 trunc from 0x02a355 to 0x031f81 (0x7c2d bytes) 000692 1744140863.213630 read 0x0135aa thru 0x022281 (0xecd8 bytes) 000694 1744140863.217291 read 0x004b42 thru 0x00b65b (0x6b1a bytes) 000695 1744140863.219142 MIRROR_EXTEND to 13 mirrors 000697 1744140863.333800 trunc from 0x031f82 to 0x01929e (0x18ce3 bytes) 000698 1744140863.357803 mapwrite 0x02072b thru 0x02ec8a (0xe560 bytes) 000699 1744140863.389667 read 0x02968c thru 0x02ad69 (0x16de bytes) 000705 1744140863.391198 mapwrite 0x025f58 thru 0x02605b (0x104 bytes) 000707 1744140863.406260 mapwrite 0x017fae thru 0x0200a0 (0x80f3 bytes) 000708 1744140863.428531 write 0x0123f4 thru 0x017472 (0x507f bytes) 000709 1744140863.429313 trunc from 0x02ec8b to 0x00aa48 (0x24242 bytes) 000710 1744140863.433842 trunc from 0x00aa49 to 0x009503 (0x1545 bytes) 000712 1744140863.438048 mapwrite 0x0044e7 thru 0x008cca (0x47e4 bytes) 000714 1744140863.455552 read 0x004c26 thru 0x009503 (0x48de bytes) 000715 1744140863.456021 mapread 0x00618d thru 0x009503 (0x3377 bytes) 000717 1744140863.456769 write 0x02d545 thru 0x032fc0 (0x5a7c bytes) 000720 1744140863.458015 trunc from 0x032fc1 to 0x00c3a2 (0x26c1e bytes) 000722 1744140863.462747 mapread 0x006f6e thru 0x00c3a2 (0x5435 bytes) 000725 1744140863.465999 trunc from 0x00c3a3 to 0x03c6f9 (0x30357 bytes) 000727 1744140863.471186 MIRROR_EXTEND to 14 mirrors 000729 1744140863.565912 write 0x0132b8 thru 0x01eb82 (0xb8cb bytes) 000730 1744140863.582031 mapwrite 0x02baab thru 0x03b877 (0xfdcd bytes) 000733 1744140863.601454 mapread 0x01a783 thru 0x01bcaf (0x152d bytes) 000734 1744140863.601981 mapread 0x0277a6 thru 0x02eae4 (0x733f bytes) 000737 1744140863.602717 mapwrite 0x01376f thru 0x01a904 (0x7196 bytes) 000741 1744140863.615229 write 0x0083d8 thru 0x016842 (0xe46b bytes) 000742 1744140863.619709 MIRROR_EXTEND to 15 mirrors 000743 1744140863.725161 write 0x00e160 thru 0x0192b1 (0xb152 bytes) 000744 1744140863.741586 mapread 0x0156f3 thru 0x02261f (0xcf2d bytes) 000745 1744140863.746343 MIRROR_EXTEND to 16 mirrors 000746 1744140863.843663 mapwrite 0x00574b thru 0x00a390 (0x4c46 bytes) 000747 1744140863.889623 mapwrite 0x013820 thru 0x0210ae (0xd88f bytes) 000748 1744140863.909337 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000749 1744140863.959566 MIRROR_SPLIT mirror 12 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000750 1744140864.003190 trunc from 0x03c6fa to 0x03bd4a (0x9af bytes) 000751 1744140864.009150 trunc from 0x03bd4b to 0x018d3f (0x2300b bytes) 000752 1744140864.012515 read 0x00d42a thru 0x01818b (0xad62 bytes) 000753 1744140864.015778 MIRROR_RESYNC 14 mirrors 000755 1744140864.309882 MIRROR_RESYNC 14 mirrors 000757 1744140864.364778 write 0x01b664 thru 0x022419 (0x6db6 bytes) 000758 1744140864.377295 write 0x003a72 thru 0x009ddb (0x636a bytes) 000760 1744140864.379797 mapread 0x01832f thru 0x01ec00 (0x68d2 bytes) 000761 1744140864.382150 read 0x015d97 thru 0x01f5ad (0x9817 bytes) 000762 1744140864.382492 mapwrite 0x02102a thru 0x02c94f (0xb926 bytes) 000763 1744140864.409120 read 0x02a9fe thru 0x02c94f (0x1f52 bytes) 000764 1744140864.409508 trunc from 0x02c950 to 0x01e8e9 (0xe066 bytes) 000765 1744140864.413205 trunc from 0x01e8ea to 0x02c39c (0xdab3 bytes) 000766 1744140864.416728 trunc from 0x02c39d to 0x037f8e (0xbbf2 bytes) 000767 1744140864.420027 trunc from 0x037f8f to 0x00d3f0 (0x2ab9e bytes) 000768 1744140864.423371 mapread 0x0009e8 thru 0x00d3f0 (0xca09 bytes) 000769 1744140864.425828 mapread 0x001247 thru 0x0098b1 (0x866b bytes) 000771 1744140864.426654 write 0x01eded thru 0x01f420 (0x634 bytes) 000773 1744140864.427529 mapwrite 0x00320d thru 0x00a806 (0x75fa bytes) 000774 1744140864.442677 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000775 1744140864.484508 mapwrite 0x02859c thru 0x02c727 (0x418c bytes) 000776 1744140864.505434 write 0x02fa00 thru 0x033792 (0x3d93 bytes) 000777 1744140864.506080 write 0x01570f thru 0x022a33 (0xd325 bytes) 000781 1744140864.510509 read 0x029d57 thru 0x03200b (0x82b5 bytes) 000783 1744140864.512250 trunc from 0x033793 to 0x0175d8 (0x1c1ba bytes) 000785 1744140864.516433 mapread 0x015aff thru 0x0175d8 (0x1ada bytes) 000786 1744140864.516916 write 0x029416 thru 0x03454b (0xb136 bytes) 000787 1744140864.517625 write 0x016cf2 thru 0x02321f (0xc52e bytes) 000789 1744140864.519707 write 0x0218a3 thru 0x029676 (0x7dd4 bytes) 000791 1744140864.522166 MIRROR_EXTEND to 14 mirrors 000792 1744140864.628808 write 0x01b30e thru 0x026867 (0xb55a bytes) 000793 1744140864.645307 mapread 0x0156b5 thru 0x01722f (0x1b7b bytes) 000795 1744140864.648049 write 0x03ec99 thru 0x03ffff (0x1367 bytes) 000796 1744140864.648461 read 0x004c9d thru 0x007a46 (0x2daa bytes) 000798 1744140864.650175 write 0x03aae3 thru 0x03e1a3 (0x36c1 bytes) 000801 1744140864.654737 write 0x019d5b thru 0x01d00f (0x32b5 bytes) 000802 1744140864.656663 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000803 1744140864.702340 mapwrite 0x0245f1 thru 0x028983 (0x4393 bytes) 000804 1744140864.718840 write 0x01afef thru 0x01c144 (0x1156 bytes) 000805 1744140864.721341 trunc from 0x040000 to 0x02bcd8 (0x14327 bytes) 000806 1744140864.728363 mapread 0x022ea4 thru 0x02bcd8 (0x8e35 bytes) 000808 1744140864.730371 MIRROR_RESYNC 13 mirrors 000809 1744140865.016081 mapwrite 0x0373e8 thru 0x03ffff (0x8c18 bytes) 000810 1744140865.056996 trunc from 0x040000 to 0x00b39e (0x34c61 bytes) 000811 1744140865.062000 read 0x006d06 thru 0x00b39e (0x4699 bytes) 000812 1744140865.064366 read 0x001814 thru 0x00502b (0x3818 bytes) 000816 1744140865.067379 mapread 0x00855a thru 0x00b39e (0x2e45 bytes) 000817 1744140865.067794 mapread 0x00b125 thru 0x00b39e (0x27a bytes) 000818 1744140865.068254 mapread 0x0006e2 thru 0x00991d (0x923c bytes) 000819 1744140865.068967 trunc from 0x00b39f to 0x02e098 (0x22cfa bytes) 000820 1744140865.074765 trunc from 0x02e099 to 0x02f5cf (0x1537 bytes) 000821 1744140865.078088 read 0x004828 thru 0x00d4ef (0x8cc8 bytes) 000823 1744140865.081087 write 0x00908a thru 0x00c7f4 (0x376b bytes) 000826 1744140865.082112 read 0x01ba56 thru 0x024e03 (0x93ae bytes) 000828 1744140865.082617 write 0x025ca6 thru 0x02febc (0xa217 bytes) 000829 1744140865.083329 write 0x008302 thru 0x00c7d0 (0x44cf bytes) 000830 1744140865.083803 trunc from 0x02febd to 0x032f07 (0x304b bytes) 000831 1744140865.087227 trunc from 0x032f08 to 0x000368 (0x32b9f bytes) 000832 1744140865.090458 mapread 0x00021e thru 0x000368 (0x14b bytes) 000837 1744140865.092034 write 0x01aa1e thru 0x024a2f (0xa012 bytes) 000838 1744140865.092819 mapwrite 0x02bc73 thru 0x030b66 (0x4ef4 bytes) 000842 1744140865.112238 write 0x029105 thru 0x02b9b3 (0x28af bytes) 000843 1744140865.112719 mapread 0x00678c thru 0x010bb2 (0xa427 bytes) 000844 1744140865.113307 mapwrite 0x00995b thru 0x0182de (0xe984 bytes) 000846 1744140865.128873 write 0x03b10f thru 0x03ffff (0x4ef1 bytes) 000849 1744140865.130037 write 0x018355 thru 0x02371e (0xb3ca bytes) 000851 1744140865.130808 mapread 0x00a1f3 thru 0x00f6c2 (0x54d0 bytes) 000852 1744140865.132771 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000853 1744140865.184422 mapread 0x02fb0c thru 0x037868 (0x7d5d bytes) 000854 1744140865.188870 read 0x0390b6 thru 0x03ba01 (0x294c bytes) 000855 1744140865.189166 mapread 0x0118a5 thru 0x011913 (0x6f bytes) 000856 1744140865.191176 mapwrite 0x0168b8 thru 0x0178c0 (0x1009 bytes) 000858 1744140865.203093 mapwrite 0x00fffc thru 0x0194c3 (0x94c8 bytes) 000859 1744140865.216229 MIRROR_RESYNC 12 mirrors 000862 1744140865.487331 write 0x0334e6 thru 0x0361cb (0x2ce6 bytes) 000863 1744140865.507918 mapwrite 0x0203f2 thru 0x02610c (0x5d1b bytes) 000864 1744140865.523341 mapread 0x003e9f thru 0x008520 (0x4682 bytes) 000866 1744140865.526215 read 0x02c835 thru 0x0307f6 (0x3fc2 bytes) 000867 1744140865.528159 mapread 0x00d16e thru 0x0133c7 (0x625a bytes) 000869 1744140865.530230 mapread 0x01d7bf thru 0x025a8d (0x82cf bytes) 000870 1744140865.530626 trunc from 0x040000 to 0x00a91f (0x356e0 bytes) 000871 1744140865.538793 write 0x011826 thru 0x01a0b8 (0x8893 bytes) 000872 1744140865.539424 write 0x0311a3 thru 0x03d5ba (0xc418 bytes) 000873 1744140865.541136 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000874 1744140865.589842 trunc from 0x03d5bb to 0x003024 (0x3a596 bytes) 000875 1744140865.593374 trunc from 0x003025 to 0x0031ce (0x1aa bytes) 000876 1744140865.596346 mapwrite 0x009231 thru 0x016426 (0xd1f6 bytes) 000877 1744140865.619966 read 0x003c08 thru 0x010db3 (0xd1ac bytes) 000879 1744140865.622104 trunc from 0x016427 to 0x010f71 (0x54b5 bytes) 000880 1744140865.625958 write 0x01e11b thru 0x02a6d7 (0xc5bd bytes) 000881 1744140865.626759 mapwrite 0x01d622 thru 0x020f67 (0x3946 bytes) 000883 1744140865.643163 read 0x00bcff thru 0x019076 (0xd378 bytes) 000884 1744140865.645253 read 0x029824 thru 0x02a22e (0xa0b bytes) 000885 1744140865.645531 write 0x02311e thru 0x028ebc (0x5d9f bytes) 000886 1744140865.647091 mapread 0x00eed0 thru 0x01c190 (0xd2c1 bytes) 000887 1744140865.647552 mapwrite 0x0356d3 thru 0x03ffff (0xa92d bytes) 000889 1744140865.669092 read 0x02f626 thru 0x03da1c (0xe3f7 bytes) 000890 1744140865.669578 trunc from 0x040000 to 0x01c25c (0x23da3 bytes) 000892 1744140865.673817 mapread 0x003e42 thru 0x00536b (0x152a bytes) 000894 1744140865.674616 read 0x015269 thru 0x017813 (0x25ab bytes) 000895 1744140865.675986 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000899 1744140865.723812 read 0x015d6e thru 0x01c25c (0x64ef bytes) 000900 1744140865.726964 MIRROR_RESYNC 10 mirrors 000902 1744140865.965113 mapread 0x010469 thru 0x0176fe (0x7296 bytes) 000906 1744140865.969694 mapread 0x00b6db thru 0x00c96f (0x1295 bytes) 000909 1744140865.971893 mapread 0x00541e thru 0x011338 (0xbf1b bytes) 000911 1744140865.972362 trunc from 0x01c25d to 0x011e4e (0xa40e bytes) 000915 1744140865.991815 write 0x00c8b2 thru 0x01760e (0xad5d bytes) 000920 1744140865.998117 MIRROR_EXTEND to 11 mirrors 000921 1744140866.096666 mapwrite 0x019127 thru 0x02581a (0xc6f4 bytes) 000922 1744140866.137392 read 0x011662 thru 0x011c71 (0x610 bytes) 000925 1744140866.139287 read 0x018175 thru 0x024626 (0xc4b2 bytes) 000928 1744140866.140047 mapread 0x0233c0 thru 0x02426d (0xeae bytes) 000929 1744140866.140470 trunc from 0x02581b to 0x00133d (0x244dd bytes) 000931 1744140866.145350 trunc from 0x00133e to 0x03c21d (0x3aee0 bytes) 000932 1744140866.148695 read 0x0339d7 thru 0x03ba70 (0x809a bytes) 000933 1744140866.153165 mapread 0x009895 thru 0x014d02 (0xb46e bytes) 000934 1744140866.153713 read 0x02f7da thru 0x0338eb (0x4112 bytes) 000935 1744140866.154051 trunc from 0x03c21e to 0x018422 (0x23dfb bytes) 000936 1744140866.159030 MIRROR_RESYNC 11 mirrors 000938 1744140866.407572 mapwrite 0x000f06 thru 0x00bdf9 (0xaef4 bytes) 000939 1744140866.450319 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000941 1744140866.499263 MIRROR_EXTEND to 11 mirrors 000943 1744140866.585216 write 0x019772 thru 0x01eaee (0x537d bytes) 000944 1744140866.599947 MIRROR_EXTEND to 12 mirrors 000947 1744140866.693906 write 0x01911f thru 0x020ba2 (0x7a84 bytes) 000948 1744140866.710157 trunc from 0x020ba3 to 0x026133 (0x5591 bytes) 000951 1744140866.718712 mapwrite 0x00eed8 thru 0x00f1f8 (0x321 bytes) 000952 1744140866.738498 MIRROR_RESYNC 12 mirrors 000953 1744140867.001354 read 0x00834a thru 0x014266 (0xbf1d bytes) 000954 1744140867.003373 mapwrite 0x00a8f0 thru 0x00d99b (0x30ac bytes) 000955 1744140867.044405 MIRROR_RESYNC 12 mirrors 000956 1744140867.302681 trunc from 0x026134 to 0x0041a7 (0x21f8c bytes) 000958 1744140867.323256 read 0x0015fc thru 0x0041a7 (0x2bac bytes) 000959 1744140867.324772 mapread 0x000137 thru 0x0041a7 (0x4071 bytes) 000960 1744140867.326038 mapwrite 0x03e496 thru 0x03ffff (0x1b6a bytes) 000961 1744140867.344724 trunc from 0x040000 to 0x02610d (0x19ef2 bytes) 000964 1744140867.348471 mapwrite 0x020ec3 thru 0x02529b (0x43d9 bytes) 000966 1744140867.362983 mapwrite 0x039a70 thru 0x03c4e1 (0x2a72 bytes) 000967 1744140867.380746 trunc from 0x03c4e2 to 0x00bf56 (0x3058b bytes) 000968 1744140867.383975 mapwrite 0x01d656 thru 0x02d074 (0xfa1f bytes) 000969 1744140867.404552 trunc from 0x02d075 to 0x02eab8 (0x1a44 bytes) 000971 1744140867.407998 read 0x00a2fe thru 0x0110fd (0x6e00 bytes) 000972 1744140867.410401 mapread 0x0249a7 thru 0x02eab8 (0xa112 bytes) 000975 1744140867.411508 read 0x01c621 thru 0x01fc12 (0x35f2 bytes) 000976 1744140867.411762 write 0x037359 thru 0x03c2e6 (0x4f8e bytes) 000977 1744140867.412442 write 0x013d88 thru 0x023503 (0xf77c bytes) 000978 1744140867.414867 MIRROR_RESYNC 12 mirrors 000980 1744140867.688796 read 0x013532 thru 0x0187e1 (0x52b0 bytes) 000982 1744140867.691913 MIRROR_EXTEND to 13 mirrors 000984 1744140867.823666 mapread 0x002936 thru 0x00817e (0x5849 bytes) 000986 1744140867.827654 write 0x032b2e thru 0x03f113 (0xc5e6 bytes) 000988 1744140867.845056 mapwrite 0x0130b5 thru 0x015472 (0x23be bytes) 000991 1744140867.862833 mapwrite 0x038ef3 thru 0x03ffff (0x710d bytes) 000992 1744140867.884390 mapwrite 0x026695 thru 0x02b08c (0x49f8 bytes) 000993 1744140867.898325 write 0x004fd3 thru 0x00c6b4 (0x76e2 bytes) 000997 1744140867.902927 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x849:0x0]: Invalid argument (22) 000998 1744140867.954581 mapread 0x0130ac thru 0x019935 (0x688a bytes) 000999 1744140867.959930 mapwrite 0x001bdc thru 0x002d56 (0x117b bytes) All operations completed A-OK! PASS 70a (13s) == sanity-flr test 71: check mirror extend parent fid ==== 15:34:29 (1744140869) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (0s) == sanity-flr test 200a: stress test ===================== 15:34:29 (1744140869) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8929920 .. Extending file size to 2857280 .. Extending file size to 3214784 .. done Extending file size to 6360192 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6415488 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7121632 .. Extending file size to 7185312 .. Extending file size to 7941856 .. failed Extending file size to 8661568 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9239968 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8848384 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9088640 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 29735 29736 29738 29739 29742 done 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) 192.168.202.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg251-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:35:33 (1744140933) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8671040 .. Extending file size to 2429056 .. done Extending file size to 7841120 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8423904 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8510592 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8576832 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8987168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9179968 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8873664 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8896736 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9000064 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 1647 1648 1650 1652 1654 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) 192.168.202.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg251-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:36:36 (1744140996) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5248 2203008 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 5248 2203008 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0415024 s, 50.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0858292 s, 24.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:36:41 (1744141001) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0287414 s, 73.0 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 15:36:42 (1744141002) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x88c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb5e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb5e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb5f:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xb61:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:36:44 (1744141004) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0263517 s, 79.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb60:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb62:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb63:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:36:46 (1744141006) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.028681 s, 73.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb62:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb64:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb62:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb64:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb65:0x0] } PASS 204b (1s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:36:47 (1744141007) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0254091 s, 82.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb63:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb66:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb63:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb66:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb67:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:37:10 (1744141030) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200689 s, 52.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb64:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb69:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb68:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb64:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb69:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb68:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb65:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:37:53 (1744141073) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb67:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0263672 s, 79.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb67:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb6a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb67:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb6a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (40s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:38:33 (1744141113) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg251-server mds-ost sync done. Creating new pool test_204f oleg251-server: Pool lustre.test_204f created Adding targets to pool oleg251-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg251-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0214461 s, 97.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb68:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb6b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xb68:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb6b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg251-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg251-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg251-server: Pool lustre.test_204f destroyed PASS 204f (53s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:39:26 (1744141166) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:39:28 (1744141168) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:39:30 (1744141170) oleg251-server: Pool lustre.test_206 created Adding targets to pool oleg251-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg251-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg251-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg251-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg251-server: Pool lustre.test_206 destroyed PASS 206 (9s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:39:39 (1744141179) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187145 s, 56.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00881792 s, 119 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xae6b5ac1 Mirror 3: 0xae6b5ac1 PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:39:41 (1744141181) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:39:42 (1744141182) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:39:43 (1744141183) Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0171437 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1830000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1830000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1830000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ba0000 Read this (4 bytes): BBBB 192.168.202.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg251-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:39:56 (1744141196) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00226718 s, 3.6 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:39:58 (1744141198) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103065 s, 102 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x8b1 lmm_fid: [0x200000404:0x8b1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x8b1 lmm_fid: [0x200000404:0x8b1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb76:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322836145 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-08 15:39:58.000000000 -0400 Modify: 2025-04-08 15:39:58.000000000 -0400 Change: 2025-04-08 15:39:59.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:40:00 (1744141200) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (0s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:40:00 (1744141200) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.116048 s, 353 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.013491 s, 304 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 8 15:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x8b3:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 8 15:40 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 710 sec ============ 15:40:02 (1744141202) === sanity-flr: start cleanup 15:40:03 (1744141203) === === sanity-flr: finish cleanup 15:40:04 (1744141204) ===