-----============= acceptance-small: sanity-flr ============----- Mon Nov 11 12:28:14 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 6 201 44c === sanity-flr: start setup 12:28:18 (1731346098) === oleg127-client.virtnet: executing check_config_client /mnt/lustre oleg127-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg127-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5c59800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5c59800.idle_timeout=debug disable quota as required oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:28:26 (1731346106) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7564] == sanity-flr test 0a: lfs mirror create with -N option == 12:28:26 (1731346106) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:28:30 (1731346110) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:28:30 (1731346110) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:28:31 (1731346111) PASS 0d (4s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:28:35 (1731346115) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:28:36 (1731346116) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 12:28:37 (1731346117) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b5c59800.stats=clear osc.lustre-OST0001-osc-ffff8800b5c59800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:28:39 (1731346119) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:28:41 (1731346121) PASS 0j (1s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:28:42 (1731346122) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:28:43 (1731346123) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 12:28:45 (1731346125) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:28:46 (1731346126) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:28:48 (1731346128) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:28:50 (1731346130) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (1s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:28:52 (1731346132) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:28:53 (1731346133) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 0.3614 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0295138 s, 35.5 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:28:55 (1731346135) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.186389 s, 101 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:28:58 (1731346138) Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5c59800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5c59800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5c59800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5c59800.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:29:22 (1731346162) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.058614 s, 53.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5c59800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5c59800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:29:35 (1731346175) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5c59800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5c59800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5c59800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5c59800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:29:59 (1731346199) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.925618 s, 56.6 MB/s -rw-r--r-- 1 root root 52428800 Nov 11 12:30 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.241s user 0m0.000s sys 0m0.122s Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b5c59800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5c59800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.253s user 0m0.002s sys 0m0.115s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b5c59800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5c59800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.244s user 0m0.000s sys 0m0.119s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5c59800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b5c59800.ost_server_uuid in FULL state after 0 sec PASS 33b (25s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:30:24 (1731346224) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:30:25 (1731346225) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:30:26 (1731346226) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 12:30:27 (1731346227) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 12:30:28 (1731346228) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.144322 s, 87.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:30:31 (1731346231) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00826452 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149135 s, 70.3 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:30:34 (1731346234) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:30:36 (1731346236) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00459674 s, 223 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00457302 s, 224 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0042712 s, 240 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00361715 s, 283 kB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 12:30:39 (1731346239) create test files with size 15756906 .. ae2d56b1dfcd3cc2d579930ce9d02cd8 - 7226bfce4695d4880c7035998c9e2d9c - 617655292cf3b30a641996c6cbb95ab5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 12:30:44 (1731346244) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:30:47 (1731346247) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:30:49 (1731346249) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.031209 s, 33.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0155714 s, 67.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 12:30:52 (1731346252) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0316524 s, 66.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.122872 s, 34.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0565207 s, 37.1 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 12:30:55 (1731346255) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 12:30:56 (1731346256) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:30:57 (1731346257) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179233 s, 58.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 12:30:59 (1731346259) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0977813 s, 32.2 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:31:02 (1731346262) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:31:05 (1731346265) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0872055 s, 120 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (2s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:31:07 (1731346267) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b6fcd000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0844014 s, 124 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b5e78800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:31:11 (1731346271) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:31:17 (1731346277) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:31:23 (1731346283) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 12:31:24 (1731346284) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0762453 s, 41.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0458291 s, 68.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: c7faa089c4755e059b54ebc158d1741e - mirror 1 checksum: f186c6bb4ab139d9150a9d7767580866 - mirror 2 checksum: c7faa089c4755e059b54ebc158d1741e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Nov 11 12:31:25 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Nov 11 12:31:25 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c7faa089c4755e059b54ebc158d1741e - mirror 1 checksum: f186c6bb4ab139d9150a9d7767580866 - mirror 2 checksum: f186c6bb4ab139d9150a9d7767580866 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Nov 11 12:31:25 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Nov 11 12:31:25 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:31:27 (1731346287) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00973566 s, 108 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 1321984.. 1323007: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (1s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:31:28 (1731346288) Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 12:31:35 (1731346295) shut down Failover mds1 to oleg127-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 12:31:49 (1731346309) targets are mounted 12:31:49 (1731346309) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215332 s, 48.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (28s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:31:56 (1731346316) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0148349 s, 1.4 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00710361 s, 2.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:32:00 (1731346320) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0165357 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00463962 s, 4.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:32:02 (1731346322) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00366807 s, 4.5 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:32:05 (1731346325) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0161538 s, 1.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:32:07 (1731346327) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0031919 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0122494 s, 334 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:32:10 (1731346330) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:32:16 (1731346336) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:32:35 (1731346355) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:32:53 (1731346373) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:33:09 (1731346389) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 12:33:11 (1731346391) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26088 mirror split pid 26089 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 26088 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 26089 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 12:34:12 (1731346452) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 6453 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1731346453.404575 write 0x0079cd thru 0x00bae1 (0x4115 bytes) fallocating to largest ever: 0x21a87 000003 1731346453.417632 fallocate 0x014f9c thru 0x021a86 (0xcaeb bytes) 000005 1731346453.419505 trunc from 0x00bae2 to 0x01797f (0xbe9e bytes) fallocating to largest ever: 0x2bc48 000006 1731346453.423073 fallocate 0x025fdc thru 0x02bc47 (0x5c6c bytes) 000007 1731346453.424784 punch from 0x009315 to 0x00ccd4 (0x39c0 bytes) 000010 1731346453.426887 mapwrite 0x016e62 thru 0x01be94 (0x5033 bytes) 000012 1731346453.437013 mapwrite 0x0067f3 thru 0x01647d (0xfc8b bytes) truncating to largest ever: 0x3d077 000013 1731346453.444646 trunc from 0x02bc48 to 0x03d076 (0x1142f bytes) 000014 1731346453.448377 trunc from 0x03d077 to 0x01ab0b (0x2256b bytes) 000015 1731346453.451903 write 0x03c895 thru 0x03ffff (0x376b bytes) 000018 1731346453.453085 trunc from 0x040000 to 0x01c293 (0x23d6c bytes) 000019 1731346453.456711 trunc from 0x01c294 to 0x030f92 (0x14cff bytes) 000020 1731346453.460336 mapwrite 0x0144a9 thru 0x01755d (0x30b5 bytes) 000021 1731346453.464828 mapwrite 0x030ff3 thru 0x03d089 (0xc097 bytes) 000022 1731346453.478045 mapwrite 0x001cff thru 0x00a5dc (0x88de bytes) 000023 1731346453.484185 MIRROR_EXTEND to 2 mirrors 000025 1731346453.577150 mapwrite 0x00d7c6 thru 0x01949e (0xbcd9 bytes) 000027 1731346453.608898 trunc from 0x03d08a to 0x035d55 (0x7334 bytes) 000028 1731346453.617448 punch from 0x0125f9 to 0x01959a (0x6fa2 bytes) 000029 1731346453.618879 read 0x035845 thru 0x035d55 (0x511 bytes) 000031 1731346453.620466 mapwrite 0x03105d thru 0x03f1f1 (0xe195 bytes) 000032 1731346453.633233 fallocate 0x002da3 thru 0x00d41e (0xa67c bytes) 000033 1731346453.635705 MIRROR_RESYNC 2 mirrors 000034 1731346453.725508 punch from 0x038b14 to 0x03c810 (0x3cfd bytes) 000036 1731346453.733048 fallocate 0x008a69 thru 0x00cd09 (0x42a1 bytes) 000038 1731346453.734485 mapread 0x00fe98 thru 0x014d1a (0x4e83 bytes) 000040 1731346453.739128 punch from 0x03d0dc to 0x03f1f1 (0x2116 bytes) 000041 1731346453.741608 MIRROR_RESYNC 2 mirrors 000043 1731346453.854824 trunc from 0x03f1f2 to 0x001219 (0x3dfd8 bytes) 000046 1731346453.867437 write 0x02ab1b thru 0x032439 (0x791f bytes) 000047 1731346453.868200 mapwrite 0x036c53 thru 0x03f4b4 (0x8862 bytes) 000048 1731346453.882944 trunc from 0x03f4b5 to 0x02714f (0x18365 bytes) 000049 1731346453.886010 read 0x01bd7c thru 0x0238bb (0x7b40 bytes) 000050 1731346453.888108 punch from 0x00f2fc to 0x01d2da (0xdfdf bytes) 000051 1731346453.889309 fallocate 0x033d6f thru 0x036afd (0x2d8f bytes) 000052 1731346453.890751 read 0x01bd6e thru 0x021e7b (0x610e bytes) 000053 1731346453.893291 mapread 0x012d8f thru 0x01cbf1 (0x9e63 bytes) 000055 1731346453.893988 write 0x028e78 thru 0x034d37 (0xbec0 bytes) 000056 1731346453.894685 mapread 0x01637a thru 0x01b7b5 (0x543c bytes) 000057 1731346453.895064 mapwrite 0x0179d6 thru 0x01a00a (0x2635 bytes) 000058 1731346453.900401 trunc from 0x034d38 to 0x00fd98 (0x24f9f bytes) 000060 1731346453.904366 fallocate 0x01aeca thru 0x023eb2 (0x8fe9 bytes) 000061 1731346453.905735 punch from 0x008774 to 0x01399e (0xb22b bytes) 000062 1731346453.907068 punch from 0x003826 to 0x00e3a3 (0xab7e bytes) 000063 1731346453.908322 read 0x010e1e thru 0x01d48a (0xc66d bytes) 000064 1731346453.911103 write 0x0375a5 thru 0x03ffff (0x8a5b bytes) 000065 1731346453.911737 punch from 0x00509c to 0x009929 (0x488e bytes) 000067 1731346453.913179 write 0x00e0dd thru 0x00e898 (0x7bc bytes) 000068 1731346453.913540 write 0x00ef85 thru 0x014032 (0x50ae bytes) 000069 1731346453.914037 write 0x039f03 thru 0x03e47e (0x457c bytes) fallocating to largest ever: 0x40000 000070 1731346453.914463 fallocate 0x03e709 thru 0x03ffff (0x18f7 bytes) 000071 1731346453.915928 trunc from 0x040000 to 0x03beb0 (0x414f bytes) 000072 1731346453.919569 write 0x000b9a thru 0x00ec81 (0xe0e8 bytes) 000073 1731346453.920332 trunc from 0x03beb1 to 0x01fb8a (0x1c326 bytes) 000074 1731346453.923905 trunc from 0x01fb8b to 0x0252af (0x5725 bytes) 000075 1731346453.927369 read 0x01c446 thru 0x0252af (0x8e6a bytes) 000076 1731346453.929443 fallocate 0x001883 thru 0x00841b (0x6b99 bytes) 000077 1731346453.930930 mapread 0x015643 thru 0x023a9f (0xe45d bytes) 000079 1731346453.931534 read 0x021f9e thru 0x0252af (0x3312 bytes) 000081 1731346453.932005 read 0x00633f thru 0x014419 (0xe0db bytes) 000082 1731346453.932270 punch from 0x020287 to 0x0252af (0x5029 bytes) 000083 1731346453.934885 MIRROR_EXTEND to 3 mirrors 000084 1731346454.032018 trunc from 0x0252b0 to 0x032b91 (0xd8e2 bytes) 000085 1731346454.045141 trunc from 0x032b92 to 0x03c07d (0x94ec bytes) 000087 1731346454.050163 punch from 0x02e867 to 0x0300da (0x1874 bytes) 000088 1731346454.051482 trunc from 0x03c07e to 0x02ba95 (0x105e8 bytes) 000089 1731346454.056056 MIRROR_RESYNC 3 mirrors 000091 1731346454.174790 trunc from 0x02ba96 to 0x0375fd (0xbb68 bytes) 000094 1731346454.185118 write 0x029c43 thru 0x02b045 (0x1403 bytes) 000095 1731346454.187613 trunc from 0x0375fe to 0x0095e4 (0x2e019 bytes) 000096 1731346454.192558 trunc from 0x0095e5 to 0x01e49d (0x14eb9 bytes) 000097 1731346454.196056 fallocate 0x013793 thru 0x018f16 (0x5784 bytes) 000098 1731346454.198623 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000099 1731346454.240128 punch from 0x011f01 to 0x01386a (0x196a bytes) 000101 1731346454.242006 read 0x00924d thru 0x00f0cf (0x5e83 bytes) 000102 1731346454.244315 fallocate 0x03591d thru 0x036755 (0xe39 bytes) 000103 1731346454.245869 mapread 0x017d3b thru 0x01e49d (0x6763 bytes) 000104 1731346454.251972 MIRROR_RESYNC 2 mirrors 000105 1731346454.335122 mapread 0x01abd7 thru 0x01e49d (0x38c7 bytes) 000106 1731346454.339072 punch from 0x01422b to 0x01c764 (0x853a bytes) 000108 1731346454.348535 fallocate 0x01db02 thru 0x028aff (0xaffe bytes) 000109 1731346454.350031 write 0x029e04 thru 0x02b7d9 (0x19d6 bytes) 000110 1731346454.351626 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000111 1731346454.380326 trunc from 0x02b7da to 0x02ada9 (0xa30 bytes) 000113 1731346454.389019 write 0x00ff6e thru 0x0143b2 (0x4445 bytes) 000115 1731346454.391903 mapwrite 0x035bec thru 0x03ffff (0xa414 bytes) 000116 1731346454.404610 punch from 0x017ee0 to 0x0256f5 (0xd816 bytes) 000117 1731346454.406137 read 0x029dd1 thru 0x02ec5b (0x4e8b bytes) 000118 1731346454.408338 mapread 0x017ced thru 0x01ddfa (0x610e bytes) 000119 1731346454.411131 write 0x03339a thru 0x036a59 (0x36c0 bytes) 000124 1731346454.413583 write 0x0353a5 thru 0x03ffff (0xac5b bytes) 000125 1731346454.414614 fallocate 0x001b76 thru 0x00dc3d (0xc0c8 bytes) 000126 1731346454.417766 MIRROR_EXTEND to 2 mirrors 000127 1731346454.531181 mapread 0x00bbfd thru 0x0172c2 (0xb6c6 bytes) 000128 1731346454.539560 trunc from 0x040000 to 0x003b48 (0x3c4b7 bytes) 000131 1731346454.552001 trunc from 0x003b49 to 0x025fc4 (0x2247c bytes) 000134 1731346454.556647 mapwrite 0x0021c7 thru 0x00937e (0x71b8 bytes) 000136 1731346454.565512 MIRROR_RESYNC 2 mirrors 000138 1731346454.637583 write 0x01caa6 thru 0x01e3f9 (0x1954 bytes) 000139 1731346454.648210 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000141 1731346454.676175 trunc from 0x025fc5 to 0x03a8dd (0x14919 bytes) 000142 1731346454.683752 mapwrite 0x021015 thru 0x028241 (0x722d bytes) 000144 1731346454.692918 punch from 0x01d9df to 0x02c6e7 (0xed09 bytes) 000145 1731346454.694397 mapwrite 0x03209b thru 0x038a0e (0x6974 bytes) 000146 1731346454.700502 read 0x00551b thru 0x0088c0 (0x33a6 bytes) 000149 1731346454.702444 mapread 0x01a7ca thru 0x02a3d1 (0xfc08 bytes) 000150 1731346454.708028 mapread 0x00a484 thru 0x01635e (0xbedb bytes) 000152 1731346454.711748 mapread 0x0214b1 thru 0x02d31d (0xbe6d bytes) 000153 1731346454.712184 mapwrite 0x025006 thru 0x034e95 (0xfe90 bytes) 000155 1731346454.717916 punch from 0x038631 to 0x03a8dd (0x22ad bytes) 000158 1731346454.719614 read 0x01f985 thru 0x022a1b (0x3097 bytes) 000160 1731346454.721155 MIRROR_EXTEND to 2 mirrors 000161 1731346454.820082 punch from 0x02e6d8 to 0x037538 (0x8e61 bytes) 000163 1731346454.829685 write 0x0018a0 thru 0x00a0f7 (0x8858 bytes) 000164 1731346454.833193 punch from 0x00518e to 0x00b4e0 (0x6353 bytes) 000165 1731346454.837073 mapread 0x015123 thru 0x01c991 (0x786f bytes) 000166 1731346454.839659 write 0x01e290 thru 0x0251b7 (0x6f28 bytes) 000167 1731346454.841372 mapread 0x01d6b3 thru 0x02468e (0x6fdc bytes) 000169 1731346454.842386 trunc from 0x03a8de to 0x02224b (0x18692 bytes) 000170 1731346454.850524 punch from 0x012dcc to 0x020ad8 (0xdd0d bytes) 000171 1731346454.853990 read 0x00941b thru 0x018f68 (0xfb4e bytes) 000172 1731346454.856500 fallocate 0x02a55e thru 0x02ea35 (0x44d8 bytes) 000173 1731346454.858015 fallocate 0x018e01 thru 0x028cf8 (0xfef8 bytes) 000174 1731346454.859606 write 0x03815d thru 0x03c1d1 (0x4075 bytes) 000175 1731346454.860472 fallocate 0x02bd90 thru 0x032d8e (0x6fff bytes) 000176 1731346454.862093 write 0x016c64 thru 0x025ee7 (0xf284 bytes) 000177 1731346454.865560 MIRROR_RESYNC 2 mirrors 000179 1731346454.956978 mapwrite 0x0178c7 thru 0x01fd68 (0x84a2 bytes) 000181 1731346454.986579 trunc from 0x03c1d2 to 0x010096 (0x2c13b bytes) 000182 1731346454.994478 read 0x00fb7f thru 0x010096 (0x518 bytes) 000183 1731346454.995896 read 0x006ff6 thru 0x010096 (0x90a1 bytes) 000184 1731346454.998049 write 0x039cab thru 0x03b9c8 (0x1d1e bytes) 000185 1731346454.998572 write 0x027c61 thru 0x0377ab (0xfb4b bytes) 000186 1731346455.001129 read 0x026fef thru 0x03184e (0xa860 bytes) 000189 1731346455.003209 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000190 1731346455.031141 fallocate 0x02e68a thru 0x03699a (0x8311 bytes) 000191 1731346455.032555 fallocate 0x0086c8 thru 0x00ebaf (0x64e8 bytes) 000192 1731346455.033969 trunc from 0x03b9c9 to 0x008562 (0x33466 bytes) 000193 1731346455.038201 read 0x001773 thru 0x008562 (0x6df0 bytes) 000194 1731346455.040437 read 0x0050ec thru 0x008562 (0x3477 bytes) 000195 1731346455.040715 fallocate 0x0101fa thru 0x011609 (0x1410 bytes) 000196 1731346455.042018 write 0x022fe5 thru 0x0303ba (0xd3d6 bytes) 000197 1731346455.042818 fallocate 0x00e573 thru 0x01a9b7 (0xc445 bytes) 000198 1731346455.044166 trunc from 0x0303bb to 0x019d99 (0x16621 bytes) 000199 1731346455.047785 fallocate 0x015343 thru 0x01e354 (0x9012 bytes) 000201 1731346455.049444 mapread 0x00b777 thru 0x01280c (0x7096 bytes) 000202 1731346455.051878 mapread 0x0189f7 thru 0x019d99 (0x13a3 bytes) 000207 1731346455.052984 mapwrite 0x020c3b thru 0x023c80 (0x3046 bytes) 000210 1731346455.064404 write 0x000617 thru 0x001d0c (0x16f6 bytes) 000211 1731346455.064981 read 0x01e527 thru 0x023364 (0x4e3e bytes) 000212 1731346455.065291 trunc from 0x023c81 to 0x00fd92 (0x13eee bytes) 000214 1731346455.068982 trunc from 0x00fd93 to 0x038b90 (0x28dfe bytes) 000216 1731346455.072777 read 0x0248ce thru 0x02b829 (0x6f5c bytes) 000217 1731346455.076046 mapwrite 0x031b43 thru 0x035b87 (0x4045 bytes) 000218 1731346455.081376 punch from 0x032cd7 to 0x0356ba (0x29e4 bytes) 000219 1731346455.082623 mapread 0x02d867 thru 0x038b90 (0xb32a bytes) 000220 1731346455.087104 write 0x0302ff thru 0x03f3ba (0xf0bc bytes) 000221 1731346455.087869 punch from 0x00e99f to 0x01691f (0x7f81 bytes) 000222 1731346455.089118 punch from 0x00e8dd to 0x01a09b (0xb7bf bytes) 000223 1731346455.090336 mapread 0x039e6c thru 0x03f3ba (0x554f bytes) 000225 1731346455.090926 mapwrite 0x02e6ad thru 0x02f11e (0xa72 bytes) 000229 1731346455.095724 fallocate 0x005f5f thru 0x013e06 (0xdea8 bytes) 000231 1731346455.097228 mapread 0x018f75 thru 0x02612b (0xd1b7 bytes) 000232 1731346455.100687 punch from 0x02cfe8 to 0x03aeb3 (0xdecc bytes) 000233 1731346455.102081 write 0x024326 thru 0x030718 (0xc3f3 bytes) 000234 1731346455.103736 read 0x023081 thru 0x02cb8f (0x9b0f bytes) 000235 1731346455.105109 MIRROR_EXTEND to 2 mirrors 000237 1731346455.205040 write 0x02d18f thru 0x038300 (0xb172 bytes) 000238 1731346455.214972 mapwrite 0x00a5ee thru 0x011b0b (0x751e bytes) 000239 1731346455.223591 mapread 0x036b4d thru 0x03f3ba (0x886e bytes) 000241 1731346455.227691 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000242 1731346455.255207 mapwrite 0x034be6 thru 0x03ffff (0xb41a bytes) 000244 1731346455.275390 fallocate 0x0309b4 thru 0x03ffff (0xf64c bytes) 000246 1731346455.277356 trunc from 0x040000 to 0x002a51 (0x3d5ae bytes) 000247 1731346455.281436 write 0x029f35 thru 0x0377fa (0xd8c6 bytes) 000248 1731346455.282739 write 0x0377ee thru 0x03ffff (0x8812 bytes) 000249 1731346455.283719 mapread 0x02979b thru 0x02da02 (0x4268 bytes) 000250 1731346455.284379 punch from 0x03da41 to 0x03e14c (0x70c bytes) 000251 1731346455.287661 fallocate 0x02a9f1 thru 0x034ff2 (0xa602 bytes) 000253 1731346455.289617 mapwrite 0x00be63 thru 0x017187 (0xb325 bytes) 000254 1731346455.299776 read 0x00bf21 thru 0x00c01d (0xfd bytes) 000255 1731346455.300099 punch from 0x008f97 to 0x0155e1 (0xc64b bytes) 000259 1731346455.302081 trunc from 0x040000 to 0x0089f2 (0x3760d bytes) 000260 1731346455.305918 mapwrite 0x013a2e thru 0x020003 (0xc5d6 bytes) 000261 1731346455.319564 fallocate 0x00958d thru 0x00ed5b (0x57cf bytes) 000263 1731346455.321278 write 0x03ce4a thru 0x03d20e (0x3c5 bytes) 000264 1731346455.321725 punch from 0x0341cb to 0x03d20e (0x9044 bytes) 000265 1731346455.324162 punch from 0x02d37b to 0x032894 (0x551a bytes) 000266 1731346455.325489 mapwrite 0x006ede thru 0x00b9e3 (0x4b06 bytes) 000267 1731346455.331672 mapread 0x001b0b thru 0x010bbd (0xf0b3 bytes) 000269 1731346455.332423 mapwrite 0x01a47a thru 0x028da6 (0xe92d bytes) 000270 1731346455.340636 punch from 0x02c828 to 0x031aef (0x52c8 bytes) 000271 1731346455.341976 mapwrite 0x000b63 thru 0x00d5fb (0xca99 bytes) 000273 1731346455.349696 fallocate 0x010404 thru 0x01c04b (0xbc48 bytes) 000275 1731346455.351647 mapwrite 0x01ac68 thru 0x0211d4 (0x656d bytes) 000276 1731346455.356466 punch from 0x017a84 to 0x019976 (0x1ef3 bytes) 000277 1731346455.357737 fallocate 0x038235 thru 0x03ffff (0x7dcb bytes) 000280 1731346455.360006 trunc from 0x040000 to 0x023529 (0x1cad6 bytes) 000281 1731346455.363793 mapwrite 0x026f61 thru 0x02c031 (0x50d1 bytes) 000282 1731346455.374668 read 0x014542 thru 0x01c1db (0x7c9a bytes) 000283 1731346455.377287 MIRROR_EXTEND to 2 mirrors 000284 1731346455.478959 punch from 0x004108 to 0x00c2a9 (0x81a2 bytes) 000285 1731346455.487962 MIRROR_RESYNC 2 mirrors 000286 1731346455.602003 punch from 0x0128a8 to 0x0198df (0x7038 bytes) 000287 1731346455.609747 read 0x017112 thru 0x020a76 (0x9965 bytes) 000288 1731346455.612109 mapwrite 0x00869a thru 0x00bc19 (0x3580 bytes) 000289 1731346455.621005 fallocate 0x002171 thru 0x006944 (0x47d4 bytes) 000290 1731346455.622712 write 0x016b0e thru 0x01ec4b (0x813e bytes) 000291 1731346455.624819 fallocate 0x02af9a thru 0x035106 (0xa16d bytes) 000292 1731346455.626225 trunc from 0x02c032 to 0x00c782 (0x1f8af bytes) 000294 1731346455.635096 punch from 0x00027e to 0x001704 (0x1487 bytes) 000296 1731346455.636792 punch from 0x0026f0 to 0x00c782 (0xa093 bytes) 000298 1731346455.639608 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000299 1731346455.665279 fallocate 0x031b9b thru 0x03500f (0x3475 bytes) 000300 1731346455.666939 read 0x014d05 thru 0x018486 (0x3782 bytes) 000301 1731346455.670209 MIRROR_EXTEND to 2 mirrors 000302 1731346455.766791 read 0x02c614 thru 0x0347dc (0x81c9 bytes) 000303 1731346455.770640 mapwrite 0x028be4 thru 0x035044 (0xc461 bytes) 000304 1731346455.795854 write 0x025bca thru 0x033763 (0xdb9a bytes) 000305 1731346455.796996 trunc from 0x035045 to 0x010e0a (0x2423a bytes) 000307 1731346455.802406 mapread 0x00581f thru 0x010e0a (0xb5ec bytes) 000308 1731346455.806417 write 0x03cede thru 0x03ffff (0x3122 bytes) 000309 1731346455.807089 fallocate 0x0339b4 thru 0x03ffff (0xc64c bytes) 000310 1731346455.808603 write 0x00ad92 thru 0x01743a (0xc6a9 bytes) 000311 1731346455.811616 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000312 1731346455.839991 trunc from 0x040000 to 0x02f787 (0x10878 bytes) 000313 1731346455.844679 fallocate 0x016b2a thru 0x01ee4b (0x8322 bytes) 000314 1731346455.846244 punch from 0x00d579 to 0x0130ff (0x5b87 bytes) 000315 1731346455.847548 punch from 0x020e1d to 0x02e5a8 (0xd78c bytes) 000316 1731346455.848770 mapwrite 0x00a74d thru 0x00d747 (0x2ffb bytes) 000319 1731346455.857392 punch from 0x002ccb to 0x0102e4 (0xd61a bytes) 000321 1731346455.858944 fallocate 0x024d28 thru 0x02c34b (0x7624 bytes) 000323 1731346455.860537 punch from 0x01c03e to 0x01f0f9 (0x30bc bytes) 000324 1731346455.861955 read 0x008d93 thru 0x00c7b9 (0x3a27 bytes) 000325 1731346455.863913 punch from 0x01ccf0 to 0x027798 (0xaaa9 bytes) 000326 1731346455.865212 fallocate 0x03e501 thru 0x03ffff (0x1aff bytes) 000327 1731346455.866600 mapread 0x00d2cf thru 0x00e066 (0xd98 bytes) 000328 1731346455.868455 read 0x01aa70 thru 0x01e57e (0x3b0f bytes) 000329 1731346455.870723 write 0x000abc thru 0x00913b (0x8680 bytes) 000331 1731346455.871491 trunc from 0x02f788 to 0x00bf29 (0x2385e bytes) 000332 1731346455.875313 punch from 0x007fe5 to 0x00bf29 (0x3f45 bytes) 000333 1731346455.878613 write 0x01237c thru 0x021e95 (0xfb1a bytes) 000334 1731346455.879394 mapread 0x0129e9 thru 0x01e88e (0xbea6 bytes) 000335 1731346455.879808 mapwrite 0x00f113 thru 0x015332 (0x6220 bytes) 000337 1731346455.889131 write 0x034883 thru 0x03be55 (0x75d3 bytes) 000338 1731346455.889762 trunc from 0x03be56 to 0x00c6b8 (0x2f79d bytes) 000340 1731346455.893966 trunc from 0x00c6b9 to 0x03ca10 (0x30358 bytes) 000341 1731346455.897569 mapread 0x00b849 thru 0x010798 (0x4f50 bytes) 000342 1731346455.905193 mapwrite 0x0198f1 thru 0x026108 (0xc818 bytes) 000343 1731346455.925452 write 0x03f43b thru 0x03ffff (0xbc5 bytes) 000344 1731346455.926146 mapwrite 0x023985 thru 0x0246e1 (0xd5d bytes) 000346 1731346455.930438 write 0x005270 thru 0x01380b (0xe59c bytes) 000347 1731346455.933124 write 0x0370bf thru 0x03dc8e (0x6bd0 bytes) 000348 1731346455.933976 mapwrite 0x0079fb thru 0x01189c (0x9ea2 bytes) 000350 1731346455.941755 MIRROR_EXTEND to 2 mirrors 000352 1731346456.057444 read 0x0288b5 thru 0x02ab2d (0x2279 bytes) 000354 1731346456.062147 read 0x029a6f thru 0x031eb1 (0x8443 bytes) 000355 1731346456.066282 read 0x0361cf thru 0x03ffff (0x9e31 bytes) 000358 1731346456.071374 mapread 0x027888 thru 0x03346e (0xbbe7 bytes) 000359 1731346456.074982 fallocate 0x00fba4 thru 0x0120c4 (0x2521 bytes) 000360 1731346456.082544 fallocate 0x018af6 thru 0x022bc6 (0xa0d1 bytes) 000361 1731346456.084040 trunc from 0x040000 to 0x0315d8 (0xea27 bytes) 000362 1731346456.089901 read 0x00b676 thru 0x00e4a8 (0x2e33 bytes) 000363 1731346456.092643 write 0x024e6d thru 0x02cd0c (0x7ea0 bytes) 000364 1731346456.093338 fallocate 0x014cc0 thru 0x018eca (0x420b bytes) 000365 1731346456.094945 mapread 0x02005f thru 0x02bfb7 (0xbf59 bytes) 000366 1731346456.095366 mapwrite 0x022be5 thru 0x028b9f (0x5fbb bytes) 000367 1731346456.100735 read 0x023a3e thru 0x024720 (0xce3 bytes) 000368 1731346456.101067 trunc from 0x0315d9 to 0x010f88 (0x20650 bytes) 000370 1731346456.106539 MIRROR_RESYNC 2 mirrors 000371 1731346456.171896 fallocate 0x00bdad thru 0x018f0a (0xd15e bytes) 000372 1731346456.179499 punch from 0x00a215 to 0x010f88 (0x6d74 bytes) 000373 1731346456.180891 punch from 0x0004d5 to 0x00958b (0x90b7 bytes) 000374 1731346456.182350 punch from 0x001484 to 0x0054f2 (0x406f bytes) 000375 1731346456.183742 mapread 0x00229c thru 0x00c4b7 (0xa21c bytes) 000376 1731346456.189317 punch from 0x010b1a to 0x010f88 (0x46f bytes) 000377 1731346456.190745 trunc from 0x010f89 to 0x01296c (0x19e4 bytes) 000378 1731346456.198569 mapwrite 0x010c87 thru 0x017940 (0x6cba bytes) 000379 1731346456.209631 mapwrite 0x030aa2 thru 0x03cc86 (0xc1e5 bytes) 000380 1731346456.222986 trunc from 0x03cc87 to 0x03573b (0x754b bytes) 000381 1731346456.226779 trunc from 0x03573c to 0x00267a (0x330c1 bytes) 000382 1731346456.230306 punch from 0x00250b to 0x00267a (0x170 bytes) 000383 1731346456.231647 write 0x000d3c thru 0x00a826 (0x9aeb bytes) 000384 1731346456.234111 punch from 0x005346 to 0x00a826 (0x54e1 bytes) 000385 1731346456.237724 write 0x003a70 thru 0x00ac0e (0x719f bytes) 000386 1731346456.239342 read 0x0017fd thru 0x008b59 (0x735d bytes) 000387 1731346456.240921 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000390 1731346456.271228 mapwrite 0x03f2b2 thru 0x03ffff (0xd4e bytes) 000391 1731346456.281564 fallocate 0x008603 thru 0x017c98 (0xf696 bytes) 000393 1731346456.283235 punch from 0x01e773 to 0x02da0f (0xf29d bytes) 000394 1731346456.284473 mapread 0x02dfea thru 0x0388cd (0xa8e4 bytes) 000395 1731346456.290087 punch from 0x03c68b to 0x03ee89 (0x27ff bytes) 000396 1731346456.291366 write 0x031015 thru 0x03f4f8 (0xe4e4 bytes) 000397 1731346456.292189 mapread 0x02e2eb thru 0x0389dd (0xa6f3 bytes) 000398 1731346456.292698 fallocate 0x026a02 thru 0x02b664 (0x4c63 bytes) 000399 1731346456.294035 trunc from 0x040000 to 0x0258f4 (0x1a70b bytes) 000402 1731346456.299329 MIRROR_EXTEND to 2 mirrors 000403 1731346456.391271 mapread 0x00f3c9 thru 0x0160fa (0x6d32 bytes) 000404 1731346456.398617 trunc from 0x0258f5 to 0x03d990 (0x1809c bytes) 000405 1731346456.411707 mapwrite 0x02639d thru 0x029a90 (0x36f4 bytes) 000406 1731346456.421091 mapwrite 0x00a660 thru 0x018abe (0xe45f bytes) 000407 1731346456.430485 punch from 0x02cb3f to 0x02e4a2 (0x1964 bytes) 000408 1731346456.431610 mapwrite 0x03bfe2 thru 0x03f4d1 (0x34f0 bytes) 000409 1731346456.442566 write 0x00c606 thru 0x010675 (0x4070 bytes) 000411 1731346456.443300 write 0x0227d1 thru 0x02ae35 (0x8665 bytes) 000412 1731346456.443896 trunc from 0x03f4d2 to 0x037b95 (0x793c bytes) 000413 1731346456.447246 mapwrite 0x03737d thru 0x0395f6 (0x227a bytes) 000415 1731346456.457247 read 0x02d3eb thru 0x036735 (0x934b bytes) 000416 1731346456.459364 fallocate 0x03583d thru 0x03ffff (0xa7c3 bytes) 000418 1731346456.461188 read 0x0268a5 thru 0x033e13 (0xd56f bytes) 000419 1731346456.463442 fallocate 0x018f50 thru 0x02293b (0x99ec bytes) 000420 1731346456.466165 MIRROR_RESYNC 2 mirrors 000421 1731346456.557730 trunc from 0x0395f7 to 0x00b52c (0x2e0ca bytes) 000423 1731346456.569627 mapread 0x00729f thru 0x00b52c (0x428e bytes) 000424 1731346456.574317 mapread 0x004b51 thru 0x00b52c (0x69dc bytes) 000425 1731346456.576274 MIRROR_RESYNC 2 mirrors 000426 1731346456.641965 mapread 0x00340f thru 0x00a7b9 (0x73ab bytes) 000429 1731346456.646781 punch from 0x006fba to 0x00b52c (0x4573 bytes) 000430 1731346456.654798 punch from 0x002225 to 0x00b52c (0x9308 bytes) 000431 1731346456.656460 mapread 0x005a1d thru 0x005c18 (0x1fc bytes) 000432 1731346456.659258 mapwrite 0x0389a1 thru 0x03ffff (0x765f bytes) 000433 1731346456.675268 mapread 0x02623c thru 0x035941 (0xf706 bytes) 000434 1731346456.679232 mapread 0x00367a thru 0x00f231 (0xbbb8 bytes) 000435 1731346456.681390 read 0x037d7b thru 0x03ffff (0x8285 bytes) 000437 1731346456.681990 punch from 0x0339b2 to 0x03ffff (0xc64e bytes) 000438 1731346456.683558 trunc from 0x040000 to 0x016a02 (0x295fd bytes) 000440 1731346456.687241 mapread 0x00b6fe thru 0x0128dd (0x71e0 bytes) 000441 1731346456.688757 MIRROR_EXTEND to 3 mirrors 000442 1731346456.779454 read 0x00d6e4 thru 0x01508f (0x79ac bytes) 000443 1731346456.782849 write 0x00fdb0 thru 0x014718 (0x4969 bytes) 000444 1731346456.793522 MIRROR_RESYNC 3 mirrors 000445 1731346456.911027 read 0x016741 thru 0x016a02 (0x2c2 bytes) 000446 1731346456.911287 write 0x011c5e thru 0x01ba2d (0x9dd0 bytes) 000447 1731346456.919317 read 0x004b3d thru 0x012a66 (0xdf2a bytes) 000449 1731346456.922183 read 0x00b96d thru 0x015339 (0x99cd bytes) 000450 1731346456.922482 punch from 0x000ade to 0x00ed5b (0xe27e bytes) 000451 1731346456.924052 punch from 0x007a44 to 0x014914 (0xced1 bytes) 000452 1731346456.926996 fallocate 0x023066 thru 0x0231b1 (0x14c bytes) 000453 1731346456.928252 punch from 0x01123e to 0x012c9a (0x1a5d bytes) 000454 1731346456.929709 mapwrite 0x003b5f thru 0x00e127 (0xa5c9 bytes) 000455 1731346456.937635 write 0x031b1d thru 0x03d4a3 (0xb987 bytes) 000456 1731346456.938515 punch from 0x01a560 to 0x025d80 (0xb821 bytes) 000457 1731346456.939711 punch from 0x00b530 to 0x012cb8 (0x7789 bytes) 000458 1731346456.940984 trunc from 0x03d4a4 to 0x03e42b (0xf88 bytes) 000459 1731346456.948722 punch from 0x013889 to 0x02038b (0xcb03 bytes) 000460 1731346456.950125 read 0x021c67 thru 0x02d497 (0xb831 bytes) 000461 1731346456.954026 MIRROR_EXTEND to 4 mirrors 000462 1731346457.065067 read 0x019a3b thru 0x02465a (0xac20 bytes) 000464 1731346457.070388 mapwrite 0x037a39 thru 0x03ccb7 (0x527f bytes) 000465 1731346457.101611 fallocate 0x009820 thru 0x012765 (0x8f46 bytes) 000466 1731346457.103419 trunc from 0x03e42c to 0x00015a (0x3e2d1 bytes) 000467 1731346457.109695 read 0x000094 thru 0x00015a (0xc7 bytes) 000469 1731346457.111953 write 0x0191b8 thru 0x025285 (0xc0ce bytes) 000470 1731346457.113344 mapread 0x00626e thru 0x010d89 (0xab1c bytes) 000473 1731346457.116927 trunc from 0x025286 to 0x03d21d (0x17f98 bytes) 000474 1731346457.120868 mapread 0x024c31 thru 0x031d27 (0xd0f7 bytes) 000475 1731346457.123828 fallocate 0x038e03 thru 0x03eac4 (0x5cc2 bytes) 000477 1731346457.126031 read 0x033876 thru 0x03890e (0x5099 bytes) 000478 1731346457.126522 mapread 0x01c8e0 thru 0x01e0b4 (0x17d5 bytes) 000479 1731346457.127169 write 0x02772b thru 0x02abf3 (0x34c9 bytes) 000480 1731346457.127928 read 0x02163b thru 0x029dd3 (0x8799 bytes) 000481 1731346457.128362 mapwrite 0x00a751 thru 0x00edfe (0x46ae bytes) 000482 1731346457.134359 write 0x029d18 thru 0x035cd7 (0xbfc0 bytes) 000483 1731346457.136955 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000484 1731346457.179469 read 0x025664 thru 0x02c2ab (0x6c48 bytes) 000485 1731346457.181726 fallocate 0x03617b thru 0x03dd47 (0x7bcd bytes) 000487 1731346457.183347 mapread 0x03610e thru 0x03eac4 (0x89b7 bytes) 000488 1731346457.188560 MIRROR_EXTEND to 4 mirrors 000489 1731346457.302996 fallocate 0x00efac thru 0x018782 (0x97d7 bytes) 000490 1731346457.311638 punch from 0x00bd2d to 0x00f010 (0x32e4 bytes) 000492 1731346457.313322 mapwrite 0x01e202 thru 0x0258b9 (0x76b8 bytes) 000493 1731346457.323966 mapwrite 0x032d96 thru 0x0348fa (0x1b65 bytes) 000494 1731346457.329936 write 0x02dcad thru 0x031870 (0x3bc4 bytes) 000495 1731346457.331778 write 0x0058e2 thru 0x007761 (0x1e80 bytes) 000497 1731346457.334709 trunc from 0x03eac5 to 0x026d50 (0x17d74 bytes) 000498 1731346457.342761 mapread 0x013b42 thru 0x015f8d (0x244c bytes) 000499 1731346457.345077 mapwrite 0x019d32 thru 0x01b273 (0x1542 bytes) 000500 1731346457.350779 mapwrite 0x00d12b thru 0x00d1da (0xb0 bytes) 000502 1731346457.354208 write 0x01bfe5 thru 0x0298d5 (0xd8f1 bytes) 000503 1731346457.360451 MIRROR_RESYNC 4 mirrors 000504 1731346457.556402 fallocate 0x01874e thru 0x020273 (0x7b26 bytes) 000505 1731346457.564268 mapread 0x014482 thru 0x01af65 (0x6ae4 bytes) 000506 1731346457.568889 read 0x006fb1 thru 0x007dce (0xe1e bytes) 000507 1731346457.570512 mapwrite 0x010358 thru 0x012330 (0x1fd9 bytes) LOG DUMP (446 total operations): 1: 1731346453.404575 WRITE 0x079cd thru 0x0bae1 (0x04115 bytes) HOLE 2: 1731346453.417632 FALLOC 0x14f9c thru 0x21a87 (0x0caeb bytes)PAST_EOF 3: 1731346453.419505 TRUNC_UP 0x0bae2 to 0x17980 4: 1731346453.423073 FALLOC 0x25fdc thru 0x2bc48 (0x05c6c bytes)EXTENDING 5: 1731346453.424784 PUNCH 0x09315 thru 0x0ccd4 (0x039c0 bytes) 6: 1731346453.426625 SKIPPED (no operation) 7: 1731346453.426887 MAPWRITE 0x16e62 thru 0x1be94 (0x05033 bytes) 8: 1731346453.437013 MAPWRITE 0x067f3 thru 0x1647d (0x0fc8b bytes) 9: 1731346453.444646 TRUNC_UP 0x2bc48 to 0x3d077 10: 1731346453.448377 TRUNC_DN 0x3d077 to 0x1ab0c 11: 1731346453.451903 WRITE 0x3c895 thru 0x3ffff (0x0376b bytes) HOLE 12: 1731346453.453085 TRUNC_DN 0x40000 to 0x1c294 13: 1731346453.456711 TRUNC_UP 0x1c294 to 0x30f93 14: 1731346453.460336 MAPWRITE 0x144a9 thru 0x1755d (0x030b5 bytes) 15: 1731346453.464828 MAPWRITE 0x30ff3 thru 0x3d089 (0x0c097 bytes) 16: 1731346453.478045 MAPWRITE 0x01cff thru 0x0a5dc (0x088de bytes) 17: 1731346453.484185 MIRROR_EXTEND to 2 mirrors 18: 1731346453.575490 SKIPPED (no operation) 19: 1731346453.577150 MAPWRITE 0x0d7c6 thru 0x1949e (0x0bcd9 bytes) 20: 1731346453.608678 SKIPPED (no operation) 21: 1731346453.608898 TRUNC_DN 0x3d08a to 0x35d56 22: 1731346453.617448 PUNCH 0x125f9 thru 0x1959a (0x06fa2 bytes) 23: 1731346453.618879 READ 0x35845 thru 0x35d55 (0x00511 bytes) 24: 1731346453.620466 MAPWRITE 0x3105d thru 0x3f1f1 (0x0e195 bytes) 25: 1731346453.633233 FALLOC 0x02da3 thru 0x0d41f (0x0a67c bytes)INTERIOR 26: 1731346453.635705 MIRROR_RESYNC 2 mirrors 27: 1731346453.725508 PUNCH 0x38b14 thru 0x3c810 (0x03cfd bytes) 28: 1731346453.732825 SKIPPED (no operation) 29: 1731346453.733048 FALLOC 0x08a69 thru 0x0cd0a (0x042a1 bytes)INTERIOR 30: 1731346453.734293 SKIPPED (no operation) 31: 1731346453.734485 MAPREAD 0x0fe98 thru 0x14d1a (0x04e83 bytes) 32: 1731346453.738919 SKIPPED (no operation) 33: 1731346453.739128 PUNCH 0x3d0dc thru 0x3f1f1 (0x02116 bytes) 34: 1731346453.741608 MIRROR_RESYNC 2 mirrors 35: 1731346453.854824 TRUNC_DN 0x3f1f2 to 0x0121a 36: 1731346453.867203 SKIPPED (no operation) 37: 1731346453.867437 WRITE 0x2ab1b thru 0x32439 (0x0791f bytes) HOLE 38: 1731346453.868200 MAPWRITE 0x36c53 thru 0x3f4b4 (0x08862 bytes) 39: 1731346453.882944 TRUNC_DN 0x3f4b5 to 0x27150 40: 1731346453.886010 READ 0x1bd7c thru 0x238bb (0x07b40 bytes) 41: 1731346453.888108 PUNCH 0x0f2fc thru 0x1d2da (0x0dfdf bytes) 42: 1731346453.889309 FALLOC 0x33d6f thru 0x36afe (0x02d8f bytes)PAST_EOF 43: 1731346453.890751 READ 0x1bd6e thru 0x21e7b (0x0610e bytes) 44: 1731346453.893291 MAPREAD 0x12d8f thru 0x1cbf1 (0x09e63 bytes) 45: 1731346453.893772 SKIPPED (no operation) 46: 1731346453.893988 WRITE 0x28e78 thru 0x34d37 (0x0bec0 bytes) HOLE 47: 1731346453.894685 MAPREAD 0x1637a thru 0x1b7b5 (0x0543c bytes) 48: 1731346453.895064 MAPWRITE 0x179d6 thru 0x1a00a (0x02635 bytes) 49: 1731346453.900401 TRUNC_DN 0x34d38 to 0x0fd99 50: 1731346453.904165 SKIPPED (no operation) 51: 1731346453.904366 FALLOC 0x1aeca thru 0x23eb3 (0x08fe9 bytes)EXTENDING 52: 1731346453.905735 PUNCH 0x08774 thru 0x1399e (0x0b22b bytes) 53: 1731346453.907068 PUNCH 0x03826 thru 0x0e3a3 (0x0ab7e bytes) 54: 1731346453.908322 READ 0x10e1e thru 0x1d48a (0x0c66d bytes) 55: 1731346453.911103 WRITE 0x375a5 thru 0x3ffff (0x08a5b bytes) HOLE 56: 1731346453.911737 PUNCH 0x0509c thru 0x09929 (0x0488e bytes) 57: 1731346453.913179 WRITE 0x0e0dd thru 0x0e898 (0x007bc bytes) 58: 1731346453.913540 WRITE 0x0ef85 thru 0x14032 (0x050ae bytes) 59: 1731346453.914037 WRITE 0x39f03 thru 0x3e47e (0x0457c bytes) 60: 1731346453.914463 FALLOC 0x3e709 thru 0x40000 (0x018f7 bytes)INTERIOR 61: 1731346453.915928 TRUNC_DN 0x40000 to 0x3beb1 62: 1731346453.919569 WRITE 0x00b9a thru 0x0ec81 (0x0e0e8 bytes) 63: 1731346453.920332 TRUNC_DN 0x3beb1 to 0x1fb8b 64: 1731346453.923905 TRUNC_UP 0x1fb8b to 0x252b0 65: 1731346453.927369 READ 0x1c446 thru 0x252af (0x08e6a bytes) 66: 1731346453.929443 FALLOC 0x01883 thru 0x0841c (0x06b99 bytes)INTERIOR 67: 1731346453.930930 MAPREAD 0x15643 thru 0x23a9f (0x0e45d bytes) 68: 1731346453.931534 READ 0x21f9e thru 0x252af (0x03312 bytes) 69: 1731346453.931795 SKIPPED (no operation) 70: 1731346453.932005 READ 0x0633f thru 0x14419 (0x0e0db bytes) 71: 1731346453.932270 PUNCH 0x20287 thru 0x252af (0x05029 bytes) 72: 1731346453.934885 MIRROR_EXTEND to 3 mirrors 73: 1731346454.032018 TRUNC_UP 0x252b0 to 0x32b92 74: 1731346454.045141 TRUNC_UP 0x32b92 to 0x3c07e 75: 1731346454.050163 PUNCH 0x2e867 thru 0x300da (0x01874 bytes) 76: 1731346454.051482 TRUNC_DN 0x3c07e to 0x2ba96 77: 1731346454.056056 MIRROR_RESYNC 3 mirrors 78: 1731346454.174595 SKIPPED (no operation) 79: 1731346454.174790 TRUNC_UP 0x2ba96 to 0x375fe 80: 1731346454.184397 SKIPPED (no operation) 81: 1731346454.185118 WRITE 0x29c43 thru 0x2b045 (0x01403 bytes) 82: 1731346454.187613 TRUNC_DN 0x375fe to 0x095e5 83: 1731346454.192558 TRUNC_UP 0x095e5 to 0x1e49e 84: 1731346454.196056 FALLOC 0x13793 thru 0x18f17 (0x05784 bytes)INTERIOR 85: 1731346454.198623 MIRROR_SPLIT mirror 2 to 2 mirrors 86: 1731346454.240128 PUNCH 0x11f01 thru 0x1386a (0x0196a bytes) 87: 1731346454.242006 READ 0x0924d thru 0x0f0cf (0x05e83 bytes) 88: 1731346454.244315 FALLOC 0x3591d thru 0x36756 (0x00e39 bytes)PAST_EOF 89: 1731346454.245869 MAPREAD 0x17d3b thru 0x1e49d (0x06763 bytes) 90: 1731346454.251972 MIRROR_RESYNC 2 mirrors 91: 1731346454.335122 MAPREAD 0x1abd7 thru 0x1e49d (0x038c7 bytes) 92: 1731346454.339072 PUNCH 0x1422b thru 0x1c764 (0x0853a bytes) 93: 1731346454.348339 SKIPPED (no operation) 94: 1731346454.348535 FALLOC 0x1db02 thru 0x28b00 (0x0affe bytes)PAST_EOF 95: 1731346454.350031 WRITE 0x29e04 thru 0x2b7d9 (0x019d6 bytes) HOLE 96: 1731346454.351626 MIRROR_SPLIT mirror 1 to 1 mirrors 97: 1731346454.380326 TRUNC_DN 0x2b7da to 0x2adaa 98: 1731346454.389019 WRITE 0x0ff6e thru 0x143b2 (0x04445 bytes) 99: 1731346454.391903 MAPWRITE 0x35bec thru 0x3ffff (0x0a414 bytes) 100: 1731346454.404610 PUNCH 0x17ee0 thru 0x256f5 (0x0d816 bytes) 101: 1731346454.406137 READ 0x29dd1 thru 0x2ec5b (0x04e8b bytes) 102: 1731346454.408338 MAPREAD 0x17ced thru 0x1ddfa (0x0610e bytes) 103: 1731346454.411131 WRITE 0x3339a thru 0x36a59 (0x036c0 bytes) 104: 1731346454.411894 SKIPPED (no operation) 105: 1731346454.412573 SKIPPED (no operation) 106: 1731346454.413583 WRITE 0x353a5 thru 0x3ffff (0x0ac5b bytes) 107: 1731346454.414614 FALLOC 0x01b76 thru 0x0dc3e (0x0c0c8 bytes)INTERIOR 108: 1731346454.417766 MIRROR_EXTEND to 2 mirrors 109: 1731346454.531181 MAPREAD 0x0bbfd thru 0x172c2 (0x0b6c6 bytes) 110: 1731346454.539560 TRUNC_DN 0x40000 to 0x03b49 111: 1731346454.551437 SKIPPED (no operation) 112: 1731346454.552001 TRUNC_UP 0x03b49 to 0x25fc5 113: 1731346454.556107 SKIPPED (no operation) 114: 1731346454.556406 SKIPPED (no operation) 115: 1731346454.556647 MAPWRITE 0x021c7 thru 0x0937e (0x071b8 bytes) 116: 1731346454.564066 SKIPPED (no operation) 117: 1731346454.565512 MIRROR_RESYNC 2 mirrors 118: 1731346454.637396 SKIPPED (no operation) 119: 1731346454.637583 WRITE 0x1caa6 thru 0x1e3f9 (0x01954 bytes) 120: 1731346454.648210 MIRROR_SPLIT mirror 1 to 1 mirrors 121: 1731346454.676175 TRUNC_UP 0x25fc5 to 0x3a8de 122: 1731346454.683752 MAPWRITE 0x21015 thru 0x28241 (0x0722d bytes) 123: 1731346454.692637 SKIPPED (no operation) 124: 1731346454.692918 PUNCH 0x1d9df thru 0x2c6e7 (0x0ed09 bytes) 125: 1731346454.694397 MAPWRITE 0x3209b thru 0x38a0e (0x06974 bytes) 126: 1731346454.700502 READ 0x0551b thru 0x088c0 (0x033a6 bytes) 127: 1731346454.701966 SKIPPED (no operation) 128: 1731346454.702198 SKIPPED (no operation) 129: 1731346454.702444 MAPREAD 0x1a7ca thru 0x2a3d1 (0x0fc08 bytes) 130: 1731346454.708028 MAPREAD 0x0a484 thru 0x1635e (0x0bedb bytes) 131: 1731346454.711525 SKIPPED (no operation) 132: 1731346454.711748 MAPREAD 0x214b1 thru 0x2d31d (0x0be6d bytes) 133: 1731346454.712184 MAPWRITE 0x25006 thru 0x34e95 (0x0fe90 bytes) 134: 1731346454.717711 SKIPPED (no operation) 135: 1731346454.717916 PUNCH 0x38631 thru 0x3a8dd (0x022ad bytes) 136: 1731346454.719051 SKIPPED (no operation) 137: 1731346454.719614 READ 0x1f985 thru 0x22a1b (0x03097 bytes) 138: 1731346454.719885 SKIPPED (no operation) 139: 1731346454.721155 MIRROR_EXTEND to 2 mirrors 140: 1731346454.820082 PUNCH 0x2e6d8 thru 0x37538 (0x08e61 bytes) 141: 1731346454.829463 SKIPPED (no operation) 142: 1731346454.829685 WRITE 0x018a0 thru 0x0a0f7 (0x08858 bytes) 143: 1731346454.833193 PUNCH 0x0518e thru 0x0b4e0 (0x06353 bytes) 144: 1731346454.837073 MAPREAD 0x15123 thru 0x1c991 (0x0786f bytes) 145: 1731346454.839659 WRITE 0x1e290 thru 0x251b7 (0x06f28 bytes) 146: 1731346454.841372 MAPREAD 0x1d6b3 thru 0x2468e (0x06fdc bytes) 147: 1731346454.842023 SKIPPED (no operation) 148: 1731346454.842386 TRUNC_DN 0x3a8de to 0x2224c 149: 1731346454.850524 PUNCH 0x12dcc thru 0x20ad8 (0x0dd0d bytes) 150: 1731346454.853990 READ 0x0941b thru 0x18f68 (0x0fb4e bytes) 151: 1731346454.856500 FALLOC 0x2a55e thru 0x2ea36 (0x044d8 bytes)PAST_EOF 152: 1731346454.858015 FALLOC 0x18e01 thru 0x28cf9 (0x0fef8 bytes)EXTENDING 153: 1731346454.859606 WRITE 0x3815d thru 0x3c1d1 (0x04075 bytes) HOLE 154: 1731346454.860472 FALLOC 0x2bd90 thru 0x32d8f (0x06fff bytes)INTERIOR 155: 1731346454.862093 WRITE 0x16c64 thru 0x25ee7 (0x0f284 bytes) 156: 1731346454.865560 MIRROR_RESYNC 2 mirrors 157: 1731346454.956859 SKIPPED (no operation) 158: 1731346454.956978 MAPWRITE 0x178c7 thru 0x1fd68 (0x084a2 bytes) 159: 1731346454.986579 TRUNC_DN 0x3c1d2 to 0x10097 160: 1731346454.994478 READ 0x0fb7f thru 0x10096 (0x00518 bytes) 161: 1731346454.995896 READ 0x06ff6 thru 0x10096 (0x090a1 bytes) 162: 1731346454.998049 WRITE 0x39cab thru 0x3b9c8 (0x01d1e bytes) HOLE 163: 1731346454.998572 WRITE 0x27c61 thru 0x377ab (0x0fb4b bytes) 164: 1731346455.001129 READ 0x26fef thru 0x3184e (0x0a860 bytes) 165: 1731346455.001490 SKIPPED (no operation) 166: 1731346455.003209 MIRROR_SPLIT mirror 1 to 1 mirrors 167: 1731346455.031141 FALLOC 0x2e68a thru 0x3699b (0x08311 bytes)INTERIOR 168: 1731346455.032555 FALLOC 0x086c8 thru 0x0ebb0 (0x064e8 bytes)INTERIOR 169: 1731346455.033969 TRUNC_DN 0x3b9c9 to 0x08563 170: 1731346455.038201 READ 0x01773 thru 0x08562 (0x06df0 bytes) 171: 1731346455.040437 READ 0x050ec thru 0x08562 (0x03477 bytes) 172: 1731346455.040715 FALLOC 0x101fa thru 0x1160a (0x01410 bytes)PAST_EOF 173: 1731346455.042018 WRITE 0x22fe5 thru 0x303ba (0x0d3d6 bytes) HOLE 174: 1731346455.042818 FALLOC 0x0e573 thru 0x1a9b8 (0x0c445 bytes)INTERIOR 175: 1731346455.044166 TRUNC_DN 0x303bb to 0x19d9a 176: 1731346455.047785 FALLOC 0x15343 thru 0x1e355 (0x09012 bytes)PAST_EOF 177: 1731346455.049444 MAPREAD 0x0b777 thru 0x1280c (0x07096 bytes) 178: 1731346455.051878 MAPREAD 0x189f7 thru 0x19d99 (0x013a3 bytes) 179: 1731346455.052220 SKIPPED (no operation) 180: 1731346455.052984 MAPWRITE 0x20c3b thru 0x23c80 (0x03046 bytes) 181: 1731346455.063889 SKIPPED (no operation) 182: 1731346455.064169 SKIPPED (no operation) 183: 1731346455.064404 WRITE 0x00617 thru 0x01d0c (0x016f6 bytes) 184: 1731346455.064981 READ 0x1e527 thru 0x23364 (0x04e3e bytes) 185: 1731346455.065291 TRUNC_DN 0x23c81 to 0x0fd93 186: 1731346455.068982 TRUNC_UP 0x0fd93 to 0x38b91 187: 1731346455.072777 READ 0x248ce thru 0x2b829 (0x06f5c bytes) 188: 1731346455.076046 MAPWRITE 0x31b43 thru 0x35b87 (0x04045 bytes) 189: 1731346455.081376 PUNCH 0x32cd7 thru 0x356ba (0x029e4 bytes) 190: 1731346455.082623 MAPREAD 0x2d867 thru 0x38b90 (0x0b32a bytes) 191: 1731346455.087104 WRITE 0x302ff thru 0x3f3ba (0x0f0bc bytes) EXTEND 192: 1731346455.087869 PUNCH 0x0e99f thru 0x1691f (0x07f81 bytes) 193: 1731346455.089118 PUNCH 0x0e8dd thru 0x1a09b (0x0b7bf bytes) 194: 1731346455.090336 MAPREAD 0x39e6c thru 0x3f3ba (0x0554f bytes) 195: 1731346455.090719 SKIPPED (no operation) 196: 1731346455.090926 MAPWRITE 0x2e6ad thru 0x2f11e (0x00a72 bytes) 197: 1731346455.094975 SKIPPED (no operation) 198: 1731346455.095724 FALLOC 0x05f5f thru 0x13e07 (0x0dea8 bytes)INTERIOR 199: 1731346455.097041 SKIPPED (no operation) 200: 1731346455.097228 MAPREAD 0x18f75 thru 0x2612b (0x0d1b7 bytes) 201: 1731346455.100687 PUNCH 0x2cfe8 thru 0x3aeb3 (0x0decc bytes) 202: 1731346455.102081 WRITE 0x24326 thru 0x30718 (0x0c3f3 bytes) 203: 1731346455.103736 READ 0x23081 thru 0x2cb8f (0x09b0f bytes) 204: 1731346455.105109 MIRROR_EXTEND to 2 mirrors 205: 1731346455.203238 SKIPPED (no operation) 206: 1731346455.205040 WRITE 0x2d18f thru 0x38300 (0x0b172 bytes) 207: 1731346455.214972 MAPWRITE 0x0a5ee thru 0x11b0b (0x0751e bytes) 208: 1731346455.223591 MAPREAD 0x36b4d thru 0x3f3ba (0x0886e bytes) 209: 1731346455.226175 SKIPPED (no operation) 210: 1731346455.227691 MIRROR_SPLIT mirror 1 to 1 mirrors 211: 1731346455.255207 MAPWRITE 0x34be6 thru 0x3ffff (0x0b41a bytes) 212: 1731346455.275390 FALLOC 0x309b4 thru 0x40000 (0x0f64c bytes)INTERIOR 213: 1731346455.277047 SKIPPED (no operation) 214: 1731346455.277356 TRUNC_DN 0x40000 to 0x02a52 215: 1731346455.281436 WRITE 0x29f35 thru 0x377fa (0x0d8c6 bytes) HOLE 216: 1731346455.282739 WRITE 0x377ee thru 0x3ffff (0x08812 bytes) EXTEND 217: 1731346455.283719 MAPREAD 0x2979b thru 0x2da02 (0x04268 bytes) 218: 1731346455.284379 PUNCH 0x3da41 thru 0x3e14c (0x0070c bytes) 219: 1731346455.287661 FALLOC 0x2a9f1 thru 0x34ff3 (0x0a602 bytes)INTERIOR 220: 1731346455.289617 MAPWRITE 0x0be63 thru 0x17187 (0x0b325 bytes) 221: 1731346455.299776 READ 0x0bf21 thru 0x0c01d (0x000fd bytes) 222: 1731346455.300099 PUNCH 0x08f97 thru 0x155e1 (0x0c64b bytes) 223: 1731346455.301358 SKIPPED (no operation) 224: 1731346455.302081 TRUNC_DN 0x40000 to 0x089f3 225: 1731346455.305918 MAPWRITE 0x13a2e thru 0x20003 (0x0c5d6 bytes) 226: 1731346455.319564 FALLOC 0x0958d thru 0x0ed5c (0x057cf bytes)INTERIOR 227: 1731346455.321088 SKIPPED (no operation) 228: 1731346455.321278 WRITE 0x3ce4a thru 0x3d20e (0x003c5 bytes) HOLE 229: 1731346455.321725 PUNCH 0x341cb thru 0x3d20e (0x09044 bytes) 230: 1731346455.324162 PUNCH 0x2d37b thru 0x32894 (0x0551a bytes) 231: 1731346455.325489 MAPWRITE 0x06ede thru 0x0b9e3 (0x04b06 bytes) 232: 1731346455.331672 MAPREAD 0x01b0b thru 0x10bbd (0x0f0b3 bytes) 233: 1731346455.332423 MAPWRITE 0x1a47a thru 0x28da6 (0x0e92d bytes) 234: 1731346455.340636 PUNCH 0x2c828 thru 0x31aef (0x052c8 bytes) 235: 1731346455.341976 MAPWRITE 0x00b63 thru 0x0d5fb (0x0ca99 bytes) 236: 1731346455.349696 FALLOC 0x10404 thru 0x1c04c (0x0bc48 bytes)INTERIOR 237: 1731346455.351281 SKIPPED (no operation) 238: 1731346455.351647 MAPWRITE 0x1ac68 thru 0x211d4 (0x0656d bytes) 239: 1731346455.356466 PUNCH 0x17a84 thru 0x19976 (0x01ef3 bytes) 240: 1731346455.357737 FALLOC 0x38235 thru 0x40000 (0x07dcb bytes)EXTENDING 241: 1731346455.360006 TRUNC_DN 0x40000 to 0x2352a 242: 1731346455.363793 MAPWRITE 0x26f61 thru 0x2c031 (0x050d1 bytes) 243: 1731346455.374668 READ 0x14542 thru 0x1c1db (0x07c9a bytes) 244: 1731346455.377287 MIRROR_EXTEND to 2 mirrors 245: 1731346455.478959 PUNCH 0x04108 thru 0x0c2a9 (0x081a2 bytes) 246: 1731346455.487962 MIRROR_RESYNC 2 mirrors 247: 1731346455.602003 PUNCH 0x128a8 thru 0x198df (0x07038 bytes) 248: 1731346455.609747 READ 0x17112 thru 0x20a76 (0x09965 bytes) 249: 1731346455.612109 MAPWRITE 0x0869a thru 0x0bc19 (0x03580 bytes) 250: 1731346455.621005 FALLOC 0x02171 thru 0x06945 (0x047d4 bytes)INTERIOR 251: 1731346455.622712 WRITE 0x16b0e thru 0x1ec4b (0x0813e bytes) 252: 1731346455.624819 FALLOC 0x2af9a thru 0x35107 (0x0a16d bytes)PAST_EOF 253: 1731346455.626225 TRUNC_DN 0x2c032 to 0x0c783 254: 1731346455.634699 SKIPPED (no operation) 255: 1731346455.635096 PUNCH 0x0027e thru 0x01704 (0x01487 bytes) 256: 1731346455.636506 SKIPPED (no operation) 257: 1731346455.636792 PUNCH 0x026f0 thru 0x0c782 (0x0a093 bytes) 258: 1731346455.638176 SKIPPED (no operation) 259: 1731346455.639608 MIRROR_SPLIT mirror 1 to 1 mirrors 260: 1731346455.665279 FALLOC 0x31b9b thru 0x35010 (0x03475 bytes)EXTENDING 261: 1731346455.666939 READ 0x14d05 thru 0x18486 (0x03782 bytes) 262: 1731346455.670209 MIRROR_EXTEND to 2 mirrors 263: 1731346455.766791 READ 0x2c614 thru 0x347dc (0x081c9 bytes) 264: 1731346455.770640 MAPWRITE 0x28be4 thru 0x35044 (0x0c461 bytes) 265: 1731346455.795854 WRITE 0x25bca thru 0x33763 (0x0db9a bytes) 266: 1731346455.796996 TRUNC_DN 0x35045 to 0x10e0b 267: 1731346455.802406 MAPREAD 0x0581f thru 0x10e0a (0x0b5ec bytes) 268: 1731346455.806417 WRITE 0x3cede thru 0x3ffff (0x03122 bytes) HOLE 269: 1731346455.807089 FALLOC 0x339b4 thru 0x40000 (0x0c64c bytes)INTERIOR 270: 1731346455.808603 WRITE 0x0ad92 thru 0x1743a (0x0c6a9 bytes) 271: 1731346455.811616 MIRROR_SPLIT mirror 1 to 1 mirrors 272: 1731346455.839991 TRUNC_DN 0x40000 to 0x2f788 273: 1731346455.844679 FALLOC 0x16b2a thru 0x1ee4c (0x08322 bytes)INTERIOR 274: 1731346455.846244 PUNCH 0x0d579 thru 0x130ff (0x05b87 bytes) 275: 1731346455.847548 PUNCH 0x20e1d thru 0x2e5a8 (0x0d78c bytes) 276: 1731346455.848770 MAPWRITE 0x0a74d thru 0x0d747 (0x02ffb bytes) 277: 1731346455.857392 PUNCH 0x02ccb thru 0x102e4 (0x0d61a bytes) 278: 1731346455.858736 SKIPPED (no operation) 279: 1731346455.858944 FALLOC 0x24d28 thru 0x2c34c (0x07624 bytes)INTERIOR 280: 1731346455.860537 PUNCH 0x1c03e thru 0x1f0f9 (0x030bc bytes) 281: 1731346455.861955 READ 0x08d93 thru 0x0c7b9 (0x03a27 bytes) 282: 1731346455.863913 PUNCH 0x1ccf0 thru 0x27798 (0x0aaa9 bytes) 283: 1731346455.865212 FALLOC 0x3e501 thru 0x40000 (0x01aff bytes)PAST_EOF 284: 1731346455.866600 MAPREAD 0x0d2cf thru 0x0e066 (0x00d98 bytes) 285: 1731346455.868455 READ 0x1aa70 thru 0x1e57e (0x03b0f bytes) 286: 1731346455.870723 WRITE 0x00abc thru 0x0913b (0x08680 bytes) 287: 1731346455.871305 SKIPPED (no operation) 288: 1731346455.871491 TRUNC_DN 0x2f788 to 0x0bf2a 289: 1731346455.875313 PUNCH 0x07fe5 thru 0x0bf29 (0x03f45 bytes) 290: 1731346455.878613 WRITE 0x1237c thru 0x21e95 (0x0fb1a bytes) HOLE 291: 1731346455.879394 MAPREAD 0x129e9 thru 0x1e88e (0x0bea6 bytes) 292: 1731346455.879808 MAPWRITE 0x0f113 thru 0x15332 (0x06220 bytes) 293: 1731346455.889131 WRITE 0x34883 thru 0x3be55 (0x075d3 bytes) HOLE 294: 1731346455.889762 TRUNC_DN 0x3be56 to 0x0c6b9 295: 1731346455.893966 TRUNC_UP 0x0c6b9 to 0x3ca11 296: 1731346455.897569 MAPREAD 0x0b849 thru 0x10798 (0x04f50 bytes) 297: 1731346455.905193 MAPWRITE 0x198f1 thru 0x26108 (0x0c818 bytes) 298: 1731346455.925452 WRITE 0x3f43b thru 0x3ffff (0x00bc5 bytes) HOLE 299: 1731346455.926146 MAPWRITE 0x23985 thru 0x246e1 (0x00d5d bytes) 300: 1731346455.930143 SKIPPED (no operation) 301: 1731346455.930438 WRITE 0x05270 thru 0x1380b (0x0e59c bytes) 302: 1731346455.933124 WRITE 0x370bf thru 0x3dc8e (0x06bd0 bytes) 303: 1731346455.933976 MAPWRITE 0x079fb thru 0x1189c (0x09ea2 bytes) 304: 1731346455.940131 SKIPPED (no operation) 305: 1731346455.941755 MIRROR_EXTEND to 2 mirrors 306: 1731346456.057444 READ 0x288b5 thru 0x2ab2d (0x02279 bytes) 307: 1731346456.062147 READ 0x29a6f thru 0x31eb1 (0x08443 bytes) 308: 1731346456.066282 READ 0x361cf thru 0x3ffff (0x09e31 bytes) 309: 1731346456.069784 SKIPPED (no operation) 310: 1731346456.071374 MAPREAD 0x27888 thru 0x3346e (0x0bbe7 bytes) 311: 1731346456.074982 FALLOC 0x0fba4 thru 0x120c5 (0x02521 bytes)INTERIOR 312: 1731346456.082544 FALLOC 0x18af6 thru 0x22bc7 (0x0a0d1 bytes)INTERIOR 313: 1731346456.084040 TRUNC_DN 0x40000 to 0x315d9 314: 1731346456.089901 READ 0x0b676 thru 0x0e4a8 (0x02e33 bytes) 315: 1731346456.092643 WRITE 0x24e6d thru 0x2cd0c (0x07ea0 bytes) 316: 1731346456.093338 FALLOC 0x14cc0 thru 0x18ecb (0x0420b bytes)INTERIOR 317: 1731346456.094945 MAPREAD 0x2005f thru 0x2bfb7 (0x0bf59 bytes) 318: 1731346456.095366 MAPWRITE 0x22be5 thru 0x28b9f (0x05fbb bytes) 319: 1731346456.100735 READ 0x23a3e thru 0x24720 (0x00ce3 bytes) 320: 1731346456.101067 TRUNC_DN 0x315d9 to 0x10f89 321: 1731346456.106539 MIRROR_RESYNC 2 mirrors 322: 1731346456.171896 FALLOC 0x0bdad thru 0x18f0b (0x0d15e bytes)PAST_EOF 323: 1731346456.179499 PUNCH 0x0a215 thru 0x10f88 (0x06d74 bytes) 324: 1731346456.180891 PUNCH 0x004d5 thru 0x0958b (0x090b7 bytes) 325: 1731346456.182350 PUNCH 0x01484 thru 0x054f2 (0x0406f bytes) 326: 1731346456.183742 MAPREAD 0x0229c thru 0x0c4b7 (0x0a21c bytes) 327: 1731346456.189317 PUNCH 0x10b1a thru 0x10f88 (0x0046f bytes) 328: 1731346456.190745 TRUNC_UP 0x10f89 to 0x1296d 329: 1731346456.198569 MAPWRITE 0x10c87 thru 0x17940 (0x06cba bytes) 330: 1731346456.209631 MAPWRITE 0x30aa2 thru 0x3cc86 (0x0c1e5 bytes) 331: 1731346456.222986 TRUNC_DN 0x3cc87 to 0x3573c 332: 1731346456.226779 TRUNC_DN 0x3573c to 0x0267b 333: 1731346456.230306 PUNCH 0x0250b thru 0x0267a (0x00170 bytes) 334: 1731346456.231647 WRITE 0x00d3c thru 0x0a826 (0x09aeb bytes) EXTEND 335: 1731346456.234111 PUNCH 0x05346 thru 0x0a826 (0x054e1 bytes) 336: 1731346456.237724 WRITE 0x03a70 thru 0x0ac0e (0x0719f bytes) EXTEND 337: 1731346456.239342 READ 0x017fd thru 0x08b59 (0x0735d bytes) 338: 1731346456.240921 MIRROR_SPLIT mirror 1 to 1 mirrors 339: 1731346456.270827 SKIPPED (no operation) 340: 1731346456.271228 MAPWRITE 0x3f2b2 thru 0x3ffff (0x00d4e bytes) 341: 1731346456.281564 FALLOC 0x08603 thru 0x17c99 (0x0f696 bytes)INTERIOR 342: 1731346456.283235 PUNCH 0x1e773 thru 0x2da0f (0x0f29d bytes) 343: 1731346456.284473 MAPREAD 0x2dfea thru 0x388cd (0x0a8e4 bytes) 344: 1731346456.290087 PUNCH 0x3c68b thru 0x3ee89 (0x027ff bytes) 345: 1731346456.291366 WRITE 0x31015 thru 0x3f4f8 (0x0e4e4 bytes) 346: 1731346456.292189 MAPREAD 0x2e2eb thru 0x389dd (0x0a6f3 bytes) 347: 1731346456.292698 FALLOC 0x26a02 thru 0x2b665 (0x04c63 bytes)INTERIOR 348: 1731346456.294035 TRUNC_DN 0x40000 to 0x258f5 349: 1731346456.297820 SKIPPED (no operation) 350: 1731346456.299329 MIRROR_EXTEND to 2 mirrors 351: 1731346456.391271 MAPREAD 0x0f3c9 thru 0x160fa (0x06d32 bytes) 352: 1731346456.398617 TRUNC_UP 0x258f5 to 0x3d991 353: 1731346456.411707 MAPWRITE 0x2639d thru 0x29a90 (0x036f4 bytes) 354: 1731346456.421091 MAPWRITE 0x0a660 thru 0x18abe (0x0e45f bytes) 355: 1731346456.430485 PUNCH 0x2cb3f thru 0x2e4a2 (0x01964 bytes) 356: 1731346456.431610 MAPWRITE 0x3bfe2 thru 0x3f4d1 (0x034f0 bytes) 357: 1731346456.442566 WRITE 0x0c606 thru 0x10675 (0x04070 bytes) 358: 1731346456.443100 SKIPPED (no operation) 359: 1731346456.443300 WRITE 0x227d1 thru 0x2ae35 (0x08665 bytes) 360: 1731346456.443896 TRUNC_DN 0x3f4d2 to 0x37b96 361: 1731346456.447246 MAPWRITE 0x3737d thru 0x395f6 (0x0227a bytes) 362: 1731346456.457247 READ 0x2d3eb thru 0x36735 (0x0934b bytes) 363: 1731346456.459364 FALLOC 0x3583d thru 0x40000 (0x0a7c3 bytes)PAST_EOF 364: 1731346456.461188 READ 0x268a5 thru 0x33e13 (0x0d56f bytes) 365: 1731346456.463442 FALLOC 0x18f50 thru 0x2293c (0x099ec bytes)INTERIOR 366: 1731346456.466165 MIRROR_RESYNC 2 mirrors 367: 1731346456.557730 TRUNC_DN 0x395f7 to 0x0b52d 368: 1731346456.569627 MAPREAD 0x0729f thru 0x0b52c (0x0428e bytes) 369: 1731346456.574317 MAPREAD 0x04b51 thru 0x0b52c (0x069dc bytes) 370: 1731346456.576274 MIRROR_RESYNC 2 mirrors 371: 1731346456.641965 MAPREAD 0x0340f thru 0x0a7b9 (0x073ab bytes) 372: 1731346456.646544 SKIPPED (no operation) 373: 1731346456.646781 PUNCH 0x06fba thru 0x0b52c (0x04573 bytes) 374: 1731346456.654798 PUNCH 0x02225 thru 0x0b52c (0x09308 bytes) 375: 1731346456.656460 MAPREAD 0x05a1d thru 0x05c18 (0x001fc bytes) 376: 1731346456.659258 MAPWRITE 0x389a1 thru 0x3ffff (0x0765f bytes) 377: 1731346456.675268 MAPREAD 0x2623c thru 0x35941 (0x0f706 bytes) 378: 1731346456.679232 MAPREAD 0x0367a thru 0x0f231 (0x0bbb8 bytes) 379: 1731346456.681390 READ 0x37d7b thru 0x3ffff (0x08285 bytes) 380: 1731346456.681741 SKIPPED (no operation) 381: 1731346456.681990 PUNCH 0x339b2 thru 0x3ffff (0x0c64e bytes) 382: 1731346456.683558 TRUNC_DN 0x40000 to 0x16a03 383: 1731346456.687241 MAPREAD 0x0b6fe thru 0x128dd (0x071e0 bytes) 384: 1731346456.688757 MIRROR_EXTEND to 3 mirrors 385: 1731346456.779454 READ 0x0d6e4 thru 0x1508f (0x079ac bytes) 386: 1731346456.782849 WRITE 0x0fdb0 thru 0x14718 (0x04969 bytes) 387: 1731346456.793522 MIRROR_RESYNC 3 mirrors 388: 1731346456.911027 READ 0x16741 thru 0x16a02 (0x002c2 bytes) 389: 1731346456.911287 WRITE 0x11c5e thru 0x1ba2d (0x09dd0 bytes) EXTEND 390: 1731346456.919317 READ 0x04b3d thru 0x12a66 (0x0df2a bytes) 391: 1731346456.921955 SKIPPED (no operation) 392: 1731346456.922183 READ 0x0b96d thru 0x15339 (0x099cd bytes) 393: 1731346456.922482 PUNCH 0x00ade thru 0x0ed5b (0x0e27e bytes) 394: 1731346456.924052 PUNCH 0x07a44 thru 0x14914 (0x0ced1 bytes) 395: 1731346456.926996 FALLOC 0x23066 thru 0x231b2 (0x0014c bytes)PAST_EOF 396: 1731346456.928252 PUNCH 0x1123e thru 0x12c9a (0x01a5d bytes) 397: 1731346456.929709 MAPWRITE 0x03b5f thru 0x0e127 (0x0a5c9 bytes) 398: 1731346456.937635 WRITE 0x31b1d thru 0x3d4a3 (0x0b987 bytes) HOLE 399: 1731346456.938515 PUNCH 0x1a560 thru 0x25d80 (0x0b821 bytes) 400: 1731346456.939711 PUNCH 0x0b530 thru 0x12cb8 (0x07789 bytes) 401: 1731346456.940984 TRUNC_UP 0x3d4a4 to 0x3e42c 402: 1731346456.948722 PUNCH 0x13889 thru 0x2038b (0x0cb03 bytes) 403: 1731346456.950125 READ 0x21c67 thru 0x2d497 (0x0b831 bytes) 404: 1731346456.954026 MIRROR_EXTEND to 4 mirrors 405: 1731346457.065067 READ 0x19a3b thru 0x2465a (0x0ac20 bytes) 406: 1731346457.068790 SKIPPED (no operation) 407: 1731346457.070388 MAPWRITE 0x37a39 thru 0x3ccb7 (0x0527f bytes) 408: 1731346457.101611 FALLOC 0x09820 thru 0x12766 (0x08f46 bytes)INTERIOR 409: 1731346457.103419 TRUNC_DN 0x3e42c to 0x0015b 410: 1731346457.109695 READ 0x00094 thru 0x0015a (0x000c7 bytes) 411: 1731346457.111953 WRITE 0x191b8 thru 0x25285 (0x0c0ce bytes) HOLE 412: 1731346457.113344 MAPREAD 0x0626e thru 0x10d89 (0x0ab1c bytes) 413: 1731346457.116125 SKIPPED (no operation) 414: 1731346457.116927 TRUNC_UP 0x25286 to 0x3d21e 415: 1731346457.120868 MAPREAD 0x24c31 thru 0x31d27 (0x0d0f7 bytes) 416: 1731346457.123828 FALLOC 0x38e03 thru 0x3eac5 (0x05cc2 bytes)EXTENDING 417: 1731346457.125644 SKIPPED (no operation) 418: 1731346457.126031 READ 0x33876 thru 0x3890e (0x05099 bytes) 419: 1731346457.126522 MAPREAD 0x1c8e0 thru 0x1e0b4 (0x017d5 bytes) 420: 1731346457.127169 WRITE 0x2772b thru 0x2abf3 (0x034c9 bytes) 421: 1731346457.127928 READ 0x2163b thru 0x29dd3 (0x08799 bytes) 422: 1731346457.128362 MAPWRITE 0x0a751 thru 0x0edfe (0x046ae bytes) 423: 1731346457.134359 WRITE 0x29d18 thru 0x35cd7 (0x0bfc0 bytes) 424: 1731346457.136955 MIRROR_SPLIT mirror 1 to 3 mirrors 425: 1731346457.179469 READ 0x25664 thru 0x2c2ab (0x06c48 bytes) 426: 1731346457.181726 FALLOC 0x3617b thru 0x3dd48 (0x07bcd bytes)INTERIOR 427: 1731346457.183347 MAPREAD 0x3610e thru 0x3eac4 (0x089b7 bytes) 428: 1731346457.188560 MIRROR_EXTEND to 4 mirrors 429: 1731346457.302996 FALLOC 0x0efac thru 0x18783 (0x097d7 bytes)INTERIOR 430: 1731346457.311638 PUNCH 0x0bd2d thru 0x0f010 (0x032e4 bytes) 431: 1731346457.312960 SKIPPED (no operation) 432: 1731346457.313322 MAPWRITE 0x1e202 thru 0x258b9 (0x076b8 bytes) 433: 1731346457.323966 MAPWRITE 0x32d96 thru 0x348fa (0x01b65 bytes) 434: 1731346457.329936 WRITE 0x2dcad thru 0x31870 (0x03bc4 bytes) 435: 1731346457.331778 WRITE 0x058e2 thru 0x07761 (0x01e80 bytes) 436: 1731346457.334709 TRUNC_DN 0x3eac5 to 0x26d51 437: 1731346457.342761 MAPREAD 0x13b42 thru 0x15f8d (0x0244c bytes) 438: 1731346457.345077 MAPWRITE 0x19d32 thru 0x1b273 (0x01542 bytes) 439: 1731346457.350779 MAPWRITE 0x0d12b thru 0x0d1da (0x000b0 bytes) 440: 1731346457.353808 SKIPPED (no operation) 441: 1731346457.354208 WRITE 0x1bfe5 thru 0x298d5 (0x0d8f1 bytes) EXTEND 442: 1731346457.360451 MIRROR_RESYNC 4 mirrors 443: 1731346457.556402 FALLOC 0x1874e thru 0x20274 (0x07b26 bytes)INTERIOR 444: 1731346457.564268 MAPREAD 0x14482 thru 0x1af65 (0x06ae4 bytes) 445: 1731346457.568889 READ 0x06fb1 thru 0x07dce (0x00e1e bytes) 446: 1731346457.570512 MAPWRITE 0x10358 thru 0x12330 (0x01fd9 bytes) Using seed 6453 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7208:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3367:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7558:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7621:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7439:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3369:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1731346458.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg127-server.1731346458.log.NhM2OE" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg127-server.1731346458.log.WQsXG5" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 70a (7s) == sanity-flr test 200a: stress test ===================== 12:34:19 (1731346459) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4021312 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7472416 .. Extending file size to 4213728 .. done Extending file size to 4579328 .. Extending file size to 6237024 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6656992 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7645056 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8734400 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7917760 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8629696 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8976224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9000384 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8867136 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 8643 8644 8646 8647 8650 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:35:23 (1731346523) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6914624 .. Extending file size to 4581632 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7928768 .. Extending file size to 8145792 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4763968 .. Extending file size to 5677408 .. failed Extending file size to 6170624 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7198720 .. Extending file size to 8396576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8426336 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8599616 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9176640 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8766752 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8779808 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8847296 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8906816 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9264800 .. failed Extending file size to 9193952 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 11460 11461 11463 11464 11467 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:36:27 (1731346587) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3780 1283908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 15964 3591056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21840 3585180 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37804 7176236 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3780 1283908 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1964 1285724 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 15964 3591056 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 21840 3585180 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37804 7176236 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0355386 s, 59.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0410491 s, 51.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:36:31 (1731346591) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.026727 s, 78.5 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 12:36:33 (1731346593) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc5b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1101:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1101:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1102:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1102:0x0] } PASS 203 (1s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:36:34 (1731346594) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0254044 s, 82.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1103:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1103:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1104:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:36:36 (1731346596) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0475909 s, 44.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1104:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1105:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1105:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1104:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1105:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1105:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1106:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:36:38 (1731346598) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0388267 s, 54.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1106:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1107:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1106:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1107:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1108:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:37:01 (1731346621) Waiting for MDT destroys to complete oleg127-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg127-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141258 s, 74.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1107:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1109:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1107:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1109:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1108:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:37:34 (1731346654) Waiting for MDT destroys to complete oleg127-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg127-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1109:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x110a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0189975 s, 110 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1109:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x110a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x1109:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x110a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (34s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:38:08 (1731346688) Waiting for MDT destroys to complete oleg127-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg127-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool oleg127-server: Pool lustre.test_204f created Adding targets to pool oleg127-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg127-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.021319 s, 98.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x110b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x110b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg127-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg127-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg127-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:38:57 (1731346737) PASS 205a (1s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:38:58 (1731346738) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:39:00 (1731346740) oleg127-server: Pool lustre.test_206 created Adding targets to pool oleg127-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg127-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg127-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg127-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg127-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:39:14 (1731346754) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187647 s, 55.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219436 s, 47.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x37d56f9b Mirror 3: 0x37d56f9b PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:39:16 (1731346756) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:39:17 (1731346757) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:39:18 (1731346758) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00674964 s, 0.6 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1560000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1560000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1560000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d90000 Read this (4 bytes): BBBB 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:39:31 (1731346771) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0025583 s, 3.2 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:39:32 (1731346772) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124071 s, 84.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc80 lmm_fid: [0x200000405:0xc80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc80 lmm_fid: [0x200000405:0xc80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x1116:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339614336 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 12:39:33.000000000 -0500 Modify: 2024-11-11 12:39:33.000000000 -0500 Change: 2024-11-11 12:39:33.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:39:34 (1731346774) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0082775 s, 127 MB/s oleg127-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg127-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg127-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg127-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 oleg127-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg127-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg127-server: O/280000401/d22/1116: File not found by ext2_lookup PASS 210b (12s) == sanity-flr test complete, duration 692 sec ============ 12:39:46 (1731346786) sanity-flr: FAIL: test_70a fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed === sanity-flr: start cleanup 12:39:46 (1731346786) === === sanity-flr: finish cleanup 12:39:48 (1731346788) ===