-----============= acceptance-small: sanity-flr ============----- Tue Dec 24 22:55:04 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg134-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:55:12 (1735098912) === oleg134-client.virtnet: executing check_config_client /mnt/lustre oleg134-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg134-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a483800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a483800.idle_timeout=debug disable quota as required oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:55:26 (1735098926) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7599] == sanity-flr test 0a: lfs mirror create with -N option == 22:55:27 (1735098927) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:55:34 (1735098934) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:55:35 (1735098935) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:55:36 (1735098936) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:55:45 (1735098945) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:55:47 (1735098947) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 22:55:48 (1735098948) error: lfs mirror create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror create: unsupported mirror flags: stale lfs mirror create: unsupported mirror flags: prefer,init lfs mirror create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a483800.stats=clear osc.lustre-OST0001-osc-ffff88012a483800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:55:51 (1735098951) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:55:55 (1735098955) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:55:58 (1735098958) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:55:59 (1735098959) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs mirror extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 22:56:03 (1735098963) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:56:06 (1735098966) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:56:09 (1735098969) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td lfs mirror extend: cannot get the last init comp error: lfs mirror extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:56:12 (1735098972) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:56:16 (1735098976) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:56:19 (1735098979) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.16636 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0578775 s, 18.1 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:56:22 (1735098982) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.0966405 s, 97.7 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:56:25 (1735098985) Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a483800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a483800.ost_server_uuid in DISCONN state after 2 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a483800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a483800.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:56:54 (1735099014) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.104958 s, 50.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a483800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a483800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:57:10 (1735099030) error: lfs mirror extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a483800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a483800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a483800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a483800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:57:36 (1735099056) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.13599 s, 46.2 MB/s -rw-r--r-- 1 root root 52428800 Dec 24 22:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.367s user 0m0.003s sys 0m0.121s Stopping /mnt/lustre-ost1 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a483800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a483800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.334s user 0m0.000s sys 0m0.120s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg134-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg134-server oleg134-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a483800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a483800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.322s user 0m0.000s sys 0m0.128s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg134-server: oleg134-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg134-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg134-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a483800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a483800.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:58:02 (1735099082) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:58:03 (1735099083) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:58:04 (1735099084) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 22:58:05 (1735099085) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 22:58:07 (1735099087) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.182609 s, 68.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:58:11 (1735099091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116656 s, 89.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169071 s, 62.0 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:58:13 (1735099093) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:58:16 (1735099096) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000674748 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000610232 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000726348 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00084362 s, 1.2 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 22:58:19 (1735099099) create test files with size 15752849 .. c8c0295ecb89df3fada0247481103976 - 0858a7075b829483855e15f03da82ccf - 2de4b67ed66abc941919c87498c09dc9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 22:58:24 (1735099104) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:58:27 (1735099107) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:58:29 (1735099109) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0283288 s, 37.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118977 s, 88.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 22:58:32 (1735099112) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0264237 s, 79.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.111276 s, 37.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0471154 s, 44.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 22:58:35 (1735099115) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 22:58:36 (1735099116) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:58:37 (1735099117) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111165 s, 94.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 22:58:39 (1735099119) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.094145 s, 33.4 MB/s lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x10:0x0]: Invalid argument (22) PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:58:42 (1735099122) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 1 id 65537, should fail split mirror_id 2 id 131073 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:58:44 (1735099124) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0787255 s, 133 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:59:04 (1735099144) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a21c3000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0792368 s, 132 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg134-client.virtnet /mnt/lustre (opts:) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a4252000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:59:08 (1735099148) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (8s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:59:16 (1735099156) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (7s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:59:23 (1735099163) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 22:59:24 (1735099164) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.072434 s, 43.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0515825 s, 61.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: eef88f88d5a69f1102806fea79b706c9 - mirror 1 checksum: 98460daac316d6835f47060e2172b249 - mirror 2 checksum: eef88f88d5a69f1102806fea79b706c9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 24 22:59:24 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Dec 24 22:59:24 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: eef88f88d5a69f1102806fea79b706c9 - mirror 1 checksum: 98460daac316d6835f47060e2172b249 - mirror 2 checksum: 98460daac316d6835f47060e2172b249 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 24 22:59:24 2024' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Dec 24 22:59:24 2024' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:59:26 (1735099166) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108783 s, 96.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:59:28 (1735099168) Failing mds1 on oleg134-server Stopping /mnt/lustre-mds1 (opts:) on oleg134-server 22:59:35 (1735099175) shut down facet: mds1 facet_host: oleg134-server facet_failover_host: oleg134-server Failover mds1 to oleg134-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 22:59:48 (1735099188) targets are mounted 22:59:48 (1735099188) facet_failover done oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00982835 s, 107 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:59:55 (1735099195) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00663219 s, 3.1 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00441705 s, 4.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:59:58 (1735099198) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0120074 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0033566 s, 6.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:00:00 (1735099200) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00218583 s, 7.5 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:00:02 (1735099202) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0126591 s, 1.6 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:00:03 (1735099203) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00294195 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0110238 s, 372 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:00:05 (1735099205) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:00:10 (1735099210) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:00:28 (1735099228) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:00:46 (1735099246) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:01:03 (1735099263) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 23:01:05 (1735099265) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 26267 mirror split pid 26268 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26267 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4860: 26268 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 23:02:06 (1735099326) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 9327 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1735099327.101192 write 0x029f01 thru 0x02d07c (0x317c bytes) 000004 1735099327.105305 mapwrite 0x02f66f thru 0x0325b1 (0x2f43 bytes) fallocating to largest ever: 0xcf09 000006 1735099327.122908 fallocate 0x0066fd thru 0x00cf08 (0x680c bytes) 000008 1735099327.124821 mapread 0x018b0b thru 0x018f62 (0x458 bytes) 000010 1735099327.127887 read 0x0316eb thru 0x0325b1 (0xec7 bytes) 000011 1735099327.128298 punch from 0x00f3a6 to 0x0187eb (0x9446 bytes) 000013 1735099327.130318 punch from 0x00de44 to 0x00f978 (0x1b35 bytes) fallocating to largest ever: 0x17e0b 000014 1735099327.131732 fallocate 0x00a3c5 thru 0x017e0a (0xda46 bytes) 000015 1735099327.133272 write 0x00721e thru 0x008683 (0x1466 bytes) 000016 1735099327.135861 mapwrite 0x007b1d thru 0x009f23 (0x2407 bytes) 000017 1735099327.139923 punch from 0x009da4 to 0x009dd8 (0x35 bytes) 000019 1735099327.141493 punch from 0x00ec61 to 0x014e27 (0x61c7 bytes) 000020 1735099327.142870 mapread 0x007fd0 thru 0x00cc43 (0x4c74 bytes) 000021 1735099327.145511 mapread 0x003e7d thru 0x00aa44 (0x6bc8 bytes) 000022 1735099327.146162 read 0x00d8d2 thru 0x01530b (0x7a3a bytes) 000023 1735099327.149453 punch from 0x030b01 to 0x0325b1 (0x1ab1 bytes) 000026 1735099327.152146 punch from 0x0070da to 0x0079be (0x8e5 bytes) 000027 1735099327.154509 fallocate 0x004659 thru 0x00d0d3 (0x8a7b bytes) truncating to largest ever: 0x2a499 000028 1735099327.155995 trunc from 0x0325b2 to 0x02a498 (0x8119 bytes) 000029 1735099327.160007 read 0x015f13 thru 0x01bdfc (0x5eea bytes) 000031 1735099327.160756 write 0x02d6d6 thru 0x03a581 (0xceac bytes) 000032 1735099327.161853 mapread 0x036a23 thru 0x03a581 (0x3b5f bytes) 000033 1735099327.162298 punch from 0x00268c to 0x005b35 (0x34aa bytes) 000034 1735099327.164410 trunc from 0x03a582 to 0x0066e9 (0x33e98 bytes) fallocating to largest ever: 0x3a783 000035 1735099327.169568 fallocate 0x035859 thru 0x03a782 (0x4f2a bytes) 000037 1735099327.171418 read 0x000875 thru 0x0051a4 (0x4930 bytes) 000039 1735099327.176250 write 0x00b701 thru 0x010ed1 (0x57d1 bytes) truncating to largest ever: 0x3d826 000041 1735099327.177648 trunc from 0x03a783 to 0x03d825 (0x30a3 bytes) 000042 1735099327.182285 read 0x0272b3 thru 0x029d12 (0x2a60 bytes) 000044 1735099327.183181 mapwrite 0x006885 thru 0x0163f7 (0xfb73 bytes) 000045 1735099327.191915 read 0x018e53 thru 0x01c25f (0x340d bytes) 000047 1735099327.192605 fallocate 0x011cb1 thru 0x01b887 (0x9bd7 bytes) 000049 1735099327.195322 mapread 0x032d7f thru 0x035081 (0x2303 bytes) 000051 1735099327.196199 mapread 0x0194a0 thru 0x020abf (0x7620 bytes) 000052 1735099327.196688 read 0x01c6a2 thru 0x025ab3 (0x9412 bytes) 000053 1735099327.197005 punch from 0x01db01 to 0x0230dd (0x55dd bytes) 000054 1735099327.198879 write 0x01ea44 thru 0x027f9f (0x955c bytes) 000056 1735099327.202524 write 0x005c2b thru 0x00a3d4 (0x47aa bytes) 000062 1735099327.205098 mapwrite 0x00a47c thru 0x0161b3 (0xbd38 bytes) 000063 1735099327.211975 mapwrite 0x0002e8 thru 0x00bec6 (0xbbdf bytes) 000064 1735099327.219788 punch from 0x01c731 to 0x02c108 (0xf9d8 bytes) 000065 1735099327.221970 trunc from 0x03d826 to 0x00a14d (0x336d8 bytes) 000066 1735099327.226557 trunc from 0x00a14e to 0x015375 (0xb228 bytes) 000067 1735099327.230466 write 0x023171 thru 0x0291c6 (0x6056 bytes) 000068 1735099327.231174 write 0x0151a4 thru 0x01d209 (0x8066 bytes) 000070 1735099327.234313 fallocate 0x02dcad thru 0x03bec6 (0xe21a bytes) 000071 1735099327.235825 trunc from 0x03bec7 to 0x03c1ad (0x2e7 bytes) 000072 1735099327.239433 write 0x014c7f thru 0x01bb9a (0x6f1c bytes) 000073 1735099327.241274 trunc from 0x03c1ae to 0x026ef8 (0x152b5 bytes) 000074 1735099327.245317 write 0x008565 thru 0x016f74 (0xea10 bytes) 000076 1735099327.246871 punch from 0x021b0c to 0x022951 (0xe46 bytes) 000077 1735099327.248576 read 0x00f058 thru 0x016e07 (0x7db0 bytes) 000078 1735099327.249041 read 0x0132f3 thru 0x01422a (0xf38 bytes) 000079 1735099327.249438 write 0x005939 thru 0x010e60 (0xb528 bytes) 000080 1735099327.250335 mapwrite 0x01eb92 thru 0x0242f0 (0x575f bytes) 000081 1735099327.259999 mapwrite 0x0296b0 thru 0x02a1b0 (0xb01 bytes) 000082 1735099327.270475 punch from 0x01d356 to 0x02a1b0 (0xce5b bytes) 000083 1735099327.272006 trunc from 0x02a1b1 to 0x03d482 (0x132d2 bytes) 000084 1735099327.275887 trunc from 0x03d483 to 0x039e21 (0x3661 bytes) 000085 1735099327.279558 fallocate 0x02b28c thru 0x0389e6 (0xd75b bytes) 000086 1735099327.281082 mapwrite 0x037d2b thru 0x03ffff (0x82d5 bytes) 000087 1735099327.294144 mapwrite 0x021fab thru 0x02f548 (0xd59e bytes) 000088 1735099327.311299 fallocate 0x01ba1a thru 0x01d5e7 (0x1bce bytes) 000089 1735099327.312725 mapwrite 0x02fa79 thru 0x03cb52 (0xd0da bytes) fallocating to largest ever: 0x40000 000090 1735099327.319826 fallocate 0x035269 thru 0x03ffff (0xad97 bytes) 000091 1735099327.322292 punch from 0x0375a6 to 0x03ffff (0x8a5a bytes) 000092 1735099327.324158 punch from 0x0389ca to 0x03ffff (0x7636 bytes) 000097 1735099327.326733 fallocate 0x0294fb thru 0x0348bf (0xb3c5 bytes) 000098 1735099327.328429 mapwrite 0x006ea8 thru 0x00c2a2 (0x53fb bytes) 000099 1735099327.334765 trunc from 0x040000 to 0x011c01 (0x2e3fe bytes) 000100 1735099327.339730 mapread 0x0056bf thru 0x0060bc (0x9fe bytes) 000104 1735099327.342724 MIRROR_EXTEND to 2 mirrors 000105 1735099327.429728 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000106 1735099327.462398 write 0x002026 thru 0x009235 (0x7210 bytes) 000107 1735099327.469495 punch from 0x00aad7 to 0x011c01 (0x712b bytes) 000109 1735099327.471529 write 0x03631f thru 0x03ffff (0x9ce1 bytes) 000110 1735099327.472223 punch from 0x015c07 to 0x023c8b (0xe085 bytes) 000111 1735099327.473762 fallocate 0x010e0f thru 0x01b25d (0xa44f bytes) 000112 1735099327.475624 mapwrite 0x01eab1 thru 0x0209de (0x1f2e bytes) 000113 1735099327.485417 punch from 0x012820 to 0x01a11c (0x78fd bytes) 000114 1735099327.487442 fallocate 0x00abda thru 0x00c27b (0x16a2 bytes) 000116 1735099327.489830 write 0x030bcb thru 0x03447e (0x38b4 bytes) 000117 1735099327.493057 trunc from 0x040000 to 0x01e352 (0x21cad bytes) 000118 1735099327.503960 write 0x03cba6 thru 0x03ffff (0x345a bytes) 000119 1735099327.504654 write 0x016f38 thru 0x01d96e (0x6a37 bytes) 000120 1735099327.507629 write 0x013a74 thru 0x022bac (0xf139 bytes) 000121 1735099327.510029 trunc from 0x040000 to 0x0334a0 (0xcb5f bytes) 000122 1735099327.514209 read 0x0317d7 thru 0x0334a0 (0x1cca bytes) 000123 1735099327.516311 mapwrite 0x003357 thru 0x0112e6 (0xdf90 bytes) 000124 1735099327.528401 trunc from 0x0334a1 to 0x0351a8 (0x1d08 bytes) 000125 1735099327.533094 trunc from 0x0351a9 to 0x0348af (0x8f9 bytes) 000126 1735099327.537614 read 0x02f68c thru 0x02f970 (0x2e5 bytes) 000127 1735099327.540523 mapread 0x025770 thru 0x029df1 (0x4682 bytes) 000128 1735099327.541073 write 0x02852c thru 0x036861 (0xe336 bytes) 000129 1735099327.542064 trunc from 0x036862 to 0x012423 (0x2443e bytes) 000130 1735099327.547943 write 0x01f521 thru 0x02e4f6 (0xefd6 bytes) 000132 1735099327.549310 mapread 0x01693e thru 0x01b9b6 (0x5079 bytes) 000133 1735099327.558455 MIRROR_EXTEND to 2 mirrors 000134 1735099327.672691 MIRROR_RESYNC 2 mirrors 000135 1735099327.714105 trunc from 0x02e4f7 to 0x024ce3 (0x9813 bytes) 000136 1735099327.733972 MIRROR_EXTEND to 3 mirrors 000137 1735099327.850408 punch from 0x00ea6c to 0x01a56e (0xbb03 bytes) 000138 1735099327.862087 read 0x022dbc thru 0x024ce3 (0x1f28 bytes) 000139 1735099327.864128 mapread 0x00922c thru 0x017c68 (0xea3d bytes) 000140 1735099327.872670 mapread 0x017115 thru 0x024735 (0xd621 bytes) 000141 1735099327.875351 punch from 0x00da8d to 0x013ad2 (0x6046 bytes) 000142 1735099327.878134 read 0x0232ae thru 0x024ce3 (0x1a36 bytes) 000143 1735099327.880505 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000145 1735099327.945482 mapread 0x008904 thru 0x0134ff (0xabfc bytes) 000146 1735099327.956660 write 0x01de76 thru 0x02776c (0x98f7 bytes) 000148 1735099327.965216 mapread 0x007239 thru 0x00bd6c (0x4b34 bytes) 000149 1735099327.965658 write 0x01d81e thru 0x01f537 (0x1d1a bytes) 000150 1735099327.966167 mapread 0x00e8a7 thru 0x019fe6 (0xb740 bytes) 000151 1735099327.966689 punch from 0x0082a0 to 0x0097fa (0x155b bytes) 000152 1735099327.968544 write 0x01f4d3 thru 0x01fac3 (0x5f1 bytes) 000153 1735099327.970655 MIRROR_RESYNC 2 mirrors 000157 1735099328.085762 write 0x021d27 thru 0x02c262 (0xa53c bytes) 000158 1735099328.097018 mapread 0x00004c thru 0x009c1c (0x9bd1 bytes) 000159 1735099328.103187 read 0x00fb69 thru 0x0112d3 (0x176b bytes) 000160 1735099328.104958 write 0x030d47 thru 0x039d9f (0x9059 bytes) 000161 1735099328.109181 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000163 1735099328.147177 mapread 0x02de03 thru 0x039d9f (0xbf9d bytes) 000164 1735099328.155744 trunc from 0x039da0 to 0x001838 (0x38567 bytes) 000165 1735099328.162344 mapwrite 0x01d76c thru 0x02a199 (0xca2e bytes) 000166 1735099328.180708 mapread 0x00a5f9 thru 0x011f90 (0x7998 bytes) 000167 1735099328.183576 fallocate 0x034f68 thru 0x0369a4 (0x1a3d bytes) 000168 1735099328.185275 mapread 0x0199eb thru 0x02090d (0x6f23 bytes) 000169 1735099328.185788 write 0x02d475 thru 0x035e31 (0x89bd bytes) 000170 1735099328.186479 write 0x01ee87 thru 0x0261ca (0x7344 bytes) 000171 1735099328.187134 punch from 0x00cb6d to 0x01b2cc (0xe760 bytes) 000172 1735099328.188861 mapread 0x001909 thru 0x010200 (0xe8f8 bytes) 000173 1735099328.193487 write 0x00d7ab thru 0x018712 (0xaf68 bytes) 000175 1735099328.194688 trunc from 0x035e32 to 0x033971 (0x24c0 bytes) 000176 1735099328.199861 punch from 0x025751 to 0x026e18 (0x16c8 bytes) 000177 1735099328.204295 mapread 0x02e658 thru 0x033971 (0x531a bytes) 000178 1735099328.204884 read 0x010faa thru 0x0207f9 (0xf850 bytes) 000180 1735099328.206964 mapread 0x0177e9 thru 0x01b8a0 (0x40b8 bytes) 000181 1735099328.207499 trunc from 0x033972 to 0x00d690 (0x262e1 bytes) 000183 1735099328.212168 fallocate 0x03be1c thru 0x03e09c (0x2281 bytes) 000184 1735099328.213755 mapread 0x003595 thru 0x00d65e (0xa0ca bytes) 000186 1735099328.214638 read 0x006903 thru 0x007ed1 (0x15cf bytes) 000187 1735099328.215040 fallocate 0x0173b1 thru 0x01ec4d (0x789d bytes) 000188 1735099328.216776 read 0x009e48 thru 0x00d690 (0x3849 bytes) 000189 1735099328.217125 fallocate 0x01e00f thru 0x0225a0 (0x4592 bytes) 000190 1735099328.218572 read 0x020641 thru 0x0225a0 (0x1f60 bytes) 000191 1735099328.221632 write 0x01772f thru 0x01ed41 (0x7613 bytes) 000194 1735099328.223170 read 0x01d6f3 thru 0x0225a0 (0x4eae bytes) 000195 1735099328.223569 punch from 0x011231 to 0x01c921 (0xb6f1 bytes) 000196 1735099328.228043 fallocate 0x02a707 thru 0x02b73b (0x1035 bytes) 000197 1735099328.229593 fallocate 0x0265c9 thru 0x02b3fa (0x4e32 bytes) 000198 1735099328.231202 fallocate 0x02ed33 thru 0x03e94f (0xfc1d bytes) 000199 1735099328.232615 mapwrite 0x01ead7 thru 0x027c58 (0x9182 bytes) 000200 1735099328.245751 write 0x01b986 thru 0x020864 (0x4edf bytes) 000201 1735099328.246455 fallocate 0x0377b3 thru 0x03ffff (0x884d bytes) 000202 1735099328.248255 mapread 0x01eb61 thru 0x02b3fa (0xc89a bytes) 000203 1735099328.248795 mapread 0x01afbf thru 0x022086 (0x70c8 bytes) 000206 1735099328.249874 mapwrite 0x0152df thru 0x020f36 (0xbc58 bytes) 000207 1735099328.256867 mapwrite 0x008ffd thru 0x012da0 (0x9da4 bytes) 000209 1735099328.263572 mapread 0x01dba6 thru 0x02894a (0xada5 bytes) 000210 1735099328.263992 read 0x01615f thru 0x02029f (0xa141 bytes) 000211 1735099328.264264 trunc from 0x02b3fb to 0x00afeb (0x2040f bytes) 000212 1735099328.269135 write 0x033d3a thru 0x03af02 (0x71c9 bytes) 000213 1735099328.269877 mapread 0x019b21 thru 0x01ab71 (0x1051 bytes) 000214 1735099328.273199 mapread 0x01600b thru 0x01d3b8 (0x73ae bytes) 000215 1735099328.273773 punch from 0x02e8f6 to 0x036b09 (0x8214 bytes) 000216 1735099328.278172 mapwrite 0x02840a thru 0x0360a8 (0xdc9f bytes) 000217 1735099328.297626 mapread 0x0255fe thru 0x02b49b (0x5e9e bytes) 000220 1735099328.299161 read 0x003c66 thru 0x0047db (0xb76 bytes) 000221 1735099328.299495 write 0x00f712 thru 0x016a48 (0x7337 bytes) 000223 1735099328.300528 read 0x01e39e thru 0x021214 (0x2e77 bytes) 000225 1735099328.301129 fallocate 0x01791a thru 0x02238c (0xaa73 bytes) 000226 1735099328.303068 mapread 0x0320ec thru 0x03af02 (0x8e17 bytes) 000227 1735099328.303579 mapwrite 0x02686d thru 0x02b20a (0x499e bytes) 000229 1735099328.310506 write 0x030a06 thru 0x03addd (0xa3d8 bytes) 000231 1735099328.311745 fallocate 0x011dac thru 0x017b1e (0x5d73 bytes) 000232 1735099328.315811 MIRROR_EXTEND to 2 mirrors 000233 1735099328.448670 read 0x0373ff thru 0x03a52d (0x312f bytes) 000235 1735099328.451555 MIRROR_EXTEND to 3 mirrors 000237 1735099328.593242 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000238 1735099328.648369 mapwrite 0x035646 thru 0x03f04f (0x9a0a bytes) 000239 1735099328.679590 fallocate 0x0049ad thru 0x00a9c3 (0x6017 bytes) 000240 1735099328.681838 write 0x0349c2 thru 0x03ffff (0xb63e bytes) 000241 1735099328.682995 fallocate 0x0363b5 thru 0x03ffff (0x9c4b bytes) 000242 1735099328.685056 punch from 0x035617 to 0x03646f (0xe59 bytes) 000243 1735099328.690100 punch from 0x024f12 to 0x03120c (0xc2fb bytes) 000245 1735099328.692588 read 0x0355a8 thru 0x0395cf (0x4028 bytes) 000246 1735099328.694722 fallocate 0x010e8b thru 0x01f993 (0xeb09 bytes) 000248 1735099328.696645 trunc from 0x040000 to 0x01bad9 (0x24526 bytes) 000249 1735099328.701440 write 0x027edf thru 0x0379e6 (0xfb08 bytes) 000250 1735099328.702526 trunc from 0x0379e7 to 0x03dbb3 (0x61cd bytes) 000251 1735099328.706160 mapwrite 0x02aa72 thru 0x02fbde (0x516d bytes) 000252 1735099328.712812 fallocate 0x004ec4 thru 0x0145b6 (0xf6f3 bytes) 000253 1735099328.715659 MIRROR_RESYNC 2 mirrors 000254 1735099328.847941 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000255 1735099328.881804 punch from 0x01a316 to 0x01c595 (0x2280 bytes) 000256 1735099328.885699 trunc from 0x03dbb4 to 0x03fdfc (0x2249 bytes) 000257 1735099328.894408 write 0x0210b7 thru 0x02bc50 (0xab9a bytes) 000259 1735099328.897270 trunc from 0x03fdfd to 0x03f128 (0xcd4 bytes) 000261 1735099328.900864 write 0x022ec9 thru 0x026659 (0x3791 bytes) 000262 1735099328.901399 fallocate 0x030a97 thru 0x037b83 (0x70ed bytes) 000263 1735099328.902784 fallocate 0x011fa0 thru 0x0126d7 (0x738 bytes) 000265 1735099328.904709 mapread 0x00577e thru 0x011c4d (0xc4d0 bytes) 000266 1735099328.909757 trunc from 0x03f129 to 0x01a1d2 (0x24f56 bytes) 000267 1735099328.914765 mapwrite 0x01b23d thru 0x01bdad (0xb71 bytes) 000268 1735099328.927029 mapwrite 0x03fef5 thru 0x03ffff (0x10b bytes) 000269 1735099328.938165 fallocate 0x025ac7 thru 0x0265df (0xb19 bytes) 000271 1735099328.940220 write 0x026273 thru 0x02e89d (0x862b bytes) 000272 1735099328.945361 MIRROR_EXTEND to 2 mirrors 000274 1735099329.088147 mapwrite 0x002df3 thru 0x00ba16 (0x8c24 bytes) 000275 1735099329.124651 punch from 0x00c2d8 to 0x00c645 (0x36e bytes) 000276 1735099329.127131 trunc from 0x040000 to 0x022ecc (0x1d133 bytes) 000277 1735099329.137803 trunc from 0x022ecd to 0x00c704 (0x167c8 bytes) 000278 1735099329.143950 punch from 0x006a67 to 0x00c704 (0x5c9e bytes) 000279 1735099329.146875 read 0x003fcd thru 0x004bff (0xc33 bytes) 000280 1735099329.147640 fallocate 0x013d88 thru 0x0233af (0xf628 bytes) 000281 1735099329.150534 mapwrite 0x024439 thru 0x028f2a (0x4af2 bytes) 000283 1735099329.170361 fallocate 0x036092 thru 0x038ab1 (0x2a20 bytes) 000284 1735099329.172341 mapwrite 0x029595 thru 0x031851 (0x82bd bytes) 000285 1735099329.183460 read 0x02f800 thru 0x030c39 (0x143a bytes) 000287 1735099329.184244 trunc from 0x038ab2 to 0x000016 (0x38a9b bytes) 000288 1735099329.190377 write 0x02da38 thru 0x036e61 (0x942a bytes) 000290 1735099329.191759 read 0x0023e7 thru 0x007f5f (0x5b79 bytes) 000291 1735099329.195239 punch from 0x035979 to 0x036e61 (0x14e9 bytes) 000292 1735099329.200598 read 0x0223f8 thru 0x026ce7 (0x48f0 bytes) 000294 1735099329.201314 punch from 0x014e6c to 0x01c4eb (0x7680 bytes) 000295 1735099329.204070 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000296 1735099329.239508 mapwrite 0x028528 thru 0x02f751 (0x722a bytes) 000297 1735099329.255401 mapwrite 0x01086d thru 0x01ee9e (0xe632 bytes) 000298 1735099329.270087 trunc from 0x036e62 to 0x014a15 (0x2244c bytes) 000299 1735099329.275791 mapwrite 0x03031f thru 0x0335b9 (0x329b bytes) 000300 1735099329.291393 read 0x00b97e thru 0x01b0de (0xf761 bytes) 000301 1735099329.294933 punch from 0x019753 to 0x01f10f (0x59bd bytes) 000306 1735099329.297829 punch from 0x02b5c2 to 0x0335b9 (0x7ff8 bytes) 000307 1735099329.300257 read 0x01a2c9 thru 0x020e4d (0x6b85 bytes) 000309 1735099329.305246 read 0x031a05 thru 0x0335b9 (0x1bb5 bytes) 000310 1735099329.305554 trunc from 0x0335ba to 0x00e5b5 (0x25004 bytes) 000311 1735099329.311198 mapread 0x0010a0 thru 0x0031bb (0x211c bytes) 000312 1735099329.311853 read 0x009417 thru 0x00e5b5 (0x519f bytes) 000313 1735099329.312258 write 0x02b0f1 thru 0x033263 (0x8173 bytes) 000314 1735099329.313109 trunc from 0x033264 to 0x013827 (0x1fa3c bytes) 000315 1735099329.318252 read 0x001a49 thru 0x0062b6 (0x486e bytes) 000317 1735099329.318972 write 0x023a14 thru 0x029593 (0x5b80 bytes) 000319 1735099329.320050 mapwrite 0x01fed8 thru 0x01ffad (0xd6 bytes) 000320 1735099329.328612 fallocate 0x0070d5 thru 0x00ac99 (0x3bc5 bytes) 000321 1735099329.330392 mapread 0x008fbb thru 0x00f925 (0x696b bytes) 000322 1735099329.330925 trunc from 0x029594 to 0x01a242 (0xf351 bytes) 000323 1735099329.337250 punch from 0x01616d to 0x01a242 (0x40d6 bytes) 000325 1735099329.339349 fallocate 0x03a119 thru 0x03ffff (0x5ee7 bytes) 000327 1735099329.341381 write 0x003aea thru 0x00e8cb (0xade2 bytes) 000328 1735099329.342230 trunc from 0x040000 to 0x008e5e (0x371a1 bytes) 000329 1735099329.347226 read 0x00163a thru 0x0064f2 (0x4eb9 bytes) 000330 1735099329.347623 fallocate 0x007b15 thru 0x00dce9 (0x61d5 bytes) 000336 1735099329.351426 write 0x00244e thru 0x007db4 (0x5967 bytes) 000337 1735099329.352098 fallocate 0x025501 thru 0x028f3d (0x3a3d bytes) 000338 1735099329.353916 write 0x0335d2 thru 0x0388d3 (0x5302 bytes) 000339 1735099329.354391 write 0x01b65d thru 0x0271ab (0xbb4f bytes) 000340 1735099329.358710 punch from 0x01f273 to 0x02d440 (0xe1ce bytes) 000341 1735099329.364214 trunc from 0x0388d4 to 0x03afdd (0x270a bytes) 000344 1735099329.369338 mapread 0x03041d thru 0x037f64 (0x7b48 bytes) 000345 1735099329.370023 mapwrite 0x001698 thru 0x011513 (0xfe7c bytes) 000347 1735099329.379373 trunc from 0x03afde to 0x036f61 (0x407c bytes) 000349 1735099329.385771 write 0x001f00 thru 0x00ba7a (0x9b7b bytes) 000352 1735099329.387262 trunc from 0x036f62 to 0x013416 (0x23b4b bytes) 000354 1735099329.393149 read 0x00f024 thru 0x013416 (0x43f3 bytes) 000355 1735099329.393589 punch from 0x00b2c0 to 0x013416 (0x8157 bytes) 000357 1735099329.399016 write 0x02f12d thru 0x03b70e (0xc5e2 bytes) 000358 1735099329.399985 trunc from 0x03b70f to 0x0111d3 (0x2a53b bytes) 000359 1735099329.404252 punch from 0x00d6db to 0x0111d3 (0x3af9 bytes) 000360 1735099329.406233 fallocate 0x036329 thru 0x038ae8 (0x27c0 bytes) 000362 1735099329.409587 MIRROR_EXTEND to 2 mirrors 000366 1735099329.503437 fallocate 0x02cf47 thru 0x031422 (0x44dc bytes) 000367 1735099329.511088 write 0x033ebb thru 0x03d402 (0x9548 bytes) 000368 1735099329.513962 mapwrite 0x029066 thru 0x02df4f (0x4eea bytes) 000369 1735099329.523220 mapread 0x025ef4 thru 0x02a8e2 (0x49ef bytes) 000370 1735099329.523840 fallocate 0x0193f8 thru 0x0245bb (0xb1c4 bytes) 000371 1735099329.525604 trunc from 0x03d403 to 0x00c03a (0x313c8 bytes) 000373 1735099329.535876 mapread 0x007900 thru 0x00c03a (0x473b bytes) 000374 1735099329.538883 fallocate 0x03315a thru 0x033f46 (0xded bytes) 000375 1735099329.540660 fallocate 0x00fb82 thru 0x013c8b (0x410a bytes) 000376 1735099329.542331 read 0x02c45f thru 0x02d397 (0xf39 bytes) 000378 1735099329.544525 read 0x02782e thru 0x0320cb (0xa89e bytes) 000379 1735099329.547989 fallocate 0x018fc1 thru 0x01f83f (0x687f bytes) 000380 1735099329.549686 fallocate 0x035d45 thru 0x039b97 (0x3e53 bytes) 000381 1735099329.551408 punch from 0x017828 to 0x02572b (0xdf04 bytes) 000382 1735099329.553246 mapread 0x01b391 thru 0x028233 (0xcea3 bytes) 000383 1735099329.568018 MIRROR_RESYNC 2 mirrors 000384 1735099329.646367 mapread 0x01e01c thru 0x023e94 (0x5e79 bytes) 000385 1735099329.653484 MIRROR_RESYNC 2 mirrors 000387 1735099329.695953 write 0x01213d thru 0x021669 (0xf52d bytes) 000388 1735099329.705418 mapwrite 0x023431 thru 0x02b0d2 (0x7ca2 bytes) 000389 1735099329.714367 punch from 0x0057e2 to 0x00ab45 (0x5364 bytes) 000391 1735099329.716129 read 0x027463 thru 0x027ceb (0x889 bytes) 000392 1735099329.716416 trunc from 0x033f47 to 0x038507 (0x45c1 bytes) 000393 1735099329.724349 read 0x000021 thru 0x006cde (0x6cbe bytes) 000394 1735099329.726782 punch from 0x01d27a to 0x024fa4 (0x7d2b bytes) 000395 1735099329.728250 mapwrite 0x02b689 thru 0x036e56 (0xb7ce bytes) 000396 1735099329.734834 write 0x03d4f1 thru 0x03e6c3 (0x11d3 bytes) 000399 1735099329.735945 fallocate 0x00f0ea thru 0x015fbb (0x6ed2 bytes) 000400 1735099329.737153 trunc from 0x03e6c4 to 0x025d69 (0x1895a bytes) 000402 1735099329.742068 trunc from 0x025d6a to 0x02cf3b (0x71d2 bytes) 000403 1735099329.746029 write 0x0007d9 thru 0x004aaf (0x42d7 bytes) 000404 1735099329.747807 MIRROR_EXTEND to 3 mirrors 000405 1735099329.879299 trunc from 0x02cf3c to 0x00fa57 (0x1d4e4 bytes) 000406 1735099329.892735 trunc from 0x00fa58 to 0x02c9b3 (0x1cf5c bytes) 000407 1735099329.898067 mapwrite 0x00c60f thru 0x00e3fa (0x1dec bytes) 000409 1735099329.907292 trunc from 0x02c9b4 to 0x024c91 (0x7d22 bytes) 000410 1735099329.911494 mapread 0x004647 thru 0x0112d8 (0xcc92 bytes) 000411 1735099329.913999 fallocate 0x01974a thru 0x01b60d (0x1ec4 bytes) 000412 1735099329.915690 trunc from 0x024c92 to 0x03aa38 (0x15da7 bytes) 000413 1735099329.919429 write 0x038e7e thru 0x03ffff (0x7182 bytes) 000414 1735099329.923519 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000415 1735099329.978521 fallocate 0x012d2d thru 0x0194d6 (0x67aa bytes) 000416 1735099329.981159 punch from 0x024417 to 0x02f145 (0xad2f bytes) 000417 1735099329.983261 read 0x004cfb thru 0x010742 (0xba48 bytes) 000418 1735099329.987005 read 0x037bbf thru 0x038371 (0x7b3 bytes) 000419 1735099329.991478 fallocate 0x03aa79 thru 0x03ffff (0x5587 bytes) 000420 1735099329.994415 mapread 0x01c8ff thru 0x027c90 (0xb392 bytes) 000421 1735099329.995049 read 0x037c6d thru 0x03c6f9 (0x4a8d bytes) 000422 1735099329.995393 mapwrite 0x0064eb thru 0x010b07 (0xa61d bytes) 000423 1735099330.005596 fallocate 0x0389b6 thru 0x03ab46 (0x2191 bytes) 000424 1735099330.008581 write 0x0107ba thru 0x01c62e (0xbe75 bytes) 000426 1735099330.015784 MIRROR_RESYNC 2 mirrors 000427 1735099330.143888 trunc from 0x040000 to 0x0070cd (0x38f32 bytes) 000428 1735099330.157417 fallocate 0x01dd3e thru 0x026c67 (0x8f2a bytes) 000430 1735099330.159718 read 0x016ddf thru 0x01df69 (0x718b bytes) 000431 1735099330.162883 mapwrite 0x02299a thru 0x02d5d1 (0xac38 bytes) 000432 1735099330.184827 mapwrite 0x03553a thru 0x03ffff (0xaac6 bytes) 000433 1735099330.202405 MIRROR_EXTEND to 3 mirrors 000435 1735099330.375161 trunc from 0x040000 to 0x0382cc (0x7d33 bytes) 000436 1735099330.408140 trunc from 0x0382cd to 0x012cd7 (0x255f5 bytes) 000437 1735099330.420349 MIRROR_EXTEND to 4 mirrors 000438 1735099330.607462 mapread 0x0072d0 thru 0x007505 (0x236 bytes) 000439 1735099330.614092 punch from 0x00517a to 0x0116e7 (0xc56e bytes) 000440 1735099330.627507 fallocate 0x00e947 thru 0x011e5b (0x3515 bytes) 000441 1735099330.630973 MIRROR_RESYNC 4 mirrors 000442 1735099330.785051 punch from 0x0052d7 to 0x00f2d8 (0xa002 bytes) 000443 1735099330.796407 mapread 0x00db62 thru 0x012cd7 (0x5176 bytes) 000444 1735099330.802768 punch from 0x00204d to 0x005535 (0x34e9 bytes) 000445 1735099330.805079 punch from 0x00edf2 to 0x0113a6 (0x25b5 bytes) 000447 1735099330.808074 trunc from 0x012cd8 to 0x027bad (0x14ed6 bytes) 000448 1735099330.819787 write 0x006c65 thru 0x0117b6 (0xab52 bytes) 000449 1735099330.826858 trunc from 0x027bae to 0x009e4b (0x1dd62 bytes) 000450 1735099330.832821 write 0x039fb2 thru 0x03ffff (0x604e bytes) 000451 1735099330.833666 trunc from 0x040000 to 0x0386d7 (0x7928 bytes) 000452 1735099330.838579 write 0x003908 thru 0x010027 (0xc720 bytes) 000453 1735099330.843042 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000454 1735099330.896144 read 0x010819 thru 0x016634 (0x5e1c bytes) 000456 1735099330.899639 mapread 0x00fd34 thru 0x015d9e (0x606b bytes) 000457 1735099330.901966 mapread 0x01b0aa thru 0x01c09b (0xff2 bytes) 000459 1735099330.909177 MIRROR_RESYNC 3 mirrors 000460 1735099331.078687 write 0x037a6e thru 0x03ffff (0x8592 bytes) 000461 1735099331.096848 punch from 0x029eb7 to 0x031558 (0x76a2 bytes) 000462 1735099331.099447 trunc from 0x040000 to 0x02582b (0x1a7d4 bytes) 000464 1735099331.106613 mapread 0x01ca5a thru 0x02467c (0x7c23 bytes) 000465 1735099331.112076 read 0x0096e0 thru 0x00cb0e (0x342f bytes) 000466 1735099331.114311 read 0x01d3c7 thru 0x01d4f4 (0x12e bytes) 000468 1735099331.115048 fallocate 0x0132d8 thru 0x01d7fa (0xa523 bytes) 000470 1735099331.117712 write 0x02ded7 thru 0x036a27 (0x8b51 bytes) 000471 1735099331.118713 fallocate 0x036696 thru 0x03af84 (0x48ef bytes) 000472 1735099331.121254 read 0x011894 thru 0x011f0d (0x67a bytes) 000473 1735099331.124711 trunc from 0x036a28 to 0x034247 (0x27e0 bytes) 000474 1735099331.133089 trunc from 0x034248 to 0x01348d (0x20dba bytes) 000476 1735099331.141373 mapwrite 0x024b29 thru 0x027cc1 (0x3199 bytes) 000477 1735099331.166556 fallocate 0x0238ce thru 0x03099d (0xd0d0 bytes) 000479 1735099331.169268 read 0x022bcc thru 0x023223 (0x658 bytes) 000480 1735099331.173585 mapread 0x01c44d thru 0x020cf9 (0x48ad bytes) 000481 1735099331.174098 punch from 0x00f5b2 to 0x014f73 (0x59c2 bytes) 000483 1735099331.176306 mapwrite 0x02b740 thru 0x0322f4 (0x6bb5 bytes) 000484 1735099331.194453 fallocate 0x01b54b thru 0x01c3e6 (0xe9c bytes) 000485 1735099331.196620 punch from 0x029f7a to 0x0322f4 (0x837b bytes) 000486 1735099331.199057 fallocate 0x02a456 thru 0x0360c4 (0xbc6f bytes) 000487 1735099331.202131 write 0x0022af thru 0x0040fb (0x1e4d bytes) 000489 1735099331.203045 read 0x02148b thru 0x02f2c7 (0xde3d bytes) 000490 1735099331.207257 read 0x011cfa thru 0x016f38 (0x523f bytes) 000492 1735099331.208016 punch from 0x01407e to 0x017a5c (0x39df bytes) 000494 1735099331.210444 mapread 0x00a48c thru 0x00f0c1 (0x4c36 bytes) 000495 1735099331.211095 write 0x01f99d thru 0x02716a (0x77ce bytes) 000496 1735099331.214329 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000497 1735099331.278290 trunc from 0x0322f5 to 0x00f346 (0x22fae bytes) 000498 1735099331.284403 read 0x00a2d6 thru 0x00f346 (0x5071 bytes) 000499 1735099331.287328 mapwrite 0x03f331 thru 0x03ffff (0xccf bytes) 000500 1735099331.300198 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000502 1735099331.338113 fallocate 0x0327e1 thru 0x03d20a (0xaa2a bytes) 000504 1735099331.340658 mapread 0x00443d thru 0x012329 (0xdeed bytes) 000505 1735099331.348570 mapread 0x03e3c2 thru 0x03ffff (0x1c3e bytes) 000506 1735099331.351533 read 0x03c9e3 thru 0x03ffff (0x361d bytes) 000507 1735099331.351989 read 0x0077da thru 0x00e958 (0x717f bytes) 000508 1735099331.352381 fallocate 0x016311 thru 0x01fb78 (0x9868 bytes) 000509 1735099331.354529 mapwrite 0x013230 thru 0x01fa63 (0xc834 bytes) 000510 1735099331.364846 fallocate 0x006d98 thru 0x00afbd (0x4226 bytes) 000511 1735099331.366403 read 0x0228c1 thru 0x024719 (0x1e59 bytes) 000513 1735099331.367139 punch from 0x0054cb to 0x008a94 (0x35ca bytes) 000514 1735099331.369459 read 0x02407b thru 0x028e4c (0x4dd2 bytes) 000515 1735099331.369793 mapread 0x0046de thru 0x005706 (0x1029 bytes) 000516 1735099331.372664 write 0x00da83 thru 0x014604 (0x6b82 bytes) 000517 1735099331.375084 mapwrite 0x0189c7 thru 0x0254d4 (0xcb0e bytes) 000518 1735099331.385695 punch from 0x03df1b to 0x03ffff (0x20e5 bytes) 000519 1735099331.390787 MIRROR_EXTEND to 2 mirrors 000521 1735099331.540336 MIRROR_EXTEND to 3 mirrors 000522 1735099331.716303 MIRROR_EXTEND to 4 mirrors 000524 1735099331.933236 read 0x001b3d thru 0x00f94c (0xde10 bytes) 000525 1735099331.937814 punch from 0x02d242 to 0x0362ec (0x90ab bytes) 000526 1735099331.955620 write 0x0205b7 thru 0x026031 (0x5a7b bytes) 000527 1735099331.961515 mapread 0x001dc6 thru 0x008b2b (0x6d66 bytes) 000529 1735099331.966529 MIRROR_RESYNC 4 mirrors 000530 1735099332.337690 punch from 0x0255bd to 0x02cc87 (0x76cb bytes) 000531 1735099332.350403 mapwrite 0x00124a thru 0x00255a (0x1311 bytes) 000532 1735099332.359097 mapwrite 0x012766 thru 0x01b241 (0x8adc bytes) 000533 1735099332.373002 fallocate 0x0173e2 thru 0x020fa5 (0x9bc4 bytes) 000534 1735099332.376756 write 0x01c76c thru 0x01cedb (0x770 bytes) 000536 1735099332.377673 mapwrite 0x02b2dc thru 0x02e62a (0x334f bytes) 000538 1735099332.389516 mapread 0x023a86 thru 0x033359 (0xf8d4 bytes) 000539 1735099332.390307 mapread 0x03e7d8 thru 0x03ffff (0x1828 bytes) 000540 1735099332.394395 write 0x03a666 thru 0x03bd48 (0x16e3 bytes) 000541 1735099332.394994 fallocate 0x02b328 thru 0x034ada (0x97b3 bytes) 000542 1735099332.397615 mapread 0x01a75a thru 0x0274fd (0xcda4 bytes) 000544 1735099332.398750 fallocate 0x034750 thru 0x038747 (0x3ff8 bytes) 000545 1735099332.400769 write 0x03dfc4 thru 0x03ffff (0x203c bytes) 000547 1735099332.403252 MIRROR_RESYNC 4 mirrors 000548 1735099332.667146 mapwrite 0x01aca2 thru 0x024c1f (0x9f7e bytes) 000549 1735099332.718809 punch from 0x00c7ec to 0x00d785 (0xf9a bytes) 000550 1735099332.722185 trunc from 0x040000 to 0x03fb97 (0x468 bytes) 000551 1735099332.736827 mapwrite 0x028f10 thru 0x032b3b (0x9c2c bytes) 000552 1735099332.749864 mapwrite 0x01ac9f thru 0x027a35 (0xcd97 bytes) 000553 1735099332.759858 fallocate 0x02cc75 thru 0x0314ea (0x4876 bytes) 000554 1735099332.761968 write 0x02b090 thru 0x0386fe (0xd66f bytes) 000555 1735099332.764731 trunc from 0x03fb98 to 0x031ad9 (0xe0be bytes) 000556 1735099332.771442 trunc from 0x031ada to 0x0271c9 (0xa910 bytes) 000557 1735099332.779270 read 0x01f52d thru 0x0271c9 (0x7c9d bytes) 000558 1735099332.779750 trunc from 0x0271ca to 0x009c2c (0x1d59d bytes) 000559 1735099332.786803 read 0x0073e8 thru 0x009c2c (0x2845 bytes) 000560 1735099332.793309 MIRROR_RESYNC 4 mirrors 000561 1735099332.965764 write 0x0044f6 thru 0x0097bd (0x52c8 bytes) 000562 1735099332.983564 fallocate 0x018025 thru 0x023ea5 (0xbe81 bytes) 000563 1735099332.987164 trunc from 0x009c2d to 0x0250d7 (0x1b4ab bytes) 000564 1735099333.003030 trunc from 0x0250d8 to 0x038565 (0x1348e bytes) 000566 1735099333.009397 mapread 0x02cb1d thru 0x0339b9 (0x6e9d bytes) 000567 1735099333.017600 MIRROR_RESYNC 4 mirrors 000569 1735099333.211743 MIRROR_EXTEND to 5 mirrors 000570 1735099333.350491 write 0x009be0 thru 0x00f9ac (0x5dcd bytes) 000571 1735099333.363787 mapwrite 0x008b86 thru 0x01230f (0x978a bytes) 000573 1735099333.377016 mapwrite 0x01d798 thru 0x027c32 (0xa49b bytes) 000574 1735099333.387358 trunc from 0x038566 to 0x03b7b4 (0x324f bytes) 000575 1735099333.396731 fallocate 0x02d5d0 thru 0x03a4d0 (0xcf01 bytes) 000576 1735099333.398294 punch from 0x014365 to 0x01d1ae (0x8e4a bytes) 000577 1735099333.399716 write 0x029340 thru 0x031bb4 (0x8875 bytes) 000580 1735099333.402359 mapread 0x00d7e2 thru 0x00eabe (0x12dd bytes) 000581 1735099333.403001 write 0x017ee3 thru 0x01ab51 (0x2c6f bytes) 000582 1735099333.405676 read 0x035653 thru 0x036aea (0x1498 bytes) 000583 1735099333.407140 mapwrite 0x00c6cd thru 0x015570 (0x8ea4 bytes) 000584 1735099333.414493 write 0x0151bf thru 0x01587f (0x6c1 bytes) 000585 1735099333.415072 trunc from 0x03b7b5 to 0x03c666 (0xeb2 bytes) 000586 1735099333.421090 punch from 0x00321f to 0x00f0b8 (0xbe9a bytes) 000587 1735099333.422761 mapwrite 0x03e737 thru 0x03ffff (0x18c9 bytes) 000589 1735099333.439924 write 0x018cbb thru 0x02732d (0xe673 bytes) 000591 1735099333.441173 mapwrite 0x03aef1 thru 0x03ffff (0x510f bytes) 000593 1735099333.450296 trunc from 0x040000 to 0x0306ff (0xf900 bytes) 000594 1735099333.458684 MIRROR_RESYNC 5 mirrors 000596 1735099333.840143 read 0x0283fa thru 0x0306ff (0x8306 bytes) 000597 1735099333.842986 read 0x00dbf8 thru 0x011cc4 (0x40cd bytes) 000598 1735099333.845914 mapwrite 0x0290e5 thru 0x037d67 (0xec83 bytes) 000599 1735099333.890667 mapwrite 0x00c2f3 thru 0x017741 (0xb44f bytes) 000600 1735099333.916728 read 0x0306e5 thru 0x037d67 (0x7683 bytes) 000601 1735099333.921062 MIRROR_RESYNC 5 mirrors 000602 1735099334.242478 fallocate 0x003d58 thru 0x010171 (0xc41a bytes) 000603 1735099334.254849 fallocate 0x038fb4 thru 0x03ffff (0x704c bytes) 000604 1735099334.258772 MIRROR_EXTEND to 6 mirrors 000605 1735099334.467388 read 0x01d19e thru 0x02a7ea (0xd64d bytes) 000607 1735099334.483871 write 0x02b018 thru 0x03af94 (0xff7d bytes) 000608 1735099334.505015 mapread 0x02589d thru 0x02dc19 (0x837d bytes) 000609 1735099334.512473 punch from 0x0341cf to 0x03af94 (0x6dc6 bytes) 000611 1735099334.538860 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000612 1735099334.627256 fallocate 0x00da72 thru 0x01b54b (0xdada bytes) 000613 1735099334.629267 punch from 0x028174 to 0x03462c (0xc4b9 bytes) 000615 1735099334.633108 fallocate 0x001196 thru 0x0027bf (0x162a bytes) 000616 1735099334.636151 mapread 0x003271 thru 0x003b8c (0x91c bytes) 000617 1735099334.639292 mapwrite 0x0342fe thru 0x03ffff (0xbd02 bytes) 000618 1735099334.672952 write 0x03e3a1 thru 0x03ffff (0x1c5f bytes) 000620 1735099334.674307 read 0x024593 thru 0x02dd30 (0x979e bytes) 000621 1735099334.681060 read 0x014e73 thru 0x021237 (0xc3c5 bytes) 000622 1735099334.687460 mapread 0x016794 thru 0x023b6e (0xd3db bytes) 000623 1735099334.688242 write 0x03513f thru 0x035bce (0xa90 bytes) 000624 1735099334.688812 mapwrite 0x03ae57 thru 0x03ffff (0x51a9 bytes) 000625 1735099334.697214 punch from 0x0296b0 to 0x03425b (0xabac bytes) 000626 1735099334.700113 punch from 0x031e03 to 0x03cb15 (0xad13 bytes) 000627 1735099334.702754 read 0x02305e thru 0x02e026 (0xafc9 bytes) 000628 1735099334.707552 trunc from 0x040000 to 0x01e43d (0x21bc2 bytes) 000631 1735099334.717652 trunc from 0x01e43e to 0x0264cc (0x808f bytes) 000632 1735099334.728835 write 0x0384b0 thru 0x03ffff (0x7b50 bytes) 000633 1735099334.729827 fallocate 0x008f71 thru 0x010629 (0x76b9 bytes) 000634 1735099334.732525 mapwrite 0x02df1e thru 0x03b9a3 (0xda86 bytes) 000635 1735099334.772429 read 0x028ed7 thru 0x031b86 (0x8cb0 bytes) 000636 1735099334.777847 write 0x02e0f6 thru 0x02f068 (0xf73 bytes) 000637 1735099334.778729 mapwrite 0x02087d thru 0x02fc82 (0xf406 bytes) 000639 1735099334.794416 mapread 0x029773 thru 0x036b9b (0xd429 bytes) 000640 1735099334.795113 trunc from 0x040000 to 0x00dbd4 (0x3242b bytes) 000642 1735099334.806028 MIRROR_RESYNC 5 mirrors 000643 1735099335.061618 mapread 0x00c827 thru 0x00dbd4 (0x13ae bytes) 000644 1735099335.064325 write 0x03c88d thru 0x03fad7 (0x324b bytes) 000645 1735099335.080521 mapwrite 0x019c01 thru 0x024c64 (0xb064 bytes) 000647 1735099335.110677 punch from 0x03d746 to 0x03e5a3 (0xe5e bytes) 000648 1735099335.113524 trunc from 0x03fad8 to 0x0075db (0x384fc bytes) 000649 1735099335.134009 write 0x018a13 thru 0x028199 (0xf787 bytes) 000650 1735099335.135507 trunc from 0x02819a to 0x023874 (0x4925 bytes) 000651 1735099335.146985 read 0x001e94 thru 0x010b46 (0xecb3 bytes) 000652 1735099335.151380 read 0x012ddb thru 0x0133b4 (0x5da bytes) 000654 1735099335.155018 mapread 0x010bc4 thru 0x01512b (0x4568 bytes) 000656 1735099335.156057 read 0x017c3f thru 0x023874 (0xbc36 bytes) 000657 1735099335.157111 punch from 0x0104c6 to 0x01dc3f (0xd77a bytes) 000658 1735099335.163414 punch from 0x0228d9 to 0x023874 (0xf9c bytes) 000662 1735099335.166768 write 0x0147d3 thru 0x01657b (0x1da9 bytes) 000663 1735099335.171523 trunc from 0x023875 to 0x00fdea (0x13a8a bytes) 000664 1735099335.179962 mapread 0x00a311 thru 0x00cf5f (0x2c4f bytes) 000665 1735099335.181161 punch from 0x00d81c to 0x00fdea (0x25cf bytes) 000666 1735099335.183353 write 0x035cf5 thru 0x03ffff (0xa30b bytes) 000667 1735099335.184279 read 0x039837 thru 0x03ffff (0x67c9 bytes) 000668 1735099335.184630 trunc from 0x040000 to 0x030a06 (0xf5f9 bytes) 000669 1735099335.193840 write 0x01238c thru 0x01b46c (0x90e1 bytes) 000671 1735099335.199744 fallocate 0x02a587 thru 0x030d24 (0x679e bytes) 000672 1735099335.205197 write 0x01c3fc thru 0x022f7d (0x6b82 bytes) 000673 1735099335.206152 trunc from 0x030a07 to 0x01162a (0x1f3dc bytes) 000674 1735099335.214490 fallocate 0x00cb60 thru 0x016d84 (0xa225 bytes) 000675 1735099335.216459 fallocate 0x017445 thru 0x01c794 (0x5350 bytes) 000676 1735099335.219148 punch from 0x0197ab to 0x01bdd3 (0x2629 bytes) 000677 1735099335.222618 read 0x004d8b thru 0x010a7f (0xbcf5 bytes) 000678 1735099335.223122 trunc from 0x01c795 to 0x02c4a4 (0xfd10 bytes) 000679 1735099335.232758 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000680 1735099335.314300 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000681 1735099335.400434 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000683 1735099335.502923 mapwrite 0x01d77a thru 0x028655 (0xaedc bytes) 000684 1735099335.529401 mapwrite 0x017bc2 thru 0x01aadb (0x2f1a bytes) 000685 1735099335.539944 write 0x0171d8 thru 0x0259af (0xe7d8 bytes) 000686 1735099335.545311 mapwrite 0x02a984 thru 0x032a54 (0x80d1 bytes) 000687 1735099335.566425 mapwrite 0x02dd9f thru 0x03bbf8 (0xde5a bytes) 000688 1735099335.600468 mapwrite 0x02de6c thru 0x035f96 (0x812b bytes) 000690 1735099335.613963 read 0x01b123 thru 0x01b944 (0x822 bytes) 000691 1735099335.615077 fallocate 0x01ba68 thru 0x020f59 (0x54f2 bytes) 000692 1735099335.622213 MIRROR_RESYNC 2 mirrors 000693 1735099335.824752 mapread 0x02bf43 thru 0x033e1f (0x7edd bytes) 000694 1735099335.836583 punch from 0x0317c8 to 0x03bbf8 (0xa431 bytes) 000695 1735099335.859174 write 0x00493d thru 0x00b329 (0x69ed bytes) 000698 1735099335.865722 read 0x00ffdb thru 0x011137 (0x115d bytes) 000699 1735099335.869326 mapwrite 0x00e686 thru 0x015bbe (0x7539 bytes) 000700 1735099335.888848 write 0x0004bd thru 0x0005df (0x123 bytes) 000702 1735099335.892938 punch from 0x0289c7 to 0x0334bf (0xaaf9 bytes) 000703 1735099335.898846 fallocate 0x030a5c thru 0x03d1c5 (0xc76a bytes) 000704 1735099335.905060 trunc from 0x03bbf9 to 0x01857c (0x2367c bytes) 000706 1735099335.945260 mapread 0x00d164 thru 0x01857c (0xb419 bytes) 000707 1735099335.946235 mapwrite 0x00cfaf thru 0x011dbb (0x4e0d bytes) 000708 1735099335.961437 mapwrite 0x020837 thru 0x02b4f0 (0xacba bytes) 000709 1735099335.999757 write 0x001623 thru 0x010366 (0xed44 bytes) 000710 1735099336.000765 mapread 0x02a0a9 thru 0x02b4f0 (0x1448 bytes) 000711 1735099336.001309 mapread 0x011a61 thru 0x01ec57 (0xd1f7 bytes) 000712 1735099336.002046 mapwrite 0x03b049 thru 0x03ffff (0x4fb7 bytes) 000713 1735099336.034016 write 0x01cc2b thru 0x02703a (0xa410 bytes) 000714 1735099336.035708 punch from 0x00ce64 to 0x0132ce (0x646b bytes) 000715 1735099336.041628 mapread 0x028a2b thru 0x032b24 (0xa0fa bytes) 000716 1735099336.049686 mapwrite 0x02509c thru 0x02b0c8 (0x602d bytes) 000717 1735099336.063564 fallocate 0x03ba54 thru 0x03efd5 (0x3582 bytes) 000719 1735099336.068432 mapread 0x021cf6 thru 0x027783 (0x5a8e bytes) 000720 1735099336.069535 read 0x025281 thru 0x027ef2 (0x2c72 bytes) 000722 1735099336.072821 punch from 0x0303e9 to 0x036d9f (0x69b7 bytes) 000723 1735099336.077856 trunc from 0x040000 to 0x0138dd (0x2c722 bytes) 000726 1735099336.094766 fallocate 0x0145c1 thru 0x01c830 (0x8270 bytes) 000727 1735099336.100822 read 0x00f523 thru 0x0138dd (0x43bb bytes) 000729 1735099336.108066 mapwrite 0x019c8c thru 0x021f55 (0x82ca bytes) 000730 1735099336.151346 fallocate 0x009570 thru 0x00f0b9 (0x5b4a bytes) 000734 1735099336.158340 mapwrite 0x0101b2 thru 0x0104de (0x32d bytes) 000735 1735099336.173767 mapwrite 0x00dfb4 thru 0x0182c2 (0xa30f bytes) 000736 1735099336.184699 write 0x02dce3 thru 0x039b5a (0xbe78 bytes) 000737 1735099336.185923 read 0x001aeb thru 0x00f7b0 (0xdcc6 bytes) 000738 1735099336.186343 mapread 0x00a2db thru 0x01365c (0x9382 bytes) 000740 1735099336.187288 trunc from 0x039b5b to 0x003f11 (0x35c49 bytes) 000741 1735099336.195773 fallocate 0x02bae1 thru 0x037946 (0xbe66 bytes) 000742 1735099336.199447 fallocate 0x0305f1 thru 0x032379 (0x1d89 bytes) 000744 1735099336.205792 MIRROR_RESYNC 2 mirrors 000745 1735099336.485554 trunc from 0x037947 to 0x033d61 (0x3be5 bytes) 000746 1735099336.523272 read 0x0166b7 thru 0x021138 (0xaa82 bytes) 000747 1735099336.527842 trunc from 0x033d62 to 0x020dc4 (0x12f9d bytes) 000748 1735099336.545502 MIRROR_RESYNC 2 mirrors 000749 1735099336.725942 mapwrite 0x0097eb thru 0x017c59 (0xe46f bytes) 000750 1735099336.839256 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000751 1735099336.957292 punch from 0x00aef3 to 0x01875a (0xd868 bytes) 000753 1735099336.962244 trunc from 0x020dc5 to 0x017999 (0x942b bytes) 000756 1735099336.986753 mapwrite 0x02c9e4 thru 0x038326 (0xb943 bytes) 000757 1735099337.037518 mapread 0x02841e thru 0x033aa7 (0xb68a bytes) 000759 1735099337.040231 punch from 0x03421f to 0x038326 (0x4108 bytes) 000760 1735099337.049483 MIRROR_EXTEND to 2 mirrors 000761 1735099337.343803 write 0x00888e thru 0x014faf (0xc722 bytes) 000762 1735099337.373954 mapread 0x00d62c thru 0x01c53b (0xef10 bytes) 000763 1735099337.379058 mapread 0x010c0d thru 0x01a0a7 (0x949b bytes) 000764 1735099337.379591 mapread 0x01f943 thru 0x02bb7d (0xc23b bytes) 000765 1735099337.384816 write 0x014ef6 thru 0x015b9f (0xcaa bytes) 000766 1735099337.385533 mapread 0x018cc6 thru 0x019adb (0xe16 bytes) 000768 1735099337.386785 fallocate 0x02049f thru 0x02d106 (0xcc68 bytes) 000769 1735099337.390611 mapwrite 0x016d24 thru 0x021a31 (0xad0e bytes) 000770 1735099337.408566 fallocate 0x021c11 thru 0x024099 (0x2489 bytes) 000772 1735099337.415381 MIRROR_RESYNC 2 mirrors 000773 1735099337.656446 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000774 1735099337.740460 read 0x004717 thru 0x00513b (0xa25 bytes) 000775 1735099337.745167 mapwrite 0x0200b3 thru 0x02912d (0x907b bytes) 000776 1735099337.776657 write 0x007bae thru 0x015c37 (0xe08a bytes) 000777 1735099337.794462 mapread 0x000b04 thru 0x00ecf9 (0xe1f6 bytes) 000779 1735099337.798783 mapwrite 0x01cacb thru 0x0297fc (0xcd32 bytes) 000780 1735099337.815418 read 0x002889 thru 0x0058f2 (0x306a bytes) 000781 1735099337.815821 write 0x00922e thru 0x00f196 (0x5f69 bytes) 000782 1735099337.816680 trunc from 0x038327 to 0x0067d6 (0x31b50 bytes) 000783 1735099337.850286 fallocate 0x030e01 thru 0x039e0f (0x900f bytes) 000784 1735099337.857687 MIRROR_EXTEND to 2 mirrors 000785 1735099338.134424 punch from 0x004310 to 0x00533b (0x102c bytes) 000786 1735099338.178704 mapread 0x004a21 thru 0x0067d6 (0x1db6 bytes) 000787 1735099338.196203 mapwrite 0x02fb60 thru 0x031998 (0x1e39 bytes) 000788 1735099338.222721 trunc from 0x031999 to 0x00e0ae (0x238ea bytes) 000789 1735099338.233808 read 0x00911b thru 0x00e0ae (0x4f94 bytes) 000790 1735099338.239194 fallocate 0x004b10 thru 0x009d96 (0x5287 bytes) 000791 1735099338.242413 trunc from 0x00e0af to 0x02c6d6 (0x1e628 bytes) 000792 1735099338.250751 punch from 0x002d46 to 0x00493e (0x1bf9 bytes) 000793 1735099338.253976 mapwrite 0x0210a1 thru 0x02f16e (0xe0ce bytes) 000794 1735099338.294767 mapwrite 0x01d1aa thru 0x01d9be (0x815 bytes) 000795 1735099338.301809 mapread 0x02c784 thru 0x02ebf8 (0x2475 bytes) 000796 1735099338.302400 punch from 0x027bfa to 0x02f16e (0x7575 bytes) 000797 1735099338.305263 mapwrite 0x010f40 thru 0x01ccfb (0xbdbc bytes) 000798 1735099338.324668 punch from 0x02f108 to 0x02f16e (0x67 bytes) 000799 1735099338.328943 read 0x02c79d thru 0x02f16e (0x29d2 bytes) 000800 1735099338.334697 trunc from 0x02f16f to 0x02fb02 (0x994 bytes) 000801 1735099338.345754 MIRROR_EXTEND to 3 mirrors 000802 1735099338.658388 mapread 0x00aa96 thru 0x0123ec (0x7957 bytes) 000803 1735099338.678662 mapwrite 0x035e9d thru 0x0395fa (0x375e bytes) 000806 1735099338.767880 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000807 1735099338.914535 trunc from 0x0395fb to 0x01f15f (0x1a49b bytes) 000808 1735099338.925588 mapwrite 0x00b770 thru 0x015ac7 (0xa358 bytes) 000809 1735099338.964146 MIRROR_EXTEND to 3 mirrors 000810 1735099339.351948 read 0x01026a thru 0x019f60 (0x9cf7 bytes) 000812 1735099339.368804 mapwrite 0x01c3da thru 0x025537 (0x915e bytes) 000813 1735099339.456253 punch from 0x018280 to 0x024fe9 (0xcd6a bytes) 000814 1735099339.466866 mapread 0x01c411 thru 0x01df6c (0x1b5c bytes) 000815 1735099339.486154 MIRROR_RESYNC 3 mirrors 000816 1735099339.895626 punch from 0x01ea01 to 0x025537 (0x6b37 bytes) 000817 1735099339.920010 mapread 0x01b468 thru 0x025537 (0xa0d0 bytes) 000818 1735099339.936039 read 0x0164b7 thru 0x021ce7 (0xb831 bytes) 000820 1735099339.936848 punch from 0x009b66 to 0x00b2f4 (0x178f bytes) 000821 1735099339.942425 MIRROR_EXTEND to 4 mirrors 000822 1735099340.247688 mapread 0x010977 thru 0x0205bb (0xfc45 bytes) 000824 1735099340.273087 read 0x00352a thru 0x01216e (0xec45 bytes) 000825 1735099340.282567 fallocate 0x02377e thru 0x02515d (0x19e0 bytes) 000826 1735099340.305705 punch from 0x019ac6 to 0x025537 (0xba72 bytes) 000827 1735099340.308647 fallocate 0x02c600 thru 0x033a03 (0x7404 bytes) 000828 1735099340.312867 trunc from 0x025538 to 0x01ac66 (0xa8d1 bytes) 000829 1735099340.329198 trunc from 0x01ac67 to 0x01273a (0x852c bytes) 000830 1735099340.347443 MIRROR_RESYNC 4 mirrors 000831 1735099340.924760 punch from 0x006b90 to 0x011c04 (0xb075 bytes) 000832 1735099340.950411 mapwrite 0x0121bb thru 0x0196d1 (0x7517 bytes) 000833 1735099340.989148 write 0x01fa72 thru 0x02d8fa (0xde89 bytes) 000836 1735099340.991012 mapwrite 0x02f2b1 thru 0x03566c (0x63bc bytes) 000837 1735099341.025322 punch from 0x02bb1c to 0x030957 (0x4e3c bytes) 000838 1735099341.030108 write 0x02d880 thru 0x034bc5 (0x7346 bytes) 000839 1735099341.038015 trunc from 0x03566d to 0x01c876 (0x18df6 bytes) 000840 1735099341.048092 trunc from 0x01c877 to 0x00d60c (0xf26a bytes) 000841 1735099341.061157 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000842 1735099341.210995 write 0x00ce49 thru 0x019c4e (0xce06 bytes) 000843 1735099341.218374 punch from 0x005228 to 0x00de1d (0x8bf6 bytes) 000844 1735099341.227040 write 0x00faa8 thru 0x01ac2f (0xb188 bytes) 000845 1735099341.230593 punch from 0x012989 to 0x014886 (0x1efe bytes) 000847 1735099341.247286 punch from 0x009b12 to 0x014123 (0xa612 bytes) 000848 1735099341.251835 write 0x02761f thru 0x028223 (0xc05 bytes) 000849 1735099341.252855 write 0x01d098 thru 0x02563f (0x85a8 bytes) 000851 1735099341.270565 fallocate 0x005087 thru 0x009522 (0x449c bytes) 000852 1735099341.279557 mapread 0x010381 thru 0x01f3bb (0xf03b bytes) 000854 1735099341.288547 read 0x008cfe thru 0x01862c (0xf92f bytes) 000855 1735099341.292417 read 0x00e70f thru 0x01a67c (0xbf6e bytes) 000857 1735099341.296733 MIRROR_EXTEND to 4 mirrors 000859 1735099341.672843 trunc from 0x028224 to 0x03be4e (0x13c2b bytes) 000860 1735099341.738142 trunc from 0x03be4f to 0x02e103 (0xdd4b bytes) 000861 1735099341.752757 mapread 0x00832e thru 0x011739 (0x940c bytes) 000863 1735099341.764787 trunc from 0x02e104 to 0x021cd2 (0xc431 bytes) 000864 1735099341.776202 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000865 1735099341.926234 read 0x010e8f thru 0x013d5a (0x2ecc bytes) 000867 1735099341.931725 punch from 0x0218a9 to 0x021cd2 (0x42a bytes) 000869 1735099341.936175 write 0x029582 thru 0x02ea60 (0x54df bytes) 000870 1735099341.943128 mapread 0x017be2 thru 0x01ebfe (0x701d bytes) 000871 1735099341.964977 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000873 1735099342.102938 MIRROR_RESYNC 2 mirrors 000874 1735099342.508049 read 0x00079a thru 0x0010a3 (0x90a bytes) 000875 1735099342.513616 trunc from 0x02ea61 to 0x017667 (0x173f9 bytes) 000876 1735099342.547422 mapwrite 0x012861 thru 0x018d17 (0x64b7 bytes) 000877 1735099342.588185 mapread 0x012ebc thru 0x0169cd (0x3b12 bytes) 000878 1735099342.589253 write 0x02e51b thru 0x038b67 (0xa64d bytes) 000879 1735099342.590879 read 0x00688e thru 0x01594d (0xf0c0 bytes) 000880 1735099342.600038 mapread 0x0023f6 thru 0x00dc72 (0xb87d bytes) 000881 1735099342.602475 fallocate 0x02bf4b thru 0x03978c (0xd842 bytes) 000882 1735099342.606868 write 0x0205ec thru 0x021342 (0xd57 bytes) 000883 1735099342.611641 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000885 1735099342.764941 trunc from 0x03978d to 0x00cdeb (0x2c9a1 bytes) 000886 1735099342.779669 write 0x0074c6 thru 0x00dcc3 (0x67fe bytes) 000888 1735099342.783241 punch from 0x00ae57 to 0x00dcc3 (0x2e6d bytes) 000889 1735099342.791507 write 0x007cc8 thru 0x012b23 (0xae5c bytes) 000891 1735099342.792954 trunc from 0x012b24 to 0x02f5cf (0x1caac bytes) 000892 1735099342.809746 trunc from 0x02f5d0 to 0x00dae2 (0x21aed bytes) 000893 1735099342.821001 mapwrite 0x0316ff thru 0x036e29 (0x572b bytes) 000894 1735099342.861013 write 0x02a725 thru 0x033353 (0x8c2f bytes) 000895 1735099342.870787 punch from 0x011e40 to 0x0178ac (0x5a6d bytes) 000897 1735099342.875077 mapread 0x024d26 thru 0x03482c (0xfb07 bytes) 000898 1735099342.875761 trunc from 0x036e2a to 0x027226 (0xfc03 bytes) 000902 1735099342.892297 read 0x00f665 thru 0x0162b9 (0x6c55 bytes) 000903 1735099342.898064 punch from 0x021437 to 0x024375 (0x2f3f bytes) 000904 1735099342.902751 write 0x00e150 thru 0x010287 (0x2138 bytes) 000907 1735099342.908021 write 0x000f93 thru 0x0043fd (0x346b bytes) 000909 1735099342.913463 fallocate 0x01bcbe thru 0x0237f7 (0x7b3a bytes) 000911 1735099342.917011 mapwrite 0x02b329 thru 0x032f50 (0x7c28 bytes) 000912 1735099342.958025 mapread 0x02ad27 thru 0x02e1ae (0x3488 bytes) 000913 1735099342.960797 read 0x02e5ef thru 0x03097c (0x238e bytes) 000914 1735099342.961211 write 0x00d3d6 thru 0x014289 (0x6eb4 bytes) 000915 1735099342.965132 write 0x02b045 thru 0x033e86 (0x8e42 bytes) 000916 1735099342.967715 trunc from 0x033e87 to 0x02b2ee (0x8b98 bytes) 000917 1735099342.984873 write 0x02c85d thru 0x03291d (0x60c1 bytes) 000918 1735099342.987573 write 0x01256f thru 0x016976 (0x4408 bytes) 000920 1735099342.991695 mapwrite 0x00cd29 thru 0x011cd0 (0x4fa8 bytes) 000921 1735099343.028202 MIRROR_EXTEND to 2 mirrors 000922 1735099343.443604 write 0x03f836 thru 0x03ffff (0x7ca bytes) 000923 1735099343.477097 punch from 0x00e8c5 to 0x01a0e2 (0xb81e bytes) 000924 1735099343.481228 read 0x004729 thru 0x009ec8 (0x57a0 bytes) 000925 1735099343.485839 fallocate 0x03e7fe thru 0x03ffff (0x1802 bytes) 000927 1735099343.499364 MIRROR_EXTEND to 3 mirrors 000928 1735099343.821327 trunc from 0x040000 to 0x03a063 (0x5f9c bytes) 000929 1735099343.891850 fallocate 0x0069ba thru 0x00945d (0x2aa4 bytes) 000930 1735099343.896206 mapread 0x01ad2a thru 0x024217 (0x94ee bytes) 000931 1735099343.923727 MIRROR_EXTEND to 4 mirrors 000932 1735099344.357249 write 0x00e922 thru 0x0161c0 (0x789f bytes) 000933 1735099344.402751 read 0x028965 thru 0x035761 (0xcdfd bytes) 000934 1735099344.413701 trunc from 0x03a064 to 0x0120d6 (0x27f8d bytes) 000935 1735099344.437401 mapread 0x011de8 thru 0x0120d6 (0x2ef bytes) 000936 1735099344.438152 mapwrite 0x006b06 thru 0x00a441 (0x393c bytes) 000937 1735099344.461604 mapwrite 0x011e41 thru 0x01609f (0x425f bytes) 000938 1735099344.502411 punch from 0x00a19a to 0x01609f (0xbf06 bytes) 000939 1735099344.517533 MIRROR_EXTEND to 5 mirrors 000941 1735099344.847024 write 0x03ef8e thru 0x03ffff (0x1072 bytes) 000942 1735099344.894963 write 0x0339d7 thru 0x03f7bc (0xbde6 bytes) 000943 1735099344.899375 fallocate 0x00dd3e thru 0x01020f (0x24d2 bytes) 000944 1735099344.902581 read 0x025bca thru 0x0272c1 (0x16f8 bytes) 000946 1735099344.905769 fallocate 0x039db2 thru 0x03ffff (0x624e bytes) 000947 1735099344.911039 trunc from 0x040000 to 0x02635a (0x19ca5 bytes) 000948 1735099344.953026 fallocate 0x015aab thru 0x01bb81 (0x60d7 bytes) 000950 1735099344.956869 read 0x012f5e thru 0x01ec02 (0xbca5 bytes) 000951 1735099344.964655 mapread 0x00d069 thru 0x017b9e (0xab36 bytes) 000952 1735099344.968486 MIRROR_EXTEND to 6 mirrors 000953 1735099345.412186 mapread 0x01335f thru 0x0228b0 (0xf552 bytes) 000954 1735099345.445849 punch from 0x01cf8d to 0x01e982 (0x19f6 bytes) 000955 1735099345.479597 mapread 0x015bb0 thru 0x0206f8 (0xab49 bytes) 000956 1735099345.509731 MIRROR_EXTEND to 7 mirrors 000957 1735099345.908563 punch from 0x00cd97 to 0x0104d6 (0x3740 bytes) 000958 1735099345.947008 mapread 0x00849a thru 0x00a4f1 (0x2058 bytes) 000959 1735099345.962640 mapread 0x002731 thru 0x01063c (0xdf0c bytes) 000960 1735099345.964087 fallocate 0x02359f thru 0x02c4ea (0x8f4c bytes) 000961 1735099345.968447 punch from 0x02038f to 0x02635a (0x5fcc bytes) 000962 1735099345.976022 punch from 0x01dcff to 0x025ad5 (0x7dd7 bytes) 000964 1735099345.981524 punch from 0x022189 to 0x02635a (0x41d2 bytes) 000965 1735099345.984687 punch from 0x0144bb to 0x0193a5 (0x4eeb bytes) 000966 1735099345.992445 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000967 1735099346.126818 write 0x028482 thru 0x0333dc (0xaf5b bytes) 000968 1735099346.136201 MIRROR_EXTEND to 7 mirrors 000969 1735099346.626319 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x924:0x0]: Invalid argument (22) 000970 1735099346.846940 mapwrite 0x02908a thru 0x038e76 (0xfded bytes) 000971 1735099346.918647 trunc from 0x038e77 to 0x00fee5 (0x28f91 bytes) 000972 1735099346.929227 punch from 0x00d877 to 0x00e2ce (0xa58 bytes) 000973 1735099346.932513 mapread 0x007d9a thru 0x00fee5 (0x814c bytes) 000974 1735099346.939675 write 0x016af2 thru 0x01744a (0x959 bytes) 000975 1735099346.940380 trunc from 0x01744b to 0x03914c (0x21d02 bytes) 000976 1735099346.948491 trunc from 0x03914d to 0x02963b (0xfb11 bytes) 000977 1735099346.957352 write 0x0166da thru 0x01c709 (0x6030 bytes) 000978 1735099346.961955 mapwrite 0x012cb8 thru 0x01a477 (0x77c0 bytes) 000979 1735099346.979400 fallocate 0x01a16b thru 0x025571 (0xb407 bytes) 000980 1735099346.981932 mapwrite 0x013ca3 thru 0x01449a (0x7f8 bytes) 000981 1735099346.988329 mapread 0x0054a3 thru 0x006767 (0x12c5 bytes) 000982 1735099346.989137 fallocate 0x02d7f0 thru 0x031157 (0x3968 bytes) 000983 1735099346.992419 write 0x011b03 thru 0x01df3e (0xc43c bytes) 000984 1735099346.996179 mapread 0x014c9f thru 0x0200b6 (0xb418 bytes) 000985 1735099347.001178 mapwrite 0x017d72 thru 0x025a9a (0xdd29 bytes) 000986 1735099347.014198 read 0x00617f thru 0x015abd (0xf93f bytes) 000987 1735099347.014689 fallocate 0x00ae94 thru 0x0129d8 (0x7b45 bytes) 000988 1735099347.017955 write 0x0154b7 thru 0x0234db (0xe025 bytes) 000989 1735099347.018857 trunc from 0x02963c to 0x001977 (0x27cc4 bytes) 000990 1735099347.033403 punch from 0x000f8b to 0x001977 (0x9ed bytes) 000991 1735099347.041151 MIRROR_EXTEND to 7 mirrors 000992 1735099347.440550 fallocate 0x016eb4 thru 0x021a34 (0xab81 bytes) 000993 1735099347.476792 write 0x02a497 thru 0x02ec68 (0x47d2 bytes) 000994 1735099347.482159 write 0x0062ea thru 0x00978c (0x34a3 bytes) 000995 1735099347.490450 punch from 0x02baf5 to 0x02ec68 (0x3174 bytes) 000996 1735099347.498439 mapread 0x00a105 thru 0x00b23d (0x1139 bytes) 000997 1735099347.506179 punch from 0x01448d to 0x02074f (0xc2c3 bytes) 000998 1735099347.510077 mapread 0x005a71 thru 0x010d55 (0xb2e5 bytes) 000999 1735099347.510928 fallocate 0x03efc8 thru 0x03ffff (0x1038 bytes) 001000 1735099347.515355 trunc from 0x040000 to 0x011346 (0x2ecb9 bytes) All operations completed A-OK! PASS 70a (24s) == sanity-flr test 200a: stress test ===================== 23:02:31 (1735099351) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4151808 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8545792 .. Extending file size to 5802816 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6673344 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6925152 .. Extending file size to 6931264 .. Extending file size to 8245280 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9130304 .. Extending file size to 9255264 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9196736 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 7228 7230 7232 7233 7234 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) 192.168.201.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:03:38 (1735099418) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8482560 .. Extending file size to 5939648 .. Extending file size to 6822784 .. Extending file size to 8803840 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7673920 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8423840 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8528992 .. Extending file size to 8700256 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8741920 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9018688 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9084608 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9388448 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 9376 9377 9379 9380 9382 done 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) 192.168.201.134@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:04:44 (1735099484) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3112 1284576 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23072 3583948 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15004 3592016 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38076 7175964 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3112 1284576 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23072 3583948 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15004 3592016 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38076 7175964 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0378051 s, 55.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0593214 s, 35.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs mirror verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:04:49 (1735099489) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0301361 s, 69.6 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 23:04:51 (1735099491) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:04:53 (1735099493) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0334346 s, 62.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xe3a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xe3a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xe3b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:04:55 (1735099495) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0287042 s, 73.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:04:58 (1735099498) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0281614 s, 74.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:05:21 (1735099521) Waiting for MDT destroys to complete oleg134-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg134-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0277754 s, 37.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xe3b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xe3d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xe3c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xe3b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xe3d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xe3c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xe3c:0x0] } PASS 204d (38s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:05:59 (1735099559) Waiting for MDT destroys to complete oleg134-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg134-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0267374 s, 78.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:06:34 (1735099594) Waiting for MDT destroys to complete oleg134-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg134-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg134-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg134-server: Pool lustre.test_204f created Adding targets to pool oleg134-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg134-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.031116 s, 67.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xe3e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xe3d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xe3e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xe3d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg134-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg134-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg134-server: Pool lustre.test_204f destroyed PASS 204f (46s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:07:20 (1735099640) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:07:22 (1735099642) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:07:24 (1735099644) oleg134-server: Pool lustre.test_206 created Adding targets to pool oleg134-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg134-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg134-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg134-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg134-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:07:37 (1735099657) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228298 s, 45.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108377 s, 96.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x7a26703a Mirror 3: 0x7a26703a PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:07:39 (1735099659) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:07:40 (1735099660) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:07:41 (1735099661) Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0135104 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x21d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xde0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xde0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xde0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1730000 Read this (4 bytes): BBBB 192.168.201.134@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg134-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:07:56 (1735099676) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00475496 s, 1.7 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:07:58 (1735099678) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116907 s, 89.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-12-24 23:07:58.000000000 -0500 Modify: 2024-12-24 23:07:58.000000000 -0500 Change: 2024-12-24 23:07:59.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:08:00 (1735099680) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158152 s, 66.3 MB/s oleg134-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg134-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg134-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg134-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 oleg134-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg134-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg134-server: O/280000401/d27/bb: File not found by ext2_lookup PASS 210b (9s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:08:10 (1735099690) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0299055 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00524393 s, 781 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 24 23:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 24 23:08 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 786 sec ============ 23:08:11 (1735099691) === sanity-flr: start cleanup 23:08:12 (1735099692) === === sanity-flr: finish cleanup 23:08:14 (1735099694) ===