-----============= acceptance-small: sanity-flr ============----- Tue Jun 10 21:31:25 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg114-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:31:30 (1749605490) === oleg114-client.virtnet: executing check_config_client /mnt/lustre oleg114-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg114-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800ad47b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800ad47b800.idle_timeout=debug disable quota as required oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:31:39 (1749605499) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7911] == sanity-flr test 0a: lfs mirror create with -N option == 21:31:40 (1749605500) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:31:43 (1749605503) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (0s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:31:44 (1749605504) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:31:44 (1749605504) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:31:49 (1749605509) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:31:50 (1749605510) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 21:31:51 (1749605511) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800ad47b800.stats=clear osc.lustre-OST0001-osc-ffff8800ad47b800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:31:53 (1749605513) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:31:55 (1749605515) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:31:57 (1749605517) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:31:58 (1749605518) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 21:32:00 (1749605520) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:32:02 (1749605522) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:32:04 (1749605524) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:32:07 (1749605527) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:32:10 (1749605530) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:32:12 (1749605532) 11+0 records in 11+0 records out 11534336 bytes (12 MB) copied, 0.292547 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0389393 s, 26.9 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:32:14 (1749605534) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0319326 s, 164 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:32:16 (1749605536) Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800ad47b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ad47b800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800ad47b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ad47b800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:32:40 (1749605560) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.037335 s, 56.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800ad47b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ad47b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (10s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:32:50 (1749605570) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800ad47b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ad47b800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800ad47b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ad47b800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:33:10 (1749605590) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.686015 s, 76.4 MB/s -rw-r--r-- 1 root root 52428800 Jun 10 21:33 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.192s user 0m0.000s sys 0m0.101s Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800ad47b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ad47b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.227s user 0m0.001s sys 0m0.113s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800ad47b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ad47b800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.218s user 0m0.000s sys 0m0.103s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800ad47b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800ad47b800.ost_server_uuid in FULL state after 0 sec PASS 33b (22s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:33:32 (1749605612) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:33:33 (1749605613) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:33:34 (1749605614) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 21:33:35 (1749605615) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 21:33:36 (1749605616) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.128402 s, 98.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:33:39 (1749605619) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00759257 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.007175 s, 146 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:33:41 (1749605621) PASS 36c (2s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:33:43 (1749605623) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000432401 s, 2.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000396007 s, 2.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00050023 s, 2.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000611459 s, 1.7 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 21:33:45 (1749605625) create test files with size 15743088 .. 800fadab702bc9376653a6c7aa52b67c - 2162e2da6913fc360d7ed2ee9b1299bb - 04f747a7297d7eea3ff5fd8bf6741c78 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 21:33:49 (1749605629) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:33:51 (1749605631) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:33:53 (1749605633) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.022448 s, 46.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154669 s, 67.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 21:33:55 (1749605635) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0319174 s, 65.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.113201 s, 37.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0508607 s, 41.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 21:33:58 (1749605638) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 21:33:59 (1749605639) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:34:00 (1749605640) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00770901 s, 136 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 21:34:01 (1749605641) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0727031 s, 43.3 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:34:03 (1749605643) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x45:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x45:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:34:05 (1749605645) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0638509 s, 164 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:34:28 (1749605668) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88013525d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0753353 s, 139 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800b6ea7000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:34:31 (1749605671) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:34:37 (1749605677) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (5s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:34:42 (1749605682) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 21:34:43 (1749605683) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0615177 s, 51.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0398456 s, 78.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 8fd54a4b9c8d87af4f295cd7d38a695c - mirror 1 checksum: 4db9a329f64e17e8a8c4325459eb80df - mirror 2 checksum: 8fd54a4b9c8d87af4f295cd7d38a695c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 10 21:34:44 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Jun 10 21:34:44 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8fd54a4b9c8d87af4f295cd7d38a695c - mirror 1 checksum: 4db9a329f64e17e8a8c4325459eb80df - mirror 2 checksum: 4db9a329f64e17e8a8c4325459eb80df - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 10 21:34:44 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Jun 10 21:34:44 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:34:45 (1749605685) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00785594 s, 133 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (1s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:34:46 (1749605686) Failing mds1 on oleg114-server Stopping /mnt/lustre-mds1 (opts:) on oleg114-server 21:34:54 (1749605694) shut down facet: mds1 facet_host: oleg114-server facet_failover_host: oleg114-server Failover mds1 to oleg114-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 21:35:07 (1749605707) targets are mounted 21:35:07 (1749605707) facet_failover done oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162096 s, 64.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:35:15 (1749605715) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00719267 s, 2.8 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00350645 s, 5.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:35:18 (1749605718) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.015468 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00477793 s, 4.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:35:21 (1749605721) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00179261 s, 9.1 MB/s PASS 50c (1s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:35:23 (1749605723) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0140035 s, 1.5 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:35:25 (1749605725) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00290047 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0125962 s, 325 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:35:27 (1749605727) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (4s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:35:31 (1749605731) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:35:49 (1749605749) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:36:06 (1749605766) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:36:23 (1749605783) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 21:36:24 (1749605784) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 26650 mirror split pid 26651 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26650 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26651 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 21:37:26 (1749605846) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 5846 skipping zero length zero range truncating to largest ever: 0x30093 000003 1749605846.955898 trunc from 00000000 to 0x030092 (0x30093 bytes) 000004 1749605846.959341 write 0x030ff8 thru 0x0384f7 (0x7500 bytes) fallocating to largest ever: 0x40000 000005 1749605846.961340 fallocate 0x03f9ac thru 0x03ffff (0x654 bytes) 000006 1749605846.963023 punch from 0x030731 to 0x035887 (0x5157 bytes) 000008 1749605846.966730 punch from 0x00771b to 0x009b89 (0x246f bytes) 000010 1749605846.968308 write 0x01e207 thru 0x02a1de (0xbfd8 bytes) 000011 1749605846.971027 trunc from 0x028d76 to 0x0384f7 (0xf782 bytes) 000013 1749605846.974385 mapread 0x0059f4 thru 0x005c4c (0x259 bytes) 000014 1749605846.977099 trunc from 0x0384f8 to 0x038e34 (0x93d bytes) 000015 1749605846.980664 write 0x029263 thru 0x02a992 (0x1730 bytes) 000016 1749605846.982825 fallocate 0x00b2e3 thru 0x00fc31 (0x494f bytes) 000017 1749605846.984257 trunc from 0x038e35 to 0x0002cf (0x38b65 bytes) 000018 1749605846.987972 mapread 0x0000a2 thru 0x0002cf (0x22e bytes) 000019 1749605846.989449 mapread 0x00002c thru 0x0002cf (0x2a4 bytes) 000020 1749605846.989899 mapread 0x00025e thru 0x0002cf (0x72 bytes) 000021 1749605846.990250 trunc from 0x0002d0 to 0x03b2c5 (0x3aff6 bytes) 000022 1749605846.993428 read 0x00ff9b thru 0x0193a6 (0x940c bytes) 000023 1749605846.995479 mapread 0x036528 thru 0x03b2c5 (0x4d9e bytes) 000024 1749605846.998041 mapread 0x011a81 thru 0x0214d6 (0xfa56 bytes) 000025 1749605847.000735 mapread 0x0102fc thru 0x01d3f2 (0xd0f7 bytes) 000026 1749605847.001398 fallocate 0x019e97 thru 0x027262 (0xd3cc bytes) 000027 1749605847.004074 MIRROR_EXTEND to 2 mirrors 000028 1749605847.098217 trunc from 0x01cf79 to 0x02a409 (0xd491 bytes) 000029 1749605847.106065 punch from 0x0307cb to 0x03b2c5 (0xaafb bytes) 000030 1749605847.107370 mapwrite 0x005a97 thru 0x00d555 (0x7abf bytes) 000031 1749605847.117100 trunc from 0x03b2c6 to 0x006da8 (0x3451d bytes) 000033 1749605847.125392 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000034 1749605847.154878 fallocate 0x01c7b4 thru 0x020345 (0x3b92 bytes) 000035 1749605847.156638 trunc from 0x020346 to 0x03b709 (0x1b3c4 bytes) 000036 1749605847.161343 punch from 0x00babd to 0x01499b (0x8edf bytes) 000037 1749605847.162629 fallocate 0x001f97 thru 0x006402 (0x446c bytes) 000038 1749605847.164096 punch from 0x009724 to 0x014dbb (0xb698 bytes) 000039 1749605847.165394 read 0x009ad5 thru 0x00a6f5 (0xc21 bytes) 000040 1749605847.167125 trunc from 0x0034fa to 0x00a53f (0x7046 bytes) 000041 1749605847.168707 fallocate 0x027d2e thru 0x02d36e (0x5641 bytes) 000042 1749605847.170296 fallocate 0x008a7e thru 0x017f37 (0xf4ba bytes) 000043 1749605847.171730 mapread 0x0346a1 thru 0x03b709 (0x7069 bytes) 000045 1749605847.176468 read 0x026d55 thru 0x02f6f3 (0x899f bytes) 000046 1749605847.179165 fallocate 0x006b5d thru 0x006cad (0x151 bytes) 000047 1749605847.181622 MIRROR_EXTEND to 2 mirrors 000048 1749605847.270627 write 0x01c19e thru 0x01df58 (0x1dbb bytes) 000049 1749605847.279545 write 0x0380a5 thru 0x03e093 (0x5fef bytes) 000051 1749605847.281882 read 0x02ebea thru 0x030d57 (0x216e bytes) 000052 1749605847.283274 mapwrite 0x0358bf thru 0x03ffff (0xa741 bytes) 000053 1749605847.297251 trunc from 0x040000 to 0x0375d7 (0x8a28 bytes) 000055 1749605847.300725 punch from 0x02bf19 to 0x0375d7 (0xb6bf bytes) 000056 1749605847.302320 trunc from 0x018785 to 0x01a89c (0x2118 bytes) 000057 1749605847.303964 trunc from 0x005a5e to 0x00737f (0x1922 bytes) 000058 1749605847.306786 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000059 1749605847.336114 read 0x011cb8 thru 0x01f0e1 (0xd42a bytes) 000060 1749605847.338767 mapwrite 0x0341df thru 0x03b14f (0x6f71 bytes) 000062 1749605847.352066 fallocate 0x00006b thru 0x008a64 (0x89fa bytes) 000063 1749605847.353487 fallocate 0x02aa57 thru 0x030937 (0x5ee1 bytes) 000065 1749605847.355096 punch from 0x023dcc to 0x02e75d (0xa992 bytes) 000066 1749605847.356327 trunc from 0x03b150 to 0x030839 (0xa916 bytes) 000067 1749605847.360064 mapwrite 0x0059f5 thru 0x0136f0 (0xdcfc bytes) 000068 1749605847.368859 trunc from 0x019e37 to 0x01c4d3 (0x269d bytes) 000069 1749605847.370251 punch from 0x0101be to 0x012f8e (0x2dd1 bytes) 000070 1749605847.371630 mapread 0x02abd3 thru 0x030839 (0x5c67 bytes) 000071 1749605847.373899 punch from 0x02415f to 0x02ab87 (0x6a29 bytes) 000072 1749605847.375223 mapread 0x008a59 thru 0x015cf8 (0xd2a0 bytes) 000074 1749605847.377766 trunc from 0x03083a to 0x009844 (0x26ff5 bytes) 000075 1749605847.381554 trunc from 0x005d02 to 0x009844 (0x3b43 bytes) 000076 1749605847.382947 punch from 0x00840b to 0x009844 (0x143a bytes) 000078 1749605847.384534 trunc from 0x009845 to 0x02b02e (0x217ea bytes) 000079 1749605847.388312 trunc from 0x017f4a to 0x020928 (0x89df bytes) 000082 1749605847.390604 fallocate 0x03feba thru 0x03ffff (0x146 bytes) 000083 1749605847.392180 mapwrite 0x00ae73 thru 0x011b0e (0x6c9c bytes) 000084 1749605847.399994 mapread 0x0281f3 thru 0x02b02e (0x2e3c bytes) 000085 1749605847.402549 fallocate 0x02756d thru 0x02ce64 (0x58f8 bytes) 000086 1749605847.404051 write 0x030c9c thru 0x03fcb3 (0xf018 bytes) 000088 1749605847.405194 mapwrite 0x003ad9 thru 0x00c168 (0x8690 bytes) 000089 1749605847.411289 trunc from 0x03fcb4 to 0x02d693 (0x12620 bytes) 000090 1749605847.415118 read 0x004aed thru 0x00dfd1 (0x94e5 bytes) 000091 1749605847.415592 trunc from 0x02d694 to 0x018f0c (0x14787 bytes) 000092 1749605847.419471 mapwrite 0x001a92 thru 0x00b0c7 (0x9636 bytes) 000093 1749605847.426280 trunc from 0x018f0d to 0x011677 (0x7895 bytes) 000094 1749605847.430082 punch from 0x002216 to 0x0107ff (0xe5ea bytes) 000095 1749605847.431520 write 0x014f73 thru 0x015ccb (0xd59 bytes) 000096 1749605847.432032 fallocate 0x001a0c thru 0x00e3cf (0xc9c4 bytes) 000097 1749605847.433678 punch from 0x00b1f0 to 0x011c15 (0x6a26 bytes) 000098 1749605847.434915 trunc from 0x015ccc to 0x0116bf (0x460c bytes) 000099 1749605847.438311 punch from 0x000e36 to 0x0012d6 (0x4a1 bytes) 000100 1749605847.439660 trunc from 0x002c33 to 0x0066c0 (0x3a8e bytes) 000101 1749605847.440922 trunc from 0x0116c0 to 0x01b565 (0x9ea6 bytes) 000102 1749605847.444500 trunc from 0x01a62b to 0x01b565 (0xf3b bytes) 000103 1749605847.445878 trunc from 0x01b566 to 0x008ada (0x12a8b bytes) 000105 1749605847.449938 write 0x010921 thru 0x01134d (0xa2d bytes) 000106 1749605847.450454 mapread 0x00e17c thru 0x01134d (0x31d2 bytes) 000108 1749605847.453196 mapread 0x005aa4 thru 0x01134d (0xb8aa bytes) 000109 1749605847.453604 mapread 0x00126b thru 0x007f20 (0x6cb6 bytes) 000110 1749605847.454020 fallocate 0x003c5f thru 0x00509b (0x143d bytes) 000111 1749605847.455447 trunc from 0x01134e to 0x009463 (0x7eea bytes) 000112 1749605847.458960 mapread 0x008d3c thru 0x009463 (0x728 bytes) 000113 1749605847.459366 punch from 0x008581 to 0x009463 (0xee3 bytes) 000114 1749605847.460782 write 0x00d005 thru 0x0197ec (0xc7e8 bytes) 000115 1749605847.461673 fallocate 0x012fad thru 0x017043 (0x4097 bytes) 000116 1749605847.463020 write 0x0041bd thru 0x0043b2 (0x1f6 bytes) 000117 1749605847.463381 mapwrite 0x0340df thru 0x0357e4 (0x1706 bytes) 000118 1749605847.475034 trunc from 0x0357e5 to 0x016064 (0x1f780 bytes) 000119 1749605847.478953 fallocate 0x00cef2 thru 0x010bc0 (0x3ccf bytes) 000120 1749605847.480336 read 0x008197 thru 0x0154e9 (0xd353 bytes) 000121 1749605847.482503 trunc from 0x016065 to 0x0004a4 (0x15bc0 bytes) 000122 1749605847.486121 punch from 0x0002d4 to 0x0004a4 (0x1d1 bytes) 000123 1749605847.487487 mapread 0x000115 thru 0x0004a4 (0x390 bytes) 000124 1749605847.489356 fallocate 0x018050 thru 0x021c48 (0x9bf9 bytes) 000125 1749605847.492289 MIRROR_EXTEND to 2 mirrors 000127 1749605847.556498 fallocate 0x036f2b thru 0x03ffff (0x90d5 bytes) 000128 1749605847.563663 trunc from 0x0004a5 to 0x039664 (0x391c0 bytes) 000129 1749605847.571746 mapwrite 0x039ad2 thru 0x03af4c (0x147b bytes) 000130 1749605847.581085 trunc from 0x03af4d to 0x00db7a (0x2d3d2 bytes) 000131 1749605847.587113 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000132 1749605847.615231 trunc from 0x00cc6f to 0x00db7a (0xf0c bytes) 000133 1749605847.616761 mapread 0x00c206 thru 0x00db7a (0x1975 bytes) 000135 1749605847.619495 trunc from 0x00db7b to 0x0377b1 (0x29c37 bytes) 000136 1749605847.623713 fallocate 0x02aeb5 thru 0x02eefc (0x4048 bytes) 000137 1749605847.626096 trunc from 0x0377b2 to 0x031918 (0x5e99 bytes) 000138 1749605847.629597 fallocate 0x0219b9 thru 0x02f40d (0xda55 bytes) 000140 1749605847.631447 trunc from 0x028992 to 0x030d67 (0x83d6 bytes) 000141 1749605847.632965 read 0x01c9be thru 0x02bce7 (0xf32a bytes) 000142 1749605847.635345 mapread 0x01671d thru 0x0245b2 (0xde96 bytes) 000143 1749605847.637850 mapread 0x0152c2 thru 0x01d88b (0x85ca bytes) 000145 1749605847.638921 trunc from 0x031919 to 0x02509e (0xc87a bytes) 000146 1749605847.642360 trunc from 0x02509f to 0x02bda8 (0x6d0a bytes) 000147 1749605847.646096 fallocate 0x0080f7 thru 0x0177b0 (0xf6ba bytes) 000148 1749605847.647855 trunc from 0x02611d to 0x02bda8 (0x5c8c bytes) 000149 1749605847.649132 write 0x0153dc thru 0x023bed (0xe812 bytes) 000150 1749605847.650162 read 0x024df1 thru 0x02b957 (0x6b67 bytes) 000151 1749605847.652617 mapwrite 0x03516a thru 0x03ffff (0xae96 bytes) 000152 1749605847.664212 write 0x030044 thru 0x03a420 (0xa3dd bytes) 000153 1749605847.665213 read 0x02d286 thru 0x02f48c (0x2207 bytes) 000154 1749605847.665627 punch from 0x01d105 to 0x01e917 (0x1813 bytes) 000157 1749605847.667705 mapwrite 0x005e33 thru 0x010d21 (0xaeef bytes) 000158 1749605847.673820 punch from 0x02760b to 0x033c1f (0xc615 bytes) 000159 1749605847.675290 trunc from 0x040000 to 0x020f0f (0x1f0f0 bytes) 000161 1749605847.678845 trunc from 0x018ce5 to 0x01acfb (0x2017 bytes) 000162 1749605847.680206 trunc from 0x020f10 to 0x0211a5 (0x296 bytes) 000163 1749605847.683440 punch from 0x017fe4 to 0x01f3a0 (0x73bd bytes) 000164 1749605847.684761 trunc from 0x017f49 to 0x0211a5 (0x925d bytes) 000165 1749605847.686077 write 0x00af89 thru 0x00afdb (0x53 bytes) 000167 1749605847.686677 punch from 0x00a9f0 to 0x0188f7 (0xdf08 bytes) 000169 1749605847.689533 mapwrite 0x007b84 thru 0x007e65 (0x2e2 bytes) 000170 1749605847.693476 write 0x013d0b thru 0x0171ec (0x34e2 bytes) 000171 1749605847.696415 mapwrite 0x02db5d thru 0x0392b3 (0xb757 bytes) 000172 1749605847.707379 mapread 0x036390 thru 0x0392b3 (0x2f24 bytes) 000174 1749605847.708085 write 0x011272 thru 0x013446 (0x21d5 bytes) 000175 1749605847.708582 mapread 0x0068b7 thru 0x009a0f (0x3159 bytes) 000176 1749605847.709055 fallocate 0x00b44e thru 0x01682b (0xb3de bytes) 000177 1749605847.710621 fallocate 0x039477 thru 0x03ffff (0x6b89 bytes) 000178 1749605847.712136 mapread 0x03c562 thru 0x03ffff (0x3a9e bytes) 000179 1749605847.714176 punch from 0x026315 to 0x030848 (0xa534 bytes) 000180 1749605847.715601 read 0x02d29d thru 0x02e02d (0xd91 bytes) 000181 1749605847.717678 mapread 0x01c119 thru 0x024a3a (0x8922 bytes) 000182 1749605847.718348 fallocate 0x019388 thru 0x0210b2 (0x7d2b bytes) 000184 1749605847.720031 mapread 0x02abc4 thru 0x03a5fa (0xfa37 bytes) 000185 1749605847.720558 punch from 0x01271f to 0x016013 (0x38f5 bytes) 000186 1749605847.723213 read 0x039298 thru 0x03ffff (0x6d68 bytes) 000187 1749605847.723531 fallocate 0x01366e thru 0x017551 (0x3ee4 bytes) 000188 1749605847.724982 trunc from 0x019824 to 0x027d05 (0xe4e2 bytes) 000189 1749605847.726377 mapread 0x01aecf thru 0x023e67 (0x8f99 bytes) 000190 1749605847.728863 write 0x032d92 thru 0x03ffff (0xd26e bytes) 000191 1749605847.729628 trunc from 0x03d5a9 to 0x03ffff (0x2a57 bytes) 000192 1749605847.732811 mapwrite 0x0234fd thru 0x02572f (0x2233 bytes) 000193 1749605847.736194 punch from 0x038548 to 0x03ffff (0x7ab8 bytes) 000195 1749605847.738054 punch from 0x028b73 to 0x0370f8 (0xe586 bytes) 000196 1749605847.739552 fallocate 0x01c861 thru 0x026719 (0x9eb9 bytes) 000198 1749605847.742468 MIRROR_EXTEND to 2 mirrors 000200 1749605847.836561 write 0x00e623 thru 0x0135c4 (0x4fa2 bytes) 000201 1749605847.846431 trunc from 0x040000 to 0x02d29f (0x12d60 bytes) 000202 1749605847.855045 fallocate 0x02c4ae thru 0x031cb6 (0x5809 bytes) 000203 1749605847.857740 MIRROR_EXTEND to 3 mirrors 000204 1749605847.959915 fallocate 0x00d2c7 thru 0x01a20a (0xcf44 bytes) 000205 1749605847.967270 read 0x0052e1 thru 0x0103db (0xb0fb bytes) 000206 1749605847.969340 punch from 0x000f50 to 0x010129 (0xf1da bytes) 000207 1749605847.970984 read 0x01b9c7 thru 0x0266a5 (0xacdf bytes) 000208 1749605847.973843 trunc from 0x016d89 to 0x01cd47 (0x5fbf bytes) 000210 1749605847.975571 trunc from 0x018b31 to 0x024351 (0xb821 bytes) 000211 1749605847.977056 punch from 0x000603 to 0x004e94 (0x4892 bytes) 000212 1749605847.978346 write 0x0167f2 thru 0x01d56d (0x6d7c bytes) 000214 1749605847.982056 mapread 0x02f55b thru 0x02fdcf (0x875 bytes) 000216 1749605847.982917 trunc from 0x031cb7 to 0x0031f5 (0x2eac1 bytes) 000217 1749605847.990490 mapwrite 0x0310ef thru 0x03ffff (0xef11 bytes) 000218 1749605848.002067 mapread 0x02f84a thru 0x038f2e (0x96e5 bytes) 000219 1749605848.004575 mapread 0x007674 thru 0x00afa3 (0x3930 bytes) 000220 1749605848.009007 read 0x01e9ca thru 0x029443 (0xaa7a bytes) 000221 1749605848.011811 mapread 0x03543a thru 0x03fa11 (0xa5d8 bytes) 000222 1749605848.012246 trunc from 0x040000 to 0x027991 (0x1866e bytes) 000223 1749605848.015557 trunc from 0x027992 to 0x0035a0 (0x243f1 bytes) 000224 1749605848.019124 fallocate 0x03c6b3 thru 0x03ffff (0x394d bytes) 000225 1749605848.021900 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000227 1749605848.062587 read 0x0020a5 thru 0x0035a0 (0x14fc bytes) 000228 1749605848.064413 mapread 0x0032ae thru 0x0035a0 (0x2f3 bytes) 000229 1749605848.064970 fallocate 0x00b422 thru 0x018867 (0xd446 bytes) 000230 1749605848.066540 read 0x001cfd thru 0x0035a0 (0x18a4 bytes) 000232 1749605848.068277 trunc from 0x0035a1 to 0x03e63f (0x3b09f bytes) 000233 1749605848.073218 mapread 0x017925 thru 0x020321 (0x89fd bytes) 000234 1749605848.077143 read 0x003e28 thru 0x005706 (0x18df bytes) 000235 1749605848.077459 write 0x00f20f thru 0x01396f (0x4761 bytes) 000237 1749605848.078306 mapread 0x01a571 thru 0x01ae8c (0x91c bytes) 000238 1749605848.078751 trunc from 0x00c915 to 0x017ae9 (0xb1d5 bytes) 000239 1749605848.082312 write 0x024553 thru 0x02ab9e (0x664c bytes) 000240 1749605848.082856 fallocate 0x0398a4 thru 0x03ffff (0x675c bytes) 000241 1749605848.084637 read 0x00e674 thru 0x01840f (0x9d9c bytes) 000242 1749605848.087086 fallocate 0x0228d8 thru 0x026e2e (0x4557 bytes) 000243 1749605848.088527 mapwrite 0x014810 thru 0x01e4fa (0x9ceb bytes) 000244 1749605848.094452 punch from 0x0196d8 to 0x01e198 (0x4ac1 bytes) 000245 1749605848.096097 trunc from 0x005d27 to 0x010290 (0xa56a bytes) 000246 1749605848.097554 read 0x03a869 thru 0x03ffff (0x5797 bytes) 000247 1749605848.099216 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000248 1749605848.129452 read 0x014f73 thru 0x01777b (0x2809 bytes) 000249 1749605848.131353 mapread 0x01dcaa thru 0x01dcfb (0x52 bytes) 000251 1749605848.133147 write 0x024f3a thru 0x032eaa (0xdf71 bytes) 000254 1749605848.137510 mapread 0x038d43 thru 0x03a4a6 (0x1764 bytes) 000255 1749605848.140002 read 0x03fa72 thru 0x03ffff (0x58e bytes) 000256 1749605848.140340 write 0x03c4e8 thru 0x03ffff (0x3b18 bytes) 000257 1749605848.140907 trunc from 0x040000 to 0x001007 (0x3eff8 bytes) 000258 1749605848.144940 mapwrite 0x03eee9 thru 0x03ff60 (0x1078 bytes) 000259 1749605848.155678 read 0x0278da thru 0x02dd1b (0x6442 bytes) 000260 1749605848.156041 trunc from 0x03ff61 to 0x009c93 (0x362cd bytes) 000261 1749605848.159386 read 0x0082e8 thru 0x009c93 (0x19ac bytes) 000262 1749605848.159823 mapwrite 0x00c568 thru 0x016af5 (0xa58e bytes) 000264 1749605848.170735 punch from 0x006d85 to 0x0091af (0x242b bytes) 000265 1749605848.171965 fallocate 0x0394fd thru 0x03a12b (0xc2f bytes) 000266 1749605848.174618 MIRROR_EXTEND to 2 mirrors 000267 1749605848.265872 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000268 1749605848.293149 read 0x02dee3 thru 0x03a12b (0xc249 bytes) 000269 1749605848.295450 punch from 0x002ab1 to 0x00c6c8 (0x9c18 bytes) 000270 1749605848.298610 punch from 0x037a1e to 0x03a12b (0x270e bytes) 000271 1749605848.300027 trunc from 0x03a12c to 0x02923f (0x10eec bytes) 000272 1749605848.308501 write 0x01a537 thru 0x01f0e3 (0x4bad bytes) 000273 1749605848.310960 trunc from 0x029240 to 0x019fb7 (0xf288 bytes) 000276 1749605848.315334 trunc from 0x019fb8 to 0x02da17 (0x13a60 bytes) 000277 1749605848.319072 mapread 0x0120cc thru 0x012965 (0x89a bytes) 000278 1749605848.321528 trunc from 0x02da18 to 0x032662 (0x4c4b bytes) 000279 1749605848.325474 punch from 0x00afdf to 0x00ec15 (0x3c37 bytes) 000280 1749605848.326796 fallocate 0x030647 thru 0x037c77 (0x7631 bytes) 000281 1749605848.328238 punch from 0x000f77 to 0x0082ff (0x7389 bytes) 000282 1749605848.329412 mapread 0x027659 thru 0x02aab4 (0x345c bytes) 000283 1749605848.332136 write 0x0293a5 thru 0x031c1b (0x8877 bytes) 000284 1749605848.333026 write 0x03e46f thru 0x03ffff (0x1b91 bytes) 000286 1749605848.334195 punch from 0x038786 to 0x038a7e (0x2f9 bytes) 000287 1749605848.336082 fallocate 0x00efdb thru 0x01d8ff (0xe925 bytes) 000288 1749605848.338049 write 0x001659 thru 0x00316d (0x1b15 bytes) 000289 1749605848.341130 trunc from 0x00b99c to 0x010e08 (0x546d bytes) 000290 1749605848.343284 punch from 0x026ba7 to 0x0304e5 (0x993f bytes) 000291 1749605848.347746 fallocate 0x025879 thru 0x033578 (0xdd00 bytes) 000292 1749605848.349276 mapread 0x026fe3 thru 0x02dbfc (0x6c1a bytes) 000294 1749605848.352126 trunc from 0x0123d4 to 0x01a362 (0x7f8f bytes) 000295 1749605848.353892 read 0x01bc7c thru 0x022a1b (0x6da0 bytes) 000296 1749605848.357640 trunc from 0x01e710 to 0x02b393 (0xcc84 bytes) 000297 1749605848.360321 trunc from 0x040000 to 0x02e0c3 (0x11f3c bytes) 000298 1749605848.365931 trunc from 0x027993 to 0x02e0c3 (0x6731 bytes) 000299 1749605848.368014 punch from 0x01783d to 0x02646f (0xec33 bytes) 000300 1749605848.370116 trunc from 0x02d096 to 0x02e0c3 (0x102e bytes) 000301 1749605848.371946 trunc from 0x02e0c4 to 0x031f03 (0x3e40 bytes) 000302 1749605848.376201 fallocate 0x007323 thru 0x014296 (0xcf74 bytes) 000303 1749605848.377758 trunc from 0x031f04 to 0x01c8fb (0x15608 bytes) 000304 1749605848.381708 write 0x0116ca thru 0x01c731 (0xb068 bytes) 000305 1749605848.383900 write 0x011348 thru 0x017111 (0x5dca bytes) 000306 1749605848.384702 trunc from 0x01c8fc to 0x0215d5 (0x4cda bytes) 000307 1749605848.390389 MIRROR_EXTEND to 2 mirrors 000308 1749605848.496399 fallocate 0x00b230 thru 0x017fd6 (0xcda7 bytes) 000309 1749605848.506425 trunc from 0x0215d6 to 0x03ba70 (0x1a49b bytes) 000310 1749605848.518960 read 0x023442 thru 0x0273fa (0x3fb9 bytes) 000311 1749605848.521571 mapread 0x036127 thru 0x0369ed (0x8c7 bytes) 000312 1749605848.523622 write 0x0143d5 thru 0x02229f (0xdecb bytes) 000314 1749605848.528243 mapwrite 0x00de38 thru 0x01d1dd (0xf3a6 bytes) 000315 1749605848.538193 write 0x0018ed thru 0x00d207 (0xb91b bytes) 000316 1749605848.540515 trunc from 0x03ba71 to 0x01f6a5 (0x1c3cb bytes) 000317 1749605848.545295 MIRROR_RESYNC 2 mirrors 000318 1749605848.614841 trunc from 0x01aa74 to 0x01f6a5 (0x4c32 bytes) 000319 1749605848.623606 trunc from 0x01f6a6 to 0x01d5e5 (0x20c0 bytes) 000320 1749605848.631878 punch from 0x010ea1 to 0x018b6f (0x7ccf bytes) 000321 1749605848.633330 mapwrite 0x00920a thru 0x00f286 (0x607d bytes) 000322 1749605848.643479 write 0x02912d thru 0x02e124 (0x4ff8 bytes) 000323 1749605848.644428 mapwrite 0x03af8e thru 0x03ffff (0x5072 bytes) 000324 1749605848.655533 mapwrite 0x03f2f6 thru 0x03ffff (0xd0a bytes) 000325 1749605848.659407 write 0x024415 thru 0x030c09 (0xc7f5 bytes) 000326 1749605848.663265 fallocate 0x00cb0e thru 0x01194e (0x4e41 bytes) 000327 1749605848.664629 fallocate 0x030abf thru 0x031a6b (0xfad bytes) 000329 1749605848.666518 trunc from 0x022b4c to 0x02b93b (0x8df0 bytes) 000330 1749605848.670260 mapwrite 0x015be8 thru 0x01f499 (0x98b2 bytes) 000331 1749605848.677077 trunc from 0x040000 to 0x01ecb9 (0x21346 bytes) 000332 1749605848.681030 mapread 0x00d299 thru 0x019df3 (0xcb5b bytes) 000333 1749605848.681575 mapwrite 0x025180 thru 0x029176 (0x3ff7 bytes) 000334 1749605848.691970 read 0x01cadd thru 0x029176 (0xc69a bytes) 000335 1749605848.692371 trunc from 0x01f525 to 0x0262c0 (0x6d9c bytes) 000336 1749605848.693750 mapread 0x0060da thru 0x011857 (0xb77e bytes) 000337 1749605848.694313 fallocate 0x004da6 thru 0x012266 (0xd4c1 bytes) 000338 1749605848.695729 read 0x0042e2 thru 0x0105c6 (0xc2e5 bytes) 000340 1749605848.696267 fallocate 0x029da9 thru 0x02c497 (0x26ef bytes) 000341 1749605848.697597 write 0x0046c8 thru 0x010b2f (0xc468 bytes) 000342 1749605848.698357 trunc from 0x02c498 to 0x00aa79 (0x21a1e bytes) 000343 1749605848.701744 write 0x03aabe thru 0x03ffff (0x5542 bytes) 000344 1749605848.702451 read 0x012cc6 thru 0x01e9d0 (0xbd0b bytes) 000345 1749605848.705415 punch from 0x02dbf7 to 0x039c10 (0xc01a bytes) 000346 1749605848.707139 trunc from 0x03f9f3 to 0x03ffff (0x60d bytes) 000347 1749605848.711135 trunc from 0x040000 to 0x00cc21 (0x333de bytes) 000348 1749605848.715482 trunc from 0x00cc22 to 0x003fab (0x8c76 bytes) 000349 1749605848.718847 mapwrite 0x00fe33 thru 0x0184d0 (0x869e bytes) 000350 1749605848.739773 read 0x008abf thru 0x011cd3 (0x9215 bytes) 000351 1749605848.742923 trunc from 0x0184d1 to 0x020eee (0x8a1e bytes) 000352 1749605848.747516 mapread 0x000d42 thru 0x0046b1 (0x3970 bytes) 000353 1749605848.749628 MIRROR_EXTEND to 3 mirrors 000355 1749605848.860339 trunc from 0x011444 to 0x0171e9 (0x5da6 bytes) 000356 1749605848.871081 trunc from 0x020eef to 0x031615 (0x10727 bytes) 000358 1749605848.884729 mapread 0x00fd6f thru 0x01e107 (0xe399 bytes) 000359 1749605848.893977 trunc from 0x031616 to 0x02d018 (0x45fd bytes) 000360 1749605848.898676 fallocate 0x02dcd7 thru 0x0314ea (0x3814 bytes) 000361 1749605848.900446 trunc from 0x02d019 to 0x001625 (0x2b9f3 bytes) 000362 1749605848.905299 mapread 0x0014dc thru 0x001625 (0x14a bytes) 000363 1749605848.907985 trunc from 0x001080 to 0x001625 (0x5a6 bytes) 000364 1749605848.910280 trunc from 0x000a43 to 0x001625 (0xbe3 bytes) 000365 1749605848.912617 trunc from 0x001626 to 0x00346b (0x1e46 bytes) 000367 1749605848.919861 punch from 0x0028a3 to 0x00346b (0xbc9 bytes) 000368 1749605848.921599 mapread 0x002e37 thru 0x00346b (0x635 bytes) 000370 1749605848.924342 mapwrite 0x0311d0 thru 0x03b7bc (0xa5ed bytes) 000371 1749605848.939483 mapread 0x0038fe thru 0x00dde5 (0xa4e8 bytes) 000372 1749605848.943096 trunc from 0x03b7bd to 0x03f185 (0x39c9 bytes) 000373 1749605848.946644 fallocate 0x03188b thru 0x033618 (0x1d8e bytes) 000374 1749605848.948052 write 0x00ec7a thru 0x019386 (0xa70d bytes) 000375 1749605848.950821 MIRROR_RESYNC 3 mirrors 000376 1749605849.091346 fallocate 0x02c644 thru 0x031061 (0x4a1e bytes) 000377 1749605849.099029 trunc from 0x032d4f to 0x03f185 (0xc437 bytes) 000380 1749605849.101180 trunc from 0x02fbdf to 0x0307fb (0xc1d bytes) 000381 1749605849.102760 read 0x0016bc thru 0x002064 (0x9a9 bytes) 000382 1749605849.104602 trunc from 0x03f186 to 0x021f91 (0x1d1f4 bytes) 000383 1749605849.109061 punch from 0x013c81 to 0x017a08 (0x3d88 bytes) 000384 1749605849.110854 trunc from 0x00d3e0 to 0x0175e4 (0xa205 bytes) 000385 1749605849.112427 read 0x006c9d thru 0x012f2b (0xc28f bytes) 000386 1749605849.116375 MIRROR_EXTEND to 4 mirrors 000387 1749605849.214473 trunc from 0x021f92 to 0x022a77 (0xae6 bytes) 000388 1749605849.228089 read 0x015969 thru 0x01f46e (0x9b06 bytes) 000389 1749605849.231647 MIRROR_RESYNC 4 mirrors 000390 1749605849.384153 punch from 0x014d80 to 0x01fbec (0xae6d bytes) 000391 1749605849.394549 MIRROR_EXTEND to 5 mirrors 000392 1749605849.491968 trunc from 0x022a78 to 0x03f07e (0x1c607 bytes) 000393 1749605849.504581 read 0x002702 thru 0x010acd (0xe3cc bytes) 000394 1749605849.507033 trunc from 0x03f07f to 0x00afa8 (0x340d6 bytes) 000395 1749605849.511669 mapread 0x006ef4 thru 0x00afa8 (0x40b5 bytes) 000396 1749605849.512091 trunc from 0x008437 to 0x00afa8 (0x2b72 bytes) 000397 1749605849.513507 trunc from 0x0092d7 to 0x00afa8 (0x1cd2 bytes) 000398 1749605849.514879 trunc from 0x00afa9 to 0x03330a (0x28362 bytes) 000399 1749605849.518806 mapread 0x012fe4 thru 0x01f815 (0xc832 bytes) 000401 1749605849.526116 write 0x028651 thru 0x03775b (0xf10b bytes) 000402 1749605849.528901 mapwrite 0x01e21f thru 0x02a4ee (0xc2d0 bytes) 000403 1749605849.535908 write 0x00a728 thru 0x014c79 (0xa552 bytes) 000404 1749605849.538653 fallocate 0x003bf1 thru 0x009b6a (0x5f7a bytes) 000405 1749605849.540248 fallocate 0x001819 thru 0x00ad17 (0x94ff bytes) 000406 1749605849.541856 fallocate 0x0277cc thru 0x036403 (0xec38 bytes) 000407 1749605849.544856 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000408 1749605849.600062 trunc from 0x011d92 to 0x01d60b (0xb87a bytes) 000409 1749605849.602271 fallocate 0x0180c4 thru 0x02375f (0xb69c bytes) 000410 1749605849.604336 read 0x01362f thru 0x0233d0 (0xfda2 bytes) 000411 1749605849.607633 mapread 0x023fba thru 0x027fe3 (0x402a bytes) 000412 1749605849.614027 mapwrite 0x019f62 thru 0x01e53f (0x45de bytes) 000413 1749605849.621539 read 0x02ac43 thru 0x02d040 (0x23fe bytes) 000414 1749605849.622170 mapread 0x031bad thru 0x03775b (0x5baf bytes) 000415 1749605849.626152 trunc from 0x037057 to 0x03775b (0x705 bytes) 000416 1749605849.628094 trunc from 0x01042c to 0x012f91 (0x2b66 bytes) 000417 1749605849.630290 write 0x02d1a1 thru 0x03be51 (0xecb1 bytes) 000418 1749605849.633905 trunc from 0x022159 to 0x02efec (0xce94 bytes) 000419 1749605849.641131 trunc from 0x03be52 to 0x00020d (0x3bc44 bytes) 000420 1749605849.649385 punch from 0x000163 to 0x00020d (0xab bytes) 000421 1749605849.652210 mapwrite 0x02a1db thru 0x035170 (0xaf96 bytes) 000422 1749605849.696351 fallocate 0x00d8e5 thru 0x016d2c (0x9448 bytes) 000423 1749605849.698466 fallocate 0x0114dc thru 0x01605e (0x4b83 bytes) 000424 1749605849.700449 read 0x02394a thru 0x024c14 (0x12cb bytes) 000425 1749605849.704229 fallocate 0x030f95 thru 0x03e9a2 (0xda0e bytes) 000426 1749605849.706237 mapread 0x01e274 thru 0x026faa (0x8d37 bytes) 000427 1749605849.706983 trunc from 0x011821 to 0x01d6ec (0xbecc bytes) 000429 1749605849.709440 punch from 0x007568 to 0x008a07 (0x14a0 bytes) 000430 1749605849.711816 punch from 0x0331a3 to 0x035170 (0x1fce bytes) 000431 1749605849.714270 fallocate 0x00a79f thru 0x00e4cb (0x3d2d bytes) 000432 1749605849.716323 trunc from 0x035171 to 0x00941a (0x2bd56 bytes) 000434 1749605849.724707 MIRROR_EXTEND to 5 mirrors 000435 1749605849.891017 mapwrite 0x0380e4 thru 0x03fbd0 (0x7aed bytes) 000436 1749605849.939383 read 0x00600f thru 0x00d825 (0x7817 bytes) 000437 1749605849.943969 mapread 0x0317c2 thru 0x03e688 (0xcec7 bytes) 000438 1749605849.948493 punch from 0x03d0c4 to 0x03f759 (0x2696 bytes) 000439 1749605849.951259 read 0x02dfbc thru 0x02e3aa (0x3ef bytes) 000440 1749605849.951857 punch from 0x032f7b to 0x03fbd0 (0xcc56 bytes) 000441 1749605849.954622 fallocate 0x01ecf1 thru 0x025d6d (0x707d bytes) 000442 1749605849.957322 read 0x001d02 thru 0x003943 (0x1c42 bytes) 000443 1749605849.962784 fallocate 0x01dff2 thru 0x01e690 (0x69f bytes) 000444 1749605849.968452 MIRROR_RESYNC 5 mirrors 000445 1749605850.171403 punch from 0x00aea7 to 0x012254 (0x73ae bytes) 000446 1749605850.180109 punch from 0x018d1a to 0x020f7e (0x8265 bytes) 000447 1749605850.181593 trunc from 0x03fbd1 to 0x03b00a (0x4bc6 bytes) 000448 1749605850.186036 mapwrite 0x018b72 thru 0x01c054 (0x34e3 bytes) 000449 1749605850.196781 MIRROR_EXTEND to 6 mirrors 000450 1749605850.300535 trunc from 0x03b00b to 0x02d43f (0xdbcb bytes) 000451 1749605850.314052 read 0x0258cf thru 0x02d43f (0x7b71 bytes) 000452 1749605850.316353 trunc from 0x02d440 to 0x007799 (0x25ca6 bytes) 000453 1749605850.321104 fallocate 0x0021fb thru 0x008e08 (0x6c0e bytes) 000455 1749605850.322911 mapwrite 0x03744f thru 0x03e75d (0x730f bytes) 000456 1749605850.335527 read 0x0007f2 thru 0x0033a4 (0x2bb3 bytes) 000457 1749605850.338283 trunc from 0x03e75e to 0x024bb5 (0x19ba8 bytes) 000458 1749605850.341668 mapread 0x009413 thru 0x00ff76 (0x6b64 bytes) 000459 1749605850.342125 mapread 0x003221 thru 0x011538 (0xe318 bytes) 000460 1749605850.342573 punch from 0x00320f to 0x00b0f5 (0x7ee7 bytes) 000461 1749605850.344091 punch from 0x0243ef to 0x024bb5 (0x7c7 bytes) 000463 1749605850.345628 mapwrite 0x03087d thru 0x03b2b7 (0xaa3b bytes) 000465 1749605850.369345 trunc from 0x03b2b8 to 0x03ab3e (0x779 bytes) 000466 1749605850.373211 punch from 0x028085 to 0x02ad29 (0x2ca5 bytes) 000467 1749605850.374728 mapwrite 0x033b9d thru 0x0348b5 (0xd19 bytes) 000468 1749605850.379393 mapread 0x009c7c thru 0x00b2c9 (0x164e bytes) 000469 1749605850.382376 write 0x01e521 thru 0x028699 (0xa179 bytes) 000470 1749605850.384319 punch from 0x0380d4 to 0x03ab3e (0x2a6b bytes) 000471 1749605850.385749 punch from 0x012e16 to 0x01ec2d (0xbe18 bytes) 000472 1749605850.389585 punch from 0x037e70 to 0x03ab3e (0x2ccf bytes) 000473 1749605850.391038 trunc from 0x03ab3f to 0x02ac6c (0xfed2 bytes) 000474 1749605850.394793 write 0x033caa thru 0x03f1ca (0xb521 bytes) 000475 1749605850.395596 mapread 0x006f8c thru 0x00e009 (0x707e bytes) 000476 1749605850.399478 read 0x0299b3 thru 0x02dedd (0x452b bytes) 000477 1749605850.401811 trunc from 0x037e46 to 0x037ecf (0x8a bytes) 000478 1749605850.405395 write 0x02d400 thru 0x02ddfc (0x9fd bytes) 000479 1749605850.405879 mapwrite 0x018660 thru 0x024aea (0xc48b bytes) 000480 1749605850.412555 mapwrite 0x005452 thru 0x00626a (0xe19 bytes) 000481 1749605850.416187 mapread 0x021462 thru 0x02b223 (0x9dc2 bytes) 000482 1749605850.416926 mapwrite 0x001e39 thru 0x011a3e (0xfc06 bytes) 000483 1749605850.422734 mapwrite 0x03221e thru 0x03f45c (0xd23f bytes) 000484 1749605850.433290 read 0x03c44d thru 0x03f45c (0x3010 bytes) 000486 1749605850.433895 mapread 0x0221d8 thru 0x02d64a (0xb473 bytes) 000487 1749605850.434301 punch from 0x03daf5 to 0x03f45c (0x1968 bytes) 000488 1749605850.435737 trunc from 0x02c721 to 0x030dde (0x46be bytes) 000489 1749605850.437427 mapwrite 0x03856e thru 0x039d61 (0x17f4 bytes) 000490 1749605850.441026 read 0x00c67e thru 0x00e199 (0x1b1c bytes) 000491 1749605850.441347 mapread 0x019709 thru 0x01aaa1 (0x1399 bytes) 000492 1749605850.441903 mapread 0x01cf3b thru 0x029618 (0xc6de bytes) 000493 1749605850.442317 write 0x031465 thru 0x0365ee (0x518a bytes) 000494 1749605850.443062 mapread 0x01b724 thru 0x02999d (0xe27a bytes) 000495 1749605850.443597 trunc from 0x02feb9 to 0x03bd03 (0xbe4b bytes) 000496 1749605850.447070 mapwrite 0x01b7c8 thru 0x027785 (0xbfbe bytes) 000497 1749605850.452278 write 0x02865d thru 0x02e9e9 (0x638d bytes) 000499 1749605850.454227 mapread 0x02eda1 thru 0x036145 (0x73a5 bytes) 000500 1749605850.456537 mapread 0x025cb1 thru 0x02b3ea (0x573a bytes) 000501 1749605850.457025 punch from 0x013f43 to 0x02376f (0xf82d bytes) 000502 1749605850.458643 trunc from 0x03f45d to 0x01833f (0x2711d bytes) 000503 1749605850.462656 mapwrite 0x00bf00 thru 0x00f9f5 (0x3af6 bytes) 000505 1749605850.468344 write 0x000d64 thru 0x002807 (0x1aa4 bytes) 000506 1749605850.468910 punch from 0x00027f to 0x00b760 (0xb4e2 bytes) 000507 1749605850.472164 mapwrite 0x012022 thru 0x0140b9 (0x2098 bytes) 000509 1749605850.478089 mapread 0x015b83 thru 0x0164da (0x958 bytes) 000510 1749605850.480544 mapread 0x010e30 thru 0x01833f (0x7510 bytes) 000511 1749605850.483287 mapread 0x00a4a3 thru 0x0175f8 (0xd156 bytes) 000512 1749605850.486179 write 0x0315a1 thru 0x03b795 (0xa1f5 bytes) 000514 1749605850.488540 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000515 1749605850.540330 MIRROR_EXTEND to 6 mirrors 000516 1749605850.636114 punch from 0x021e01 to 0x030943 (0xeb43 bytes) 000517 1749605850.644167 write 0x037d3d thru 0x03ffff (0x82c3 bytes) 000518 1749605850.647295 trunc from 0x040000 to 0x008e29 (0x371d6 bytes) 000519 1749605850.654932 mapwrite 0x007746 thru 0x00c35f (0x4c1a bytes) 000520 1749605850.666656 trunc from 0x00c360 to 0x007712 (0x4c4d bytes) 000521 1749605850.671298 MIRROR_EXTEND to 7 mirrors 000523 1749605850.753721 read 0x006954 thru 0x007712 (0xdbf bytes) 000524 1749605850.757295 MIRROR_EXTEND to 8 mirrors 000526 1749605850.832071 trunc from 0x007713 to 0x02e07a (0x26968 bytes) 000527 1749605850.845233 MIRROR_EXTEND to 9 mirrors 000528 1749605850.933024 mapread 0x010aac thru 0x018f2e (0x8483 bytes) 000529 1749605850.939761 fallocate 0x0138ca thru 0x019c8b (0x63c2 bytes) 000530 1749605850.947010 punch from 0x00253e to 0x00b846 (0x9309 bytes) 000531 1749605850.948300 read 0x020a13 thru 0x024cb7 (0x42a5 bytes) 000532 1749605850.950139 write 0x01260e thru 0x0193bb (0x6dae bytes) 000533 1749605850.953210 fallocate 0x02ca03 thru 0x0357df (0x8ddd bytes) 000535 1749605850.955143 trunc from 0x0357e0 to 0x0386df (0x2f00 bytes) 000536 1749605850.962334 read 0x02a07d thru 0x02d522 (0x34a6 bytes) 000537 1749605850.964874 read 0x02027b thru 0x02df52 (0xdcd8 bytes) 000538 1749605850.965229 trunc from 0x027eae to 0x02d398 (0x54eb bytes) 000539 1749605850.966545 punch from 0x00071b to 0x00bbc8 (0xb4ae bytes) 000540 1749605850.969255 MIRROR_RESYNC 9 mirrors 000541 1749605851.341242 punch from 0x011378 to 0x01b103 (0x9d8c bytes) 000542 1749605851.349191 fallocate 0x0256c3 thru 0x031092 (0xb9d0 bytes) 000543 1749605851.350492 write 0x0126a9 thru 0x01cd7a (0xa6d2 bytes) 000544 1749605851.353204 trunc from 0x01837b to 0x01a516 (0x219c bytes) 000545 1749605851.356305 mapwrite 0x008744 thru 0x011726 (0x8fe3 bytes) 000546 1749605851.365049 punch from 0x031822 to 0x0386df (0x6ebe bytes) 000548 1749605851.366634 fallocate 0x01bc7c thru 0x027312 (0xb697 bytes) 000549 1749605851.368123 write 0x00715c thru 0x00b8d6 (0x477b bytes) 000550 1749605851.368697 read 0x01e800 thru 0x02ba44 (0xd245 bytes) 000551 1749605851.370763 mapread 0x02970c thru 0x02c9f1 (0x32e6 bytes) 000552 1749605851.372960 trunc from 0x0386e0 to 0x011e0c (0x268d3 bytes) 000554 1749605851.377909 trunc from 0x011e0d to 0x01433c (0x2530 bytes) 000555 1749605851.381468 punch from 0x00ce45 to 0x01433c (0x74f8 bytes) 000556 1749605851.382830 trunc from 0x01433d to 0x000e98 (0x134a4 bytes) 000557 1749605851.386196 read 0x00000b thru 0x000e98 (0xe8e bytes) 000559 1749605851.389092 MIRROR_EXTEND to 10 mirrors 000560 1749605851.456167 trunc from 0x000e99 to 0x01ceb3 (0x1c01b bytes) 000561 1749605851.469781 mapwrite 0x001aba thru 0x004489 (0x29d0 bytes) 000562 1749605851.479045 write 0x00a92d thru 0x00dd68 (0x343c bytes) 000563 1749605851.479913 mapread 0x005fb3 thru 0x00d7dd (0x782b bytes) 000564 1749605851.480643 mapread 0x018253 thru 0x01ceb3 (0x4c61 bytes) 000566 1749605851.483621 trunc from 0x01ceb4 to 0x0195f5 (0x38be bytes) 000567 1749605851.489648 MIRROR_EXTEND to 11 mirrors 000568 1749605851.594245 trunc from 0x0195f6 to 0x015a46 (0x3baf bytes) 000569 1749605851.609523 mapwrite 0x02810e thru 0x02a080 (0x1f73 bytes) 000570 1749605851.622310 fallocate 0x00c3a4 thru 0x01919d (0xcdfa bytes) 000571 1749605851.623976 fallocate 0x01f749 thru 0x0244fe (0x4db6 bytes) 000572 1749605851.625555 trunc from 0x02a081 to 0x03bc89 (0x11c09 bytes) 000573 1749605851.629210 mapwrite 0x01d813 thru 0x0291ac (0xb99a bytes) 000574 1749605851.638094 punch from 0x004769 to 0x00a2ac (0x5b44 bytes) 000575 1749605851.639776 fallocate 0x018d85 thru 0x0233f1 (0xa66d bytes) 000577 1749605851.641641 mapread 0x0287c0 thru 0x03512b (0xc96c bytes) 000579 1749605851.645867 mapread 0x0225c7 thru 0x031369 (0xeda3 bytes) 000580 1749605851.648147 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000581 1749605851.695052 trunc from 0x002592 to 0x00301d (0xa8c bytes) 000582 1749605851.696601 write 0x02fd33 thru 0x034539 (0x4807 bytes) 000583 1749605851.700177 trunc from 0x00e72f to 0x017571 (0x8e43 bytes) 000584 1749605851.701573 mapwrite 0x004f18 thru 0x00de83 (0x8f6c bytes) 000585 1749605851.710179 fallocate 0x02c27a thru 0x031d8d (0x5b14 bytes) 000586 1749605851.711884 mapwrite 0x035246 thru 0x03f8cb (0xa686 bytes) 000587 1749605851.724576 write 0x00883f thru 0x010b4c (0x830e bytes) 000588 1749605851.725326 fallocate 0x015e7b thru 0x024b16 (0xec9c bytes) 000589 1749605851.726855 punch from 0x014675 to 0x0203ad (0xbd39 bytes) 000590 1749605851.728255 trunc from 0x0097bb to 0x00a517 (0xd5d bytes) 000591 1749605851.731580 mapread 0x011341 thru 0x01b11e (0x9dde bytes) 000592 1749605851.736823 MIRROR_EXTEND to 11 mirrors 000593 1749605851.837347 trunc from 0x03f8cc to 0x03a277 (0x5654 bytes) 000594 1749605851.851719 trunc from 0x03a278 to 0x0368bd (0x39ba bytes) 000595 1749605851.855928 read 0x011937 thru 0x0216ac (0xfd76 bytes) 000596 1749605851.858114 punch from 0x035355 to 0x0368bd (0x1569 bytes) 000597 1749605851.859493 trunc from 0x0216c7 to 0x02e416 (0xcd50 bytes) 000598 1749605851.860874 trunc from 0x0368be to 0x036afb (0x23e bytes) 000599 1749605851.864400 trunc from 0x01ae10 to 0x0211eb (0x63dc bytes) 000600 1749605851.865951 trunc from 0x0032fb to 0x012509 (0xf20f bytes) 000601 1749605851.867557 read 0x00d0f0 thru 0x0120c3 (0x4fd4 bytes) 000602 1749605851.871507 MIRROR_RESYNC 11 mirrors 000603 1749605852.125041 trunc from 0x036afc to 0x00fad6 (0x27025 bytes) 000604 1749605852.133409 write 0x03ae96 thru 0x03ffff (0x516a bytes) 000605 1749605852.134432 mapwrite 0x00bdc6 thru 0x01a371 (0xe5ac bytes) 000606 1749605852.146040 fallocate 0x03d302 thru 0x03ffff (0x2cfe bytes) 000607 1749605852.147395 trunc from 0x0179d4 to 0x01a1fc (0x2829 bytes) 000608 1749605852.148817 fallocate 0x00a50f thru 0x012225 (0x7d17 bytes) 000609 1749605852.150291 mapwrite 0x01d138 thru 0x02a595 (0xd45e bytes) 000610 1749605852.159958 fallocate 0x01b99a thru 0x02abae (0xf215 bytes) 000611 1749605852.161366 trunc from 0x040000 to 0x02b2dd (0x14d22 bytes) 000612 1749605852.166140 mapread 0x0167c4 thru 0x0222a2 (0xbadf bytes) 000613 1749605852.167868 mapwrite 0x034df5 thru 0x03f3dd (0xa5e9 bytes) 000614 1749605852.181399 fallocate 0x02cf94 thru 0x03b5bc (0xe629 bytes) 000615 1749605852.182797 write 0x0259b1 thru 0x0284e7 (0x2b37 bytes) 000616 1749605852.183262 mapread 0x0363b3 thru 0x038b56 (0x27a4 bytes) 000617 1749605852.183674 trunc from 0x03f3de to 0x01b052 (0x2438b bytes) 000618 1749605852.188123 trunc from 0x01b053 to 0x026934 (0xb8e2 bytes) 000619 1749605852.191818 write 0x02891f thru 0x02998e (0x1070 bytes) 000620 1749605852.192383 fallocate 0x00bbc4 thru 0x00f55f (0x399c bytes) 000621 1749605852.193832 fallocate 0x03ceb5 thru 0x03ffff (0x314b bytes) 000622 1749605852.195247 trunc from 0x02998f to 0x033713 (0x9d85 bytes) 000623 1749605852.198527 mapwrite 0x039de2 thru 0x03c9b2 (0x2bd1 bytes) 000624 1749605852.208514 trunc from 0x03c9b3 to 0x0016ec (0x3b2c6 bytes) 000625 1749605852.212082 write 0x02e2b2 thru 0x039c7a (0xb9c9 bytes) 000626 1749605852.212898 read 0x033c44 thru 0x039c7a (0x6037 bytes) 000627 1749605852.213169 read 0x01871f thru 0x0194c8 (0xdaa bytes) 000628 1749605852.215358 punch from 0x030a95 to 0x031022 (0x58e bytes) 000629 1749605852.218406 trunc from 0x02c77f to 0x033002 (0x6884 bytes) 000630 1749605852.219872 trunc from 0x039c7b to 0x038c7b (0xfff bytes) 000631 1749605852.223378 trunc from 0x00f0c8 to 0x0157d6 (0x670f bytes) 000633 1749605852.224975 fallocate 0x017b3d thru 0x01b27b (0x373f bytes) 000634 1749605852.226340 mapread 0x026c1e thru 0x028282 (0x1665 bytes) 000635 1749605852.226841 mapwrite 0x023a4e thru 0x02ee13 (0xb3c6 bytes) 000637 1749605852.236824 write 0x03c35b thru 0x03d37a (0x1020 bytes) 000638 1749605852.237542 mapread 0x02f560 thru 0x039d63 (0xa804 bytes) 000639 1749605852.245588 read 0x012623 thru 0x01aa37 (0x8415 bytes) 000640 1749605852.248267 mapread 0x017fee thru 0x01c216 (0x4229 bytes) 000641 1749605852.248901 punch from 0x030679 to 0x0334ea (0x2e72 bytes) 000643 1749605852.250848 mapread 0x016541 thru 0x019fb6 (0x3a76 bytes) 000644 1749605852.252577 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000645 1749605852.302976 punch from 0x002226 to 0x011ac3 (0xf89e bytes) 000646 1749605852.304646 trunc from 0x03d37b to 0x02f8b9 (0xdac1 bytes) 000647 1749605852.309645 punch from 0x0231eb to 0x02f8b9 (0xc6cf bytes) 000648 1749605852.311092 fallocate 0x01c6c1 thru 0x027222 (0xab62 bytes) 000649 1749605852.312678 mapwrite 0x03b650 thru 0x03cd90 (0x1741 bytes) 000650 1749605852.324767 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000651 1749605852.368750 punch from 0x02c3aa to 0x0340ea (0x7d41 bytes) 000652 1749605852.370353 trunc from 0x03cd91 to 0x01eab9 (0x1e2d7 bytes) 000653 1749605852.376236 punch from 0x017b42 to 0x01eab9 (0x6f78 bytes) 000654 1749605852.378282 trunc from 0x01eaba to 0x02dc89 (0xf1d0 bytes) 000656 1749605852.383377 trunc from 0x02dc8a to 0x0383dd (0xa754 bytes) 000657 1749605852.387781 mapread 0x021b9a thru 0x030ad5 (0xef3c bytes) 000658 1749605852.396413 read 0x02730f thru 0x02f88e (0x8580 bytes) 000659 1749605852.397037 mapread 0x01319c thru 0x0175ed (0x4452 bytes) 000660 1749605852.399400 mapread 0x0245a2 thru 0x033f16 (0xf975 bytes) 000661 1749605852.399923 mapread 0x01a677 thru 0x01bf30 (0x18ba bytes) 000662 1749605852.401582 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000663 1749605852.447640 trunc from 0x0132f1 to 0x022aab (0xf7bb bytes) 000664 1749605852.449403 write 0x03b06d thru 0x03ffff (0x4f93 bytes) 000665 1749605852.450986 fallocate 0x02cf44 thru 0x03351b (0x65d8 bytes) 000666 1749605852.453417 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000667 1749605852.496069 read 0x0353b1 thru 0x03fc8a (0xa8da bytes) 000668 1749605852.498286 write 0x004513 thru 0x009660 (0x514e bytes) 000669 1749605852.502173 trunc from 0x0349e5 to 0x03ffff (0xb61b bytes) 000670 1749605852.504183 mapread 0x02c646 thru 0x0375d1 (0xaf8c bytes) 000671 1749605852.509437 read 0x0393bc thru 0x03d85f (0x44a4 bytes) 000673 1749605852.510157 trunc from 0x007c79 to 0x016d80 (0xf108 bytes) 000675 1749605852.515079 read 0x0317cc thru 0x03948c (0x7cc1 bytes) 000676 1749605852.515775 punch from 0x0197f2 to 0x01f1f5 (0x5a04 bytes) 000677 1749605852.517830 mapwrite 0x00be5b thru 0x01afe6 (0xf18c bytes) 000678 1749605852.529899 mapread 0x00e1f2 thru 0x010075 (0x1e84 bytes) 000679 1749605852.530510 mapread 0x02bd45 thru 0x033816 (0x7ad2 bytes) 000681 1749605852.531340 mapread 0x023e47 thru 0x02bda7 (0x7f61 bytes) 000682 1749605852.532032 trunc from 0x040000 to 0x01c52c (0x23ad3 bytes) 000683 1749605852.537315 MIRROR_RESYNC 7 mirrors 000684 1749605852.743094 write 0x025fd7 thru 0x034cc8 (0xecf2 bytes) 000685 1749605852.751656 trunc from 0x00221d to 0x004900 (0x26e4 bytes) 000686 1749605852.753208 read 0x02e85d thru 0x034cc8 (0x646c bytes) 000688 1749605852.753830 trunc from 0x02475b to 0x032de9 (0xe68f bytes) 000689 1749605852.757346 write 0x014c67 thru 0x0156fb (0xa95 bytes) 000690 1749605852.759865 trunc from 0x02c648 to 0x034396 (0x7d4f bytes) 000691 1749605852.761209 mapread 0x031120 thru 0x0343f7 (0x32d8 bytes) 000692 1749605852.763343 trunc from 0x008061 to 0x00f4fa (0x749a bytes) 000693 1749605852.764767 trunc from 0x02a85b to 0x034cc8 (0xa46e bytes) 000694 1749605852.766081 read 0x0071a3 thru 0x00fbb4 (0x8a12 bytes) 000696 1749605852.769555 read 0x02df7a thru 0x034cc8 (0x6d4f bytes) 000697 1749605852.770075 punch from 0x0160ec to 0x01f922 (0x9837 bytes) 000698 1749605852.771530 read 0x033f09 thru 0x034cc8 (0xdc0 bytes) 000699 1749605852.772024 mapread 0x01140a thru 0x013e26 (0x2a1d bytes) 000700 1749605852.772585 write 0x02d35d thru 0x035d0b (0x89af bytes) 000701 1749605852.774512 MIRROR_RESYNC 7 mirrors 000703 1749605852.958157 trunc from 0x035d0c to 0x02c31f (0x99ec bytes) 000704 1749605852.970403 trunc from 0x02c320 to 0x0319cd (0x56ae bytes) 000706 1749605852.974834 mapwrite 0x00b064 thru 0x014a72 (0x9a0f bytes) 000707 1749605852.984910 punch from 0x025581 to 0x02a07b (0x4afb bytes) 000708 1749605852.986293 mapread 0x01570d thru 0x01f42e (0x9d22 bytes) 000709 1749605852.989738 trunc from 0x01668e to 0x01beb0 (0x5823 bytes) 000710 1749605852.991060 mapread 0x0064b7 thru 0x00ede6 (0x8930 bytes) 000711 1749605852.991582 trunc from 0x0319ce to 0x00bde2 (0x25beb bytes) 000712 1749605852.995256 mapread 0x000f61 thru 0x007de7 (0x6e87 bytes) 000713 1749605852.998407 write 0x00640f thru 0x00a1bb (0x3dad bytes) 000714 1749605852.999949 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000715 1749605853.043854 punch from 0x00acaa to 0x00bde2 (0x1139 bytes) 000716 1749605853.045226 trunc from 0x00bde3 to 0x003f3a (0x7ea8 bytes) 000717 1749605853.049964 mapread 0x003bcf thru 0x003f3a (0x36c bytes) 000718 1749605853.051758 mapwrite 0x003b4c thru 0x013623 (0xfad8 bytes) 000719 1749605853.066596 fallocate 0x03c000 thru 0x03e9c2 (0x29c3 bytes) 000720 1749605853.068255 write 0x007191 thru 0x00d86a (0x66da bytes) 000721 1749605853.068841 fallocate 0x01aff6 thru 0x01c495 (0x14a0 bytes) 000722 1749605853.070421 read 0x006864 thru 0x00a936 (0x40d3 bytes) 000723 1749605853.072093 MIRROR_EXTEND to 7 mirrors 000724 1749605853.151773 write 0x01a105 thru 0x01d193 (0x308f bytes) 000725 1749605853.160167 mapread 0x00e3f0 thru 0x013803 (0x5414 bytes) 000726 1749605853.164530 mapwrite 0x02bef8 thru 0x030439 (0x4542 bytes) 000727 1749605853.178804 punch from 0x02a84b to 0x02bcc3 (0x1479 bytes) 000728 1749605853.180077 punch from 0x00e730 to 0x011c36 (0x3507 bytes) 000729 1749605853.181478 mapwrite 0x0104aa thru 0x0180c4 (0x7c1b bytes) 000731 1749605853.187734 write 0x02ef3e thru 0x031673 (0x2736 bytes) 000732 1749605853.188204 read 0x030ce6 thru 0x030fa1 (0x2bc bytes) 000733 1749605853.188474 trunc from 0x031674 to 0x009c98 (0x279db bytes) 000735 1749605853.192663 fallocate 0x018e8b thru 0x018f98 (0x10e bytes) 000738 1749605853.194592 mapread 0x0063e2 thru 0x009c98 (0x38b7 bytes) 000739 1749605853.196644 read 0x001116 thru 0x009c98 (0x8b83 bytes) 000740 1749605853.196974 mapwrite 0x039c45 thru 0x03ffff (0x63bb bytes) 000742 1749605853.207879 read 0x02c976 thru 0x03a7c1 (0xde4c bytes) 000743 1749605853.208209 mapwrite 0x02933a thru 0x034d59 (0xba20 bytes) 000744 1749605853.212986 fallocate 0x013d8a thru 0x01aac6 (0x6d3d bytes) 000745 1749605853.214780 write 0x0397d9 thru 0x03b065 (0x188d bytes) 000746 1749605853.216669 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000747 1749605853.261062 punch from 0x009a1c to 0x016d7c (0xd361 bytes) 000748 1749605853.262602 punch from 0x026dd0 to 0x029622 (0x2853 bytes) 000749 1749605853.263900 mapwrite 0x011460 thru 0x0166f9 (0x529a bytes) 000750 1749605853.272471 write 0x011dc4 thru 0x01ebfb (0xce38 bytes) 000751 1749605853.275587 punch from 0x0214e3 to 0x02f4d7 (0xdff5 bytes) 000753 1749605853.277900 mapread 0x03b444 thru 0x03ffff (0x4bbc bytes) 000754 1749605853.280144 punch from 0x027b5f to 0x027d56 (0x1f8 bytes) 000755 1749605853.281571 fallocate 0x03e0c3 thru 0x03ffff (0x1f3d bytes) 000756 1749605853.282977 trunc from 0x040000 to 0x03d50a (0x2af5 bytes) 000757 1749605853.286448 write 0x023295 thru 0x02b57f (0x82eb bytes) 000758 1749605853.288686 mapwrite 0x0328d3 thru 0x03307c (0x7aa bytes) 000759 1749605853.295376 punch from 0x022f9b to 0x02ee05 (0xbe6b bytes) 000760 1749605853.296996 trunc from 0x03d50b to 0x02c71e (0x10dec bytes) 000761 1749605853.302314 MIRROR_EXTEND to 7 mirrors 000762 1749605853.404657 mapwrite 0x016b6a thru 0x0223be (0xb855 bytes) 000763 1749605853.438800 punch from 0x020d33 to 0x02533c (0x460a bytes) 000764 1749605853.440099 mapwrite 0x00331c thru 0x00be95 (0x8b7a bytes) 000765 1749605853.447217 mapwrite 0x006ae6 thru 0x0147de (0xdcf9 bytes) 000766 1749605853.455141 MIRROR_EXTEND to 8 mirrors 000767 1749605853.554310 write 0x03a823 thru 0x03ffff (0x57dd bytes) 000768 1749605853.564245 read 0x03c891 thru 0x03ffff (0x376f bytes) 000769 1749605853.565004 trunc from 0x01c4b6 to 0x025006 (0x8b51 bytes) 000771 1749605853.566780 mapread 0x031118 thru 0x03cadb (0xb9c4 bytes) 000775 1749605853.572512 trunc from 0x02b84b to 0x034c91 (0x9447 bytes) 000776 1749605853.574282 mapread 0x02cbc8 thru 0x0370c0 (0xa4f9 bytes) 000778 1749605853.579399 mapread 0x017d30 thru 0x01b837 (0x3b08 bytes) 000779 1749605853.583105 write 0x0015fc thru 0x010b2e (0xf533 bytes) 000780 1749605853.588895 mapread 0x005860 thru 0x00ee19 (0x95ba bytes) 000781 1749605853.589458 trunc from 0x03bf0b to 0x03ffff (0x40f5 bytes) 000783 1749605853.594485 fallocate 0x00bc9b thru 0x01abd0 (0xef36 bytes) 000784 1749605853.596332 trunc from 0x040000 to 0x00ff0d (0x300f2 bytes) 000785 1749605853.605855 mapwrite 0x01684a thru 0x01e76c (0x7f23 bytes) 000786 1749605853.618198 mapread 0x010a0e thru 0x016e3d (0x6430 bytes) 000787 1749605853.618676 read 0x008b54 thru 0x016923 (0xddd0 bytes) 000788 1749605853.619113 mapwrite 0x006bb6 thru 0x007068 (0x4b3 bytes) 000789 1749605853.622895 punch from 0x00d170 to 0x0109ed (0x387e bytes) 000790 1749605853.624675 punch from 0x004ace to 0x00e36e (0x98a1 bytes) 000791 1749605853.627522 MIRROR_EXTEND to 9 mirrors 000792 1749605853.722943 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000793 1749605853.795015 mapwrite 0x00ceef thru 0x012576 (0x5688 bytes) 000794 1749605853.831294 write 0x024c1c thru 0x02e725 (0x9b0a bytes) 000795 1749605853.833833 read 0x0283f6 thru 0x02e725 (0x6330 bytes) 000796 1749605853.834417 read 0x00d563 thru 0x018c35 (0xb6d3 bytes) 000797 1749605853.838025 punch from 0x0014fb to 0x005b01 (0x4607 bytes) 000798 1749605853.840101 trunc from 0x02e726 to 0x01e756 (0xffcf bytes) 000799 1749605853.848912 fallocate 0x021d55 thru 0x031b13 (0xfdbf bytes) 000800 1749605853.850585 read 0x02a5c4 thru 0x031b13 (0x7550 bytes) 000801 1749605853.853391 mapwrite 0x006663 thru 0x00c1ce (0x5b6c bytes) 000802 1749605853.859612 write 0x02cdd1 thru 0x03bb54 (0xed84 bytes) 000803 1749605853.860422 mapread 0x008aad thru 0x01748f (0xe9e3 bytes) 000804 1749605853.860879 fallocate 0x00309e thru 0x00640c (0x336f bytes) 000805 1749605853.862217 trunc from 0x008e61 to 0x00acab (0x1e4b bytes) 000806 1749605853.863549 read 0x00460b thru 0x014474 (0xfe6a bytes) 000807 1749605853.865104 write 0x031a3c thru 0x033c43 (0x2208 bytes) 000809 1749605853.865838 write 0x0131b8 thru 0x020f29 (0xdd72 bytes) 000810 1749605853.866718 write 0x00a70d thru 0x00e2d3 (0x3bc7 bytes) 000811 1749605853.867210 mapwrite 0x0184f8 thru 0x01ae44 (0x294d bytes) 000812 1749605853.872297 trunc from 0x03bb55 to 0x01186e (0x2a2e6 bytes) 000813 1749605853.876629 write 0x00d5ab thru 0x0190cb (0xbb21 bytes) 000814 1749605853.877828 write 0x03fdcf thru 0x03ffff (0x231 bytes) 000815 1749605853.878492 mapread 0x01705a thru 0x025e2d (0xedd4 bytes) 000817 1749605853.891512 punch from 0x01bc26 to 0x0240b3 (0x848e bytes) 000818 1749605853.892894 mapwrite 0x03020c thru 0x03cc17 (0xca0c bytes) 000819 1749605853.912449 MIRROR_RESYNC 8 mirrors 000820 1749605854.216456 trunc from 0x040000 to 0x039b6d (0x6492 bytes) 000822 1749605854.224715 trunc from 0x039b6e to 0x0380bb (0x1ab2 bytes) 000823 1749605854.228414 trunc from 0x006643 to 0x00a546 (0x3f04 bytes) 000824 1749605854.230849 MIRROR_EXTEND to 9 mirrors 000825 1749605854.332472 trunc from 0x02081b to 0x0247b3 (0x3f99 bytes) 000826 1749605854.340101 trunc from 0x0380bc to 0x0298e1 (0xe7da bytes) 000827 1749605854.345286 mapwrite 0x030a08 thru 0x03fec1 (0xf4ba bytes) 000828 1749605854.360601 trunc from 0x021524 to 0x02565c (0x4139 bytes) 000829 1749605854.361937 mapwrite 0x00d1b1 thru 0x011269 (0x40b9 bytes) 000830 1749605854.367830 trunc from 0x01d5f3 to 0x01fc10 (0x261e bytes) 000831 1749605854.370199 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000832 1749605854.414284 mapwrite 0x01bb7b thru 0x029606 (0xda8c bytes) 000833 1749605854.425941 punch from 0x0268ce to 0x02c8c9 (0x5ffc bytes) 000834 1749605854.427316 punch from 0x03630a to 0x038875 (0x256c bytes) 000836 1749605854.430045 MIRROR_EXTEND to 9 mirrors 000837 1749605854.532830 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000838 1749605854.597217 mapread 0x0321b1 thru 0x0362be (0x410e bytes) 000839 1749605854.607090 punch from 0x013224 to 0x020a0c (0xd7e9 bytes) 000840 1749605854.615613 trunc from 0x03fec2 to 0x003acd (0x3c3f4 bytes) 000841 1749605854.624639 fallocate 0x0212c8 thru 0x024a89 (0x37c2 bytes) 000842 1749605854.626296 read 0x0090c7 thru 0x00f8d7 (0x6811 bytes) 000843 1749605854.628447 mapread 0x01fef2 thru 0x024a89 (0x4b98 bytes) 000844 1749605854.630720 mapread 0x012621 thru 0x0157c4 (0x31a4 bytes) 000845 1749605854.633158 read 0x01590b thru 0x023ff8 (0xe6ee bytes) 000846 1749605854.633541 trunc from 0x016df9 to 0x01c3eb (0x55f3 bytes) 000848 1749605854.635200 trunc from 0x01c90e to 0x024a89 (0x817c bytes) 000849 1749605854.637808 MIRROR_EXTEND to 9 mirrors 000850 1749605854.735513 mapread 0x01ed61 thru 0x024a89 (0x5d29 bytes) 000851 1749605854.744200 write 0x025ef7 thru 0x02e472 (0x857c bytes) 000852 1749605854.754092 fallocate 0x0091d7 thru 0x00f734 (0x655e bytes) 000853 1749605854.755575 trunc from 0x02a60f to 0x02be57 (0x1849 bytes) 000855 1749605854.759022 write 0x024384 thru 0x02a030 (0x5cad bytes) 000856 1749605854.761422 mapwrite 0x019afe thru 0x020a36 (0x6f39 bytes) 000857 1749605854.769664 trunc from 0x02e473 to 0x011e31 (0x1c641 bytes) 000858 1749605854.778205 trunc from 0x002ead to 0x006c7c (0x3dd0 bytes) 000859 1749605854.779605 read 0x00f26d thru 0x011e31 (0x2bc5 bytes) 000860 1749605854.781157 punch from 0x0011fc to 0x0083b3 (0x71b8 bytes) 000861 1749605854.782646 mapread 0x007b00 thru 0x00a352 (0x2853 bytes) 000862 1749605854.785458 mapwrite 0x013add thru 0x01ffc6 (0xc4ea bytes) 000863 1749605854.800051 write 0x022838 thru 0x02e875 (0xc03e bytes) 000864 1749605854.800992 read 0x01021c thru 0x01d491 (0xd276 bytes) 000867 1749605854.801896 trunc from 0x02e876 to 0x03ee0f (0x1059a bytes) 000868 1749605854.805566 fallocate 0x00426c thru 0x004d71 (0xb06 bytes) 000869 1749605854.806950 fallocate 0x0287aa thru 0x02d1c9 (0x4a20 bytes) 000870 1749605854.808739 fallocate 0x039036 thru 0x03ffff (0x6fca bytes) 000871 1749605854.810258 mapwrite 0x000bc0 thru 0x0011ba (0x5fb bytes) 000872 1749605854.817826 read 0x01ebc4 thru 0x0204a3 (0x18e0 bytes) 000874 1749605854.818732 punch from 0x011535 to 0x01bcc8 (0xa794 bytes) 000875 1749605854.820106 trunc from 0x03ee10 to 0x011748 (0x2d6c7 bytes) 000876 1749605854.824545 write 0x03c1d9 thru 0x03ffff (0x3e27 bytes) 000877 1749605854.825259 punch from 0x01f820 to 0x021c39 (0x241a bytes) 000878 1749605854.826878 write 0x0365e7 thru 0x03ffff (0x9a19 bytes) 000879 1749605854.829819 fallocate 0x03ab6b thru 0x03e9cc (0x3e62 bytes) 000880 1749605854.831180 write 0x008728 thru 0x00f8d6 (0x71af bytes) 000881 1749605854.831751 punch from 0x0039b9 to 0x007da2 (0x43ea bytes) 000882 1749605854.832935 fallocate 0x034bda thru 0x034d89 (0x1b0 bytes) 000883 1749605854.835279 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000884 1749605854.884826 mapread 0x012cf6 thru 0x01c795 (0x9aa0 bytes) 000885 1749605854.892123 mapread 0x015f33 thru 0x02106f (0xb13d bytes) 000886 1749605854.892986 mapwrite 0x036926 thru 0x03cbd1 (0x62ac bytes) 000887 1749605854.901594 mapwrite 0x031419 thru 0x036634 (0x521c bytes) 000888 1749605854.906553 trunc from 0x006694 to 0x00f19b (0x8b08 bytes) 000889 1749605854.908009 trunc from 0x015369 to 0x024809 (0xf4a1 bytes) 000890 1749605854.909297 read 0x0374fe thru 0x03e6c9 (0x71cc bytes) 000891 1749605854.909608 mapwrite 0x015363 thru 0x01a874 (0x5512 bytes) 000892 1749605854.915474 fallocate 0x032a6e thru 0x03b855 (0x8de8 bytes) 000893 1749605854.917032 fallocate 0x02031d thru 0x020426 (0x10a bytes) 000894 1749605854.918897 write 0x0341aa thru 0x03ffff (0xbe56 bytes) 000895 1749605854.921249 punch from 0x03001d to 0x03ace5 (0xacc9 bytes) 000896 1749605854.925357 read 0x018ab7 thru 0x01fbd8 (0x7122 bytes) 000897 1749605854.928796 trunc from 0x017442 to 0x018865 (0x1424 bytes) 000898 1749605854.930394 read 0x03b369 thru 0x03c815 (0x14ad bytes) 000899 1749605854.930794 punch from 0x0228a2 to 0x023c92 (0x13f1 bytes) 000902 1749605854.932638 punch from 0x031719 to 0x03ffff (0xe8e7 bytes) 000903 1749605854.935400 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000904 1749605854.978505 read 0x0190f7 thru 0x01c521 (0x342b bytes) 000905 1749605854.980348 mapwrite 0x038150 thru 0x03a959 (0x280a bytes) 000906 1749605854.990044 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000908 1749605855.040384 mapwrite 0x006a44 thru 0x016980 (0xff3d bytes) 000909 1749605855.053357 read 0x029031 thru 0x02fd19 (0x6ce9 bytes) 000910 1749605855.057008 MIRROR_EXTEND to 7 mirrors 000911 1749605855.165978 read 0x03600a thru 0x039761 (0x3758 bytes) 000912 1749605855.169828 mapread 0x030460 thru 0x032e0e (0x29af bytes) 000913 1749605855.179297 mapread 0x006e20 thru 0x00b8f0 (0x4ad1 bytes) 000914 1749605855.185151 trunc from 0x040000 to 0x039911 (0x66ee bytes) 000915 1749605855.199132 write 0x021468 thru 0x02bd96 (0xa92f bytes) 000916 1749605855.202504 read 0x00e6b5 thru 0x0166ff (0x804b bytes) 000917 1749605855.205171 mapread 0x00359d thru 0x011886 (0xe2ea bytes) 000918 1749605855.205756 fallocate 0x018596 thru 0x0217bb (0x9226 bytes) 000919 1749605855.207137 write 0x0393c6 thru 0x03ffff (0x6c3a bytes) 000920 1749605855.207668 read 0x00aa23 thru 0x0143c3 (0x99a1 bytes) 000921 1749605855.208040 read 0x034c52 thru 0x03cb8f (0x7f3e bytes) 000922 1749605855.209414 MIRROR_EXTEND to 8 mirrors 000924 1749605855.327291 punch from 0x010719 to 0x01eebf (0xe7a7 bytes) 000925 1749605855.336116 fallocate 0x03b847 thru 0x03ffff (0x47b9 bytes) 000926 1749605855.337602 trunc from 0x040000 to 0x015e9c (0x2a163 bytes) 000927 1749605855.346563 punch from 0x001cca to 0x00a9a2 (0x8cd9 bytes) 000928 1749605855.348124 trunc from 0x0101f2 to 0x015e9c (0x5cab bytes) 000929 1749605855.349661 mapwrite 0x0294e8 thru 0x0373a8 (0xdec1 bytes) 000930 1749605855.366141 trunc from 0x0373a9 to 0x01fad0 (0x178d8 bytes) 000931 1749605855.370402 fallocate 0x008bf7 thru 0x0132ae (0xa6b8 bytes) 000932 1749605855.371981 punch from 0x0131cd to 0x016ee9 (0x3d1d bytes) 000935 1749605855.374401 fallocate 0x003eab thru 0x0067db (0x2931 bytes) 000936 1749605855.375988 punch from 0x012c8a to 0x016e57 (0x41ce bytes) 000937 1749605855.377295 mapwrite 0x0214fa thru 0x027ea6 (0x69ad bytes) 000938 1749605855.389907 punch from 0x00e672 to 0x011f76 (0x3905 bytes) 000939 1749605855.391354 read 0x01b9ed thru 0x0246f7 (0x8d0b bytes) 000940 1749605855.392010 punch from 0x00bb31 to 0x019791 (0xdc61 bytes) 000941 1749605855.393506 fallocate 0x02f0a7 thru 0x0370f6 (0x8050 bytes) 000943 1749605855.395509 read 0x009637 thru 0x0189f3 (0xf3bd bytes) 000944 1749605855.398993 fallocate 0x01bc55 thru 0x01fb94 (0x3f40 bytes) 000945 1749605855.402069 MIRROR_RESYNC 8 mirrors 000946 1749605855.707545 trunc from 0x022431 to 0x02def3 (0xbac3 bytes) 000947 1749605855.715163 mapread 0x00f759 thru 0x01a29d (0xab45 bytes) 000948 1749605855.721356 trunc from 0x0370f7 to 0x017864 (0x1f892 bytes) 000950 1749605855.730485 write 0x010537 thru 0x0202cd (0xfd97 bytes) 000951 1749605855.732499 MIRROR_EXTEND to 9 mirrors 000952 1749605855.835069 MIRROR_EXTEND to 10 mirrors 000953 1749605855.934112 mapread 0x0048c3 thru 0x014710 (0xfe4e bytes) 000954 1749605855.940553 punch from 0x0087ad to 0x0100ec (0x7940 bytes) 000955 1749605855.950499 trunc from 0x00f8d6 to 0x014b70 (0x529b bytes) 000956 1749605855.952360 trunc from 0x00d875 to 0x01378d (0x5f19 bytes) 000958 1749605855.954566 fallocate 0x030957 thru 0x03ffff (0xf6a9 bytes) 000959 1749605855.958062 MIRROR_EXTEND to 11 mirrors 000960 1749605856.089543 fallocate 0x03ac0b thru 0x03ffff (0x53f5 bytes) 000961 1749605856.099870 mapwrite 0x00fa54 thru 0x01a766 (0xad13 bytes) 000962 1749605856.116052 mapwrite 0x020e01 thru 0x02549f (0x469f bytes) 000963 1749605856.125238 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000964 1749605856.173794 fallocate 0x0185e9 thru 0x01e869 (0x6281 bytes) 000965 1749605856.175212 trunc from 0x037191 to 0x03ffff (0x8e6f bytes) 000966 1749605856.176570 fallocate 0x011239 thru 0x01db54 (0xc91c bytes) 000967 1749605856.179490 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000968 1749605856.224083 write 0x000023 thru 0x00386f (0x384d bytes) 000969 1749605856.227851 punch from 0x0104ca to 0x0177e6 (0x731d bytes) 000970 1749605856.229798 punch from 0x00fc33 to 0x01f7aa (0xfb78 bytes) 000971 1749605856.232120 MIRROR_RESYNC 9 mirrors 000972 1749605856.714976 trunc from 0x0039ff to 0x0113c9 (0xd9cb bytes) 000974 1749605856.723871 write 0x01db59 thru 0x029f1c (0xc3c4 bytes) 000975 1749605856.726600 trunc from 0x040000 to 0x0378fd (0x8702 bytes) 000976 1749605856.731172 mapwrite 0x03df23 thru 0x03ffff (0x20dd bytes) 000978 1749605856.742594 read 0x011e1b thru 0x020321 (0xe507 bytes) 000979 1749605856.745065 punch from 0x03aa00 to 0x03ffff (0x5600 bytes) 000981 1749605856.746881 mapwrite 0x03d230 thru 0x03ffff (0x2dd0 bytes) 000982 1749605856.752529 fallocate 0x00cdb4 thru 0x010556 (0x37a3 bytes) 000983 1749605856.754098 read 0x001855 thru 0x0024df (0xc8b bytes) 000984 1749605856.757744 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xbb4:0x0]: Invalid argument (22) 000985 1749605856.804154 punch from 0x020446 to 0x028e44 (0x89ff bytes) 000986 1749605856.806000 read 0x02d3cc thru 0x03c53e (0xf173 bytes) 000987 1749605856.808632 read 0x02d787 thru 0x02f8d8 (0x2152 bytes) 000988 1749605856.808937 read 0x00329e thru 0x00de70 (0xabd3 bytes) 000989 1749605856.812232 punch from 0x021beb to 0x02e584 (0xc99a bytes) 000990 1749605856.813873 fallocate 0x034651 thru 0x03afde (0x698e bytes) 000991 1749605856.815510 trunc from 0x040000 to 0x00f699 (0x30966 bytes) 000992 1749605856.820752 punch from 0x0023bf to 0x007db5 (0x59f7 bytes) 000993 1749605856.822095 trunc from 0x00f69a to 0x00e7b5 (0xee4 bytes) 000995 1749605856.825859 mapwrite 0x023c24 thru 0x031f4e (0xe32b bytes) 000996 1749605856.852479 fallocate 0x02bf3d thru 0x032ade (0x6ba2 bytes) 000997 1749605856.854018 write 0x004c6b thru 0x012590 (0xd926 bytes) 000998 1749605856.856955 read 0x00780f thru 0x013901 (0xc0f3 bytes) 000999 1749605856.859314 fallocate 0x00a2f4 thru 0x0117d1 (0x74de bytes) 001000 1749605856.860948 fallocate 0x01814c thru 0x019ff9 (0x1eae bytes) All operations completed A-OK! PASS 70a (11s) == sanity-flr test 71: check mirror extend parent fid ==== 21:37:37 (1749605857) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0xbf0:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 21:37:40 (1749605860) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4919168 .. Extending file size to 3434560 .. done Extending file size to 7397760 .. Extending file size to 7068960 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8605632 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9331968 .. failed Extending file size to 7464512 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8373536 .. done Extending file size to 8787776 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8806336 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8809152 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8999968 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 5477 5478 5480 5481 5484 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) 192.168.201.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (63s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:38:44 (1749605924) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8751968 .. Extending file size to 7929248 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8248832 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8895552 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8870528 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9185696 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8979008 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9038304 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9232896 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9219456 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 9811 9812 9813 9815 9817 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) 192.168.201.114@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:39:47 (1749605987) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2776 1284912 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3548 1284140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14852 3592168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22780 3584240 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37632 7176408 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2776 1284912 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3548 1284140 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14852 3592168 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22780 3584240 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37632 7176408 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0286588 s, 73.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0458467 s, 45.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) PASS 200c (4s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:39:51 (1749605991) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0240231 s, 87.3 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 21:39:53 (1749605993) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:39:55 (1749605995) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0362698 s, 57.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:39:57 (1749605997) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0210579 s, 99.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:39:59 (1749605999) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0194152 s, 108 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb1:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:40:23 (1749606023) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219779 s, 47.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1072:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1074:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1073:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1072:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1074:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1073:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1073:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:40:58 (1749606058) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.035208 s, 59.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (34s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:41:32 (1749606092) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg114-server: Pool lustre.test_204f created Adding targets to pool oleg114-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg114-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0202437 s, 104 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1075:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1074:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1075:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1074:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg114-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg114-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg114-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:42:19 (1749606139) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:42:21 (1749606141) PASS 205b (1s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:42:22 (1749606142) oleg114-server: Pool lustre.test_206 created Adding targets to pool oleg114-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg114-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg114-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg114-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg114-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:42:37 (1749606157) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184429 s, 56.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0087452 s, 120 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3c29f5ef Mirror 3: 0x3c29f5ef PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:42:39 (1749606159) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:42:40 (1749606160) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:42:41 (1749606161) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00969167 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2560000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2560000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2560000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c00000 Read this (4 bytes): BBBB 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:42:54 (1749606174) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00322988 s, 2.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:42:56 (1749606176) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00945132 s, 111 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xba:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-10 21:42:56.000000000 -0400 Modify: 2025-06-10 21:42:56.000000000 -0400 Change: 2025-06-10 21:42:57.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:42:58 (1749606178) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00852042 s, 123 MB/s oleg114-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg114-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg114-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg114-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg114-server: O/280000401/d27/bb: File not found by ext2_lookup PASS 210b (10s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:43:08 (1749606188) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0268473 s, 1.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00539914 s, 759 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 10 21:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 10 21:43 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 704 sec ============ 21:43:10 (1749606190) === sanity-flr: start cleanup 21:43:11 (1749606191) === === sanity-flr: finish cleanup 21:43:12 (1749606192) ===