-----============= acceptance-small: sanity-flr ============----- Wed May 28 17:50:03 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg232-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:50:07 (1748469007) === oleg232-client.virtnet: executing check_config_client /mnt/lustre oleg232-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg232-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b4004000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b4004000.idle_timeout=debug disable quota as required oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:50:17 (1748469017) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7934] == sanity-flr test 0a: lfs mirror create with -N option == 17:50:17 (1748469017) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:50:22 (1748469022) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:50:23 (1748469023) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:50:24 (1748469024) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:50:31 (1748469031) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:50:32 (1748469032) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 17:50:33 (1748469033) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b4004000.stats=clear osc.lustre-OST0001-osc-ffff8800b4004000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:50:35 (1748469035) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:50:38 (1748469038) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:50:40 (1748469040) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:50:41 (1748469041) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 17:50:44 (1748469044) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:50:46 (1748469046) PASS 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:50:47 (1748469047) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:50:49 (1748469049) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:50:52 (1748469052) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:50:53 (1748469053) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.190118 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0383762 s, 27.3 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:50:56 (1748469056) 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.148816 s, 141 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:50:58 (1748469058) Stopping /mnt/lustre-ost1 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b4004000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b4004000.ost_server_uuid in DISCONN state after 3 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg232-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b4004000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b4004000.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg232-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:51:27 (1748469087) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.111134 s, 75.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b4004000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b4004000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg232-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:51:41 (1748469101) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b4004000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b4004000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg232-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b4004000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b4004000.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg232-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (19s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:52:00 (1748469120) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.659508 s, 79.5 MB/s -rw-r--r-- 1 root root 52428800 May 28 17:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.214s user 0m0.000s sys 0m0.095s Stopping /mnt/lustre-ost1 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b4004000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b4004000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.216s user 0m0.001s sys 0m0.104s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg232-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b4004000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b4004000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.247s user 0m0.000s sys 0m0.113s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg232-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b4004000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b4004000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:52:21 (1748469141) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:52:22 (1748469142) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:52:22 (1748469142) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 17:52:23 (1748469143) PASS 35 (1s) == sanity-flr test 36a: write to mirrored files ========== 17:52:24 (1748469144) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.103131 s, 122 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:52:27 (1748469147) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00721791 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00781159 s, 134 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:52:28 (1748469148) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:52:31 (1748469151) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000619903 s, 1.7 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000383085 s, 2.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000512798 s, 2.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000486522 s, 2.1 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 17:52:33 (1748469153) create test files with size 15734380 .. de59cc742dd7f96e327be4040ea7c952 - 6cf6e38faaf1548869926f9fee0a6e1d - 84b05503c3aecdeec44951c94d9df7d2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 17:52:37 (1748469157) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:52:39 (1748469159) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:52:41 (1748469161) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017561 s, 59.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x37:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00935524 s, 112 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (1s) == sanity-flr test 41: lfs mirror resync check =========== 17:52:42 (1748469162) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0170099 s, 123 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0710424 s, 59.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0286314 s, 73.2 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 17:52:44 (1748469164) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 17:52:45 (1748469165) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:52:46 (1748469166) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0079763 s, 131 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (1s) == sanity-flr test 44a: lfs mirror split check =========== 17:52:47 (1748469167) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0668837 s, 47.0 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:52:49 (1748469169) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x44:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:52:52 (1748469172) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.064863 s, 162 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:53:15 (1748469195) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800b66fc800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0601027 s, 174 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88013528f800.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:53:18 (1748469198) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (9s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:53:27 (1748469207) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (8s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:53:35 (1748469215) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 17:53:36 (1748469216) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0834683 s, 37.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0649256 s, 48.5 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 9a87d1c3fa531477663735ddc0f62553 - mirror 1 checksum: e26d27efc124303ea6cae81d4e78c47c - mirror 2 checksum: 9a87d1c3fa531477663735ddc0f62553 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 28 17:53:37 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 28 17:53:37 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9a87d1c3fa531477663735ddc0f62553 - mirror 1 checksum: e26d27efc124303ea6cae81d4e78c47c - mirror 2 checksum: e26d27efc124303ea6cae81d4e78c47c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 28 17:53:37 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 28 17:53:37 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:53:39 (1748469219) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169019 s, 62.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:53:41 (1748469221) Failing mds1 on oleg232-server Stopping /mnt/lustre-mds1 (opts:) on oleg232-server 17:53:49 (1748469229) shut down facet: mds1 facet_host: oleg232-server facet_failover_host: oleg232-server Failover mds1 to oleg232-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 17:54:02 (1748469242) targets are mounted 17:54:02 (1748469242) facet_failover done oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0197461 s, 53.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:54:10 (1748469250) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0104233 s, 2.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00564643 s, 3.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:54:13 (1748469253) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0120468 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00502399 s, 4.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:54:15 (1748469255) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.0021663 s, 7.6 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:54:17 (1748469257) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00938353 s, 2.2 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:54:19 (1748469259) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00232674 s, 1.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00978425 s, 419 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:54:22 (1748469262) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:54:27 (1748469267) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:54:45 (1748469285) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:55:02 (1748469302) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:55:20 (1748469320) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (1s) == sanity-flr test 70: mirror create and split race ====== 17:55:21 (1748469321) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 26733 mirror split pid 26734 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7665: 26734 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26733 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 17:56:23 (1748469383) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 9383 skipping zero length punch hole skipping zero length zero range skipping zero size read fallocating to largest ever: 0x40000 000005 1748469383.973759 fallocate 0x036088 thru 0x03ffff (0x9f78 bytes) 000006 1748469383.975837 write 0x0243a1 thru 0x02e9f4 (0xa654 bytes) 000007 1748469383.976802 write 0x024082 thru 0x02890f (0x488e bytes) 000008 1748469383.977372 write 0x008a2e thru 0x0107ea (0x7dbd bytes) 000009 1748469383.981596 punch from 0x028e93 to 0x02e9f4 (0x5b62 bytes) 000011 1748469383.986825 trunc from 0x01cf91 to 0x022c1d (0x5c8d bytes) 000012 1748469383.988255 punch from 0x00d52e to 0x01818b (0xac5e bytes) 000013 1748469383.989688 trunc from 0x02e9f5 to 0x031141 (0x274d bytes) 000014 1748469383.993717 trunc from 0x031142 to 0x00561c (0x2bb25 bytes) 000015 1748469383.997607 mapwrite 0x008e7f thru 0x00b5c9 (0x274b bytes) 000016 1748469384.011176 write 0x026913 thru 0x032f5a (0xc648 bytes) 000017 1748469384.012164 read 0x029127 thru 0x030596 (0x7470 bytes) 000018 1748469384.012616 trunc from 0x011786 to 0x01a9ce (0x9249 bytes) 000019 1748469384.014353 trunc from 0x03023f to 0x032f5a (0x2d1c bytes) 000020 1748469384.018445 trunc from 0x032f5b to 0x0052e8 (0x2dc72 bytes) 000021 1748469384.023389 read 0x002561 thru 0x0052e8 (0x2d88 bytes) 000022 1748469384.025767 punch from 0x002806 to 0x0052e8 (0x2ae3 bytes) 000023 1748469384.027246 trunc from 0x0052e9 to 0x004b47 (0x7a1 bytes) 000024 1748469384.031448 mapread 0x00068b thru 0x00169e (0x1014 bytes) 000025 1748469384.032308 mapwrite 0x0177b4 thru 0x01c863 (0x50b0 bytes) 000026 1748469384.045161 mapwrite 0x03fb62 thru 0x03ffff (0x49e bytes) 000028 1748469384.056428 mapread 0x03d7b2 thru 0x03f7b1 (0x2000 bytes) 000030 1748469384.057066 fallocate 0x02b377 thru 0x031df9 (0x6a83 bytes) 000031 1748469384.058834 mapread 0x011d69 thru 0x01bb18 (0x9db0 bytes) 000032 1748469384.059338 trunc from 0x040000 to 0x0348ae (0xb751 bytes) 000034 1748469384.064220 punch from 0x01c8d5 to 0x02afcf (0xe6fb bytes) 000037 1748469384.067136 mapwrite 0x037935 thru 0x03ffff (0x86cb bytes) 000041 1748469384.093914 trunc from 0x040000 to 0x020aea (0x1f515 bytes) 000042 1748469384.100035 mapread 0x00fb85 thru 0x010bcb (0x1047 bytes) 000046 1748469384.102671 trunc from 0x0002b1 to 0x00661b (0x636b bytes) 000047 1748469384.105009 read 0x013bb6 thru 0x020867 (0xccb2 bytes) 000048 1748469384.108396 read 0x0205d2 thru 0x020aea (0x519 bytes) 000049 1748469384.108855 read 0x016a63 thru 0x01cfe3 (0x6581 bytes) 000050 1748469384.109294 write 0x0019cf thru 0x00d2a3 (0xb8d5 bytes) 000051 1748469384.110486 write 0x02f564 thru 0x02ff91 (0xa2e bytes) 000052 1748469384.111189 punch from 0x016632 to 0x01f40e (0x8ddd bytes) 000053 1748469384.113524 write 0x014a0a thru 0x0238ad (0xeea4 bytes) 000054 1748469384.116298 mapread 0x0163f3 thru 0x025cac (0xf8ba bytes) 000055 1748469384.120489 MIRROR_EXTEND to 2 mirrors 000056 1748469384.231452 write 0x00f86f thru 0x010735 (0xec7 bytes) 000057 1748469384.244250 trunc from 0x0150b0 to 0x023d8b (0xecdc bytes) 000058 1748469384.246186 punch from 0x00f95a to 0x01cf9f (0xd646 bytes) 000059 1748469384.249230 trunc from 0x02ff92 to 0x016de1 (0x191b0 bytes) 000060 1748469384.259438 fallocate 0x00f79c thru 0x01b8e5 (0xc14a bytes) 000061 1748469384.261479 mapread 0x006a02 thru 0x01417a (0xd779 bytes) 000063 1748469384.268257 read 0x008b2e thru 0x009e92 (0x1365 bytes) 000064 1748469384.268847 mapwrite 0x026fed thru 0x0284e8 (0x14fc bytes) 000065 1748469384.283340 MIRROR_EXTEND to 3 mirrors 000066 1748469384.391675 MIRROR_RESYNC 3 mirrors 000067 1748469384.496163 trunc from 0x013415 to 0x0185e0 (0x51cc bytes) 000068 1748469384.505993 write 0x0104db thru 0x010865 (0x38b bytes) 000069 1748469384.507895 mapread 0x0046c9 thru 0x0073bf (0x2cf7 bytes) 000070 1748469384.511960 mapread 0x01ef61 thru 0x0267ae (0x784e bytes) 000071 1748469384.516187 mapread 0x000b37 thru 0x00c838 (0xbd02 bytes) 000072 1748469384.518213 trunc from 0x0284e9 to 0x03ff6d (0x17a85 bytes) 000073 1748469384.528164 trunc from 0x03ff6e to 0x03f7c5 (0x7a8 bytes) 000074 1748469384.532150 mapread 0x028d87 thru 0x037bbb (0xee35 bytes) 000076 1748469384.538738 read 0x02e7b4 thru 0x0379b5 (0x9202 bytes) 000077 1748469384.539461 mapwrite 0x03efd7 thru 0x03ffff (0x1029 bytes) 000078 1748469384.550968 MIRROR_RESYNC 3 mirrors 000079 1748469384.695944 trunc from 0x040000 to 0x000eaf (0x3f150 bytes) 000080 1748469384.708344 fallocate 0x023ec6 thru 0x0279a7 (0x3ae2 bytes) 000081 1748469384.710865 MIRROR_RESYNC 3 mirrors 000082 1748469384.819491 mapread 0x0058af thru 0x01006a (0xa7bc bytes) 000083 1748469384.827164 trunc from 0x0279a8 to 0x000d16 (0x26c91 bytes) 000084 1748469384.844378 mapread 0x00062d thru 0x000d16 (0x6ea bytes) 000085 1748469384.846985 write 0x0363e4 thru 0x037231 (0xe4e bytes) 000086 1748469384.847912 trunc from 0x037232 to 0x02bd91 (0xb4a0 bytes) 000087 1748469384.854928 mapwrite 0x02e22f thru 0x039802 (0xb5d4 bytes) 000088 1748469384.875632 mapwrite 0x02e4bd thru 0x036b1d (0x8661 bytes) 000090 1748469384.884714 trunc from 0x039803 to 0x005312 (0x344f0 bytes) 000091 1748469384.891451 mapread 0x00356f thru 0x005312 (0x1da4 bytes) 000092 1748469384.897764 MIRROR_RESYNC 3 mirrors 000093 1748469385.003299 fallocate 0x029bfd thru 0x035760 (0xbb64 bytes) 000094 1748469385.016288 trunc from 0x00a1bf to 0x019db1 (0xfbf3 bytes) 000095 1748469385.017697 fallocate 0x0233bf thru 0x02c612 (0x9254 bytes) 000096 1748469385.020508 MIRROR_RESYNC 3 mirrors 000097 1748469385.182920 read 0x02a0e5 thru 0x02a1ac (0xc8 bytes) 000098 1748469385.184775 punch from 0x005b6a to 0x006781 (0xc18 bytes) 000099 1748469385.193355 read 0x02e0da thru 0x035760 (0x7687 bytes) 000100 1748469385.197093 punch from 0x0341fc to 0x035760 (0x1565 bytes) 000101 1748469385.198924 write 0x0088c1 thru 0x016ef8 (0xe638 bytes) 000102 1748469385.200309 trunc from 0x0152a4 to 0x0212b1 (0xc00e bytes) 000103 1748469385.206234 trunc from 0x035761 to 0x02cd02 (0x8a5e bytes) 000104 1748469385.214303 trunc from 0x02cd03 to 0x02bb0f (0x11f3 bytes) 000105 1748469385.219007 trunc from 0x00cd18 to 0x00db6e (0xe57 bytes) 000106 1748469385.220905 trunc from 0x01ebab to 0x02115a (0x25b0 bytes) 000107 1748469385.222412 fallocate 0x00d4fc thru 0x0125dc (0x50e1 bytes) 000108 1748469385.224161 read 0x013f1c thru 0x023a82 (0xfb67 bytes) 000109 1748469385.227059 trunc from 0x021460 to 0x02bb0f (0xa6b0 bytes) 000111 1748469385.229383 mapread 0x007fec thru 0x00d4cc (0x54e1 bytes) 000112 1748469385.229974 write 0x02d15f thru 0x030af0 (0x3992 bytes) 000113 1748469385.232660 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000114 1748469385.313763 punch from 0x029c48 to 0x030af0 (0x6ea9 bytes) 000116 1748469385.317114 fallocate 0x020448 thru 0x021a59 (0x1612 bytes) 000117 1748469385.320086 write 0x039407 thru 0x03ffff (0x6bf9 bytes) 000119 1748469385.324060 read 0x01b489 thru 0x0209b7 (0x552f bytes) 000120 1748469385.328250 trunc from 0x02c454 to 0x03941e (0xcfcb bytes) 000121 1748469385.335462 fallocate 0x038b18 thru 0x03d9fd (0x4ee6 bytes) 000122 1748469385.340628 MIRROR_EXTEND to 3 mirrors 000123 1748469385.536102 mapwrite 0x00157b thru 0x008ad3 (0x7559 bytes) 000124 1748469385.578191 MIRROR_EXTEND to 4 mirrors 000125 1748469385.768108 trunc from 0x040000 to 0x002d4d (0x3d2b2 bytes) 000126 1748469385.795307 punch from 0x0000c3 to 0x002d4d (0x2c8b bytes) 000127 1748469385.797766 write 0x0368b7 thru 0x03bd2d (0x5477 bytes) 000128 1748469385.798990 mapread 0x035afe thru 0x039bf0 (0x40f3 bytes) 000129 1748469385.802457 trunc from 0x03217f to 0x03b696 (0x9518 bytes) 000131 1748469385.809853 punch from 0x006dba to 0x013f05 (0xd14c bytes) 000132 1748469385.812466 trunc from 0x03bd2e to 0x00cbf3 (0x2f13a bytes) 000133 1748469385.820411 read 0x007b8f thru 0x00cbf3 (0x5065 bytes) 000134 1748469385.824206 punch from 0x005c9f to 0x00cbf3 (0x6f55 bytes) 000136 1748469385.827227 read 0x007f3a thru 0x00bbec (0x3cb3 bytes) 000137 1748469385.831362 mapread 0x00654b thru 0x00873c (0x21f2 bytes) 000138 1748469385.836878 punch from 0x006775 to 0x00cbf3 (0x647f bytes) 000139 1748469385.839475 fallocate 0x02e201 thru 0x0343f3 (0x61f3 bytes) 000141 1748469385.842491 mapread 0x01a6a3 thru 0x021227 (0x6b85 bytes) 000142 1748469385.846072 mapwrite 0x001cb0 thru 0x001fa7 (0x2f8 bytes) 000143 1748469385.851741 trunc from 0x0309b6 to 0x0343f3 (0x3a3e bytes) 000144 1748469385.853608 mapread 0x0151d4 thru 0x020187 (0xafb4 bytes) 000145 1748469385.854252 fallocate 0x018c79 thru 0x0209c0 (0x7d48 bytes) 000146 1748469385.856476 trunc from 0x0343f4 to 0x011091 (0x23362 bytes) 000147 1748469385.862015 mapread 0x00eec7 thru 0x011091 (0x21cb bytes) 000148 1748469385.862895 read 0x004df5 thru 0x00fb22 (0xad2e bytes) 000149 1748469385.866063 fallocate 0x02200b thru 0x02c38c (0xa382 bytes) 000151 1748469385.869031 trunc from 0x02c38d to 0x031f56 (0x5bca bytes) 000153 1748469385.874949 fallocate 0x023987 thru 0x02f837 (0xbeb1 bytes) 000154 1748469385.879139 MIRROR_RESYNC 4 mirrors 000155 1748469386.087250 trunc from 0x011d21 to 0x0123c5 (0x6a5 bytes) 000156 1748469386.100825 mapwrite 0x0051eb thru 0x0101e9 (0xafff bytes) 000157 1748469386.115992 fallocate 0x01a9f2 thru 0x02716d (0xc77c bytes) 000158 1748469386.118005 trunc from 0x024943 to 0x031f56 (0xd614 bytes) 000159 1748469386.119883 read 0x028b9b thru 0x02cd7f (0x41e5 bytes) 000160 1748469386.122636 trunc from 0x005948 to 0x009535 (0x3bee bytes) 000161 1748469386.125782 MIRROR_RESYNC 4 mirrors 000162 1748469386.373738 MIRROR_RESYNC 4 mirrors 000163 1748469386.433904 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000166 1748469386.492845 trunc from 0x02098d to 0x021998 (0x100c bytes) 000167 1748469386.500208 punch from 0x020b9f to 0x0268f8 (0x5d5a bytes) 000168 1748469386.501558 mapread 0x01120b thru 0x016b7f (0x5975 bytes) 000169 1748469386.505736 mapwrite 0x02a55d thru 0x02de74 (0x3918 bytes) 000170 1748469386.511918 trunc from 0x031f57 to 0x02cd1c (0x523a bytes) 000173 1748469386.521161 fallocate 0x03e8b3 thru 0x03ffff (0x174d bytes) 000174 1748469386.522575 write 0x0395b9 thru 0x03c038 (0x2a80 bytes) 000175 1748469386.524952 read 0x0216fd thru 0x0273f7 (0x5cfb bytes) 000176 1748469386.526453 fallocate 0x009f05 thru 0x01727e (0xd37a bytes) 000177 1748469386.527957 mapread 0x025915 thru 0x031d2d (0xc419 bytes) 000178 1748469386.529807 fallocate 0x030916 thru 0x038d5e (0x8449 bytes) 000179 1748469386.531203 mapwrite 0x021d74 thru 0x02b538 (0x97c5 bytes) 000182 1748469386.538208 fallocate 0x02f8cc thru 0x038a2f (0x9164 bytes) 000183 1748469386.539884 punch from 0x0202bd to 0x02db8d (0xd8d1 bytes) 000185 1748469386.542077 trunc from 0x03b852 to 0x03ffff (0x47ae bytes) 000186 1748469386.543570 mapread 0x02ed1c thru 0x0310fa (0x23df bytes) 000187 1748469386.543987 trunc from 0x01a17c to 0x01eb4a (0x49cf bytes) 000188 1748469386.545287 trunc from 0x031375 to 0x03da19 (0xc6a5 bytes) 000189 1748469386.547727 MIRROR_EXTEND to 4 mirrors 000190 1748469386.656455 mapwrite 0x013b67 thru 0x015818 (0x1cb2 bytes) 000191 1748469386.686682 write 0x039643 thru 0x03aa55 (0x1413 bytes) 000192 1748469386.689760 write 0x036de1 thru 0x03ffff (0x921f bytes) 000193 1748469386.690394 trunc from 0x02cc39 to 0x0328cf (0x5c97 bytes) 000195 1748469386.692364 trunc from 0x01fdf9 to 0x02f1db (0xf3e3 bytes) 000196 1748469386.693917 write 0x019280 thru 0x0219c5 (0x8746 bytes) 000197 1748469386.696844 MIRROR_EXTEND to 5 mirrors 000199 1748469386.812128 trunc from 0x040000 to 0x00bb6f (0x34490 bytes) 000200 1748469386.828291 fallocate 0x016123 thru 0x023a11 (0xd8ef bytes) 000201 1748469386.830023 read 0x000484 thru 0x002019 (0x1b96 bytes) 000202 1748469386.831921 mapread 0x0162ba thru 0x01df49 (0x7c90 bytes) 000203 1748469386.836991 read 0x0167be thru 0x023a11 (0xd254 bytes) 000204 1748469386.838587 MIRROR_EXTEND to 6 mirrors 000206 1748469386.936214 fallocate 0x013276 thru 0x022e6a (0xfbf5 bytes) 000207 1748469386.944808 mapread 0x015f02 thru 0x0162ee (0x3ed bytes) 000208 1748469386.947444 fallocate 0x02862c thru 0x02c4d1 (0x3ea6 bytes) 000209 1748469386.948851 read 0x006719 thru 0x00aeb2 (0x479a bytes) 000210 1748469386.951057 write 0x0287a0 thru 0x0333fb (0xac5c bytes) 000211 1748469386.952904 trunc from 0x010dde to 0x01c343 (0xb566 bytes) 000212 1748469386.954464 read 0x027352 thru 0x02ca4d (0x56fc bytes) 000213 1748469386.957203 trunc from 0x0333fc to 0x039dba (0x69bf bytes) 000214 1748469386.965765 mapread 0x027947 thru 0x036897 (0xef51 bytes) 000215 1748469386.968047 mapwrite 0x00cc56 thru 0x01a88a (0xdc35 bytes) 000216 1748469386.975359 mapread 0x01df30 thru 0x02dc0a (0xfcdb bytes) 000217 1748469386.976004 trunc from 0x039dbb to 0x01c492 (0x1d928 bytes) 000218 1748469386.979523 punch from 0x00c581 to 0x00f079 (0x2af9 bytes) 000219 1748469386.980949 write 0x03b278 thru 0x03ffff (0x4d88 bytes) 000220 1748469386.981588 fallocate 0x025dad thru 0x02dba1 (0x7df5 bytes) 000221 1748469386.983034 read 0x02b679 thru 0x031b4b (0x64d3 bytes) 000222 1748469386.986819 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000223 1748469387.036515 mapwrite 0x013c67 thru 0x0215c6 (0xd960 bytes) 000224 1748469387.049494 fallocate 0x00dfe7 thru 0x01957a (0xb594 bytes) 000225 1748469387.051357 trunc from 0x011865 to 0x01d7cd (0xbf69 bytes) 000226 1748469387.054395 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000227 1748469387.099931 write 0x000536 thru 0x00ebc5 (0xe690 bytes) 000228 1748469387.104020 trunc from 0x00b520 to 0x00f863 (0x4344 bytes) 000229 1748469387.107715 read 0x01a670 thru 0x01dffe (0x398f bytes) 000230 1748469387.109348 punch from 0x0352a9 to 0x036ce3 (0x1a3b bytes) 000231 1748469387.111753 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000232 1748469387.151486 trunc from 0x040000 to 0x0021a7 (0x3de58 bytes) 000233 1748469387.155960 read 0x001b2d thru 0x0021a7 (0x67b bytes) 000234 1748469387.157600 read 0x0001dc thru 0x0021a7 (0x1fcc bytes) 000235 1748469387.160355 MIRROR_RESYNC 3 mirrors 000236 1748469387.232729 trunc from 0x001644 to 0x0021a7 (0xb64 bytes) 000238 1748469387.240839 read 0x000aad thru 0x0021a7 (0x16fb bytes) 000239 1748469387.242233 mapwrite 0x014510 thru 0x01b22a (0x6d1b bytes) 000240 1748469387.259534 mapread 0x0134c9 thru 0x01b22a (0x7d62 bytes) 000241 1748469387.260027 fallocate 0x0149e7 thru 0x01c5eb (0x7c05 bytes) 000242 1748469387.261400 trunc from 0x0148e8 to 0x01b22a (0x6943 bytes) 000243 1748469387.262815 trunc from 0x01b22b to 0x028822 (0xd5f8 bytes) 000244 1748469387.266622 trunc from 0x028823 to 0x016376 (0x124ac bytes) 000245 1748469387.271062 trunc from 0x011a86 to 0x016376 (0x48f1 bytes) 000246 1748469387.272357 trunc from 0x016377 to 0x01c92a (0x65b4 bytes) 000247 1748469387.275748 fallocate 0x02b94d thru 0x03a1f2 (0xe8a6 bytes) 000248 1748469387.277221 trunc from 0x01c92b to 0x032f0f (0x165e5 bytes) 000249 1748469387.280749 punch from 0x01d5ef to 0x02c4ce (0xeee0 bytes) 000250 1748469387.282401 punch from 0x029428 to 0x029a34 (0x60d bytes) 000251 1748469387.283984 fallocate 0x00a66b thru 0x011440 (0x6dd6 bytes) 000252 1748469387.285659 fallocate 0x026a79 thru 0x02a13a (0x36c2 bytes) 000253 1748469387.287176 read 0x0050f4 thru 0x00ae22 (0x5d2f bytes) 000254 1748469387.291713 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000256 1748469387.335178 fallocate 0x007585 thru 0x007b37 (0x5b3 bytes) 000257 1748469387.338039 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000259 1748469387.364374 fallocate 0x027f24 thru 0x02ea36 (0x6b13 bytes) 000260 1748469387.366010 trunc from 0x032f10 to 0x022722 (0x107ed bytes) 000261 1748469387.370755 write 0x019caf thru 0x029500 (0xf852 bytes) 000263 1748469387.373222 mapread 0x018baf thru 0x02231c (0x976e bytes) 000264 1748469387.375003 punch from 0x00e692 to 0x01d7d3 (0xf142 bytes) 000265 1748469387.379403 read 0x0245c9 thru 0x029500 (0x4f38 bytes) 000266 1748469387.379667 read 0x0200c8 thru 0x029500 (0x9439 bytes) 000268 1748469387.380143 trunc from 0x029501 to 0x01b0b6 (0xe44a bytes) 000269 1748469387.383769 punch from 0x00d8e2 to 0x00f384 (0x1aa3 bytes) 000270 1748469387.385446 mapwrite 0x008550 thru 0x015cc4 (0xd775 bytes) 000271 1748469387.393587 write 0x0154bd thru 0x023c9f (0xe7e3 bytes) 000273 1748469387.394641 mapread 0x00ba9a thru 0x00f5b9 (0x3b20 bytes) 000275 1748469387.395259 read 0x023c75 thru 0x023c9f (0x2b bytes) 000277 1748469387.395908 read 0x021cfe thru 0x023c9f (0x1fa2 bytes) 000278 1748469387.396225 write 0x02ea44 thru 0x031a62 (0x301f bytes) 000279 1748469387.396955 punch from 0x02b263 to 0x031a62 (0x6800 bytes) 000280 1748469387.400360 mapwrite 0x02c7f1 thru 0x0343c8 (0x7bd8 bytes) 000282 1748469387.412446 mapread 0x02abbc thru 0x031617 (0x6a5c bytes) 000284 1748469387.413267 mapread 0x013abe thru 0x01782f (0x3d72 bytes) 000285 1748469387.413656 write 0x0319aa thru 0x03b77f (0x9dd6 bytes) 000286 1748469387.414406 mapwrite 0x0342ee thru 0x03506c (0xd7f bytes) 000287 1748469387.420487 trunc from 0x01b305 to 0x02a790 (0xf48c bytes) 000288 1748469387.423254 MIRROR_EXTEND to 2 mirrors 000289 1748469387.540089 trunc from 0x001f25 to 0x00f15f (0xd23b bytes) 000290 1748469387.547987 write 0x03dffe thru 0x03ffff (0x2002 bytes) 000291 1748469387.549564 trunc from 0x040000 to 0x000788 (0x3f877 bytes) 000292 1748469387.558010 mapwrite 0x019669 thru 0x0289fc (0xf394 bytes) 000293 1748469387.574126 fallocate 0x022ccb thru 0x030a67 (0xdd9d bytes) 000294 1748469387.577232 MIRROR_EXTEND to 3 mirrors 000295 1748469387.720462 fallocate 0x03e375 thru 0x03ffff (0x1c8b bytes) 000296 1748469387.732235 trunc from 0x040000 to 0x013878 (0x2c787 bytes) 000297 1748469387.749872 trunc from 0x00edc7 to 0x013878 (0x4ab2 bytes) 000298 1748469387.754817 MIRROR_EXTEND to 4 mirrors 000299 1748469387.921338 punch from 0x01164e to 0x013878 (0x222b bytes) 000300 1748469387.936069 fallocate 0x01ee60 thru 0x0218d9 (0x2a7a bytes) 000301 1748469387.938710 mapwrite 0x02f8db thru 0x031384 (0x1aaa bytes) 000302 1748469387.968186 fallocate 0x01389b thru 0x0166a3 (0x2e09 bytes) 000303 1748469387.970207 punch from 0x01c9a1 to 0x0284fc (0xbb5c bytes) 000304 1748469387.971993 mapread 0x002ae2 thru 0x00683a (0x3d59 bytes) 000305 1748469387.975151 mapread 0x0118f3 thru 0x017054 (0x5762 bytes) 000306 1748469387.979130 trunc from 0x031385 to 0x00a0e3 (0x272a1 bytes) 000307 1748469387.986196 fallocate 0x02d46e thru 0x030adb (0x366e bytes) 000308 1748469387.989282 read 0x02d3b6 thru 0x030adb (0x3726 bytes) 000309 1748469387.992358 trunc from 0x006e4e to 0x008a37 (0x1bea bytes) 000310 1748469387.994982 write 0x030818 thru 0x03e5fe (0xdde7 bytes) 000311 1748469387.997412 fallocate 0x023b6f thru 0x02a6c9 (0x6b5b bytes) 000312 1748469388.000105 trunc from 0x03457f to 0x03a7bc (0x623e bytes) 000313 1748469388.007248 mapread 0x01ab4d thru 0x01dbda (0x308e bytes) 000314 1748469388.012112 fallocate 0x015680 thru 0x01bc2e (0x65af bytes) 000316 1748469388.015816 trunc from 0x036f3e to 0x03dbf3 (0x6cb6 bytes) 000317 1748469388.018226 read 0x037021 thru 0x03e5fe (0x75de bytes) 000319 1748469388.022989 trunc from 0x03e5ff to 0x004863 (0x39d9b bytes) 000320 1748469388.030287 punch from 0x00244f to 0x004863 (0x2415 bytes) 000321 1748469388.032721 trunc from 0x004864 to 0x028762 (0x23eff bytes) 000322 1748469388.040386 trunc from 0x01a8c1 to 0x01b5c3 (0xd03 bytes) 000323 1748469388.043115 fallocate 0x01b923 thru 0x01e674 (0x2d52 bytes) 000324 1748469388.045983 mapread 0x0192be thru 0x01e1f6 (0x4f39 bytes) 000325 1748469388.060187 fallocate 0x02cef9 thru 0x03b52f (0xe637 bytes) 000326 1748469388.063221 mapread 0x022a07 thru 0x028762 (0x5d5c bytes) 000327 1748469388.079702 mapread 0x001111 thru 0x00faba (0xe9aa bytes) 000328 1748469388.111594 mapread 0x025584 thru 0x028762 (0x31df bytes) 000329 1748469388.112409 read 0x0006f0 thru 0x0032b9 (0x2bca bytes) 000330 1748469388.112976 trunc from 0x028763 to 0x03fee0 (0x1777e bytes) 000331 1748469388.122276 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000332 1748469388.204419 punch from 0x01fa90 to 0x025951 (0x5ec2 bytes) 000333 1748469388.206832 trunc from 0x02ca61 to 0x031528 (0x4ac8 bytes) 000334 1748469388.209147 fallocate 0x007488 thru 0x00c103 (0x4c7c bytes) 000335 1748469388.211980 trunc from 0x03fee1 to 0x02f162 (0x10d7e bytes) 000336 1748469388.218937 mapread 0x022050 thru 0x0287eb (0x679c bytes) 000337 1748469388.225240 read 0x007b5f thru 0x0131fd (0xb69f bytes) 000338 1748469388.228203 mapwrite 0x01d453 thru 0x01e9cc (0x157a bytes) 000339 1748469388.238933 trunc from 0x02f163 to 0x03fa71 (0x1090f bytes) 000340 1748469388.245940 write 0x03fd9b thru 0x03ffff (0x265 bytes) 000341 1748469388.250746 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000342 1748469388.308447 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000343 1748469388.345346 trunc from 0x02701f to 0x02da88 (0x6a6a bytes) 000344 1748469388.347422 trunc from 0x040000 to 0x037b80 (0x847f bytes) 000345 1748469388.354910 mapwrite 0x027f15 thru 0x03208e (0xa17a bytes) 000346 1748469388.372638 write 0x037e4a thru 0x03ffff (0x81b6 bytes) 000348 1748469388.375563 punch from 0x023268 to 0x025dbb (0x2b54 bytes) 000349 1748469388.377418 trunc from 0x006525 to 0x006bc6 (0x6a2 bytes) 000351 1748469388.379574 fallocate 0x023c23 thru 0x028658 (0x4a36 bytes) 000352 1748469388.381354 fallocate 0x0276f5 thru 0x0350d8 (0xd9e4 bytes) 000353 1748469388.383182 read 0x009da0 thru 0x00d93f (0x3ba0 bytes) 000354 1748469388.387089 trunc from 0x040000 to 0x03c0b5 (0x3f4a bytes) 000356 1748469388.392546 mapwrite 0x00b96e thru 0x0167ed (0xae80 bytes) 000357 1748469388.404724 fallocate 0x03794a thru 0x039380 (0x1a37 bytes) 000358 1748469388.406639 write 0x033e99 thru 0x03ffff (0xc167 bytes) 000359 1748469388.407977 mapread 0x0061ef thru 0x00e0bc (0x7ece bytes) 000360 1748469388.411680 mapread 0x036ae6 thru 0x03ffff (0x951a bytes) 000361 1748469388.413002 mapwrite 0x015cea thru 0x023934 (0xdc4b bytes) 000362 1748469388.432985 mapread 0x01b10e thru 0x0265a0 (0xb493 bytes) 000363 1748469388.433905 fallocate 0x004322 thru 0x00a7c1 (0x64a0 bytes) 000365 1748469388.436423 trunc from 0x02831c to 0x036fc1 (0xeca6 bytes) 000366 1748469388.438634 trunc from 0x040000 to 0x0237fd (0x1c802 bytes) 000367 1748469388.444392 fallocate 0x00dfe5 thru 0x01a14b (0xc167 bytes) 000369 1748469388.446863 read 0x0176ed thru 0x01d499 (0x5dad bytes) 000370 1748469388.447344 trunc from 0x02181c to 0x0237fd (0x1fe2 bytes) 000371 1748469388.449216 write 0x021b68 thru 0x028200 (0x6699 bytes) 000373 1748469388.453089 trunc from 0x028201 to 0x014b78 (0x13688 bytes) 000374 1748469388.458836 mapwrite 0x02abbb thru 0x02ae18 (0x25e bytes) 000375 1748469388.474408 trunc from 0x00890b to 0x011d7e (0x9474 bytes) 000376 1748469388.476537 fallocate 0x03b828 thru 0x03ffff (0x47d8 bytes) 000377 1748469388.478567 write 0x036c74 thru 0x03ffff (0x938c bytes) 000378 1748469388.479625 punch from 0x0371f3 to 0x03adc9 (0x3bd7 bytes) 000379 1748469388.484683 read 0x0099f4 thru 0x00d49d (0x3aaa bytes) 000380 1748469388.489278 punch from 0x03d75c to 0x03ffff (0x28a4 bytes) 000381 1748469388.492459 punch from 0x000d81 to 0x00b785 (0xaa05 bytes) 000383 1748469388.494945 mapread 0x000593 thru 0x005ccf (0x573d bytes) 000384 1748469388.505408 fallocate 0x01baf0 thru 0x02629e (0xa7af bytes) 000385 1748469388.508628 write 0x0240ec thru 0x02e494 (0xa3a9 bytes) 000386 1748469388.509516 read 0x0191f9 thru 0x025897 (0xc69f bytes) 000387 1748469388.509981 mapread 0x011068 thru 0x01c0d6 (0xb06f bytes) 000388 1748469388.510605 mapread 0x037a9f thru 0x03ffff (0x8561 bytes) 000391 1748469388.515207 mapread 0x02c5cf thru 0x02c881 (0x2b3 bytes) 000392 1748469388.516037 fallocate 0x012644 thru 0x0157c4 (0x3181 bytes) 000393 1748469388.518116 fallocate 0x00c516 thru 0x018971 (0xc45c bytes) 000394 1748469388.521481 trunc from 0x040000 to 0x020d26 (0x1f2d9 bytes) 000395 1748469388.526670 fallocate 0x00bc10 thru 0x01721c (0xb60d bytes) 000396 1748469388.529514 fallocate 0x026825 thru 0x02a3cf (0x3bab bytes) 000397 1748469388.531389 mapwrite 0x016e6d thru 0x023e08 (0xcf9c bytes) 000398 1748469388.546188 mapwrite 0x02b083 thru 0x02d3fb (0x2379 bytes) 000399 1748469388.560741 punch from 0x02b79b to 0x02d3fb (0x1c61 bytes) 000400 1748469388.562475 trunc from 0x01b753 to 0x02a01d (0xe8cb bytes) 000402 1748469388.564756 write 0x028b6a thru 0x033fde (0xb475 bytes) 000404 1748469388.567832 read 0x019aa8 thru 0x01f8b1 (0x5e0a bytes) 000405 1748469388.572839 MIRROR_EXTEND to 2 mirrors 000406 1748469388.702871 fallocate 0x02f89e thru 0x03229b (0x29fe bytes) 000407 1748469388.713808 MIRROR_RESYNC 2 mirrors 000408 1748469388.806178 mapread 0x00376b thru 0x008ce7 (0x557d bytes) 000409 1748469388.811386 punch from 0x017f9f to 0x020ad4 (0x8b36 bytes) 000411 1748469388.820131 trunc from 0x033fdf to 0x01a629 (0x199b5 bytes) 000412 1748469388.828264 fallocate 0x03dd97 thru 0x03ffff (0x2269 bytes) 000413 1748469388.829744 read 0x001ff9 thru 0x011ef6 (0xfefe bytes) 000414 1748469388.833672 MIRROR_RESYNC 2 mirrors 000415 1748469388.958823 fallocate 0x013acb thru 0x0176e3 (0x3c19 bytes) 000416 1748469388.975126 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000417 1748469389.028213 mapwrite 0x014c58 thru 0x022446 (0xd7ef bytes) 000418 1748469389.063242 mapwrite 0x012429 thru 0x01a0a1 (0x7c79 bytes) 000420 1748469389.071377 read 0x0123d7 thru 0x0182bc (0x5ee6 bytes) 000422 1748469389.072159 write 0x0047b4 thru 0x00d934 (0x9181 bytes) 000423 1748469389.076015 trunc from 0x022447 to 0x01e508 (0x3f3e bytes) 000424 1748469389.081999 fallocate 0x0346eb thru 0x03cc7f (0x8595 bytes) 000425 1748469389.084507 trunc from 0x00817e to 0x00e5a0 (0x6423 bytes) 000426 1748469389.091226 write 0x03740b thru 0x03ffff (0x8bf5 bytes) 000428 1748469389.093036 mapwrite 0x014d02 thru 0x01dcf3 (0x8ff2 bytes) 000429 1748469389.104516 read 0x0206b0 thru 0x02a0f2 (0x9a43 bytes) 000430 1748469389.109006 read 0x00169f thru 0x00c6bb (0xb01d bytes) 000431 1748469389.109401 write 0x021d89 thru 0x028982 (0x6bfa bytes) 000433 1748469389.110550 fallocate 0x00437d thru 0x005a3f (0x16c3 bytes) 000434 1748469389.112644 trunc from 0x040000 to 0x02a2e1 (0x15d1e bytes) 000435 1748469389.119737 mapwrite 0x03d017 thru 0x03ffff (0x2fe9 bytes) 000436 1748469389.137734 trunc from 0x02c239 to 0x039e26 (0xdbee bytes) 000437 1748469389.139664 write 0x017388 thru 0x01bfb7 (0x4c30 bytes) 000438 1748469389.140294 punch from 0x01a797 to 0x025789 (0xaff3 bytes) 000439 1748469389.146061 mapwrite 0x0213c9 thru 0x030416 (0xf04e bytes) 000440 1748469389.178166 mapwrite 0x020f4a thru 0x028d58 (0x7e0f bytes) 000441 1748469389.190263 fallocate 0x03f6ce thru 0x03ffff (0x932 bytes) 000442 1748469389.192740 read 0x0010c9 thru 0x00ce85 (0xbdbd bytes) 000443 1748469389.193182 mapread 0x0006ad thru 0x006525 (0x5e79 bytes) 000445 1748469389.194096 trunc from 0x040000 to 0x02be13 (0x141ec bytes) 000446 1748469389.201102 read 0x006b51 thru 0x007288 (0x738 bytes) 000448 1748469389.202416 read 0x023136 thru 0x02be13 (0x8cde bytes) 000449 1748469389.202890 punch from 0x006ea4 to 0x010d08 (0x9e65 bytes) 000450 1748469389.205989 fallocate 0x0286e4 thru 0x0301ff (0x7b1c bytes) 000451 1748469389.209169 trunc from 0x019cce to 0x02233a (0x866d bytes) 000452 1748469389.212192 trunc from 0x00c037 to 0x0173c0 (0xb38a bytes) 000453 1748469389.214729 mapwrite 0x029d81 thru 0x038404 (0xe684 bytes) 000454 1748469389.237040 read 0x0359ee thru 0x038404 (0x2a17 bytes) 000456 1748469389.238482 punch from 0x02d4c1 to 0x038404 (0xaf44 bytes) 000457 1748469389.240991 mapwrite 0x00d6ac thru 0x018c4a (0xb59f bytes) 000458 1748469389.249520 trunc from 0x0151ba to 0x02453d (0xf384 bytes) 000459 1748469389.252049 write 0x0155ba thru 0x01c560 (0x6fa7 bytes) 000460 1748469389.256042 mapread 0x00f761 thru 0x010999 (0x1239 bytes) 000462 1748469389.258043 fallocate 0x03e22a thru 0x03ffff (0x1dd6 bytes) 000463 1748469389.260695 write 0x027bbf thru 0x02e342 (0x6784 bytes) 000464 1748469389.261535 mapwrite 0x01e364 thru 0x026183 (0x7e20 bytes) 000465 1748469389.271241 trunc from 0x040000 to 0x0076af (0x38950 bytes) 000467 1748469389.278685 mapread 0x006e86 thru 0x0076af (0x82a bytes) 000468 1748469389.279883 write 0x0271ef thru 0x02a2a7 (0x30b9 bytes) 000469 1748469389.280857 write 0x00dd2b thru 0x01bd51 (0xe027 bytes) 000470 1748469389.284660 mapwrite 0x002306 thru 0x008d89 (0x6a84 bytes) 000471 1748469389.294563 mapwrite 0x014b5a thru 0x01b46e (0x6915 bytes) 000472 1748469389.301806 read 0x007bfe thru 0x012cac (0xb0af bytes) 000473 1748469389.305985 trunc from 0x01602d to 0x019450 (0x3424 bytes) 000475 1748469389.307754 mapread 0x00a121 thru 0x01552a (0xb40a bytes) 000476 1748469389.308410 mapread 0x00617d thru 0x00fb0d (0x9991 bytes) 000477 1748469389.308987 trunc from 0x01b829 to 0x01b9f9 (0x1d1 bytes) 000479 1748469389.311278 read 0x01a166 thru 0x01e797 (0x4632 bytes) 000480 1748469389.314178 write 0x00c6e3 thru 0x0107e9 (0x4107 bytes) 000481 1748469389.315631 fallocate 0x030030 thru 0x03705b (0x702c bytes) 000483 1748469389.319361 trunc from 0x03705c to 0x030917 (0x6744 bytes) 000485 1748469389.326185 write 0x0180f1 thru 0x01f9b8 (0x78c8 bytes) 000486 1748469389.327285 trunc from 0x030918 to 0x024bde (0xbd39 bytes) 000487 1748469389.332797 write 0x01f77d thru 0x02c08e (0xc912 bytes) 000488 1748469389.334372 write 0x02dade thru 0x038b60 (0xb083 bytes) 000489 1748469389.335818 mapread 0x0028e6 thru 0x00d67c (0xad97 bytes) 000491 1748469389.337255 fallocate 0x00bc67 thru 0x00ccd1 (0x106b bytes) 000492 1748469389.339841 fallocate 0x00378d thru 0x009ac3 (0x6337 bytes) 000493 1748469389.341959 mapwrite 0x039806 thru 0x03b9fa (0x21f5 bytes) 000494 1748469389.358579 read 0x00540b thru 0x00bcf5 (0x68eb bytes) 000495 1748469389.359202 mapread 0x015eda thru 0x01bdaa (0x5ed1 bytes) 000496 1748469389.360003 read 0x032445 thru 0x03b9fa (0x95b6 bytes) 000497 1748469389.360576 trunc from 0x00acde to 0x017f1f (0xd242 bytes) 000498 1748469389.362718 write 0x00f030 thru 0x0127ad (0x377e bytes) 000499 1748469389.366123 trunc from 0x033560 to 0x038a80 (0x5521 bytes) 000500 1748469389.368488 fallocate 0x013c72 thru 0x0222e4 (0xe673 bytes) 000501 1748469389.370658 write 0x0267cf thru 0x02a77d (0x3faf bytes) 000504 1748469389.372035 fallocate 0x01e29a thru 0x023f1a (0x5c81 bytes) 000506 1748469389.375686 punch from 0x039877 to 0x03b9fa (0x2184 bytes) 000509 1748469389.380237 mapread 0x019e2f thru 0x02848d (0xe65f bytes) 000510 1748469389.381781 trunc from 0x01f601 to 0x027423 (0x7e23 bytes) 000511 1748469389.388686 trunc from 0x01ad76 to 0x02ab2f (0xfdba bytes) 000513 1748469389.392031 trunc from 0x02c3a1 to 0x02d15c (0xdbc bytes) 000514 1748469389.396580 MIRROR_EXTEND to 2 mirrors 000515 1748469389.571576 mapread 0x034397 thru 0x03b9fa (0x7664 bytes) 000516 1748469389.579687 write 0x00287d thru 0x009a0b (0x718f bytes) 000518 1748469389.598905 trunc from 0x02d483 to 0x0352b2 (0x7e30 bytes) 000519 1748469389.601723 write 0x01db56 thru 0x02a03b (0xc4e6 bytes) 000520 1748469389.608920 punch from 0x0071f1 to 0x00747a (0x28a bytes) 000521 1748469389.616707 trunc from 0x030de3 to 0x03b066 (0xa284 bytes) 000522 1748469389.619311 mapread 0x0348d4 thru 0x03b9fa (0x7127 bytes) 000523 1748469389.623404 fallocate 0x00c6b8 thru 0x019406 (0xcd4f bytes) 000524 1748469389.625957 trunc from 0x03b9fb to 0x024b9b (0x16e5f bytes) 000527 1748469389.640475 fallocate 0x027bec thru 0x02bb73 (0x3f88 bytes) 000528 1748469389.642702 trunc from 0x00beec to 0x01b2aa (0xf3bf bytes) 000530 1748469389.645043 trunc from 0x02bb74 to 0x02d03c (0x14c9 bytes) 000531 1748469389.651012 read 0x00e6e4 thru 0x019615 (0xaf32 bytes) 000533 1748469389.655851 mapread 0x002833 thru 0x00d9bd (0xb18b bytes) 000534 1748469389.659393 read 0x01e88e thru 0x025d44 (0x74b7 bytes) 000536 1748469389.665444 MIRROR_RESYNC 2 mirrors 000537 1748469389.802648 trunc from 0x02d03d to 0x00c113 (0x20f29 bytes) 000538 1748469389.825102 read 0x00a482 thru 0x00c113 (0x1c92 bytes) 000539 1748469389.829060 trunc from 0x00c114 to 0x0098e0 (0x2833 bytes) 000540 1748469389.836001 write 0x01d151 thru 0x021bf4 (0x4aa4 bytes) 000541 1748469389.839119 MIRROR_EXTEND to 3 mirrors 000542 1748469390.014183 trunc from 0x021bf5 to 0x03e5a7 (0x1c9b3 bytes) 000545 1748469390.041442 mapread 0x012029 thru 0x018b3a (0x6b12 bytes) 000547 1748469390.051118 punch from 0x0065e6 to 0x00b311 (0x4d2c bytes) 000548 1748469390.053878 punch from 0x02d574 to 0x030e05 (0x3892 bytes) 000549 1748469390.056295 punch from 0x01f4dc to 0x02c991 (0xd4b6 bytes) 000550 1748469390.059937 trunc from 0x00b141 to 0x016047 (0xaf07 bytes) 000551 1748469390.065061 MIRROR_EXTEND to 4 mirrors 000552 1748469390.271783 fallocate 0x009daf thru 0x013593 (0x97e5 bytes) 000553 1748469390.287640 fallocate 0x016249 thru 0x018266 (0x201e bytes) 000554 1748469390.290890 mapwrite 0x03e029 thru 0x03ffff (0x1fd7 bytes) 000555 1748469390.320103 read 0x00e7df thru 0x01749a (0x8cbc bytes) 000556 1748469390.326965 MIRROR_RESYNC 4 mirrors 000557 1748469390.706648 trunc from 0x021437 to 0x02c202 (0xadcc bytes) 000558 1748469390.722104 punch from 0x01b70f to 0x023b3b (0x842d bytes) 000559 1748469390.724511 trunc from 0x02b1e4 to 0x0319db (0x67f8 bytes) 000560 1748469390.729168 MIRROR_RESYNC 4 mirrors 000562 1748469391.155807 trunc from 0x030bfb to 0x0368ce (0x5cd4 bytes) 000563 1748469391.170151 mapwrite 0x018917 thru 0x01bd15 (0x33ff bytes) 000564 1748469391.187255 punch from 0x00ecdc to 0x01144e (0x2773 bytes) 000565 1748469391.189706 trunc from 0x040000 to 0x01ed40 (0x212bf bytes) 000566 1748469391.199080 fallocate 0x028a04 thru 0x03863f (0xfc3c bytes) 000567 1748469391.201923 mapread 0x0038fd thru 0x008095 (0x4799 bytes) 000568 1748469391.206727 read 0x0021a5 thru 0x007072 (0x4ece bytes) 000569 1748469391.207350 read 0x009f38 thru 0x01841f (0xe4e8 bytes) 000570 1748469391.211972 trunc from 0x00acce to 0x019d0c (0xf03f bytes) 000571 1748469391.214893 trunc from 0x02308c to 0x026df8 (0x3d6d bytes) 000572 1748469391.217632 write 0x0260d0 thru 0x02af03 (0x4e34 bytes) 000573 1748469391.220687 write 0x036d9c thru 0x03bcbb (0x4f20 bytes) 000574 1748469391.223993 MIRROR_EXTEND to 5 mirrors 000575 1748469391.425455 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000576 1748469391.545393 trunc from 0x00e798 to 0x0152ba (0x6b23 bytes) 000577 1748469391.560613 mapread 0x0132a2 thru 0x01b5f9 (0x8358 bytes) 000578 1748469391.569431 punch from 0x03af2d to 0x03bcbb (0xd8f bytes) 000579 1748469391.572223 trunc from 0x03bcbc to 0x018c54 (0x23067 bytes) 000580 1748469391.591105 punch from 0x011493 to 0x0177e7 (0x6355 bytes) 000581 1748469391.593930 punch from 0x0046e4 to 0x00de2a (0x9747 bytes) 000583 1748469391.596807 write 0x029aa1 thru 0x0382a4 (0xe804 bytes) 000585 1748469391.598971 mapread 0x00ad3e thru 0x0126b2 (0x7975 bytes) 000586 1748469391.606180 mapwrite 0x007c76 thru 0x016e2b (0xf1b6 bytes) 000587 1748469391.618491 fallocate 0x02f7d1 thru 0x034210 (0x4a40 bytes) 000588 1748469391.623248 MIRROR_EXTEND to 5 mirrors 000589 1748469391.810882 fallocate 0x03cdd1 thru 0x03ffff (0x322f bytes) 000590 1748469391.821644 fallocate 0x0000b8 thru 0x00f466 (0xf3af bytes) 000591 1748469391.823617 trunc from 0x0382a5 to 0x0161f5 (0x220af bytes) 000592 1748469391.837716 mapwrite 0x00917d thru 0x00c270 (0x30f4 bytes) 000593 1748469391.851602 trunc from 0x00c9ee to 0x00dcf3 (0x1306 bytes) 000594 1748469391.853693 write 0x011fc4 thru 0x01ae2c (0x8e69 bytes) 000595 1748469391.854921 write 0x0351bf thru 0x03ffff (0xae41 bytes) 000596 1748469391.856239 fallocate 0x01af48 thru 0x02aac1 (0xfb7a bytes) 000597 1748469391.858873 fallocate 0x03f261 thru 0x03ffff (0xd9f bytes) 000598 1748469391.861166 read 0x02f72f thru 0x0321be (0x2a90 bytes) 000599 1748469391.863596 read 0x026698 thru 0x02f21d (0x8b86 bytes) 000600 1748469391.867382 mapread 0x016aa4 thru 0x01eb11 (0x806e bytes) 000601 1748469391.868164 trunc from 0x040000 to 0x018414 (0x27beb bytes) 000602 1748469391.874641 write 0x01ecb7 thru 0x026b93 (0x7edd bytes) 000603 1748469391.875564 trunc from 0x026b94 to 0x03351c (0xc989 bytes) 000604 1748469391.883167 MIRROR_RESYNC 5 mirrors 000606 1748469392.171455 mapread 0x019463 thru 0x01fffa (0x6b98 bytes) 000607 1748469392.178884 punch from 0x029352 to 0x02d14c (0x3dfb bytes) 000608 1748469392.191372 punch from 0x02fed7 to 0x03351c (0x3646 bytes) 000609 1748469392.193446 mapread 0x025e0b thru 0x0310ab (0xb2a1 bytes) 000610 1748469392.199892 trunc from 0x0136f7 to 0x0160a6 (0x29b0 bytes) 000611 1748469392.202299 punch from 0x00afde to 0x013fa2 (0x8fc5 bytes) 000612 1748469392.204255 write 0x02fce1 thru 0x037112 (0x7432 bytes) 000613 1748469392.205245 trunc from 0x037113 to 0x00f540 (0x27bd2 bytes) 000614 1748469392.213882 mapread 0x00167b thru 0x00df7a (0xc900 bytes) 000615 1748469392.219662 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000616 1748469392.284604 write 0x01c993 thru 0x01e26b (0x18d9 bytes) 000617 1748469392.287238 trunc from 0x01e26c to 0x0063eb (0x17e80 bytes) 000618 1748469392.292302 mapread 0x0060c6 thru 0x0063eb (0x326 bytes) 000619 1748469392.295190 punch from 0x0043e4 to 0x0063eb (0x2008 bytes) 000620 1748469392.297494 fallocate 0x02cd3a thru 0x02fabc (0x2d83 bytes) 000621 1748469392.299861 fallocate 0x02025a thru 0x02179a (0x1541 bytes) 000622 1748469392.301838 mapwrite 0x037cba thru 0x03ffff (0x8346 bytes) 000623 1748469392.323249 fallocate 0x0197b8 thru 0x01b364 (0x1bad bytes) 000624 1748469392.325225 trunc from 0x015254 to 0x015de6 (0xb93 bytes) 000625 1748469392.326958 mapread 0x03c5b1 thru 0x03ffff (0x3a4f bytes) 000626 1748469392.327778 write 0x00a71f thru 0x0193f1 (0xecd3 bytes) 000627 1748469392.335082 write 0x01d6f9 thru 0x02a418 (0xcd20 bytes) 000628 1748469392.340306 read 0x021e06 thru 0x0280ed (0x62e8 bytes) 000629 1748469392.340968 write 0x02a2eb thru 0x02c952 (0x2668 bytes) 000630 1748469392.344480 mapread 0x029a4b thru 0x03044d (0x6a03 bytes) 000631 1748469392.345601 read 0x03a740 thru 0x03ffff (0x58c0 bytes) 000632 1748469392.346453 mapwrite 0x02b408 thru 0x036fb1 (0xbbaa bytes) 000633 1748469392.357106 mapwrite 0x0182c8 thru 0x019cc4 (0x19fd bytes) 000634 1748469392.361875 mapread 0x009f8d thru 0x00c0a4 (0x2118 bytes) 000635 1748469392.362870 mapread 0x0295a0 thru 0x02abe6 (0x1647 bytes) 000636 1748469392.363850 trunc from 0x040000 to 0x01f2b9 (0x20d46 bytes) 000637 1748469392.369434 read 0x018880 thru 0x01d071 (0x47f2 bytes) 000638 1748469392.369966 trunc from 0x01f2ba to 0x019edc (0x53dd bytes) 000639 1748469392.374869 read 0x01192a thru 0x012406 (0xadd bytes) 000640 1748469392.375295 mapwrite 0x018093 thru 0x01da40 (0x59ae bytes) 000642 1748469392.391593 punch from 0x011d6c to 0x01da40 (0xbcd5 bytes) 000643 1748469392.394120 punch from 0x016bd4 to 0x01da40 (0x6e6d bytes) 000644 1748469392.396366 trunc from 0x0120ed to 0x01da40 (0xb954 bytes) 000645 1748469392.398610 fallocate 0x007d2c thru 0x00dda3 (0x6078 bytes) 000646 1748469392.401007 trunc from 0x01da41 to 0x017b82 (0x5ebe bytes) 000648 1748469392.408008 trunc from 0x017b83 to 0x03ad4f (0x231cd bytes) 000649 1748469392.414383 mapread 0x01ec80 thru 0x028fc4 (0xa345 bytes) 000650 1748469392.419613 trunc from 0x03ad50 to 0x01bce8 (0x1f067 bytes) 000651 1748469392.427133 mapwrite 0x036bf9 thru 0x03c31f (0x5727 bytes) 000652 1748469392.460018 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000653 1748469392.529957 trunc from 0x03c320 to 0x0216a3 (0x1ac7c bytes) 000654 1748469392.537152 trunc from 0x00aac3 to 0x0139d0 (0x8f0e bytes) 000655 1748469392.541391 MIRROR_RESYNC 3 mirrors 000656 1748469392.769203 mapwrite 0x0158e4 thru 0x024b1a (0xf237 bytes) 000657 1748469392.807136 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000658 1748469392.876140 trunc from 0x0220af to 0x024b1a (0x2a6c bytes) 000659 1748469392.878223 read 0x023706 thru 0x024b1a (0x1415 bytes) 000660 1748469392.880786 punch from 0x01ae31 to 0x0204ff (0x56cf bytes) 000661 1748469392.883229 fallocate 0x001a50 thru 0x0058e4 (0x3e95 bytes) 000662 1748469392.886406 trunc from 0x024b1b to 0x034069 (0xf54f bytes) 000663 1748469392.893691 trunc from 0x023f2e to 0x02dd4d (0x9e20 bytes) 000664 1748469392.895912 fallocate 0x036c41 thru 0x03ffff (0x93bf bytes) 000665 1748469392.898011 fallocate 0x031717 thru 0x03d3a7 (0xbc91 bytes) 000666 1748469392.900104 read 0x0278ee thru 0x02c14f (0x4862 bytes) 000667 1748469392.905031 punch from 0x014be1 to 0x01dd52 (0x9172 bytes) 000668 1748469392.907699 read 0x03e22d thru 0x03ffff (0x1dd3 bytes) 000669 1748469392.908098 mapread 0x00f81a thru 0x015d40 (0x6527 bytes) 000670 1748469392.910540 write 0x006392 thru 0x008a46 (0x26b5 bytes) 000671 1748469392.911032 mapread 0x013902 thru 0x02280a (0xef09 bytes) 000672 1748469392.911656 mapread 0x005044 thru 0x0146cd (0xf68a bytes) 000673 1748469392.912104 mapwrite 0x01b544 thru 0x029c3b (0xe6f8 bytes) 000674 1748469392.919456 trunc from 0x01dbcc to 0x01f1ad (0x15e2 bytes) 000675 1748469392.920955 read 0x0077b8 thru 0x0123eb (0xac34 bytes) 000676 1748469392.921247 fallocate 0x034952 thru 0x03a02d (0x56dc bytes) 000677 1748469392.922746 read 0x0258ea thru 0x03291b (0xd032 bytes) 000678 1748469392.924421 MIRROR_RESYNC 2 mirrors 000680 1748469393.031263 trunc from 0x040000 to 0x0005e3 (0x3fa1c bytes) 000681 1748469393.046658 trunc from 0x0005e4 to 0x016a09 (0x16426 bytes) 000682 1748469393.052391 read 0x00755b thru 0x00e8dc (0x7382 bytes) 000683 1748469393.055128 trunc from 0x016a0a to 0x0352ac (0x1e8a3 bytes) 000684 1748469393.059553 read 0x026f85 thru 0x0352ac (0xe328 bytes) 000685 1748469393.064636 fallocate 0x01d8e2 thru 0x023daf (0x64ce bytes) 000686 1748469393.067897 trunc from 0x02dd74 to 0x0352ac (0x7539 bytes) 000687 1748469393.071447 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000688 1748469393.112846 write 0x0072e7 thru 0x00bf10 (0x4c2a bytes) 000689 1748469393.118152 mapwrite 0x03c532 thru 0x03ffff (0x3ace bytes) 000690 1748469393.132040 mapwrite 0x030ca4 thru 0x0395c4 (0x8921 bytes) 000691 1748469393.140950 mapread 0x038cb2 thru 0x03d5bf (0x490e bytes) 000692 1748469393.141539 mapread 0x02bef5 thru 0x03915f (0xd26b bytes) 000693 1748469393.142304 fallocate 0x0017ff thru 0x00a05d (0x885f bytes) 000694 1748469393.144618 write 0x03be4c thru 0x03ffff (0x41b4 bytes) 000696 1748469393.145504 punch from 0x009673 to 0x016549 (0xced7 bytes) 000697 1748469393.147555 trunc from 0x01b627 to 0x022778 (0x7152 bytes) 000698 1748469393.149185 fallocate 0x039c5f thru 0x03fa1c (0x5dbe bytes) 000699 1748469393.151030 read 0x018da5 thru 0x01fa8c (0x6ce8 bytes) 000700 1748469393.153091 trunc from 0x040000 to 0x00b590 (0x34a6f bytes) 000701 1748469393.157560 fallocate 0x007be7 thru 0x00d308 (0x5722 bytes) 000702 1748469393.160665 mapread 0x00ab1d thru 0x00b590 (0xa74 bytes) 000703 1748469393.163663 mapwrite 0x02e9c1 thru 0x030290 (0x18d0 bytes) 000704 1748469393.179690 punch from 0x005ccb to 0x00cea0 (0x71d6 bytes) 000705 1748469393.182207 write 0x01c8ab thru 0x02bbb0 (0xf306 bytes) 000706 1748469393.186115 punch from 0x025874 to 0x027e64 (0x25f1 bytes) 000707 1748469393.191029 mapread 0x023a3a thru 0x026ff0 (0x35b7 bytes) 000708 1748469393.192876 read 0x01640f thru 0x01c55c (0x614e bytes) 000709 1748469393.193267 trunc from 0x023f64 to 0x02e610 (0xa6ad bytes) 000710 1748469393.194947 mapread 0x02ae40 thru 0x030290 (0x5451 bytes) 000712 1748469393.199238 trunc from 0x018ef8 to 0x027cd3 (0xeddc bytes) 000713 1748469393.201692 punch from 0x026ae0 to 0x030290 (0x97b1 bytes) 000717 1748469393.206258 write 0x024a7f thru 0x02aea5 (0x6427 bytes) 000718 1748469393.209766 read 0x0301db thru 0x030290 (0xb6 bytes) 000719 1748469393.212640 read 0x02ca45 thru 0x030290 (0x384c bytes) 000720 1748469393.213261 fallocate 0x0110a3 thru 0x01918b (0x80e9 bytes) 000722 1748469393.216047 write 0x01e51e thru 0x028910 (0xa3f3 bytes) 000723 1748469393.217026 mapread 0x0216ab thru 0x030290 (0xebe6 bytes) 000724 1748469393.217992 read 0x0242c6 thru 0x030290 (0xbfcb bytes) 000725 1748469393.218387 write 0x00e9b7 thru 0x018a91 (0xa0db bytes) 000726 1748469393.219079 fallocate 0x02192f thru 0x02c30c (0xa9de bytes) 000727 1748469393.220710 mapread 0x0072db thru 0x012fb1 (0xbcd7 bytes) 000728 1748469393.221170 punch from 0x00c584 to 0x01ac68 (0xe6e5 bytes) 000730 1748469393.225178 mapread 0x01b538 thru 0x023540 (0x8009 bytes) 000731 1748469393.225623 punch from 0x019faf to 0x01edc4 (0x4e16 bytes) 000732 1748469393.228983 MIRROR_EXTEND to 2 mirrors 000733 1748469393.324943 fallocate 0x03275e thru 0x03ffff (0xd8a2 bytes) 000734 1748469393.331956 write 0x014fb4 thru 0x022d56 (0xdda3 bytes) 000735 1748469393.335494 fallocate 0x0283ed thru 0x02c275 (0x3e89 bytes) 000736 1748469393.337173 punch from 0x0298c2 to 0x030290 (0x69cf bytes) 000737 1748469393.338662 write 0x01ef63 thru 0x027cb9 (0x8d57 bytes) 000738 1748469393.340293 mapread 0x02be98 thru 0x030290 (0x43f9 bytes) 000739 1748469393.343623 MIRROR_EXTEND to 3 mirrors 000740 1748469393.443518 punch from 0x024b31 to 0x02b73d (0x6c0d bytes) 000741 1748469393.451313 trunc from 0x030291 to 0x005acf (0x2a7c1 bytes) 000743 1748469393.460867 trunc from 0x004a1a to 0x005acf (0x10b6 bytes) 000745 1748469393.462681 mapread 0x003e09 thru 0x005acf (0x1cc7 bytes) 000746 1748469393.466690 read 0x002194 thru 0x005acf (0x393c bytes) 000747 1748469393.467018 read 0x00478f thru 0x005acf (0x1341 bytes) 000748 1748469393.467300 trunc from 0x0027da to 0x005acf (0x32f6 bytes) 000749 1748469393.468660 mapread 0x002d02 thru 0x003629 (0x928 bytes) 000750 1748469393.471241 trunc from 0x005ad0 to 0x0375f3 (0x31b24 bytes) 000751 1748469393.474664 trunc from 0x0375f4 to 0x007197 (0x3045c bytes) 000752 1748469393.479114 mapwrite 0x0373f1 thru 0x03ffff (0x8c0f bytes) 000753 1748469393.490620 trunc from 0x017856 to 0x01f408 (0x7bb3 bytes) 000754 1748469393.491955 read 0x01582b thru 0x01739b (0x1b71 bytes) 000755 1748469393.494347 trunc from 0x038195 to 0x03ffff (0x7e6b bytes) 000756 1748469393.495766 mapwrite 0x00ee95 thru 0x01b23f (0xc3ab bytes) 000757 1748469393.501802 punch from 0x017237 to 0x0252d1 (0xe09b bytes) 000758 1748469393.503203 write 0x002979 thru 0x004937 (0x1fbf bytes) 000759 1748469393.504753 punch from 0x03fa38 to 0x03ffff (0x5c8 bytes) 000760 1748469393.507373 MIRROR_RESYNC 3 mirrors 000761 1748469393.642411 MIRROR_RESYNC 3 mirrors 000762 1748469393.680013 write 0x01be22 thru 0x01fac5 (0x3ca4 bytes) 000763 1748469393.688296 trunc from 0x01bbb4 to 0x021550 (0x599d bytes) 000764 1748469393.691601 fallocate 0x00f4c1 thru 0x013409 (0x3f49 bytes) 000765 1748469393.694140 MIRROR_RESYNC 3 mirrors 000766 1748469393.848021 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000767 1748469393.899634 mapread 0x00916f thru 0x018750 (0xf5e2 bytes) 000768 1748469393.906648 trunc from 0x040000 to 0x008221 (0x37dde bytes) 000770 1748469393.920386 mapread 0x002ef9 thru 0x008221 (0x5329 bytes) 000773 1748469393.923446 read 0x0072e7 thru 0x008221 (0xf3b bytes) 000774 1748469393.923861 mapread 0x00754f thru 0x008221 (0xcd3 bytes) 000776 1748469393.924661 fallocate 0x005882 thru 0x00c89c (0x701b bytes) 000778 1748469393.926653 trunc from 0x001961 to 0x0062e6 (0x4986 bytes) 000779 1748469393.929152 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000782 1748469393.960339 trunc from 0x00c89d to 0x03711d (0x2a881 bytes) 000784 1748469393.966123 trunc from 0x008290 to 0x00bec9 (0x3c3a bytes) 000785 1748469393.967793 fallocate 0x02e572 thru 0x030388 (0x1e17 bytes) 000786 1748469393.969375 mapread 0x02395a thru 0x02cc26 (0x92cd bytes) 000787 1748469393.975282 mapread 0x0341eb thru 0x03711d (0x2f33 bytes) 000788 1748469393.977616 trunc from 0x0106dd to 0x016099 (0x59bd bytes) 000789 1748469393.980127 MIRROR_EXTEND to 2 mirrors 000790 1748469394.091792 fallocate 0x02e6f0 thru 0x036bcd (0x84de bytes) 000791 1748469394.099173 punch from 0x01d00d to 0x021508 (0x44fc bytes) 000792 1748469394.100531 punch from 0x012bd8 to 0x020203 (0xd62c bytes) 000793 1748469394.101953 fallocate 0x012c10 thru 0x01cd40 (0xa131 bytes) 000794 1748469394.103535 mapread 0x007e60 thru 0x011792 (0x9933 bytes) 000795 1748469394.109446 fallocate 0x020f0b thru 0x025c7b (0x4d71 bytes) 000796 1748469394.110901 trunc from 0x005740 to 0x00be0a (0x66cb bytes) 000797 1748469394.112232 fallocate 0x029efd thru 0x037fb0 (0xe0b4 bytes) 000798 1748469394.114052 mapread 0x025814 thru 0x025c73 (0x460 bytes) 000799 1748469394.116520 trunc from 0x025150 to 0x0322a8 (0xd159 bytes) 000800 1748469394.118087 trunc from 0x02f76d to 0x034f08 (0x579c bytes) 000801 1748469394.119205 write 0x016ade thru 0x02137f (0xa8a2 bytes) 000802 1748469394.121021 punch from 0x001e64 to 0x00ba51 (0x9bee bytes) 000803 1748469394.122496 write 0x020a91 thru 0x03085c (0xfdcc bytes) 000804 1748469394.124960 trunc from 0x03711e to 0x031659 (0x5ac4 bytes) 000805 1748469394.129657 punch from 0x00827c to 0x013574 (0xb2f9 bytes) 000807 1748469394.131580 fallocate 0x0220c2 thru 0x0315b1 (0xf4f0 bytes) 000808 1748469394.134182 MIRROR_RESYNC 2 mirrors 000809 1748469394.221286 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000812 1748469394.250383 write 0x02d0e4 thru 0x02e6f4 (0x1611 bytes) 000813 1748469394.255478 write 0x00d58c thru 0x014a3c (0x74b1 bytes) 000814 1748469394.257695 mapwrite 0x035fb2 thru 0x03ffff (0xa04e bytes) 000816 1748469394.271600 read 0x015265 thru 0x01f4cf (0xa26b bytes) 000817 1748469394.273665 mapread 0x006a0b thru 0x012475 (0xba6b bytes) 000818 1748469394.274126 trunc from 0x004edf to 0x00c21c (0x733e bytes) 000819 1748469394.275391 fallocate 0x024288 thru 0x029f20 (0x5c99 bytes) 000820 1748469394.276753 fallocate 0x00fb8b thru 0x017288 (0x76fe bytes) 000822 1748469394.278416 read 0x00618c thru 0x00c703 (0x6578 bytes) 000823 1748469394.281042 fallocate 0x027263 thru 0x0297bf (0x255d bytes) 000824 1748469394.282298 trunc from 0x02ac2d to 0x0337a9 (0x8b7d bytes) 000825 1748469394.283664 trunc from 0x01c73f to 0x023840 (0x7102 bytes) 000826 1748469394.284947 fallocate 0x009d6c thru 0x00b7cc (0x1a61 bytes) 000827 1748469394.286434 trunc from 0x040000 to 0x02a2aa (0x15d55 bytes) 000828 1748469394.289834 write 0x00ebde thru 0x012825 (0x3c48 bytes) 000829 1748469394.290437 trunc from 0x005eef to 0x014715 (0xe827 bytes) 000830 1748469394.295000 MIRROR_EXTEND to 2 mirrors 000831 1748469394.389283 fallocate 0x017f89 thru 0x019776 (0x17ee bytes) 000832 1748469394.397355 mapwrite 0x00fbb6 thru 0x0117cb (0x1c16 bytes) 000833 1748469394.405932 read 0x00b2b6 thru 0x00c88e (0x15d9 bytes) 000835 1748469394.406539 mapwrite 0x00390e thru 0x004be4 (0x12d7 bytes) 000836 1748469394.411893 trunc from 0x02a2ab to 0x031974 (0x76ca bytes) 000837 1748469394.422019 trunc from 0x031975 to 0x03a028 (0x86b4 bytes) 000838 1748469394.425494 fallocate 0x020e2c thru 0x029f93 (0x9168 bytes) 000839 1748469394.427877 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000840 1748469394.456672 mapread 0x00bca2 thru 0x0196d4 (0xda33 bytes) 000841 1748469394.464179 mapread 0x021374 thru 0x02ff96 (0xec23 bytes) 000842 1748469394.467954 trunc from 0x000f65 to 0x00ec68 (0xdd04 bytes) 000843 1748469394.470563 trunc from 0x03a029 to 0x025ff3 (0x14035 bytes) 000844 1748469394.475361 mapwrite 0x0201f3 thru 0x02ff77 (0xfd85 bytes) 000845 1748469394.489497 trunc from 0x02ff78 to 0x02ab2e (0x5449 bytes) 000847 1748469394.493825 mapread 0x0273c9 thru 0x02910d (0x1d45 bytes) 000849 1748469394.494902 fallocate 0x00c4ac thru 0x00c83b (0x390 bytes) 000850 1748469394.496518 fallocate 0x01a9b0 thru 0x02361d (0x8c6e bytes) 000851 1748469394.497857 read 0x013713 thru 0x0227bc (0xf0aa bytes) 000852 1748469394.498176 write 0x011837 thru 0x012d27 (0x14f1 bytes) 000853 1748469394.498778 trunc from 0x01473d to 0x01fef2 (0xb7b6 bytes) 000854 1748469394.500354 mapwrite 0x00e59e thru 0x0134d3 (0x4f36 bytes) 000855 1748469394.507298 trunc from 0x01c173 to 0x02363b (0x74c9 bytes) 000856 1748469394.508856 mapread 0x011856 thru 0x0168e9 (0x5094 bytes) 000857 1748469394.509245 read 0x014dbe thru 0x01d2a7 (0x84ea bytes) 000858 1748469394.513121 trunc from 0x02ab2f to 0x012856 (0x182d8 bytes) 000860 1748469394.517025 punch from 0x00dc4e to 0x012856 (0x4c09 bytes) 000861 1748469394.518298 punch from 0x008613 to 0x012856 (0xa244 bytes) 000862 1748469394.520870 mapwrite 0x03ff5c thru 0x03ffff (0xa4 bytes) 000863 1748469394.532258 mapwrite 0x0119f1 thru 0x014327 (0x2937 bytes) 000864 1748469394.543627 fallocate 0x021284 thru 0x024779 (0x34f6 bytes) 000865 1748469394.544966 mapread 0x0115ab thru 0x01be79 (0xa8cf bytes) 000867 1748469394.552975 punch from 0x03fea7 to 0x03ffff (0x159 bytes) 000868 1748469394.555356 punch from 0x000d0b to 0x0059c7 (0x4cbd bytes) 000869 1748469394.556841 read 0x00d09e thru 0x0111a5 (0x4108 bytes) 000870 1748469394.560649 trunc from 0x040000 to 0x015c72 (0x2a38d bytes) 000872 1748469394.564840 trunc from 0x015c73 to 0x001917 (0x1435b bytes) 000873 1748469394.568367 fallocate 0x013116 thru 0x01f15c (0xc047 bytes) 000874 1748469394.569695 read 0x00d008 thru 0x016116 (0x910f bytes) 000875 1748469394.572435 trunc from 0x01f15d to 0x025d09 (0x6bad bytes) 000876 1748469394.575814 write 0x009ee7 thru 0x00f7db (0x58f5 bytes) 000877 1748469394.576371 trunc from 0x025d0a to 0x011083 (0x14c86 bytes) 000878 1748469394.579780 fallocate 0x03267c thru 0x034096 (0x1a1b bytes) 000879 1748469394.581141 punch from 0x00a9ca to 0x011083 (0x66ba bytes) 000880 1748469394.584311 fallocate 0x0141d6 thru 0x022727 (0xe552 bytes) 000881 1748469394.586936 trunc from 0x0162ec to 0x016e87 (0xb9c bytes) 000882 1748469394.589592 punch from 0x01314f to 0x018e02 (0x5cb4 bytes) 000886 1748469394.591997 punch from 0x00fd5f to 0x01aab5 (0xad57 bytes) 000887 1748469394.594483 fallocate 0x03dffb thru 0x03ffff (0x2005 bytes) 000888 1748469394.596051 read 0x021745 thru 0x022727 (0xfe3 bytes) 000889 1748469394.598580 read 0x016537 thru 0x022727 (0xc1f1 bytes) 000890 1748469394.598978 trunc from 0x016ddc to 0x022727 (0xb94c bytes) 000891 1748469394.600454 trunc from 0x022728 to 0x005d99 (0x1c98e bytes) 000892 1748469394.604099 write 0x015806 thru 0x022761 (0xcf5c bytes) 000893 1748469394.605124 mapread 0x0169f2 thru 0x017a83 (0x1092 bytes) 000894 1748469394.605678 mapwrite 0x03c802 thru 0x03ffff (0x37fe bytes) 000896 1748469394.617830 mapwrite 0x03b3be thru 0x03ffff (0x4c42 bytes) 000897 1748469394.622277 trunc from 0x013c04 to 0x01c761 (0x8b5e bytes) 000898 1748469394.623839 mapread 0x00f82b thru 0x0116d5 (0x1eab bytes) 000899 1748469394.624419 mapwrite 0x02bdb6 thru 0x02bf35 (0x180 bytes) 000900 1748469394.627470 punch from 0x01ecbf to 0x023179 (0x44bb bytes) 000901 1748469394.628924 mapwrite 0x0358d6 thru 0x03ffff (0xa72a bytes) 000902 1748469394.633781 read 0x0158cd thru 0x018651 (0x2d85 bytes) 000903 1748469394.636244 fallocate 0x024927 thru 0x02498c (0x66 bytes) 000905 1748469394.638270 fallocate 0x03ebb9 thru 0x03ffff (0x1447 bytes) 000907 1748469394.641117 punch from 0x002270 to 0x010645 (0xe3d6 bytes) 000910 1748469394.642933 write 0x02253d thru 0x022fa6 (0xa6a bytes) 000912 1748469394.643773 trunc from 0x040000 to 0x03d592 (0x2a6d bytes) 000913 1748469394.647189 punch from 0x02b366 to 0x032ca8 (0x7943 bytes) 000914 1748469394.648577 read 0x019571 thru 0x01f0cf (0x5b5f bytes) 000915 1748469394.648944 write 0x0021d2 thru 0x00299a (0x7c9 bytes) 000916 1748469394.651865 read 0x006831 thru 0x00fdfb (0x95cb bytes) 000917 1748469394.652289 trunc from 0x03d593 to 0x0397ef (0x3da3 bytes) 000918 1748469394.656019 punch from 0x013057 to 0x021428 (0xe3d2 bytes) 000921 1748469394.658496 write 0x0183fd thru 0x01f32d (0x6f31 bytes) 000922 1748469394.661472 mapread 0x00d429 thru 0x01c67c (0xf254 bytes) 000923 1748469394.662084 read 0x02ac79 thru 0x0327eb (0x7b73 bytes) 000926 1748469394.662994 write 0x038d49 thru 0x03ffff (0x72b7 bytes) 000927 1748469394.663632 mapwrite 0x03486c thru 0x037744 (0x2ed9 bytes) 000928 1748469394.669484 trunc from 0x02f5d7 to 0x033bb2 (0x45dc bytes) 000930 1748469394.672727 fallocate 0x012c4a thru 0x01ddef (0xb1a6 bytes) 000931 1748469394.674511 read 0x02364d thru 0x0330dd (0xfa91 bytes) 000935 1748469394.678270 punch from 0x00e24c to 0x016dab (0x8b60 bytes) 000937 1748469394.680334 mapwrite 0x013e52 thru 0x018f34 (0x50e3 bytes) 000938 1748469394.688747 punch from 0x005956 to 0x0068bd (0xf68 bytes) 000939 1748469394.690143 fallocate 0x00e787 thru 0x011bc7 (0x3441 bytes) 000940 1748469394.692679 read 0x03898d thru 0x03ffff (0x7673 bytes) 000941 1748469394.693072 fallocate 0x01dca0 thru 0x02a31d (0xc67e bytes) 000942 1748469394.695554 trunc from 0x040000 to 0x030831 (0xf7ce bytes) 000943 1748469394.699584 mapread 0x02aacc thru 0x030831 (0x5d66 bytes) 000945 1748469394.700338 mapwrite 0x01b518 thru 0x0245cc (0x90b5 bytes) 000946 1748469394.705767 punch from 0x0303ad to 0x030831 (0x485 bytes) 000947 1748469394.707081 write 0x02d43e thru 0x03a64c (0xd20f bytes) 000948 1748469394.707899 mapread 0x031960 thru 0x036e01 (0x54a2 bytes) 000949 1748469394.708471 fallocate 0x02fa31 thru 0x03d09c (0xd66c bytes) 000950 1748469394.710076 trunc from 0x022d26 to 0x02690f (0x3bea bytes) 000951 1748469394.712571 mapwrite 0x001956 thru 0x005883 (0x3f2e bytes) 000952 1748469394.719168 trunc from 0x03d09d to 0x0332ef (0x9dad bytes) 000953 1748469394.723087 punch from 0x02443d to 0x02c7b6 (0x837a bytes) 000954 1748469394.724496 punch from 0x028e70 to 0x02e004 (0x5195 bytes) 000955 1748469394.725860 mapread 0x001b1b thru 0x0032db (0x17c1 bytes) 000956 1748469394.727454 fallocate 0x024bdd thru 0x030eb6 (0xc2da bytes) 000957 1748469394.728964 trunc from 0x0332f0 to 0x026dca (0xc525 bytes) 000958 1748469394.732552 punch from 0x00ac6c to 0x00f589 (0x491e bytes) 000959 1748469394.734033 fallocate 0x035cf5 thru 0x03f013 (0x931f bytes) 000960 1748469394.736479 mapwrite 0x008020 thru 0x011a0d (0x99ee bytes) 000961 1748469394.748970 write 0x008a66 thru 0x00a45e (0x19f9 bytes) 000962 1748469394.749622 fallocate 0x01d9b7 thru 0x02356e (0x5bb8 bytes) 000963 1748469394.750918 mapread 0x01776d thru 0x02645b (0xecef bytes) 000964 1748469394.754644 mapwrite 0x02fca0 thru 0x0366e1 (0x6a42 bytes) 000965 1748469394.775570 fallocate 0x03be55 thru 0x03fdd3 (0x3f7f bytes) 000966 1748469394.777005 mapread 0x0080d2 thru 0x009fa5 (0x1ed4 bytes) 000967 1748469394.777454 punch from 0x031f57 to 0x037633 (0x56dd bytes) 000969 1748469394.779053 read 0x031382 thru 0x03fdd3 (0xea52 bytes) 000970 1748469394.782482 write 0x03fb89 thru 0x03ffff (0x477 bytes) 000971 1748469394.782948 write 0x00467e thru 0x00be81 (0x7804 bytes) 000972 1748469394.783534 write 0x020121 thru 0x0293b2 (0x9292 bytes) 000973 1748469394.784210 punch from 0x036634 to 0x03ffff (0x99cc bytes) 000974 1748469394.787973 trunc from 0x040000 to 0x01e632 (0x219cd bytes) 000976 1748469394.793209 MIRROR_EXTEND to 2 mirrors 000977 1748469394.878291 write 0x0332d3 thru 0x03765e (0x438c bytes) 000978 1748469394.885959 punch from 0x033d41 to 0x03475f (0xa1f bytes) 000979 1748469394.889276 mapwrite 0x02c422 thru 0x037670 (0xb24f bytes) 000980 1748469394.906639 trunc from 0x02cedb to 0x034a62 (0x7b88 bytes) 000981 1748469394.909393 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x891:0x0]: Invalid argument (22) 000983 1748469394.937767 punch from 0x000303 to 0x009410 (0x910e bytes) 000984 1748469394.939271 mapread 0x0236bb thru 0x02a5af (0x6ef5 bytes) 000985 1748469394.943231 write 0x02fa0d thru 0x03edcb (0xf3bf bytes) 000986 1748469394.945161 fallocate 0x0305a4 thru 0x0375c9 (0x7026 bytes) 000987 1748469394.946603 read 0x01942e thru 0x019924 (0x4f7 bytes) 000988 1748469394.947952 mapread 0x013d12 thru 0x022de4 (0xf0d3 bytes) 000989 1748469394.952728 mapread 0x0396df thru 0x03edcb (0x56ed bytes) 000990 1748469394.953234 fallocate 0x0088e2 thru 0x00f09f (0x67be bytes) 000991 1748469394.954762 mapwrite 0x004186 thru 0x00a1f4 (0x606f bytes) 000992 1748469394.961310 trunc from 0x03616e to 0x03c541 (0x63d4 bytes) 000993 1748469394.962693 write 0x024470 thru 0x02a50c (0x609d bytes) 000997 1748469394.964234 mapread 0x01d0cf thru 0x0212c9 (0x41fb bytes) 000998 1748469394.964702 punch from 0x016cd4 to 0x0218cd (0xabfa bytes) 000999 1748469394.966045 trunc from 0x03edcc to 0x02a1c6 (0x14c05 bytes) 001000 1748469394.971170 MIRROR_EXTEND to 2 mirrors All operations completed A-OK! PASS 70a (13s) == sanity-flr test 71: check mirror extend parent fid ==== 17:56:36 (1748469396) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x56:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (2s) == sanity-flr test 200a: stress test ===================== 17:56:38 (1748469398) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4233056 .. Extending file size to 4394144 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7338944 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5621184 .. done Extending file size to 7960480 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8880160 .. Extending file size to 7818048 .. done Extending file size to 8003168 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9181088 .. Extending file size to 9129632 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9207008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9177632 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9263104 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 2899 2900 2901 2904 2906 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) 192.168.202.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:57:42 (1748469462) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3371936 .. Extending file size to 3636928 .. failed Extending file size to 9233408 .. Extending file size to 3998848 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 5286560 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8788224 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8917088 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9020896 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9119872 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 7028 7029 7030 7033 7035 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) 192.168.202.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:58:46 (1748469526) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2672 1285016 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3020 1284668 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17676 3589344 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19988 3587032 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37664 7176376 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2672 1285016 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 3020 1284668 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17676 3589344 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19988 3587032 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37664 7176376 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0433491 s, 48.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0732268 s, 28.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:58:52 (1748469532) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0423672 s, 49.5 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 17:58:55 (1748469535) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:58:57 (1748469537) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0343019 s, 61.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:59:00 (1748469540) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0440582 s, 47.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb1:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:59:02 (1748469542) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0205785 s, 102 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:59:26 (1748469566) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0269093 s, 39.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbb2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbb5:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:59:59 (1748469599) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0220964 s, 94.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (32s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:00:31 (1748469631) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg232-server: Pool lustre.test_204f created Adding targets to pool oleg232-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg232-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0364382 s, 57.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg232-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg232-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg232-server: Pool lustre.test_204f destroyed PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:01:21 (1748469681) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:01:23 (1748469683) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:01:25 (1748469685) oleg232-server: Pool lustre.test_206 created Adding targets to pool oleg232-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg232-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg232-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg232-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg232-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:01:39 (1748469699) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221721 s, 47.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00973286 s, 108 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x70b586cf Mirror 3: 0x70b586cf PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:01:41 (1748469701) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (0s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:01:41 (1748469701) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:01:42 (1748469702) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.00928824 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2220000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1510000 Read this (4 bytes): BBBB 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:01:57 (1748469717) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00529153 s, 1.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:01:59 (1748469719) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159828 s, 65.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x81 lmm_fid: [0x200000405:0x81:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9a:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611265 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-28 18:02:00.000000000 -0400 Modify: 2025-05-28 18:02:00.000000000 -0400 Change: 2025-05-28 18:02:00.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:02:02 (1748469722) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144244 s, 72.7 MB/s oleg232-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg232-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg232-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg232-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg232-server: O/280000401/d26/ba: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:02:15 (1748469735) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0201545 s, 2.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00587179 s, 698 kB/s size after second write -rw-r--r-- 1 root root 4096 May 28 18:02 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x86:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 28 18:02 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 734 sec ============ 18:02:17 (1748469737) === sanity-flr: start cleanup 18:02:17 (1748469737) === === sanity-flr: finish cleanup 18:02:20 (1748469740) ===