-----============= acceptance-small: sanity-flr ============----- Fri Jul 25 06:44:38 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg219-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:44:55 (1753440295) === oleg219-client.virtnet: executing check_config_client /mnt/lustre oleg219-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg219-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f001081f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f001081f800.idle_timeout=debug disable quota as required oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:45:26 (1753440326) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8204] == sanity-flr test 0a: lfs mirror create with -N option == 06:45:28 (1753440328) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:45:40 (1753440340) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:45:42 (1753440342) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:45:45 (1753440345) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:46:00 (1753440360) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:46:01 (1753440361) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 06:46:03 (1753440363) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f001081f800.stats=clear osc.lustre-OST0001-osc-ffff8f001081f800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:46:08 (1753440368) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:46:14 (1753440374) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:46:19 (1753440379) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:46:21 (1753440381) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 06:46:26 (1753440386) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:46:31 (1753440391) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:46:35 (1753440395) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:46:42 (1753440402) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:46:49 (1753440409) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:46:53 (1753440413) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140515 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809959 s, 12.9 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:46:58 (1753440418) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.178511 s, 76.4 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:47:03 (1753440423) Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f001081f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f001081f800.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f001081f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f001081f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (46s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:47:49 (1753440469) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0553658 s, 37.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f001081f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f001081f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (24s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:48:13 (1753440493) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f001081f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f001081f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f001081f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f001081f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (44s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:48:58 (1753440538) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50186 s, 34.9 MB/s -rw-r--r-- 1 root root 52428800 Jul 25 06:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.605s user 0m0.001s sys 0m0.147s Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f001081f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f001081f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.514s user 0m0.000s sys 0m0.165s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f001081f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f001081f800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.699s user 0m0.002s sys 0m0.166s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f001081f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f001081f800.ost_server_uuid in FULL state after 0 sec PASS 33b (51s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:49:48 (1753440588) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:49:50 (1753440590) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:49:51 (1753440591) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 06:49:53 (1753440593) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 06:49:58 (1753440598) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.34768 s, 36.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:50:05 (1753440605) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157106 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192624 s, 54.4 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:50:09 (1753440609) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:50:14 (1753440614) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000733594 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000726077 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00071177 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000743528 s, 1.4 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 06:50:18 (1753440618) create test files with size 15752418 .. c86b7527aa1ec8cfcc0caed746e63fb4 - 74cde7f24db992efae295d3ce422ba07 - 90ac230d93e1aa3bb7382d1d7ede6e1d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 06:50:28 (1753440628) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:50:34 (1753440634) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:50:37 (1753440637) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592661 s, 17.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147138 s, 71.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x42:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 06:50:41 (1753440641) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0423913 s, 49.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.152427 s, 27.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.085718 s, 24.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 06:50:46 (1753440646) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 06:50:48 (1753440648) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:50:49 (1753440649) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156526 s, 67.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 06:50:53 (1753440653) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.151226 s, 20.8 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:50:57 (1753440657) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4d:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4d:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:51:02 (1753440662) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.128106 s, 81.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:51:17 (1753440677) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8f00080ff800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.121043 s, 86.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8f001081a000.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:51:24 (1753440684) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (19s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:51:43 (1753440703) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (17s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:52:00 (1753440720) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 06:52:02 (1753440722) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0937259 s, 33.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0753947 s, 41.7 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: cf0649a154e421f2d69eb06a5adf0a2a - mirror 1 checksum: ed775798ab296bd49abc8052acec1bc6 - mirror 2 checksum: cf0649a154e421f2d69eb06a5adf0a2a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Jul 25 06:52:03 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Jul 25 06:52:03 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cf0649a154e421f2d69eb06a5adf0a2a - mirror 1 checksum: ed775798ab296bd49abc8052acec1bc6 - mirror 2 checksum: ed775798ab296bd49abc8052acec1bc6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Jul 25 06:52:03 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Jul 25 06:52:03 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:52:07 (1753440727) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187682 s, 55.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:52:10 (1753440730) Failing mds1 on oleg219-server Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 06:52:19 (1753440739) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server Failover mds1 to oleg219-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 06:52:36 (1753440756) targets are mounted 06:52:36 (1753440756) facet_failover done oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197059 s, 53.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 50A (35s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:52:45 (1753440765) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0247228 s, 828 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00796653 s, 2.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:52:50 (1753440770) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.025127 s, 815 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00638782 s, 3.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:52:55 (1753440775) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00357067 s, 4.6 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:52:58 (1753440778) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0322123 s, 636 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:53:02 (1753440782) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00691825 s, 592 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0256754 s, 160 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:53:06 (1753440786) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:53:17 (1753440797) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:53:38 (1753440818) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:53:58 (1753440838) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:54:18 (1753440858) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 06:54:22 (1753440862) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55856 mirror split pid 55857 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55856 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 55857 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 06:55:26 (1753440926) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 927 skipping zero size read 000002 1753440927.178680 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x276a0 000003 1753440927.360838 fallocate 0x02057c thru 0x02769f (0x7124 bytes) 000004 1753440927.379861 MIRROR_EXTEND to 3 mirrors 000005 1753440927.732377 trunc from 0x016cc3 to 0x025820 (0xeb5e bytes) 000008 1753440927.762137 write 0x03b087 thru 0x03dc28 (0x2ba2 bytes) 000009 1753440927.768847 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000010 1753440927.927725 write 0x026cc5 thru 0x02e125 (0x7461 bytes) fallocating to largest ever: 0x40000 000011 1753440927.942794 fallocate 0x035266 thru 0x03ffff (0xad9a bytes) 000012 1753440927.949566 trunc from 0x040000 to 0x0109ce (0x2f631 bytes) 000013 1753440927.966204 trunc from 0x0109cf to 0x023348 (0x1297a bytes) 000014 1753440927.978563 trunc from 0x022bcf to 0x023348 (0x77a bytes) 000015 1753440927.985591 write 0x01abfb thru 0x01f65f (0x4a65 bytes) 000016 1753440928.001718 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000017 1753440928.093269 mapread 0x01a948 thru 0x022027 (0x76e0 bytes) 000019 1753440928.105652 trunc from 0x007983 to 0x007e29 (0x4a7 bytes) 000020 1753440928.109486 trunc from 0x023349 to 0x00a1b7 (0x19191 bytes) 000022 1753440928.125462 punch from 0x000c52 to 0x0064fe (0x58ad bytes) 000023 1753440928.130195 mapwrite 0x01bd25 thru 0x0267c1 (0xaa9d bytes) 000024 1753440928.164146 punch from 0x00a201 to 0x0181e2 (0xdfe2 bytes) 000025 1753440928.167672 write 0x02e7f2 thru 0x0375a8 (0x8db7 bytes) 000027 1753440928.169534 trunc from 0x0058b3 to 0x014f89 (0xf6d7 bytes) 000028 1753440928.173266 punch from 0x005f64 to 0x008be5 (0x2c82 bytes) 000029 1753440928.176963 fallocate 0x005ad9 thru 0x00ca1b (0x6f43 bytes) 000030 1753440928.182132 mapread 0x02793d thru 0x036832 (0xeef6 bytes) 000031 1753440928.188023 mapwrite 0x013199 thru 0x01971d (0x6585 bytes) 000032 1753440928.211187 write 0x023366 thru 0x02b303 (0x7f9e bytes) 000033 1753440928.215730 MIRROR_EXTEND to 2 mirrors 000034 1753440928.558293 fallocate 0x02141f thru 0x0268d1 (0x54b3 bytes) 000035 1753440928.575864 mapread 0x01abe1 thru 0x0295e6 (0xea06 bytes) 000036 1753440928.601705 punch from 0x00e4ba to 0x00eb01 (0x648 bytes) 000038 1753440928.608676 punch from 0x02bdbc to 0x0375a8 (0xb7ed bytes) 000039 1753440928.615205 fallocate 0x023b53 thru 0x0248c6 (0xd74 bytes) 000040 1753440928.619149 write 0x03cc1d thru 0x03ffff (0x33e3 bytes) 000041 1753440928.622416 write 0x00bb06 thru 0x016426 (0xa921 bytes) 000042 1753440928.632368 fallocate 0x02ada6 thru 0x02ee01 (0x405c bytes) 000043 1753440928.639379 fallocate 0x0173c1 thru 0x01ed91 (0x79d1 bytes) 000044 1753440928.642923 mapwrite 0x03a39f thru 0x03ffff (0x5c61 bytes) 000045 1753440928.672714 punch from 0x0087e5 to 0x010969 (0x8185 bytes) 000046 1753440928.678634 punch from 0x0048b6 to 0x013c3d (0xf388 bytes) 000047 1753440928.684160 trunc from 0x040000 to 0x018f21 (0x270de bytes) 000048 1753440928.705343 trunc from 0x016daf to 0x018f21 (0x2173 bytes) 000049 1753440928.710509 read 0x00f0e2 thru 0x0139a5 (0x48c4 bytes) 000050 1753440928.717847 trunc from 0x018f22 to 0x03bf1b (0x22ffa bytes) 000051 1753440928.731551 read 0x003de5 thru 0x00d74f (0x996b bytes) 000052 1753440928.737626 read 0x02902c thru 0x02f851 (0x6826 bytes) 000053 1753440928.738227 fallocate 0x0060f2 thru 0x00b945 (0x5854 bytes) 000054 1753440928.741661 fallocate 0x03e154 thru 0x03ffff (0x1eac bytes) 000055 1753440928.746020 mapwrite 0x00f8dd thru 0x016351 (0x6a75 bytes) 000056 1753440928.770335 read 0x0316ef thru 0x03bf1b (0xa82d bytes) 000058 1753440928.771466 mapread 0x01554b thru 0x024cb0 (0xf766 bytes) 000059 1753440928.772419 fallocate 0x017394 thru 0x019b1b (0x2788 bytes) 000060 1753440928.776501 trunc from 0x03bf1c to 0x0019f1 (0x3a52a bytes) 000061 1753440928.788883 fallocate 0x00252f thru 0x00aebd (0x898f bytes) 000062 1753440928.796509 read 0x00616d thru 0x00aebd (0x4d51 bytes) 000063 1753440928.808914 mapread 0x009410 thru 0x00aebd (0x1aae bytes) 000064 1753440928.809781 punch from 0x009a7a to 0x00aebd (0x1444 bytes) 000065 1753440928.816203 mapwrite 0x0393ef thru 0x03f103 (0x5d15 bytes) 000067 1753440928.874891 fallocate 0x03797d thru 0x03ffff (0x8683 bytes) 000068 1753440928.882583 punch from 0x02c437 to 0x02e556 (0x2120 bytes) 000069 1753440928.887509 read 0x00b6a1 thru 0x010525 (0x4e85 bytes) 000070 1753440928.888158 write 0x00b720 thru 0x00d3d2 (0x1cb3 bytes) 000071 1753440928.889007 trunc from 0x040000 to 0x03f1dd (0xe22 bytes) 000072 1753440928.908783 trunc from 0x03f1de to 0x034e01 (0xa3dc bytes) 000073 1753440928.922639 read 0x0299a5 thru 0x034e01 (0xb45d bytes) 000074 1753440928.931728 write 0x015b0e thru 0x0227df (0xccd2 bytes) 000075 1753440928.933353 punch from 0x0098f3 to 0x012892 (0x8fa0 bytes) 000076 1753440928.945869 trunc from 0x010fc2 to 0x012ba3 (0x1be2 bytes) 000077 1753440928.949865 punch from 0x01eff8 to 0x02e177 (0xf180 bytes) 000078 1753440928.953934 read 0x02ec6e thru 0x034e01 (0x6194 bytes) 000079 1753440928.963686 mapread 0x0058f3 thru 0x0150b4 (0xf7c2 bytes) 000080 1753440928.964706 mapwrite 0x03d265 thru 0x03ffff (0x2d9b bytes) 000081 1753440929.018886 trunc from 0x040000 to 0x033387 (0xcc78 bytes) 000082 1753440929.033987 trunc from 0x033388 to 0x02be81 (0x7506 bytes) 000084 1753440929.049813 mapwrite 0x021429 thru 0x02f248 (0xde20 bytes) 000086 1753440929.097036 mapread 0x01c5ba thru 0x01d02c (0xa73 bytes) 000087 1753440929.097874 trunc from 0x009c3b to 0x0177bc (0xdb82 bytes) 000088 1753440929.102486 mapwrite 0x00c370 thru 0x01bd05 (0xf996 bytes) 000089 1753440929.158095 trunc from 0x0166b2 to 0x0186cc (0x201b bytes) 000091 1753440929.162344 read 0x02b538 thru 0x02f248 (0x3d11 bytes) 000093 1753440929.163302 read 0x008d1c thru 0x00ea42 (0x5d27 bytes) 000094 1753440929.168273 read 0x02e398 thru 0x02f248 (0xeb1 bytes) 000095 1753440929.168856 mapwrite 0x01734c thru 0x0194f9 (0x21ae bytes) 000096 1753440929.182675 punch from 0x015ada to 0x01a27a (0x47a1 bytes) 000097 1753440929.187655 read 0x00850c thru 0x016537 (0xe02c bytes) 000099 1753440929.193546 mapwrite 0x029313 thru 0x0365c2 (0xd2b0 bytes) 000100 1753440929.228009 mapwrite 0x02cc87 thru 0x02fba1 (0x2f1b bytes) 000101 1753440929.236741 punch from 0x01ca63 to 0x02723b (0xa7d9 bytes) 000102 1753440929.239529 write 0x024155 thru 0x026cd3 (0x2b7f bytes) 000104 1753440929.244879 read 0x02585f thru 0x02cdbb (0x755d bytes) 000105 1753440929.249705 read 0x0145b5 thru 0x019a8b (0x54d7 bytes) 000107 1753440929.250817 trunc from 0x02656b to 0x02dcc0 (0x7756 bytes) 000108 1753440929.257484 mapread 0x004cb1 thru 0x005c2a (0xf7a bytes) 000109 1753440929.258385 punch from 0x000a80 to 0x009716 (0x8c97 bytes) 000110 1753440929.261560 write 0x00ffcd thru 0x018145 (0x8179 bytes) 000111 1753440929.262773 trunc from 0x030e6c to 0x0365c2 (0x5757 bytes) 000112 1753440929.266358 trunc from 0x0365c3 to 0x03e4eb (0x7f29 bytes) 000113 1753440929.285326 trunc from 0x03e4ec to 0x01d885 (0x20c66 bytes) 000114 1753440929.305033 fallocate 0x026320 thru 0x02eb28 (0x8809 bytes) 000115 1753440929.311563 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000116 1753440929.444497 write 0x0074af thru 0x009c16 (0x2768 bytes) 000117 1753440929.468253 trunc from 0x0257d1 to 0x027173 (0x19a3 bytes) 000118 1753440929.473384 read 0x00406d thru 0x00990d (0x58a1 bytes) 000120 1753440929.481322 fallocate 0x010f48 thru 0x015812 (0x48cb bytes) 000121 1753440929.489038 read 0x020c61 thru 0x0286e3 (0x7a83 bytes) 000123 1753440929.496657 punch from 0x0192ea to 0x01aafa (0x1811 bytes) 000124 1753440929.501625 mapwrite 0x03ba64 thru 0x03ffff (0x459c bytes) 000125 1753440929.538910 write 0x01b7e3 thru 0x01e065 (0x2883 bytes) 000126 1753440929.539925 trunc from 0x01e65d to 0x029595 (0xaf39 bytes) 000128 1753440929.549979 mapwrite 0x029042 thru 0x02b0d6 (0x2095 bytes) 000129 1753440929.563397 write 0x02cf9e thru 0x036a15 (0x9a78 bytes) 000130 1753440929.564751 trunc from 0x040000 to 0x024d20 (0x1b2df bytes) 000131 1753440929.581750 fallocate 0x032553 thru 0x034b59 (0x2607 bytes) 000132 1753440929.584739 trunc from 0x01fd1e to 0x0250e7 (0x53ca bytes) 000133 1753440929.589716 read 0x01b86d thru 0x02aabd (0xf251 bytes) 000136 1753440929.598944 mapwrite 0x02a35f thru 0x02dcc7 (0x3969 bytes) 000137 1753440929.615522 write 0x0305da thru 0x035bcd (0x55f4 bytes) 000138 1753440929.616714 trunc from 0x035bce to 0x01c0ea (0x19ae3 bytes) 000139 1753440929.632762 mapwrite 0x0220d1 thru 0x02bea1 (0x9dd1 bytes) 000140 1753440929.705772 mapwrite 0x028b33 thru 0x0364da (0xd9a8 bytes) 000141 1753440929.764785 write 0x032dc7 thru 0x03ffff (0xd239 bytes) 000142 1753440929.771094 MIRROR_EXTEND to 2 mirrors 000143 1753440930.192286 trunc from 0x0114b9 to 0x01a225 (0x8d6d bytes) 000144 1753440930.215221 trunc from 0x02e4d4 to 0x039218 (0xad45 bytes) 000145 1753440930.218311 read 0x00b855 thru 0x00ec34 (0x33e0 bytes) 000146 1753440930.222675 trunc from 0x040000 to 0x0096e1 (0x3691e bytes) 000147 1753440930.241611 MIRROR_EXTEND to 3 mirrors 000148 1753440930.538881 MIRROR_EXTEND to 4 mirrors 000149 1753440930.806460 MIRROR_RESYNC 4 mirrors 000150 1753440931.100257 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000151 1753440931.311381 punch from 0x004e9d to 0x0096e1 (0x4845 bytes) 000152 1753440931.340298 trunc from 0x0096e2 to 0x010e05 (0x7724 bytes) 000153 1753440931.363995 read 0x00044f thru 0x00eb74 (0xe726 bytes) 000154 1753440931.372930 mapwrite 0x033834 thru 0x035f80 (0x274d bytes) 000157 1753440931.409030 trunc from 0x035f81 to 0x02e796 (0x77ea bytes) 000158 1753440931.423860 read 0x017d39 thru 0x01a3b8 (0x2680 bytes) 000159 1753440931.441187 trunc from 0x02e797 to 0x01945f (0x15337 bytes) 000160 1753440931.458347 fallocate 0x023223 thru 0x02352f (0x30d bytes) 000162 1753440931.462565 mapread 0x00b220 thru 0x016fb8 (0xbd99 bytes) 000163 1753440931.463799 write 0x0252be thru 0x02b88b (0x65ce bytes) 000165 1753440931.466185 mapwrite 0x0037a4 thru 0x00607d (0x28da bytes) 000166 1753440931.482808 fallocate 0x01ffb6 thru 0x02f8e9 (0xf934 bytes) 000167 1753440931.487372 trunc from 0x02b88c to 0x022a1c (0x8e6f bytes) 000169 1753440931.502006 trunc from 0x022a1d to 0x025dbd (0x33a1 bytes) 000170 1753440931.515541 fallocate 0x018dc7 thru 0x01c2e0 (0x351a bytes) 000171 1753440931.521400 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000174 1753440931.697510 read 0x00758a thru 0x015e34 (0xe8ab bytes) 000175 1753440931.704002 trunc from 0x01be29 to 0x021103 (0x52db bytes) 000176 1753440931.707807 mapread 0x0168b0 thru 0x025dbd (0xf50e bytes) 000177 1753440931.738177 mapwrite 0x03f251 thru 0x03fbc8 (0x978 bytes) 000178 1753440931.774152 trunc from 0x036b11 to 0x03fbc8 (0x90b8 bytes) 000179 1753440931.778672 write 0x019c97 thru 0x01ff95 (0x62ff bytes) 000180 1753440931.779815 mapread 0x0188a2 thru 0x01ec9b (0x63fa bytes) 000181 1753440931.780619 punch from 0x0260af to 0x02f731 (0x9683 bytes) 000182 1753440931.784994 fallocate 0x019d20 thru 0x0256fd (0xb9de bytes) 000183 1753440931.790662 read 0x02bcb0 thru 0x0362e2 (0xa633 bytes) 000184 1753440931.805396 fallocate 0x02f5c2 thru 0x03e90c (0xf34b bytes) 000185 1753440931.810215 read 0x00cc6b thru 0x012f88 (0x631e bytes) 000187 1753440931.817846 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000188 1753440931.930465 MIRROR_EXTEND to 2 mirrors 000190 1753440932.361290 mapwrite 0x001036 thru 0x0080dc (0x70a7 bytes) 000191 1753440932.486260 trunc from 0x03fbc9 to 0x010142 (0x2fa86 bytes) 000192 1753440932.510877 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000193 1753440932.618806 trunc from 0x010143 to 0x02a0f6 (0x19fb4 bytes) 000195 1753440932.635967 read 0x023538 thru 0x02a0f6 (0x6bbf bytes) 000196 1753440932.642259 mapread 0x00f3f3 thru 0x013676 (0x4284 bytes) 000197 1753440932.656439 trunc from 0x000d77 to 0x00bd81 (0xb00b bytes) 000198 1753440932.660462 punch from 0x02212f to 0x0293f9 (0x72cb bytes) 000202 1753440932.667763 fallocate 0x0098ab thru 0x0126bf (0x8e15 bytes) 000203 1753440932.672190 punch from 0x0073f6 to 0x00b157 (0x3d62 bytes) 000206 1753440932.678981 read 0x0253d6 thru 0x02a0f6 (0x4d21 bytes) 000208 1753440932.694346 trunc from 0x02248b to 0x02a0f6 (0x7c6c bytes) 000209 1753440932.702176 mapread 0x003241 thru 0x008b30 (0x58f0 bytes) 000210 1753440932.703125 punch from 0x015785 to 0x0256cf (0xff4b bytes) 000211 1753440932.709925 write 0x03626b thru 0x03ffff (0x9d95 bytes) 000212 1753440932.711422 read 0x011953 thru 0x01c29a (0xa948 bytes) 000215 1753440932.721618 trunc from 0x01a537 to 0x025586 (0xb050 bytes) 000216 1753440932.730316 read 0x00de14 thru 0x01a467 (0xc654 bytes) 000217 1753440932.738823 mapread 0x021171 thru 0x0250ac (0x3f3c bytes) 000218 1753440932.739747 punch from 0x01d2a9 to 0x01ea75 (0x17cd bytes) 000219 1753440932.743190 read 0x0096e2 thru 0x018e6e (0xf78d bytes) 000221 1753440932.744310 mapread 0x02da84 thru 0x02df7b (0x4f8 bytes) 000222 1753440932.745142 mapread 0x01927d thru 0x0211ee (0x7f72 bytes) 000225 1753440932.751594 trunc from 0x01fb26 to 0x01ff86 (0x461 bytes) 000226 1753440932.760905 MIRROR_EXTEND to 2 mirrors 000227 1753440933.137650 mapwrite 0x0268b2 thru 0x02cd9d (0x64ec bytes) 000228 1753440933.251923 mapread 0x013757 thru 0x02220e (0xeab8 bytes) 000229 1753440933.266584 trunc from 0x040000 to 0x03f3fa (0xc05 bytes) 000230 1753440933.289014 MIRROR_RESYNC 2 mirrors 000231 1753440933.573566 write 0x0102d7 thru 0x01a60c (0xa336 bytes) 000232 1753440933.613988 mapwrite 0x0344a7 thru 0x03cbcd (0x8727 bytes) 000233 1753440933.638672 punch from 0x01fe4c to 0x02f282 (0xf437 bytes) 000234 1753440933.643137 read 0x034363 thru 0x036d81 (0x2a1f bytes) 000235 1753440933.643747 trunc from 0x03f3fb to 0x03d92b (0x1acf bytes) 000237 1753440933.666171 trunc from 0x000278 to 0x006529 (0x62b2 bytes) 000238 1753440933.673614 MIRROR_EXTEND to 3 mirrors 000239 1753440934.033663 fallocate 0x035f88 thru 0x03d989 (0x7a02 bytes) 000240 1753440934.062767 read 0x03c09c thru 0x03d989 (0x18ee bytes) 000241 1753440934.068644 punch from 0x03b86e to 0x03d989 (0x211c bytes) 000242 1753440934.073723 fallocate 0x03f99d thru 0x03ffff (0x663 bytes) 000243 1753440934.077420 trunc from 0x03d98a to 0x00b69d (0x322ec bytes) 000244 1753440934.096252 mapread 0x007a82 thru 0x00b69d (0x3c1c bytes) 000245 1753440934.119316 mapread 0x004d69 thru 0x009577 (0x480f bytes) 000247 1753440934.120557 mapwrite 0x000f0d thru 0x0026ee (0x17e2 bytes) 000248 1753440934.142725 fallocate 0x009448 thru 0x014c98 (0xb851 bytes) 000249 1753440934.153942 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000250 1753440934.338125 fallocate 0x006e2b thru 0x00a597 (0x376d bytes) 000251 1753440934.343476 write 0x027c8d thru 0x0331db (0xb54f bytes) 000252 1753440934.349480 write 0x03922b thru 0x03ffff (0x6dd5 bytes) 000253 1753440934.350591 fallocate 0x01b9a8 thru 0x0248c6 (0x8f1f bytes) 000254 1753440934.354699 trunc from 0x040000 to 0x0120f6 (0x2df09 bytes) 000255 1753440934.373273 read 0x00f5fe thru 0x0120f6 (0x2af9 bytes) 000256 1753440934.381029 trunc from 0x00da65 to 0x0120f6 (0x4692 bytes) 000257 1753440934.388373 trunc from 0x0120f7 to 0x0148b5 (0x27bf bytes) 000258 1753440934.408373 trunc from 0x0148b6 to 0x01e57b (0x9cc6 bytes) 000259 1753440934.418197 write 0x010161 thru 0x01a4de (0xa37e bytes) 000260 1753440934.427410 write 0x027664 thru 0x02f5e2 (0x7f7f bytes) 000262 1753440934.429450 punch from 0x02665e to 0x02f5e2 (0x8f85 bytes) 000264 1753440934.445660 trunc from 0x02f5e3 to 0x031806 (0x2224 bytes) 000265 1753440934.460731 trunc from 0x031807 to 0x02a11a (0x76ec bytes) 000266 1753440934.478692 trunc from 0x02a11b to 0x02b792 (0x1678 bytes) 000267 1753440934.494716 trunc from 0x02b793 to 0x00b8c8 (0x1feca bytes) 000268 1753440934.510083 mapread 0x001f52 thru 0x00b8c8 (0x9977 bytes) 000269 1753440934.516319 trunc from 0x00b8c9 to 0x0027a2 (0x9126 bytes) 000270 1753440934.534183 write 0x01b6fb thru 0x01ec92 (0x3598 bytes) 000271 1753440934.535334 punch from 0x018da8 to 0x01b6ef (0x2948 bytes) 000272 1753440934.549308 punch from 0x012f2d to 0x01ba8c (0x8b60 bytes) 000273 1753440934.562734 MIRROR_RESYNC 2 mirrors 000275 1753440934.896331 write 0x003015 thru 0x0070c1 (0x40ad bytes) 000276 1753440934.931258 trunc from 0x016976 to 0x01ec92 (0x831d bytes) 000277 1753440934.934855 write 0x0245fb thru 0x03176a (0xd170 bytes) 000278 1753440934.936432 read 0x03002d thru 0x03176a (0x173e bytes) 000280 1753440934.937391 fallocate 0x01a343 thru 0x020f68 (0x6c26 bytes) 000281 1753440934.942678 trunc from 0x03176b to 0x01792d (0x19e3d bytes) 000282 1753440934.961726 mapread 0x003e19 thru 0x00a18b (0x6373 bytes) 000283 1753440934.970025 mapread 0x002f42 thru 0x0051fc (0x22bb bytes) 000284 1753440934.971118 trunc from 0x01792e to 0x0100b4 (0x7879 bytes) 000286 1753440934.988640 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000288 1753440935.130219 mapwrite 0x003ee5 thru 0x00f9e1 (0xbafd bytes) 000289 1753440935.185270 write 0x02253b thru 0x03062a (0xe0f0 bytes) 000290 1753440935.191267 trunc from 0x03062b to 0x00ad2d (0x258fd bytes) 000291 1753440935.214320 punch from 0x004649 to 0x009859 (0x5211 bytes) 000292 1753440935.218036 write 0x01d63d thru 0x01e3a5 (0xd69 bytes) 000293 1753440935.219121 fallocate 0x00b7f4 thru 0x019322 (0xdb2f bytes) 000294 1753440935.223998 trunc from 0x01b184 to 0x01e3a5 (0x3222 bytes) 000295 1753440935.234946 read 0x00fc6a thru 0x013d3f (0x40d6 bytes) 000296 1753440935.242327 trunc from 0x01e3a6 to 0x00e3fc (0xffa9 bytes) 000297 1753440935.256625 trunc from 0x0099af to 0x00e3fc (0x4a4e bytes) 000298 1753440935.259673 trunc from 0x00e3fd to 0x009b96 (0x4866 bytes) 000299 1753440935.275035 mapwrite 0x0278d2 thru 0x0322a6 (0xa9d5 bytes) 000300 1753440935.342475 trunc from 0x005279 to 0x00e143 (0x8ecb bytes) 000302 1753440935.350443 fallocate 0x016739 thru 0x01cfb8 (0x6880 bytes) 000303 1753440935.357767 fallocate 0x013eb2 thru 0x017d73 (0x3ec2 bytes) 000304 1753440935.369905 mapwrite 0x02946b thru 0x02c6ec (0x3282 bytes) 000305 1753440935.386357 mapwrite 0x02b226 thru 0x032c12 (0x79ed bytes) 000307 1753440935.432307 read 0x019a4e thru 0x01cb04 (0x30b7 bytes) 000310 1753440935.447156 fallocate 0x0096ab thru 0x00dbdc (0x4532 bytes) 000311 1753440935.451959 write 0x019489 thru 0x023416 (0x9f8e bytes) 000312 1753440935.453167 mapwrite 0x03a6ba thru 0x03b211 (0xb58 bytes) 000313 1753440935.480563 mapread 0x011cb8 thru 0x017a38 (0x5d81 bytes) 000314 1753440935.481412 fallocate 0x0189b8 thru 0x01cd3a (0x4383 bytes) 000315 1753440935.488868 MIRROR_EXTEND to 2 mirrors 000316 1753440935.826852 MIRROR_RESYNC 2 mirrors 000317 1753440935.921172 mapwrite 0x031bd9 thru 0x03261f (0xa47 bytes) 000318 1753440935.995823 trunc from 0x03b212 to 0x032a53 (0x87be bytes) 000319 1753440936.008204 trunc from 0x01be76 to 0x02729e (0xb429 bytes) 000320 1753440936.012966 mapread 0x03137a thru 0x032a53 (0x16da bytes) 000321 1753440936.014336 punch from 0x0322b8 to 0x032a53 (0x79c bytes) 000322 1753440936.017618 trunc from 0x032a54 to 0x0288d3 (0xa180 bytes) 000323 1753440936.026976 write 0x03a6d5 thru 0x03ef12 (0x483e bytes) 000324 1753440936.032237 MIRROR_EXTEND to 3 mirrors 000325 1753440936.400229 mapwrite 0x026b50 thru 0x031fe1 (0xb492 bytes) 000326 1753440936.481226 mapread 0x014461 thru 0x02196c (0xd50c bytes) 000327 1753440936.495873 mapwrite 0x020303 thru 0x0250a2 (0x4da0 bytes) 000328 1753440936.508707 fallocate 0x02fe5d thru 0x03516d (0x5311 bytes) 000329 1753440936.512407 read 0x0109c8 thru 0x013c52 (0x328b bytes) 000330 1753440936.512960 mapread 0x019e92 thru 0x01d2a7 (0x3416 bytes) 000331 1753440936.513737 mapread 0x00a235 thru 0x00b743 (0x150f bytes) 000332 1753440936.523449 fallocate 0x02bf98 thru 0x030313 (0x437c bytes) 000333 1753440936.527785 mapwrite 0x00e003 thru 0x0198fc (0xb8fa bytes) 000334 1753440936.549847 trunc from 0x03ef13 to 0x02eb15 (0x103fd bytes) 000335 1753440936.570375 trunc from 0x02eb16 to 0x00c668 (0x224ad bytes) 000338 1753440936.583197 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000339 1753440936.740626 mapread 0x00afac thru 0x00c668 (0x16bd bytes) 000340 1753440936.756114 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000341 1753440936.834710 trunc from 0x003422 to 0x00c178 (0x8d57 bytes) 000342 1753440936.841681 mapwrite 0x03afce thru 0x03fe68 (0x4e9b bytes) 000343 1753440936.881444 write 0x0145bf thru 0x02204e (0xda90 bytes) 000344 1753440936.893724 write 0x014579 thru 0x01a2bc (0x5d44 bytes) 000346 1753440936.895550 trunc from 0x03fe69 to 0x020a47 (0x1f421 bytes) 000347 1753440936.908537 trunc from 0x020a48 to 0x01eb20 (0x1f27 bytes) 000348 1753440936.920297 mapwrite 0x03253b thru 0x03c61e (0xa0e4 bytes) 000350 1753440936.983549 trunc from 0x028250 to 0x02fee7 (0x7c98 bytes) 000351 1753440936.990319 fallocate 0x0261d6 thru 0x02b435 (0x5260 bytes) 000352 1753440936.998991 write 0x021587 thru 0x026210 (0x4c8a bytes) 000354 1753440937.007876 punch from 0x0237f5 to 0x027bd1 (0x43dd bytes) 000355 1753440937.031676 trunc from 0x029415 to 0x03007d (0x6c69 bytes) 000356 1753440937.035849 trunc from 0x03c61f to 0x0180d5 (0x24549 bytes) 000358 1753440937.057367 fallocate 0x038027 thru 0x03f995 (0x796f bytes) 000359 1753440937.062286 write 0x00a690 thru 0x00c315 (0x1c86 bytes) 000361 1753440937.070242 mapread 0x01a8b2 thru 0x0235df (0x8d2e bytes) 000362 1753440937.080460 mapwrite 0x00eadd thru 0x013610 (0x4b34 bytes) 000363 1753440937.098563 mapread 0x02bddb thru 0x02d0dd (0x1303 bytes) 000364 1753440937.108650 punch from 0x03dd71 to 0x03f995 (0x1c25 bytes) 000365 1753440937.115299 write 0x009780 thru 0x00f7f4 (0x6075 bytes) 000366 1753440937.116341 punch from 0x01c762 to 0x01d611 (0xeb0 bytes) 000367 1753440937.122731 mapread 0x00dacb thru 0x017910 (0x9e46 bytes) 000368 1753440937.123635 fallocate 0x005eb6 thru 0x00b52f (0x567a bytes) 000369 1753440937.129169 read 0x0330ed thru 0x03d5bc (0xa4d0 bytes) 000371 1753440937.135829 write 0x02857e thru 0x0371e7 (0xec6a bytes) 000373 1753440937.137756 write 0x0206f3 thru 0x023eb5 (0x37c3 bytes) 000374 1753440937.138729 write 0x029565 thru 0x0347ec (0xb288 bytes) 000375 1753440937.139863 write 0x03f25d thru 0x03ffff (0xda3 bytes) 000379 1753440937.147018 read 0x0018b9 thru 0x005055 (0x379d bytes) 000380 1753440937.157381 mapread 0x02a60f thru 0x032b0f (0x8501 bytes) 000381 1753440937.158329 mapwrite 0x01230d thru 0x01747a (0x516e bytes) 000382 1753440937.173970 fallocate 0x0048ba thru 0x00b903 (0x704a bytes) 000383 1753440937.177366 punch from 0x007301 to 0x008ade (0x17de bytes) 000384 1753440937.181273 write 0x0202a2 thru 0x022d47 (0x2aa6 bytes) 000386 1753440937.182651 mapread 0x03231a thru 0x03ffff (0xdce6 bytes) 000387 1753440937.183581 punch from 0x01fc71 to 0x024b03 (0x4e93 bytes) 000388 1753440937.189643 read 0x01d82b thru 0x01eda4 (0x157a bytes) 000389 1753440937.190233 trunc from 0x014c0e to 0x0232ca (0xe6bd bytes) 000390 1753440937.193261 fallocate 0x02a593 thru 0x031950 (0x73be bytes) 000392 1753440937.196922 write 0x0003ba thru 0x005bfe (0x5845 bytes) 000393 1753440937.197885 trunc from 0x033da3 to 0x035ba8 (0x1e06 bytes) 000394 1753440937.202333 write 0x00ad18 thru 0x00ce45 (0x212e bytes) 000395 1753440937.203161 mapwrite 0x0147ba thru 0x01b5ae (0x6df5 bytes) 000397 1753440937.223963 write 0x034d61 thru 0x03e4b8 (0x9758 bytes) 000398 1753440937.225228 mapread 0x0018ea thru 0x003bc4 (0x22db bytes) 000399 1753440937.228398 MIRROR_EXTEND to 2 mirrors 000400 1753440937.652776 trunc from 0x040000 to 0x034367 (0xbc98 bytes) 000401 1753440937.690977 mapread 0x02dfd9 thru 0x034367 (0x638f bytes) 000402 1753440937.707773 trunc from 0x0136a8 to 0x019c88 (0x65e1 bytes) 000403 1753440937.715559 punch from 0x005594 to 0x0077aa (0x2217 bytes) 000404 1753440937.720102 punch from 0x016c4b to 0x01986f (0x2c25 bytes) 000405 1753440937.723788 mapwrite 0x034898 thru 0x0367ea (0x1f53 bytes) 000406 1753440937.763795 MIRROR_RESYNC 2 mirrors 000407 1753440938.115523 fallocate 0x017810 thru 0x01d22f (0x5a20 bytes) 000408 1753440938.134247 trunc from 0x01fea4 to 0x02f4eb (0xf648 bytes) 000409 1753440938.138260 mapwrite 0x00de91 thru 0x014dc3 (0x6f33 bytes) 000410 1753440938.180719 MIRROR_RESYNC 2 mirrors 000411 1753440938.468093 punch from 0x025f30 to 0x032fbc (0xd08d bytes) 000412 1753440938.488347 read 0x02d87f thru 0x034e04 (0x7586 bytes) 000413 1753440938.495350 read 0x00f8fe thru 0x01a59c (0xac9f bytes) 000414 1753440938.509276 MIRROR_RESYNC 2 mirrors 000415 1753440938.891890 punch from 0x030e75 to 0x0367ea (0x5976 bytes) 000416 1753440938.918855 trunc from 0x0367eb to 0x03ee42 (0x8658 bytes) 000417 1753440938.933885 read 0x01f03b thru 0x024238 (0x51fe bytes) 000418 1753440938.943224 punch from 0x031852 to 0x03b9de (0xa18d bytes) 000419 1753440938.948861 trunc from 0x0129d8 to 0x01bcbe (0x92e7 bytes) 000420 1753440938.952470 trunc from 0x01263a to 0x020005 (0xd9cc bytes) 000423 1753440938.968040 MIRROR_EXTEND to 3 mirrors 000424 1753440939.355003 mapread 0x02054f thru 0x02f132 (0xebe4 bytes) 000426 1753440939.390536 MIRROR_RESYNC 3 mirrors 000427 1753440939.786483 read 0x037f89 thru 0x03ee42 (0x6eba bytes) 000428 1753440939.793300 write 0x007189 thru 0x01641f (0xf297 bytes) 000429 1753440939.836905 trunc from 0x03ee43 to 0x025e06 (0x1903c bytes) 000430 1753440939.867647 trunc from 0x025e07 to 0x024b4b (0x12bb bytes) 000431 1753440939.879228 punch from 0x010d46 to 0x020791 (0xfa4c bytes) 000432 1753440939.895635 MIRROR_EXTEND to 4 mirrors 000433 1753440940.317090 write 0x00c613 thru 0x0162a4 (0x9c92 bytes) 000434 1753440940.352595 trunc from 0x01cd15 to 0x01f3ee (0x26da bytes) 000435 1753440940.356609 punch from 0x003922 to 0x007a00 (0x40df bytes) 000436 1753440940.363999 MIRROR_EXTEND to 5 mirrors 000437 1753440940.702727 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000438 1753440940.906752 trunc from 0x0159a6 to 0x0203ef (0xaa4a bytes) 000439 1753440940.932889 trunc from 0x024b4c to 0x01065b (0x144f0 bytes) 000440 1753440940.953605 punch from 0x001ded to 0x00e4c2 (0xc6d6 bytes) 000441 1753440940.960379 MIRROR_RESYNC 4 mirrors 000442 1753440941.405909 fallocate 0x029727 thru 0x038cff (0xf5d9 bytes) 000443 1753440941.426845 write 0x00b911 thru 0x016572 (0xac62 bytes) 000444 1753440941.431829 fallocate 0x00e1a1 thru 0x016726 (0x8586 bytes) 000445 1753440941.437882 fallocate 0x0092d2 thru 0x013f1d (0xac4c bytes) 000446 1753440941.442348 mapwrite 0x03d8c8 thru 0x03e54c (0xc85 bytes) 000447 1753440941.481475 mapwrite 0x01caf0 thru 0x01d11d (0x62e bytes) 000448 1753440941.502265 read 0x006523 thru 0x0082bf (0x1d9d bytes) 000449 1753440941.506888 trunc from 0x03e54d to 0x034d23 (0x9829 bytes) 000450 1753440941.521777 mapwrite 0x01da77 thru 0x025c5a (0x81e4 bytes) 000451 1753440941.539147 write 0x03c7bc thru 0x03e00e (0x1853 bytes) 000452 1753440941.540151 trunc from 0x0070cc to 0x013b65 (0xca9a bytes) 000453 1753440941.544950 write 0x016596 thru 0x01c2ac (0x5d17 bytes) 000454 1753440941.546189 fallocate 0x037985 thru 0x03ffff (0x867b bytes) 000455 1753440941.549698 trunc from 0x040000 to 0x03141f (0xebe0 bytes) 000457 1753440941.561049 mapread 0x013ad3 thru 0x0194b3 (0x59e1 bytes) 000458 1753440941.571724 punch from 0x00a3c2 to 0x00b4c7 (0x1106 bytes) 000459 1753440941.576664 trunc from 0x031420 to 0x026c8f (0xa790 bytes) 000460 1753440941.592882 read 0x026633 thru 0x026c8f (0x65d bytes) 000461 1753440941.600698 trunc from 0x0110c7 to 0x01b6c9 (0xa603 bytes) 000462 1753440941.608120 punch from 0x00f901 to 0x015487 (0x5b87 bytes) 000463 1753440941.611254 write 0x008254 thru 0x00ee2c (0x6bd9 bytes) 000464 1753440941.612393 mapread 0x00e574 thru 0x00f728 (0x11b5 bytes) 000465 1753440941.622794 read 0x026616 thru 0x026c8f (0x67a bytes) 000467 1753440941.625362 mapwrite 0x002d66 thru 0x004fc8 (0x2263 bytes) 000468 1753440941.652116 mapread 0x00efc8 thru 0x01e933 (0xf96c bytes) 000469 1753440941.660847 read 0x007fc6 thru 0x0107cf (0x880a bytes) 000470 1753440941.661472 fallocate 0x01d7c8 thru 0x021e52 (0x468b bytes) 000471 1753440941.664623 write 0x03c28d thru 0x03ffff (0x3d73 bytes) 000473 1753440941.666013 mapread 0x01765b thru 0x02326e (0xbc14 bytes) 000475 1753440941.667374 trunc from 0x040000 to 0x034d76 (0xb289 bytes) 000476 1753440941.683181 punch from 0x00d451 to 0x01b462 (0xe012 bytes) 000477 1753440941.686818 mapread 0x0170f9 thru 0x021f65 (0xae6d bytes) 000478 1753440941.695199 mapread 0x01ca75 thru 0x027fcd (0xb559 bytes) 000479 1753440941.696429 fallocate 0x00259b thru 0x00895b (0x63c1 bytes) 000480 1753440941.702663 write 0x038be2 thru 0x03c039 (0x3458 bytes) 000481 1753440941.703745 mapread 0x021a08 thru 0x029f94 (0x858d bytes) 000482 1753440941.705168 trunc from 0x03c03a to 0x01501b (0x2701e bytes) 000483 1753440941.724130 trunc from 0x01501c to 0x01e431 (0x9416 bytes) 000484 1753440941.741780 write 0x002b77 thru 0x0104b4 (0xd93e bytes) 000485 1753440941.743218 trunc from 0x01e432 to 0x0268fb (0x84ca bytes) 000486 1753440941.761727 MIRROR_EXTEND to 5 mirrors 000487 1753440942.082147 mapread 0x01d6df thru 0x024f44 (0x7866 bytes) 000488 1753440942.107551 mapwrite 0x034a02 thru 0x03da82 (0x9081 bytes) 000489 1753440942.159387 read 0x01cb82 thru 0x02a9ce (0xde4d bytes) 000491 1753440942.166574 fallocate 0x007ff2 thru 0x014837 (0xc846 bytes) 000492 1753440942.170864 write 0x01527e thru 0x01b0b8 (0x5e3b bytes) 000493 1753440942.178488 trunc from 0x03a155 to 0x03ac95 (0xb41 bytes) 000494 1753440942.181876 trunc from 0x00bdaf to 0x019579 (0xd7cb bytes) 000495 1753440942.190604 punch from 0x005f06 to 0x00ae00 (0x4efb bytes) 000496 1753440942.195024 read 0x00a028 thru 0x014781 (0xa75a bytes) 000497 1753440942.203448 trunc from 0x009dcf to 0x019adc (0xfd0e bytes) 000498 1753440942.207621 punch from 0x035e79 to 0x03da82 (0x7c0a bytes) 000499 1753440942.211445 mapread 0x00c4ab thru 0x00d688 (0x11de bytes) 000500 1753440942.219211 write 0x0105a4 thru 0x019dd2 (0x982f bytes) 000501 1753440942.224889 read 0x034e46 thru 0x03da82 (0x8c3d bytes) 000502 1753440942.231160 read 0x023cd6 thru 0x024fff (0x132a bytes) 000503 1753440942.231978 punch from 0x01cfc5 to 0x01e547 (0x1583 bytes) 000504 1753440942.236441 mapread 0x0131a8 thru 0x01d550 (0xa3a9 bytes) 000505 1753440942.241352 read 0x0257cb thru 0x0335eb (0xde21 bytes) 000506 1753440942.242013 mapwrite 0x01a3db thru 0x024023 (0x9c49 bytes) 000507 1753440942.257397 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000508 1753440942.390784 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000509 1753440942.508744 fallocate 0x03453a thru 0x03f01f (0xaae6 bytes) 000510 1753440942.512727 mapwrite 0x00074a thru 0x003480 (0x2d37 bytes) 000511 1753440942.544650 MIRROR_EXTEND to 4 mirrors 000512 1753440942.860555 trunc from 0x0202bd to 0x02d767 (0xd4ab bytes) 000513 1753440942.884922 mapwrite 0x01002d thru 0x010902 (0x8d6 bytes) 000514 1753440942.903666 MIRROR_RESYNC 4 mirrors 000515 1753440943.438014 trunc from 0x0379f8 to 0x03da82 (0x608b bytes) 000516 1753440943.455792 fallocate 0x010f44 thru 0x01ae39 (0x9ef6 bytes) 000517 1753440943.458376 trunc from 0x03da83 to 0x0395c6 (0x44bc bytes) 000518 1753440943.469511 mapwrite 0x01694c thru 0x01ba35 (0x50ea bytes) 000519 1753440943.496608 trunc from 0x005206 to 0x00b453 (0x624e bytes) 000520 1753440943.500027 mapread 0x008f80 thru 0x0106d3 (0x7754 bytes) 000521 1753440943.506649 mapwrite 0x01ff9e thru 0x02a562 (0xa5c5 bytes) 000522 1753440943.529908 punch from 0x032d67 to 0x0395c6 (0x6860 bytes) 000523 1753440943.534502 fallocate 0x02de3d thru 0x039d23 (0xbee7 bytes) 000524 1753440943.538247 trunc from 0x039d24 to 0x00081c (0x39507 bytes) 000525 1753440943.545453 mapread 0x0000e9 thru 0x00081c (0x734 bytes) 000526 1753440943.549164 write 0x03b928 thru 0x03c68f (0xd68 bytes) 000527 1753440943.550141 read 0x035c8d thru 0x03c68f (0x6a03 bytes) 000528 1753440943.554544 mapwrite 0x00faef thru 0x012fd8 (0x34ea bytes) 000529 1753440943.570736 write 0x02b209 thru 0x032c7a (0x7a72 bytes) 000531 1753440943.576090 trunc from 0x03c690 to 0x02f0d1 (0xd5be bytes) 000532 1753440943.587862 mapwrite 0x02dbe4 thru 0x030157 (0x2574 bytes) 000534 1753440943.622566 mapwrite 0x0272b0 thru 0x02bc82 (0x49d3 bytes) 000535 1753440943.637321 trunc from 0x030158 to 0x03ea43 (0xe8ec bytes) 000536 1753440943.650226 trunc from 0x03ae35 to 0x03ea43 (0x3c0f bytes) 000537 1753440943.658888 MIRROR_RESYNC 4 mirrors 000539 1753440944.086256 trunc from 0x0242ab to 0x02e8f1 (0xa647 bytes) 000541 1753440944.104301 mapread 0x03630c thru 0x03b651 (0x5346 bytes) 000542 1753440944.113185 trunc from 0x0270b6 to 0x02c853 (0x579e bytes) 000543 1753440944.116332 mapwrite 0x02e2e8 thru 0x037c8a (0x99a3 bytes) 000544 1753440944.134801 trunc from 0x03ea44 to 0x02bb76 (0x12ecd bytes) 000545 1753440944.153787 MIRROR_EXTEND to 5 mirrors 000546 1753440944.476346 trunc from 0x02bb77 to 0x02faf5 (0x3f7f bytes) 000547 1753440944.503244 write 0x00fdab thru 0x011dc2 (0x2018 bytes) 000548 1753440944.509001 mapread 0x01d97d thru 0x02d41b (0xfa9f bytes) 000549 1753440944.519674 punch from 0x00e796 to 0x01007f (0x18ea bytes) 000550 1753440944.528607 fallocate 0x00413a thru 0x011265 (0xd12c bytes) 000551 1753440944.531892 trunc from 0x001527 to 0x008956 (0x7430 bytes) 000552 1753440944.535215 mapwrite 0x01d128 thru 0x02cbac (0xfa85 bytes) 000554 1753440944.554452 punch from 0x0291b5 to 0x02faf5 (0x6941 bytes) 000556 1753440944.558648 mapwrite 0x030c5e thru 0x0325dd (0x1980 bytes) 000558 1753440944.588917 write 0x01f264 thru 0x02eddc (0xfb79 bytes) 000559 1753440944.590423 trunc from 0x017cca to 0x0193a6 (0x16dd bytes) 000560 1753440944.598215 MIRROR_RESYNC 5 mirrors 000561 1753440945.192224 read 0x006a3a thru 0x00ffff (0x95c6 bytes) 000562 1753440945.201726 MIRROR_EXTEND to 6 mirrors 000563 1753440945.581581 write 0x005738 thru 0x011535 (0xbdfe bytes) 000564 1753440945.623279 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000565 1753440945.754520 trunc from 0x009e84 to 0x0193b6 (0xf533 bytes) 000566 1753440945.757961 punch from 0x02396c to 0x02c628 (0x8cbd bytes) 000568 1753440945.764766 fallocate 0x030618 thru 0x0349b6 (0x439f bytes) 000569 1753440945.769221 write 0x039b8a thru 0x03a832 (0xca9 bytes) 000570 1753440945.772931 fallocate 0x00a1f8 thru 0x013e43 (0x9c4c bytes) 000571 1753440945.776987 write 0x039709 thru 0x03ffff (0x68f7 bytes) 000572 1753440945.778265 mapwrite 0x03847c thru 0x03ffff (0x7b84 bytes) 000573 1753440945.796762 fallocate 0x005255 thru 0x00e6e1 (0x948d bytes) 000574 1753440945.805992 fallocate 0x00790d thru 0x012f75 (0xb669 bytes) 000575 1753440945.811470 mapread 0x0079d4 thru 0x009eab (0x24d8 bytes) 000576 1753440945.821168 mapread 0x006fff thru 0x01095a (0x995c bytes) 000577 1753440945.822122 punch from 0x034d43 to 0x03a8b5 (0x5b73 bytes) 000578 1753440945.825972 read 0x038f29 thru 0x03ffff (0x70d7 bytes) 000579 1753440945.834487 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000580 1753440945.957962 mapread 0x02729f thru 0x030113 (0x8e75 bytes) 000581 1753440945.967969 punch from 0x03a384 to 0x03ffff (0x5c7c bytes) 000582 1753440945.973751 fallocate 0x00dabb thru 0x0185f8 (0xab3e bytes) 000583 1753440945.979285 read 0x011c66 thru 0x01e4e2 (0xc87d bytes) 000584 1753440945.988007 MIRROR_EXTEND to 5 mirrors 000585 1753440946.319520 trunc from 0x023cb7 to 0x0273d7 (0x3721 bytes) 000586 1753440946.339454 mapread 0x0009f0 thru 0x0022d6 (0x18e7 bytes) 000588 1753440946.349237 punch from 0x022b7b to 0x02c6b3 (0x9b39 bytes) 000589 1753440946.352812 punch from 0x001271 to 0x00ad2b (0x9abb bytes) 000590 1753440946.356338 fallocate 0x004a6f thru 0x011e6c (0xd3fe bytes) 000591 1753440946.361134 trunc from 0x040000 to 0x019f01 (0x260fe bytes) 000592 1753440946.379720 mapread 0x001d55 thru 0x007db3 (0x605f bytes) 000593 1753440946.388521 fallocate 0x01db23 thru 0x02692a (0x8e08 bytes) 000594 1753440946.393978 write 0x012c4b thru 0x0184ff (0x58b5 bytes) 000595 1753440946.403083 trunc from 0x025993 to 0x02692a (0xf98 bytes) 000596 1753440946.407699 trunc from 0x02692b to 0x0251f0 (0x173a bytes) 000597 1753440946.426386 trunc from 0x0251f1 to 0x039784 (0x14594 bytes) 000598 1753440946.441594 MIRROR_EXTEND to 6 mirrors 000599 1753440946.809048 fallocate 0x03d27d thru 0x03ffff (0x2d83 bytes) 000600 1753440946.826846 trunc from 0x040000 to 0x0394cf (0x6b30 bytes) 000602 1753440946.841449 trunc from 0x0394d0 to 0x00e31c (0x2b1b3 bytes) 000603 1753440946.856823 trunc from 0x00e31d to 0x00a9a6 (0x3976 bytes) 000604 1753440946.865025 fallocate 0x01b09b thru 0x02a021 (0xef87 bytes) 000605 1753440946.872600 MIRROR_EXTEND to 7 mirrors 000607 1753440947.213984 punch from 0x009900 to 0x00a9a6 (0x10a7 bytes) 000609 1753440947.245129 mapread 0x00548c thru 0x007ad7 (0x264c bytes) 000610 1753440947.259221 mapwrite 0x00c92a thru 0x00ff68 (0x363f bytes) 000611 1753440947.285454 fallocate 0x030ce9 thru 0x037d36 (0x704e bytes) 000612 1753440947.289888 punch from 0x018f46 to 0x01e9e2 (0x5a9d bytes) 000613 1753440947.293456 mapwrite 0x02438f thru 0x0243c5 (0x37 bytes) 000614 1753440947.309984 mapread 0x01ef7a thru 0x029542 (0xa5c9 bytes) 000615 1753440947.312131 mapwrite 0x03b9eb thru 0x03ffff (0x4615 bytes) 000617 1753440947.350042 read 0x0317aa thru 0x0337f0 (0x2047 bytes) 000618 1753440947.354663 write 0x015bc0 thru 0x016d7e (0x11bf bytes) 000619 1753440947.360668 punch from 0x007d04 to 0x0105cb (0x88c8 bytes) 000620 1753440947.364326 punch from 0x024716 to 0x028df7 (0x46e2 bytes) 000621 1753440947.369396 MIRROR_RESYNC 7 mirrors 000622 1753440948.243648 mapwrite 0x03a751 thru 0x03b4f5 (0xda5 bytes) 000623 1753440948.319730 MIRROR_RESYNC 7 mirrors 000625 1753440949.350159 trunc from 0x00816e to 0x00baba (0x394d bytes) 000627 1753440949.377538 mapread 0x000a66 thru 0x002217 (0x17b2 bytes) 000628 1753440949.390753 MIRROR_RESYNC 7 mirrors 000629 1753440950.365792 trunc from 0x01df6c to 0x023d07 (0x5d9c bytes) 000630 1753440950.392177 trunc from 0x02f57e to 0x03c5ca (0xd04d bytes) 000631 1753440950.402050 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000632 1753440950.515874 trunc from 0x040000 to 0x02c1d9 (0x13e26 bytes) 000633 1753440950.528252 trunc from 0x01f907 to 0x02c1d9 (0xc8d3 bytes) 000635 1753440950.533610 read 0x02889e thru 0x02b000 (0x2763 bytes) 000636 1753440950.537369 trunc from 0x01f197 to 0x01f974 (0x7de bytes) 000638 1753440950.546456 trunc from 0x024452 to 0x02a3ae (0x5f5d bytes) 000639 1753440950.552212 MIRROR_EXTEND to 7 mirrors 000640 1753440950.841350 trunc from 0x02c1da to 0x0103ac (0x1be2d bytes) 000641 1753440950.867846 trunc from 0x0037cc to 0x00b6d0 (0x7f05 bytes) 000642 1753440950.873922 punch from 0x001e86 to 0x00af58 (0x90d3 bytes) 000643 1753440950.880095 write 0x03ea66 thru 0x03ffff (0x159a bytes) 000644 1753440950.881021 mapwrite 0x032b1f thru 0x03b1fd (0x86df bytes) 000645 1753440950.919634 write 0x013eea thru 0x01f050 (0xb167 bytes) 000647 1753440950.931401 mapwrite 0x0114b3 thru 0x019c8e (0x87dc bytes) 000648 1753440950.945749 read 0x01cabd thru 0x026654 (0x9b98 bytes) 000649 1753440950.951281 trunc from 0x040000 to 0x02cca0 (0x1335f bytes) 000650 1753440950.967233 read 0x01a3c3 thru 0x01cbf2 (0x2830 bytes) 000651 1753440950.967841 trunc from 0x00abac to 0x00b0e9 (0x53e bytes) 000652 1753440950.971626 punch from 0x00240c to 0x0109ba (0xe5af bytes) 000654 1753440950.975998 read 0x006e9b thru 0x00eb77 (0x7cdd bytes) 000655 1753440950.983134 mapread 0x008210 thru 0x015ff1 (0xdde2 bytes) 000656 1753440950.984110 trunc from 0x02cca1 to 0x0315b4 (0x4914 bytes) 000659 1753440951.000448 mapread 0x01edd2 thru 0x022843 (0x3a72 bytes) 000660 1753440951.001909 trunc from 0x0315b5 to 0x008445 (0x2916f bytes) 000661 1753440951.010897 trunc from 0x004291 to 0x008445 (0x41b5 bytes) 000664 1753440951.014939 fallocate 0x013098 thru 0x017833 (0x479c bytes) 000665 1753440951.018397 trunc from 0x004464 to 0x008445 (0x3fe2 bytes) 000666 1753440951.021592 trunc from 0x008446 to 0x02ccc7 (0x24882 bytes) 000667 1753440951.032166 punch from 0x01e018 to 0x026c16 (0x8bff bytes) 000668 1753440951.035996 trunc from 0x02ccc8 to 0x03cf15 (0x1024e bytes) 000669 1753440951.046509 write 0x01c252 thru 0x02246c (0x621b bytes) 000670 1753440951.057297 read 0x0152bc thru 0x01a8d3 (0x5618 bytes) 000671 1753440951.067696 punch from 0x01ca3e to 0x020564 (0x3b27 bytes) 000672 1753440951.081811 mapread 0x00fbc1 thru 0x010f4f (0x138f bytes) 000673 1753440951.082799 write 0x000ceb thru 0x00a8cf (0x9be5 bytes) 000674 1753440951.083931 write 0x00ddb1 thru 0x011bfe (0x3e4e bytes) 000675 1753440951.084782 mapwrite 0x03dbbf thru 0x03ffff (0x2441 bytes) 000676 1753440951.136570 trunc from 0x00a96c to 0x00b6f4 (0xd89 bytes) 000677 1753440951.140789 fallocate 0x038681 thru 0x03ffff (0x797f bytes) 000678 1753440951.144578 fallocate 0x026dc9 thru 0x02ec46 (0x7e7e bytes) 000679 1753440951.151614 fallocate 0x022118 thru 0x02b372 (0x925b bytes) 000680 1753440951.157269 write 0x001685 thru 0x011541 (0xfebd bytes) 000681 1753440951.159604 read 0x03fed4 thru 0x03ffff (0x12c bytes) 000682 1753440951.160795 punch from 0x008bc1 to 0x0120e4 (0x9524 bytes) 000683 1753440951.179330 MIRROR_EXTEND to 8 mirrors 000685 1753440951.591981 mapwrite 0x009948 thru 0x00adf1 (0x14aa bytes) 000686 1753440951.685743 write 0x03a5d1 thru 0x03c36d (0x1d9d bytes) 000687 1753440951.697897 fallocate 0x0391a4 thru 0x03ffff (0x6e5c bytes) 000688 1753440951.702460 trunc from 0x01c5bb to 0x01e9d6 (0x241c bytes) 000689 1753440951.709684 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000690 1753440951.860293 read 0x019599 thru 0x027a94 (0xe4fc bytes) 000691 1753440951.866855 punch from 0x005dd1 to 0x012083 (0xc2b3 bytes) 000692 1753440951.872440 trunc from 0x040000 to 0x03db86 (0x2479 bytes) 000693 1753440951.890593 punch from 0x020ba0 to 0x020d64 (0x1c5 bytes) 000694 1753440951.895611 mapwrite 0x0005d2 thru 0x005f8d (0x59bc bytes) 000695 1753440951.943487 mapwrite 0x01d76d thru 0x026414 (0x8ca8 bytes) 000696 1753440951.972622 mapread 0x037478 thru 0x03db86 (0x670f bytes) 000697 1753440951.982624 punch from 0x03868e to 0x03db86 (0x54f9 bytes) 000698 1753440951.987871 punch from 0x025777 to 0x029af9 (0x4383 bytes) 000699 1753440951.994290 punch from 0x01f976 to 0x020243 (0x8ce bytes) 000700 1753440951.998205 mapwrite 0x01bc4d thru 0x01ec29 (0x2fdd bytes) 000701 1753440952.017191 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000702 1753440952.197611 MIRROR_RESYNC 6 mirrors 000703 1753440952.951045 punch from 0x00ab4d to 0x0184af (0xd963 bytes) 000704 1753440952.972664 trunc from 0x01df2d to 0x01fc88 (0x1d5c bytes) 000705 1753440952.975817 write 0x02716c thru 0x0281aa (0x103f bytes) 000706 1753440952.985697 mapread 0x01a4d2 thru 0x023f9e (0x9acd bytes) 000707 1753440952.997680 trunc from 0x03db87 to 0x0075c8 (0x365be bytes) 000708 1753440953.015312 mapread 0x00631c thru 0x0075c8 (0x12ad bytes) 000709 1753440953.022792 punch from 0x001791 to 0x0075c8 (0x5e38 bytes) 000710 1753440953.026883 mapwrite 0x0154dc thru 0x019638 (0x415d bytes) 000711 1753440953.068492 mapread 0x000c00 thru 0x007111 (0x6512 bytes) 000712 1753440953.077320 read 0x0001dc thru 0x0014ef (0x1314 bytes) 000713 1753440953.077948 trunc from 0x0015e1 to 0x00c786 (0xb1a6 bytes) 000714 1753440953.082870 mapread 0x003733 thru 0x008d8c (0x565a bytes) 000715 1753440953.088823 mapread 0x00d7f9 thru 0x00f738 (0x1f40 bytes) 000716 1753440953.089809 write 0x0186d5 thru 0x025eef (0xd81b bytes) 000717 1753440953.091303 mapread 0x00dab8 thru 0x01d302 (0xf84b bytes) 000718 1753440953.092385 read 0x0021e8 thru 0x00c67e (0xa497 bytes) 000719 1753440953.093179 trunc from 0x016b78 to 0x024076 (0xd4ff bytes) 000721 1753440953.109931 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000722 1753440953.260897 fallocate 0x003ba9 thru 0x003e03 (0x25b bytes) 000724 1753440953.266600 mapwrite 0x0390a7 thru 0x03ffff (0x6f59 bytes) 000725 1753440953.339667 MIRROR_RESYNC 5 mirrors 000726 1753440954.037266 mapread 0x036b25 thru 0x03ffff (0x94db bytes) 000727 1753440954.058242 MIRROR_RESYNC 5 mirrors 000728 1753440954.172378 trunc from 0x040000 to 0x025d13 (0x1a2ec bytes) 000730 1753440954.204528 trunc from 0x00a672 to 0x00fb2b (0x54ba bytes) 000731 1753440954.210148 read 0x011caa thru 0x01a772 (0x8ac9 bytes) 000732 1753440954.216595 mapread 0x00f740 thru 0x01bd4a (0xc60b bytes) 000733 1753440954.231878 read 0x016212 thru 0x01b5cd (0x53bc bytes) 000734 1753440954.232455 write 0x0350c3 thru 0x03ffff (0xaf3d bytes) 000735 1753440954.233745 write 0x0223f7 thru 0x02b6eb (0x92f5 bytes) 000736 1753440954.245118 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000737 1753440954.374505 read 0x00c2f3 thru 0x01ae9a (0xeba8 bytes) 000738 1753440954.379658 read 0x0398f9 thru 0x03cb3a (0x3242 bytes) 000739 1753440954.390827 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000740 1753440954.533222 trunc from 0x040000 to 0x01a049 (0x25fb6 bytes) 000741 1753440954.548704 read 0x00a612 thru 0x00b419 (0xe08 bytes) 000742 1753440954.552415 write 0x011757 thru 0x01f52f (0xddd9 bytes) 000743 1753440954.556907 punch from 0x00229a to 0x011e7c (0xfbe3 bytes) 000744 1753440954.570790 mapread 0x008afc thru 0x014f8e (0xc493 bytes) 000745 1753440954.583235 write 0x00909a thru 0x014efe (0xbe65 bytes) 000746 1753440954.584538 write 0x00458d thru 0x00ce56 (0x88ca bytes) 000748 1753440954.586190 fallocate 0x03e6fd thru 0x03ffff (0x1903 bytes) 000749 1753440954.591266 trunc from 0x040000 to 0x035778 (0xa887 bytes) 000750 1753440954.613471 punch from 0x0332fe to 0x035778 (0x247b bytes) 000751 1753440954.617452 mapwrite 0x013961 thru 0x023030 (0xf6d0 bytes) 000752 1753440954.646836 read 0x00cd91 thru 0x01150b (0x477b bytes) 000753 1753440954.647480 mapwrite 0x01d77f thru 0x01eee1 (0x1763 bytes) 000754 1753440954.660267 punch from 0x024444 to 0x02d807 (0x93c4 bytes) 000755 1753440954.663941 write 0x03f2a6 thru 0x03ffff (0xd5a bytes) 000756 1753440954.665304 read 0x011dfe thru 0x01e84a (0xca4d bytes) 000758 1753440954.666318 mapwrite 0x00228c thru 0x00e809 (0xc57e bytes) 000759 1753440954.682315 mapread 0x013fb4 thru 0x01b15b (0x71a8 bytes) 000760 1753440954.683207 trunc from 0x033cf7 to 0x039b79 (0x5e83 bytes) 000762 1753440954.688673 mapread 0x02d599 thru 0x035ec3 (0x892b bytes) 000763 1753440954.696834 fallocate 0x0227b0 thru 0x0312e0 (0xeb31 bytes) 000764 1753440954.707799 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000765 1753440954.841360 trunc from 0x040000 to 0x030fcd (0xf032 bytes) 000766 1753440954.855327 read 0x0082b9 thru 0x0142d8 (0xc020 bytes) 000767 1753440954.867150 MIRROR_EXTEND to 3 mirrors 000768 1753440955.172124 write 0x006be6 thru 0x01211d (0xb538 bytes) 000769 1753440955.198484 MIRROR_RESYNC 3 mirrors 000770 1753440955.543928 punch from 0x020c92 to 0x02392e (0x2c9d bytes) 000772 1753440955.568112 trunc from 0x030fce to 0x0375cf (0x6602 bytes) 000773 1753440955.585637 trunc from 0x0293b0 to 0x0343d4 (0xb025 bytes) 000774 1753440955.590558 trunc from 0x0375d0 to 0x02dd32 (0x989d bytes) 000775 1753440955.601244 punch from 0x01f28d to 0x026149 (0x6ebd bytes) 000776 1753440955.607688 read 0x01fa3b thru 0x02176d (0x1d33 bytes) 000777 1753440955.612502 mapwrite 0x00d082 thru 0x01b97a (0xe8f9 bytes) 000778 1753440955.642723 punch from 0x004083 to 0x01356a (0xf4e8 bytes) 000779 1753440955.646039 punch from 0x01b3cb to 0x02913f (0xdd75 bytes) 000780 1753440955.648823 trunc from 0x029bbc to 0x02dd32 (0x4177 bytes) 000781 1753440955.651798 mapwrite 0x00deb0 thru 0x00fdd5 (0x1f26 bytes) 000782 1753440955.667938 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000783 1753440955.836796 write 0x03cc27 thru 0x03ffff (0x33d9 bytes) 000784 1753440955.840651 mapwrite 0x0203c9 thru 0x02a665 (0xa29d bytes) 000785 1753440955.887289 trunc from 0x02deab to 0x03a820 (0xc976 bytes) 000786 1753440955.891466 read 0x03ce21 thru 0x03ffff (0x31df bytes) 000787 1753440955.892097 punch from 0x0303f2 to 0x03fe72 (0xfa81 bytes) 000789 1753440955.899221 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000790 1753440955.986508 read 0x00a4ed thru 0x00f839 (0x534d bytes) 000791 1753440955.999323 MIRROR_EXTEND to 2 mirrors 000792 1753440956.353361 MIRROR_EXTEND to 3 mirrors 000793 1753440956.747409 trunc from 0x0397bc to 0x03ffff (0x6844 bytes) 000794 1753440956.768474 fallocate 0x031fc6 thru 0x03ffff (0xe03a bytes) 000796 1753440956.772905 mapread 0x03209e thru 0x03f7f0 (0xd753 bytes) 000797 1753440956.795686 mapwrite 0x000dbd thru 0x00e466 (0xd6aa bytes) 000798 1753440956.819105 trunc from 0x02d7d5 to 0x0325d0 (0x4dfc bytes) 000799 1753440956.822649 punch from 0x004c2e to 0x00adcc (0x619f bytes) 000800 1753440956.826116 mapwrite 0x01a5be thru 0x02098b (0x63ce bytes) 000801 1753440956.845239 trunc from 0x03b327 to 0x03d2d1 (0x1fab bytes) 000802 1753440956.848668 read 0x02c74c thru 0x038309 (0xbbbe bytes) 000803 1753440956.855480 read 0x02a8bf thru 0x0373b1 (0xcaf3 bytes) 000804 1753440956.865140 MIRROR_RESYNC 3 mirrors 000805 1753440957.311200 punch from 0x022b14 to 0x0319d2 (0xeebf bytes) 000806 1753440957.329973 punch from 0x01b5d5 to 0x02510d (0x9b39 bytes) 000807 1753440957.333440 punch from 0x03cd90 to 0x03ffff (0x3270 bytes) 000808 1753440957.336811 trunc from 0x040000 to 0x00b32f (0x34cd0 bytes) 000809 1753440957.351643 fallocate 0x02f6ae thru 0x02ffaf (0x902 bytes) 000810 1753440957.355453 punch from 0x0237b3 to 0x0266a1 (0x2eef bytes) 000811 1753440957.358911 read 0x00ef04 thru 0x01af2d (0xc02a bytes) 000814 1753440957.371659 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000815 1753440957.481467 read 0x006f53 thru 0x0093d7 (0x2485 bytes) 000816 1753440957.487934 fallocate 0x0010ac thru 0x006480 (0x53d5 bytes) 000817 1753440957.491632 read 0x02f408 thru 0x02ffaf (0xba8 bytes) 000818 1753440957.500840 MIRROR_RESYNC 2 mirrors 000819 1753440957.827230 read 0x01eeb4 thru 0x0284ed (0x963a bytes) 000820 1753440957.831838 trunc from 0x025f0b to 0x02fd43 (0x9e39 bytes) 000821 1753440957.855577 read 0x002e8b thru 0x00c197 (0x930d bytes) 000822 1753440957.864798 fallocate 0x002995 thru 0x003d49 (0x13b5 bytes) 000823 1753440957.868854 punch from 0x02064a to 0x025b45 (0x54fc bytes) 000824 1753440957.875903 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000825 1753440957.963238 read 0x01692c thru 0x02614a (0xf81f bytes) 000826 1753440957.972977 mapwrite 0x019b20 thru 0x01e8c2 (0x4da3 bytes) 000829 1753440957.992275 write 0x0141f4 thru 0x02050a (0xc317 bytes) 000830 1753440958.003761 read 0x020f60 thru 0x027849 (0x68ea bytes) 000831 1753440958.013393 read 0x02863a thru 0x02fae7 (0x74ae bytes) 000832 1753440958.014395 mapwrite 0x0139e0 thru 0x0219c0 (0xdfe1 bytes) 000833 1753440958.032477 trunc from 0x0045f6 to 0x007ec2 (0x38cd bytes) 000835 1753440958.037464 punch from 0x026c4f to 0x02ffaf (0x9361 bytes) 000836 1753440958.042622 write 0x01d04a thru 0x025158 (0x810f bytes) 000839 1753440958.045106 punch from 0x01a0a5 to 0x020bdd (0x6b39 bytes) 000840 1753440958.068798 read 0x019d93 thru 0x0244cc (0xa73a bytes) 000841 1753440958.075618 trunc from 0x02ffb0 to 0x0151ba (0x1adf5 bytes) 000843 1753440958.092596 trunc from 0x0151bb to 0x025c80 (0x10ac6 bytes) 000844 1753440958.105698 mapwrite 0x001d94 thru 0x0022f4 (0x561 bytes) 000845 1753440958.118355 mapwrite 0x011cc4 thru 0x01734a (0x5687 bytes) 000846 1753440958.134957 read 0x01533e thru 0x01e5fa (0x92bd bytes) 000847 1753440958.140871 trunc from 0x0239e8 to 0x025c80 (0x2299 bytes) 000849 1753440958.144967 mapread 0x00aade thru 0x012e65 (0x8388 bytes) 000850 1753440958.145927 trunc from 0x025c81 to 0x016451 (0xf82f bytes) 000851 1753440958.157189 write 0x015c23 thru 0x017b38 (0x1f16 bytes) 000852 1753440958.158921 trunc from 0x00ecec to 0x017b38 (0x8e4d bytes) 000853 1753440958.170852 fallocate 0x0232f5 thru 0x031028 (0xdd34 bytes) 000854 1753440958.174397 trunc from 0x017b39 to 0x00c8a8 (0xb290 bytes) 000855 1753440958.186485 mapread 0x00474e thru 0x00a82e (0x60e1 bytes) 000856 1753440958.187373 trunc from 0x00634a to 0x0082fc (0x1fb3 bytes) 000858 1753440958.190852 fallocate 0x022479 thru 0x02c3b1 (0x9f39 bytes) 000859 1753440958.197881 MIRROR_EXTEND to 2 mirrors 000860 1753440958.522892 punch from 0x02b375 to 0x02c3b1 (0x103d bytes) 000861 1753440958.541949 read 0x01b201 thru 0x02256f (0x736f bytes) 000862 1753440958.552526 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000863 1753440958.659139 read 0x01cbd0 thru 0x02c3b1 (0xf7e2 bytes) 000864 1753440958.665772 fallocate 0x01e380 thru 0x02b3e1 (0xd062 bytes) 000865 1753440958.671194 fallocate 0x031495 thru 0x038931 (0x749d bytes) 000866 1753440958.674748 mapread 0x00d710 thru 0x01b44e (0xdd3f bytes) 000868 1753440958.693336 mapread 0x00e8e6 thru 0x014d40 (0x645b bytes) 000869 1753440958.694194 mapread 0x02a050 thru 0x02c3b1 (0x2362 bytes) 000870 1753440958.694872 mapwrite 0x017dbd thru 0x024792 (0xc9d6 bytes) 000871 1753440958.715544 write 0x02bb64 thru 0x0356a1 (0x9b3e bytes) 000872 1753440958.721396 fallocate 0x025e6a thru 0x026df3 (0xf8a bytes) 000874 1753440958.725192 write 0x0083cc thru 0x010fc6 (0x8bfb bytes) 000877 1753440958.727224 mapwrite 0x00a9be thru 0x01398a (0x8fcd bytes) 000878 1753440958.739018 read 0x014178 thru 0x01d6e9 (0x9572 bytes) 000879 1753440958.739960 trunc from 0x0017ea to 0x00760b (0x5e22 bytes) 000880 1753440958.745399 fallocate 0x03d565 thru 0x03ffff (0x2a9b bytes) 000881 1753440958.753274 mapread 0x009f3a thru 0x011203 (0x72ca bytes) 000882 1753440958.754861 mapwrite 0x000f53 thru 0x003ca3 (0x2d51 bytes) 000883 1753440958.774723 punch from 0x01a2b4 to 0x02a26e (0xffbb bytes) 000884 1753440958.781187 mapwrite 0x0023be thru 0x011d2b (0xf96e bytes) 000885 1753440958.798916 fallocate 0x001fe0 thru 0x00a32c (0x834d bytes) 000886 1753440958.807334 punch from 0x01ad0e to 0x024ecd (0xa1c0 bytes) 000887 1753440958.811821 punch from 0x00c27e to 0x01c0ab (0xfe2e bytes) 000888 1753440958.817942 fallocate 0x0060f7 thru 0x0129b3 (0xc8bd bytes) 000890 1753440958.825258 trunc from 0x0356a2 to 0x03508f (0x612 bytes) 000891 1753440958.840680 punch from 0x01cc01 to 0x022081 (0x5481 bytes) 000892 1753440958.845476 write 0x038eab thru 0x03ffff (0x7155 bytes) 000893 1753440958.846859 trunc from 0x040000 to 0x00b56f (0x34a90 bytes) 000894 1753440958.856221 punch from 0x00303a to 0x00b56f (0x8536 bytes) 000895 1753440958.859867 mapwrite 0x0020fe thru 0x00c02f (0x9f32 bytes) 000896 1753440958.934618 write 0x001957 thru 0x00a00e (0x86b8 bytes) 000897 1753440958.936045 punch from 0x007a80 to 0x00c02f (0x45b0 bytes) 000898 1753440958.957240 MIRROR_EXTEND to 2 mirrors 000899 1753440959.284003 fallocate 0x026868 thru 0x028ce3 (0x247c bytes) 000900 1753440959.304852 read 0x020070 thru 0x023016 (0x2fa7 bytes) 000901 1753440959.316790 read 0x01c833 thru 0x023f58 (0x7726 bytes) 000902 1753440959.327528 mapread 0x00484c thru 0x00937a (0x4b2f bytes) 000903 1753440959.328540 punch from 0x0239f2 to 0x026f94 (0x35a3 bytes) 000904 1753440959.334149 mapread 0x022938 thru 0x028704 (0x5dcd bytes) 000905 1753440959.373767 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x36f:0x0]: Invalid argument (22) 000906 1753440959.481006 trunc from 0x0243d6 to 0x028ce3 (0x490e bytes) 000907 1753440959.484740 punch from 0x002333 to 0x002753 (0x421 bytes) 000908 1753440959.488972 read 0x0137e3 thru 0x01d563 (0x9d81 bytes) 000910 1753440959.496828 mapread 0x005332 thru 0x007d7e (0x2a4d bytes) 000911 1753440959.513307 write 0x0204b0 thru 0x026e0f (0x6960 bytes) 000912 1753440959.522542 mapwrite 0x00f71c thru 0x01a468 (0xad4d bytes) 000913 1753440959.541768 fallocate 0x0131f5 thru 0x017d2a (0x4b36 bytes) 000914 1753440959.546285 trunc from 0x028ce4 to 0x007c8c (0x21057 bytes) 000915 1753440959.563891 mapread 0x006121 thru 0x007c8c (0x1b6c bytes) 000917 1753440959.565658 trunc from 0x007c8d to 0x03d41d (0x35791 bytes) 000918 1753440959.575821 trunc from 0x03d41e to 0x01415c (0x292c1 bytes) 000919 1753440959.585464 trunc from 0x008718 to 0x01415c (0xba45 bytes) 000921 1753440959.590127 read 0x005f4f thru 0x01415c (0xe20e bytes) 000922 1753440959.596695 fallocate 0x03d890 thru 0x03ffff (0x2770 bytes) 000923 1753440959.601506 mapread 0x0059d4 thru 0x011b93 (0xc1c0 bytes) 000924 1753440959.602821 mapwrite 0x014c41 thru 0x0160fa (0x14ba bytes) 000925 1753440959.624022 punch from 0x01fc85 to 0x02308d (0x3409 bytes) 000926 1753440959.627599 mapwrite 0x0155a9 thru 0x0157af (0x207 bytes) 000927 1753440959.639355 write 0x03f54e thru 0x03ffff (0xab2 bytes) 000928 1753440959.640214 trunc from 0x03175b to 0x03fb59 (0xe3ff bytes) 000929 1753440959.647596 punch from 0x0037f9 to 0x012d0d (0xf515 bytes) 000930 1753440959.650810 fallocate 0x0199ee thru 0x021e1d (0x8430 bytes) 000931 1753440959.653977 mapread 0x025ac9 thru 0x02dc65 (0x819d bytes) 000932 1753440959.657142 fallocate 0x039e18 thru 0x03fa12 (0x5bfb bytes) 000933 1753440959.660605 fallocate 0x02fb1d thru 0x034fb2 (0x5496 bytes) 000935 1753440959.664354 fallocate 0x014b32 thru 0x01f9a5 (0xae74 bytes) 000936 1753440959.668152 mapwrite 0x03e2dd thru 0x03ffff (0x1d23 bytes) 000937 1753440959.685213 punch from 0x012db6 to 0x01b2fc (0x8547 bytes) 000938 1753440959.689586 read 0x0093dc thru 0x01429a (0xaebf bytes) 000939 1753440959.703555 MIRROR_EXTEND to 2 mirrors 000940 1753440960.049691 read 0x022420 thru 0x02e76e (0xc34f bytes) 000941 1753440960.064606 MIRROR_EXTEND to 3 mirrors 000942 1753440960.398499 mapread 0x038de6 thru 0x03ffff (0x721a bytes) 000943 1753440960.431462 read 0x038e6a thru 0x03b9ec (0x2b83 bytes) 000944 1753440960.438852 trunc from 0x040000 to 0x01b1a9 (0x24e56 bytes) 000945 1753440960.483960 MIRROR_EXTEND to 4 mirrors 000946 1753440960.752425 mapread 0x017fce thru 0x01b1a9 (0x31dc bytes) 000947 1753440960.778385 punch from 0x00d606 to 0x013980 (0x637b bytes) 000948 1753440960.806767 MIRROR_RESYNC 4 mirrors 000949 1753440961.192883 read 0x0084e3 thru 0x0136e5 (0xb203 bytes) 000950 1753440961.197948 punch from 0x017f92 to 0x01b1a9 (0x3218 bytes) 000951 1753440961.218619 MIRROR_EXTEND to 5 mirrors 000952 1753440961.559321 read 0x01722f thru 0x01b1a9 (0x3f7b bytes) 000953 1753440961.571712 read 0x00533e thru 0x005ac9 (0x78c bytes) 000955 1753440961.589133 mapread 0x0026c8 thru 0x00ee01 (0xc73a bytes) 000956 1753440961.601836 trunc from 0x01b1aa to 0x012b4b (0x865e bytes) 000957 1753440961.625531 read 0x00f06c thru 0x012b4b (0x3ae0 bytes) 000958 1753440961.629854 mapread 0x00b7c0 thru 0x012b4b (0x738c bytes) 000960 1753440961.631257 write 0x01f817 thru 0x027824 (0x800e bytes) 000961 1753440961.632394 fallocate 0x027ce4 thru 0x027efb (0x218 bytes) 000962 1753440961.637153 MIRROR_RESYNC 5 mirrors 000963 1753440962.011162 trunc from 0x0129ea to 0x01660e (0x3c25 bytes) 000964 1753440962.026039 read 0x0234f1 thru 0x027824 (0x4334 bytes) 000965 1753440962.030870 fallocate 0x01ba44 thru 0x01c878 (0xe35 bytes) 000966 1753440962.034679 write 0x0026e7 thru 0x00f787 (0xd0a1 bytes) 000967 1753440962.041556 mapwrite 0x01b678 thru 0x02ac87 (0xf610 bytes) 000968 1753440962.081892 mapread 0x0175c9 thru 0x02483f (0xd277 bytes) 000969 1753440962.084626 trunc from 0x02ac88 to 0x013c50 (0x17037 bytes) 000970 1753440962.099581 trunc from 0x012fac to 0x013c50 (0xca5 bytes) 000971 1753440962.102624 trunc from 0x005c0e to 0x00ca17 (0x6e0a bytes) 000972 1753440962.107045 trunc from 0x013c51 to 0x035948 (0x21cf8 bytes) 000973 1753440962.121576 write 0x03886b thru 0x03ffff (0x7795 bytes) 000974 1753440962.122917 trunc from 0x040000 to 0x00707f (0x38f80 bytes) 000975 1753440962.135543 read 0x003e44 thru 0x00707f (0x323c bytes) 000976 1753440962.143342 mapread 0x004170 thru 0x005eb3 (0x1d44 bytes) 000977 1753440962.144166 read 0x003d2a thru 0x00707f (0x3356 bytes) 000978 1753440962.144718 write 0x029d63 thru 0x02c554 (0x27f2 bytes) 000979 1753440962.145814 punch from 0x02aa4b to 0x02c554 (0x1b0a bytes) 000980 1753440962.158597 mapwrite 0x0054b6 thru 0x0137e0 (0xe32b bytes) 000981 1753440962.186704 fallocate 0x0041a4 thru 0x006f13 (0x2d70 bytes) 000982 1753440962.191598 read 0x01a75c thru 0x01d369 (0x2c0e bytes) 000983 1753440962.192164 fallocate 0x007478 thru 0x010324 (0x8ead bytes) 000985 1753440962.199078 write 0x010363 thru 0x017522 (0x71c0 bytes) 000986 1753440962.200187 trunc from 0x02c555 to 0x00f43e (0x1d116 bytes) 000987 1753440962.221283 trunc from 0x005d69 to 0x008ff0 (0x3288 bytes) 000988 1753440962.225880 mapread 0x005804 thru 0x008739 (0x2f36 bytes) 000989 1753440962.232294 mapwrite 0x022188 thru 0x0286d1 (0x654a bytes) 000990 1753440962.273229 mapwrite 0x004ec8 thru 0x00ce95 (0x7fce bytes) 000991 1753440962.290603 read 0x019bf9 thru 0x01e338 (0x4740 bytes) 000992 1753440962.292941 fallocate 0x038c21 thru 0x0397a1 (0xb81 bytes) 000993 1753440962.296892 fallocate 0x0009f8 thru 0x002410 (0x1a19 bytes) 000994 1753440962.300615 mapread 0x038c0b thru 0x0397a1 (0xb97 bytes) 000995 1753440962.306886 punch from 0x01c911 to 0x02bf3b (0xf62b bytes) 000996 1753440962.310385 mapwrite 0x012a45 thru 0x01d73f (0xacfb bytes) 000997 1753440962.335094 mapread 0x0038cb thru 0x00505e (0x1794 bytes) 000998 1753440962.335921 mapread 0x02ff3c thru 0x0397a1 (0x9866 bytes) 000999 1753440962.336766 write 0x02d87b thru 0x0318c9 (0x404f bytes) 001000 1753440962.337707 mapwrite 0x0303fc thru 0x032900 (0x2505 bytes) All operations completed A-OK! PASS 70a (38s) == sanity-flr test 71: check mirror extend parent fid ==== 06:56:04 (1753440964) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x56:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 200a: stress test ===================== 06:56:09 (1753440969) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2743840 .. Extending file size to 2095488 .. failed Extending file size to 3444000 .. Extending file size to 8981600 .. Extending file size to 5819616 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6916992 .. Extending file size to 8819840 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9240320 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9276960 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9320928 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60427 60428 60429 60432 60434 done 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) 192.168.202.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:57:27 (1753441047) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6203072 .. Extending file size to 3500896 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6573856 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8065984 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8758112 .. Extending file size to 8174528 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8974272 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9058368 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8989952 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9091680 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9036096 .. Waiting 63270 63271 63272 63273 63276 done 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) 192.168.202.119@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:58:39 (1753441119) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22464 3584556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14612 3592408 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37076 7176964 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 22464 3584556 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14612 3592408 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37076 7176964 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0790043 s, 26.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18889 s, 11.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:58:53 (1753441133) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.073275 s, 28.6 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 06:58:59 (1753441139) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xad:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:59:04 (1753441144) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0550266 s, 38.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:59:10 (1753441150) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115112 s, 18.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:59:16 (1753441156) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0867014 s, 24.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4ff:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:59:45 (1753441185) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610222 s, 17.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x501:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x500:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x501:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x500:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4fe:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:00:32 (1753441232) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0606196 s, 34.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:01:20 (1753441280) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg219-server: Pool lustre.test_204f created Adding targets to pool oleg219-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg219-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0785898 s, 26.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x502:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x502:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg219-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg219-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg219-server: Pool lustre.test_204f destroyed PASS 204f (64s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:02:24 (1753441344) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:02:29 (1753441349) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:02:34 (1753441354) oleg219-server: Pool lustre.test_206 created Adding targets to pool oleg219-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg219-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg219-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg219-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg219-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:02:55 (1753441375) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582733 s, 18.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322469 s, 32.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x8a3bdf8d Mirror 3: 0x8a3bdf8d PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:03:03 (1753441383) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:03:06 (1753441386) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:03:08 (1753441388) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0393931 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1530000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1230000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1230000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1230000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xab0000 Read this (4 bytes): BBBB 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:03:33 (1753441413) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0182667 s, 448 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:03:36 (1753441416) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251313 s, 41.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-07-25 07:03:37.000000000 -0400 Modify: 2025-07-25 07:03:37.000000000 -0400 Change: 2025-07-25 07:03:38.422876997 -0400 Birth: 2025-07-25 07:03:37.128000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:03:41 (1753441421) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036126 s, 29.0 MB/s oleg219-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg219-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg219-server: O/280000401/d26/ba: File not found by ext2_lookup PASS 210b (18s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:03:59 (1753441439) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0892175 s, 459 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0184503 s, 222 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 25 07:04 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 25 07:04 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test complete, duration 1163 sec =========== 07:04:03 (1753441443) === sanity-flr: start cleanup 07:04:04 (1753441444) === === sanity-flr: finish cleanup 07:04:08 (1753441448) ===