-----============= acceptance-small: sanity-flr ============----- Sun Aug 31 17:15:04 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg147-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:15:10 (1756674910) === oleg147-client.virtnet: executing check_config_client /mnt/lustre oleg147-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg147-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff999b19645800.idle_timeout=debug osc.lustre-OST0001-osc-ffff999b19645800.idle_timeout=debug disable quota as required oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:15:22 (1756674922) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8176] == sanity-flr test 0a: lfs mirror create with -N option == 17:15:23 (1756674923) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:15:27 (1756674927) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:15:29 (1756674929) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:15:30 (1756674930) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:15:36 (1756674936) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:15:37 (1756674937) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 17:15:38 (1756674938) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff999b19645800.stats=clear osc.lustre-OST0001-osc-ffff999b19645800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:15:41 (1756674941) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:15:44 (1756674944) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:15:46 (1756674946) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:15:47 (1756674947) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 17:15:50 (1756674950) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:15:52 (1756674952) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:15:54 (1756674954) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:15:57 (1756674957) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:16:00 (1756674960) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:16:03 (1756674963) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.341219 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235379 s, 44.5 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:16:05 (1756674965) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120597 s, 86.9 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:16:08 (1756674968) Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff999b19645800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff999b19645800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff999b19645800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff999b19645800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (25s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:16:33 (1756674993) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.137326 s, 53.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff999b19645800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff999b19645800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:16:46 (1756675006) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff999b19645800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff999b19645800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff999b19645800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff999b19645800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:17:10 (1756675030) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.866232 s, 60.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 31 17:17 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.217s user 0m0.000s sys 0m0.136s Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff999b19645800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff999b19645800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.231s user 0m0.000s sys 0m0.136s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff999b19645800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff999b19645800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.305s user 0m0.000s sys 0m0.156s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff999b19645800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff999b19645800.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:17:38 (1756675058) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:17:39 (1756675059) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:17:40 (1756675060) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 17:17:41 (1756675061) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 17:17:43 (1756675063) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.153085 s, 82.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:17:47 (1756675067) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147578 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177122 s, 59.2 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:17:50 (1756675070) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:17:53 (1756675073) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0008647 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000807447 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000873801 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000696658 s, 1.5 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 17:17:56 (1756675076) create test files with size 15756616 .. 1ca529243ad8bc6fba4d9aea32a6dc22 - 0d1181d8ecf7b0a172f45eaa9cf8274c - 6d3a72781829dbde35ec0dc8f43957f0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 17:18:02 (1756675082) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:18:05 (1756675085) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:18:07 (1756675087) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334756 s, 31.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x37:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00979464 s, 107 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 17:18:10 (1756675090) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.023867 s, 87.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0873382 s, 48.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0344858 s, 60.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (3s) == sanity-flr test 42: lfs mirror verify ================= 17:18:13 (1756675093) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 17:18:14 (1756675094) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:18:15 (1756675095) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00940074 s, 112 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 17:18:17 (1756675097) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.079679 s, 39.5 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:18:20 (1756675100) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x43:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x43:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:18:23 (1756675103) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0871931 s, 120 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (21s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:18:44 (1756675124) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff999b20e01800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.102746 s, 102 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff999b29b25000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:18:49 (1756675129) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:19:02 (1756675142) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:19:16 (1756675156) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 17:19:18 (1756675158) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.067849 s, 46.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0523201 s, 60.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 3a9a0f864923f390d2ca038e8758a765 - mirror 1 checksum: d101fb58e5dbf187902d870b3502f551 - mirror 2 checksum: 3a9a0f864923f390d2ca038e8758a765 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Aug 31 17:19:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Aug 31 17:19:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3a9a0f864923f390d2ca038e8758a765 - mirror 1 checksum: d101fb58e5dbf187902d870b3502f551 - mirror 2 checksum: d101fb58e5dbf187902d870b3502f551 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Aug 31 17:19:17 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Aug 31 17:19:17 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:19:21 (1756675161) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109917 s, 95.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:19:24 (1756675164) Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 17:19:31 (1756675171) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 17:19:46 (1756675186) targets are mounted 17:19:46 (1756675186) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102581 s, 102 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (30s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:19:54 (1756675194) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0102366 s, 2.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00506804 s, 4.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:19:57 (1756675197) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0123239 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00499863 s, 4.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:20:00 (1756675200) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00228084 s, 7.2 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:20:03 (1756675203) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0126544 s, 1.6 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:20:06 (1756675206) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00338719 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0144084 s, 284 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:20:09 (1756675209) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:20:18 (1756675218) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:20:38 (1756675238) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:20:57 (1756675257) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:21:15 (1756675275) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 17:21:17 (1756675277) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 55815 mirror split pid 55817 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 55815 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 55817 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 17:22:20 (1756675340) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 5340 skipping zero length punch hole skipping zero size read truncating to largest ever: 0x27f54 000003 1756675341.052490 trunc from 00000000 to 0x027f53 (0x27f54 bytes) 000004 1756675341.057019 trunc from 0x01cbe2 to 0x02229e (0x56bd bytes) truncating to largest ever: 0x3875f 000005 1756675341.058702 trunc from 0x027f54 to 0x03875e (0x1080b bytes) 000006 1756675341.062905 punch from 0x018310 to 0x02543c (0xd12d bytes) 000007 1756675341.064650 mapwrite 0x01576e thru 0x01c1d9 (0x6a6c bytes) 000008 1756675341.076833 mapread 0x035fc5 thru 0x03875e (0x279a bytes) 000009 1756675341.079866 punch from 0x00b79d to 0x0104e1 (0x4d45 bytes) 000010 1756675341.081661 punch from 0x01d803 to 0x01f6ac (0x1eaa bytes) 000011 1756675341.083424 read 0x03637c thru 0x03875e (0x23e3 bytes) 000012 1756675341.084079 read 0x02821e thru 0x0351c0 (0xcfa3 bytes) 000013 1756675341.087882 mapread 0x01dc58 thru 0x01ffad (0x2356 bytes) 000014 1756675341.088473 fallocate 0x021fd8 thru 0x026cd7 (0x4d00 bytes) 000016 1756675341.091512 punch from 0x0384eb to 0x03875e (0x274 bytes) 000018 1756675341.093499 write 0x0244a6 thru 0x02fe17 (0xb972 bytes) 000019 1756675341.094336 fallocate 0x0299dc thru 0x02ba35 (0x205a bytes) 000020 1756675341.096367 mapwrite 0x01c5ca thru 0x026134 (0x9b6b bytes) 000021 1756675341.103808 mapread 0x02c9ae thru 0x03875e (0xbdb1 bytes) 000022 1756675341.106296 mapread 0x020c84 thru 0x0291cb (0x8548 bytes) 000023 1756675341.106921 trunc from 0x03875f to 0x0271e7 (0x11577 bytes) 000024 1756675341.111841 read 0x004bcc thru 0x0091f0 (0x4625 bytes) 000025 1756675341.112265 trunc from 0x024c76 to 0x0271e7 (0x2572 bytes) 000026 1756675341.113851 write 0x0373a0 thru 0x03ffff (0x8c60 bytes) 000027 1756675341.114725 mapwrite 0x016ac5 thru 0x020c3a (0xa176 bytes) 000030 1756675341.122394 punch from 0x020950 to 0x02f3ac (0xea5d bytes) 000031 1756675341.124088 trunc from 0x040000 to 0x00cb5a (0x334a5 bytes) 000032 1756675341.128913 mapread 0x000aa9 thru 0x002232 (0x178a bytes) 000033 1756675341.129513 punch from 0x0030a7 to 0x00cb5a (0x9ab4 bytes) 000034 1756675341.131230 mapread 0x0029b8 thru 0x00b21d (0x8866 bytes) 000035 1756675341.134228 mapread 0x000623 thru 0x00121e (0xbfc bytes) 000036 1756675341.134754 trunc from 0x000f39 to 0x00cb5a (0xbc22 bytes) 000037 1756675341.136547 mapwrite 0x0291d1 thru 0x02b2b9 (0x20e9 bytes) 000038 1756675341.148814 mapread 0x00f516 thru 0x01deac (0xe997 bytes) 000041 1756675341.150266 fallocate 0x02c1f5 thru 0x038588 (0xc394 bytes) 000043 1756675341.152899 mapwrite 0x034ecb thru 0x03ffff (0xb135 bytes) 000044 1756675341.167001 punch from 0x02aa1c to 0x034964 (0x9f49 bytes) 000045 1756675341.169045 fallocate 0x008f59 thru 0x01878e (0xf836 bytes) 000046 1756675341.170929 fallocate 0x02ec87 thru 0x034574 (0x58ee bytes) 000047 1756675341.172899 trunc from 0x03642e to 0x039928 (0x34fb bytes) 000048 1756675341.174577 fallocate 0x0229e7 thru 0x029074 (0x668e bytes) 000049 1756675341.176293 fallocate 0x01ef79 thru 0x02b1f8 (0xc280 bytes) 000050 1756675341.178239 read 0x01a4f6 thru 0x01dcb8 (0x37c3 bytes) 000051 1756675341.178673 write 0x03e0df thru 0x03ffff (0x1f21 bytes) 000052 1756675341.179653 write 0x031a87 thru 0x03afec (0x9566 bytes) 000053 1756675341.183264 mapwrite 0x01c68f thru 0x01f788 (0x30fa bytes) 000054 1756675341.189105 read 0x0098f3 thru 0x012089 (0x8797 bytes) 000055 1756675341.189555 read 0x013f5f thru 0x01c2ef (0x8391 bytes) 000056 1756675341.189897 mapread 0x026f43 thru 0x02f84e (0x890c bytes) 000057 1756675341.190502 punch from 0x030df1 to 0x0311d7 (0x3e7 bytes) 000058 1756675341.192149 trunc from 0x035b57 to 0x036039 (0x4e3 bytes) 000059 1756675341.193821 trunc from 0x040000 to 0x027eb1 (0x1814e bytes) 000060 1756675341.198870 write 0x02d9ae thru 0x033a80 (0x60d3 bytes) 000061 1756675341.199719 read 0x00ca10 thru 0x017757 (0xad48 bytes) 000063 1756675341.200470 mapread 0x025613 thru 0x0320d5 (0xcac3 bytes) 000064 1756675341.203372 punch from 0x002317 to 0x0055d1 (0x32bb bytes) 000065 1756675341.205054 mapwrite 0x00eec4 thru 0x016118 (0x7255 bytes) 000066 1756675341.211281 read 0x0197d7 thru 0x0207f5 (0x701f bytes) 000067 1756675341.211656 write 0x027616 thru 0x034189 (0xcb74 bytes) 000068 1756675341.212509 mapwrite 0x03bc85 thru 0x03e516 (0x2892 bytes) 000069 1756675341.224548 fallocate 0x033fa0 thru 0x034e14 (0xe75 bytes) 000070 1756675341.226408 trunc from 0x03e517 to 0x02ca57 (0x11abf bytes) 000071 1756675341.231090 fallocate 0x02a19d thru 0x03289c (0x8700 bytes) 000072 1756675341.232838 read 0x017aa9 thru 0x01f815 (0x7d6d bytes) 000073 1756675341.233251 fallocate 0x020cf9 thru 0x02356f (0x2877 bytes) 000074 1756675341.235004 trunc from 0x02ca58 to 0x01c352 (0x10705 bytes) 000075 1756675341.239492 trunc from 0x01c353 to 0x02f2d6 (0x12f84 bytes) 000076 1756675341.243609 trunc from 0x001170 to 0x010c4a (0xfadb bytes) 000077 1756675341.245537 trunc from 0x0175bc to 0x01e37c (0x6dc1 bytes) 000079 1756675341.247903 trunc from 0x01a985 to 0x029872 (0xeeee bytes) 000080 1756675341.249712 punch from 0x005f0a to 0x013753 (0xd84a bytes) 000081 1756675341.251435 fallocate 0x0123e2 thru 0x019e05 (0x7a24 bytes) 000082 1756675341.254633 MIRROR_EXTEND to 2 mirrors 000083 1756675341.423964 MIRROR_EXTEND to 3 mirrors 000084 1756675341.584093 write 0x03c9fd thru 0x03ffff (0x3603 bytes) 000085 1756675341.593842 trunc from 0x040000 to 0x0198a5 (0x2675a bytes) 000086 1756675341.604044 write 0x032903 thru 0x03ffff (0xd6fd bytes) 000087 1756675341.606622 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000090 1756675341.668254 mapread 0x029cc7 thru 0x037755 (0xda8f bytes) 000091 1756675341.679145 MIRROR_EXTEND to 3 mirrors 000092 1756675341.841350 write 0x01df6f thru 0x02329a (0x532c bytes) 000093 1756675341.857937 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000094 1756675341.916924 fallocate 0x01bdcf thru 0x01e7e8 (0x2a1a bytes) 000095 1756675341.918946 punch from 0x03c132 to 0x03ffff (0x3ece bytes) 000096 1756675341.920788 read 0x001896 thru 0x00ea22 (0xd18d bytes) 000097 1756675341.925122 MIRROR_RESYNC 2 mirrors 000098 1756675342.103063 punch from 0x0254fb to 0x02c228 (0x6d2e bytes) 000099 1756675342.113323 fallocate 0x00d209 thru 0x0188d3 (0xb6cb bytes) 000100 1756675342.115597 trunc from 0x040000 to 0x00b83f (0x347c0 bytes) 000102 1756675342.121968 read 0x00ab9b thru 0x00b83f (0xca5 bytes) 000103 1756675342.124246 read 0x002ab0 thru 0x006f42 (0x4493 bytes) 000104 1756675342.128266 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000105 1756675342.162469 punch from 0x004c97 to 0x00a2aa (0x5614 bytes) 000106 1756675342.164430 write 0x00f7ac thru 0x01a3e9 (0xac3e bytes) 000107 1756675342.168848 MIRROR_EXTEND to 2 mirrors 000108 1756675342.350082 read 0x019561 thru 0x01a3e9 (0xe89 bytes) 000109 1756675342.356156 punch from 0x00b275 to 0x00d073 (0x1dff bytes) 000111 1756675342.374017 punch from 0x014eae to 0x01a3e9 (0x553c bytes) 000113 1756675342.377785 mapread 0x017072 thru 0x01a3e9 (0x3378 bytes) 000114 1756675342.386286 punch from 0x00946f to 0x015f03 (0xca95 bytes) 000115 1756675342.389154 read 0x003ade thru 0x00f164 (0xb687 bytes) 000117 1756675342.395501 punch from 0x0151c5 to 0x01593a (0x776 bytes) 000118 1756675342.398445 punch from 0x017ae9 to 0x01a3e9 (0x2901 bytes) truncating to largest ever: 0x3e119 000119 1756675342.401406 trunc from 0x01a3ea to 0x03e118 (0x23d2f bytes) 000120 1756675342.421690 fallocate 0x02f4e6 thru 0x03571d (0x6238 bytes) 000121 1756675342.427403 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000122 1756675342.487330 MIRROR_EXTEND to 2 mirrors 000123 1756675342.756130 mapwrite 0x01fd2b thru 0x02ad33 (0xb009 bytes) 000124 1756675342.827453 MIRROR_RESYNC 2 mirrors 000125 1756675343.110292 read 0x00f14b thru 0x01e3b2 (0xf268 bytes) 000126 1756675343.118276 MIRROR_RESYNC 2 mirrors 000127 1756675343.214248 MIRROR_RESYNC 2 mirrors 000128 1756675343.295965 trunc from 0x02e3ec to 0x0366eb (0x8300 bytes) 000129 1756675343.308541 MIRROR_EXTEND to 3 mirrors 000130 1756675343.481907 read 0x026d1b thru 0x029d22 (0x3008 bytes) 000131 1756675343.488520 mapread 0x035620 thru 0x03d947 (0x8328 bytes) 000132 1756675343.497747 read 0x00a90c thru 0x01435d (0x9a52 bytes) 000133 1756675343.502904 mapwrite 0x01533c thru 0x01800e (0x2cd3 bytes) 000134 1756675343.535399 write 0x035014 thru 0x03ffff (0xafec bytes) 000135 1756675343.539643 mapread 0x03751a thru 0x03ffff (0x8ae6 bytes) 000136 1756675343.540347 trunc from 0x040000 to 0x02ea4e (0x115b1 bytes) 000138 1756675343.546570 trunc from 0x02cb50 to 0x02ea4e (0x1eff bytes) 000139 1756675343.548667 mapread 0x010d88 thru 0x019354 (0x85cd bytes) 000140 1756675343.559526 MIRROR_RESYNC 3 mirrors 000141 1756675343.772310 fallocate 0x02ba7f thru 0x035ca8 (0xa22a bytes) 000142 1756675343.782131 mapwrite 0x00c794 thru 0x00cc4e (0x4bb bytes) 000143 1756675343.788223 trunc from 0x035ca9 to 0x02c90b (0x939d bytes) 000144 1756675343.799025 read 0x025143 thru 0x02b5e0 (0x649e bytes) fallocating to largest ever: 0x40000 000145 1756675343.801907 fallocate 0x03d942 thru 0x03ffff (0x26be bytes) 000146 1756675343.803660 trunc from 0x0037fc to 0x00d353 (0x9b58 bytes) 000147 1756675343.806965 MIRROR_RESYNC 3 mirrors 000148 1756675344.018208 mapread 0x023ad6 thru 0x02a41b (0x6946 bytes) 000150 1756675344.024869 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000151 1756675344.094141 write 0x0333ca thru 0x0384b0 (0x50e7 bytes) 000153 1756675344.107553 write 0x02ffe6 thru 0x034fe1 (0x4ffc bytes) 000154 1756675344.110606 trunc from 0x040000 to 0x007496 (0x38b69 bytes) 000155 1756675344.121879 punch from 0x00572c to 0x007496 (0x1d6b bytes) 000156 1756675344.123589 punch from 0x005f48 to 0x007496 (0x154f bytes) 000157 1756675344.125178 trunc from 0x007497 to 0x0025df (0x4eb7 bytes) 000158 1756675344.129423 fallocate 0x017316 thru 0x01deae (0x6b99 bytes) 000159 1756675344.133061 MIRROR_RESYNC 2 mirrors 000160 1756675344.261008 trunc from 0x008fa2 to 0x0148eb (0xb94a bytes) 000161 1756675344.268580 fallocate 0x01ce68 thru 0x026556 (0x96ef bytes) 000162 1756675344.270399 fallocate 0x0171e2 thru 0x02403e (0xce5d bytes) 000163 1756675344.272226 trunc from 0x00cf6a to 0x01185e (0x48f5 bytes) 000165 1756675344.274134 trunc from 0x02403f to 0x03f5a7 (0x1b569 bytes) 000166 1756675344.280395 mapread 0x029046 thru 0x02e5c6 (0x5581 bytes) 000168 1756675344.287184 mapwrite 0x013b35 thru 0x01bd5a (0x8226 bytes) 000169 1756675344.297597 mapwrite 0x011e6a thru 0x01a821 (0x89b8 bytes) 000170 1756675344.305585 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000171 1756675344.343389 fallocate 0x0370bb thru 0x03974e (0x2694 bytes) 000172 1756675344.345188 trunc from 0x00aeff to 0x01a85e (0xf960 bytes) 000173 1756675344.346936 mapwrite 0x02ab8d thru 0x02d905 (0x2d79 bytes) 000175 1756675344.356466 fallocate 0x02a297 thru 0x0306d3 (0x643d bytes) 000177 1756675344.358832 mapwrite 0x00da7b thru 0x014bf5 (0x717b bytes) 000178 1756675344.367121 write 0x030638 thru 0x03f6b6 (0xf07f bytes) 000179 1756675344.370602 punch from 0x013746 to 0x01746a (0x3d25 bytes) 000180 1756675344.372304 trunc from 0x03f6b7 to 0x036c01 (0x8ab5 bytes) 000181 1756675344.376535 fallocate 0x03bfad thru 0x03ffff (0x4053 bytes) 000182 1756675344.378360 mapwrite 0x02be30 thru 0x030254 (0x4425 bytes) 000184 1756675344.384617 fallocate 0x037619 thru 0x03ffff (0x89e7 bytes) 000186 1756675344.386837 mapwrite 0x03a5c9 thru 0x03ffff (0x5a37 bytes) 000187 1756675344.394496 write 0x009af0 thru 0x01306f (0x9580 bytes) 000188 1756675344.396584 read 0x0144d4 thru 0x018742 (0x426f bytes) 000189 1756675344.399259 trunc from 0x040000 to 0x00114e (0x3eeb1 bytes) 000190 1756675344.405428 MIRROR_EXTEND to 2 mirrors 000192 1756675344.541062 read 0x00051f thru 0x00114e (0xc30 bytes) 000193 1756675344.545206 punch from 0x0008b5 to 0x000cdd (0x429 bytes) 000194 1756675344.555863 mapwrite 0x035ced thru 0x03c4e6 (0x67fa bytes) 000195 1756675344.579721 punch from 0x00eda1 to 0x019ccb (0xaf2b bytes) 000196 1756675344.581551 write 0x038b8c thru 0x03ffff (0x7474 bytes) 000197 1756675344.582754 trunc from 0x039110 to 0x03d4ca (0x43bb bytes) 000199 1756675344.587456 write 0x001a69 thru 0x0076ac (0x5c44 bytes) 000200 1756675344.590729 mapwrite 0x01dd18 thru 0x02c325 (0xe60e bytes) 000202 1756675344.604510 mapread 0x00aa7d thru 0x01366f (0x8bf3 bytes) 000203 1756675344.607360 write 0x00f4b2 thru 0x0139da (0x4529 bytes) 000204 1756675344.608309 write 0x01b9bf thru 0x0278fa (0xbf3c bytes) 000205 1756675344.609371 write 0x00f6b0 thru 0x010cb0 (0x1601 bytes) 000208 1756675344.610466 trunc from 0x03748b to 0x03d869 (0x63df bytes) 000209 1756675344.614018 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000210 1756675344.654859 fallocate 0x012832 thru 0x018b7e (0x634d bytes) 000211 1756675344.657246 fallocate 0x00aa79 thru 0x01572f (0xacb7 bytes) 000213 1756675344.659973 punch from 0x003360 to 0x005d6b (0x2a0c bytes) 000214 1756675344.661804 read 0x011736 thru 0x01d486 (0xbd51 bytes) 000216 1756675344.664858 read 0x020fa4 thru 0x0280fc (0x7159 bytes) 000217 1756675344.668304 read 0x0007f8 thru 0x0040c3 (0x38cc bytes) 000218 1756675344.668676 fallocate 0x00ecd0 thru 0x013342 (0x4673 bytes) 000219 1756675344.670467 mapwrite 0x036b2f thru 0x03ffff (0x94d1 bytes) 000220 1756675344.677171 mapwrite 0x009ec5 thru 0x00d76b (0x38a7 bytes) 000221 1756675344.682721 trunc from 0x020554 to 0x029a4e (0x94fb bytes) 000222 1756675344.684722 read 0x0013e7 thru 0x00ad9f (0x99b9 bytes) 000223 1756675344.685430 trunc from 0x018496 to 0x01a1c2 (0x1d2d bytes) 000224 1756675344.687244 mapread 0x0101f6 thru 0x014cf9 (0x4b04 bytes) 000225 1756675344.687910 mapwrite 0x023a88 thru 0x025acf (0x2048 bytes) 000226 1756675344.694902 mapwrite 0x00380c thru 0x011ace (0xe2c3 bytes) 000228 1756675344.703003 trunc from 0x040000 to 0x03946b (0x6b94 bytes) 000229 1756675344.709333 mapread 0x0302b7 thru 0x038730 (0x847a bytes) 000230 1756675344.709972 mapwrite 0x020978 thru 0x028919 (0x7fa2 bytes) 000231 1756675344.716260 mapread 0x01654e thru 0x017f06 (0x19b9 bytes) 000232 1756675344.716769 trunc from 0x03946c to 0x00ba07 (0x2da64 bytes) 000233 1756675344.721937 trunc from 0x003f33 to 0x00ba07 (0x7ad5 bytes) 000234 1756675344.723776 trunc from 0x00ba08 to 0x01fa7f (0x14078 bytes) 000235 1756675344.728548 mapwrite 0x039069 thru 0x03ffff (0x6f97 bytes) 000236 1756675344.747939 punch from 0x027d22 to 0x036238 (0xe517 bytes) 000237 1756675344.749767 mapwrite 0x00af94 thru 0x00d873 (0x28e0 bytes) 000238 1756675344.759557 read 0x031929 thru 0x03e1cd (0xc8a5 bytes) 000239 1756675344.763048 mapwrite 0x00c5a0 thru 0x019cf3 (0xd754 bytes) 000240 1756675344.770239 trunc from 0x004189 to 0x004fe0 (0xe58 bytes) 000242 1756675344.772730 fallocate 0x024df0 thru 0x025a2a (0xc3b bytes) 000243 1756675344.774845 mapwrite 0x01d77c thru 0x02bd6a (0xe5ef bytes) 000244 1756675344.781902 trunc from 0x040000 to 0x002777 (0x3d888 bytes) 000245 1756675344.789093 MIRROR_EXTEND to 2 mirrors 000247 1756675344.984052 punch from 0x002651 to 0x002777 (0x127 bytes) 000248 1756675344.999592 trunc from 0x0000dc to 0x002777 (0x269c bytes) 000250 1756675345.001541 read 0x0001bf thru 0x002777 (0x25b9 bytes) 000251 1756675345.003589 punch from 0x000a7b to 0x002777 (0x1cfd bytes) 000252 1756675345.005412 trunc from 0x002778 to 0x01537f (0x12c08 bytes) 000253 1756675345.017373 punch from 0x00cd01 to 0x01537f (0x867f bytes) 000254 1756675345.019641 write 0x009b63 thru 0x0145ad (0xaa4b bytes) 000255 1756675345.025357 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000257 1756675345.071332 read 0x00273c thru 0x00b869 (0x912e bytes) 000258 1756675345.074547 write 0x013cc4 thru 0x01519a (0x14d7 bytes) 000259 1756675345.079440 punch from 0x002b9c to 0x01059b (0xda00 bytes) 000260 1756675345.081545 mapread 0x001881 thru 0x00f878 (0xdff8 bytes) 000261 1756675345.085164 fallocate 0x00f887 thru 0x014277 (0x49f1 bytes) 000262 1756675345.087261 write 0x003099 thru 0x003d50 (0xcb8 bytes) 000263 1756675345.087920 write 0x037368 thru 0x03a9c7 (0x3660 bytes) 000264 1756675345.089072 read 0x029ef3 thru 0x02bae0 (0x1bee bytes) 000265 1756675345.092545 fallocate 0x02ecd2 thru 0x03d2e7 (0xe616 bytes) 000266 1756675345.094755 mapwrite 0x03a20f thru 0x03ffff (0x5df1 bytes) 000267 1756675345.108892 trunc from 0x01c236 to 0x021544 (0x530f bytes) 000268 1756675345.112226 MIRROR_EXTEND to 2 mirrors 000269 1756675345.302656 mapread 0x03a196 thru 0x03ffff (0x5e6a bytes) 000270 1756675345.313950 trunc from 0x035048 to 0x03813a (0x30f3 bytes) 000271 1756675345.326666 trunc from 0x040000 to 0x02415a (0x1bea5 bytes) 000272 1756675345.340706 mapwrite 0x034cce thru 0x03ffff (0xb332 bytes) 000273 1756675345.357629 write 0x0202aa thru 0x026662 (0x63b9 bytes) 000274 1756675345.361090 trunc from 0x032928 to 0x03a40f (0x7ae8 bytes) 000275 1756675345.363282 read 0x022050 thru 0x0248d9 (0x288a bytes) 000276 1756675345.365239 MIRROR_EXTEND to 3 mirrors 000277 1756675345.551745 mapread 0x0044e6 thru 0x00ab2b (0x6646 bytes) 000278 1756675345.560726 punch from 0x03d320 to 0x03ffff (0x2ce0 bytes) 000279 1756675345.571537 trunc from 0x040000 to 0x020d9e (0x1f261 bytes) 000280 1756675345.583815 read 0x0054fa thru 0x006528 (0x102f bytes) 000281 1756675345.586021 trunc from 0x020d9f to 0x00d9f3 (0x133ab bytes) 000282 1756675345.590556 write 0x001756 thru 0x007bfa (0x64a5 bytes) 000283 1756675345.593487 fallocate 0x00380d thru 0x0119bf (0xe1b3 bytes) 000284 1756675345.595428 fallocate 0x0041cb thru 0x00880e (0x4644 bytes) 000286 1756675345.597531 mapwrite 0x034a98 thru 0x03d49f (0x8a08 bytes) 000287 1756675345.614063 MIRROR_EXTEND to 4 mirrors 000288 1756675345.783632 trunc from 0x03d4a0 to 0x03357f (0x9f20 bytes) 000290 1756675345.805414 MIRROR_EXTEND to 5 mirrors 000291 1756675345.970813 trunc from 0x033580 to 0x01d5ac (0x15fd3 bytes) 000293 1756675345.990787 trunc from 0x010d98 to 0x01c9c2 (0xbc2b bytes) 000294 1756675345.992616 fallocate 0x00358c thru 0x01309d (0xfb12 bytes) 000295 1756675345.996246 MIRROR_EXTEND to 6 mirrors 000296 1756675346.172269 read 0x014a41 thru 0x01d5ac (0x8b6c bytes) 000297 1756675346.177620 trunc from 0x0028de to 0x00bc31 (0x9354 bytes) 000298 1756675346.188828 fallocate 0x03fadd thru 0x03ffff (0x523 bytes) 000299 1756675346.191434 trunc from 0x02ec50 to 0x037fa7 (0x9358 bytes) 000301 1756675346.193931 write 0x0183b8 thru 0x0244c1 (0xc10a bytes) 000302 1756675346.199924 write 0x0059dd thru 0x0151c6 (0xf7ea bytes) 000303 1756675346.205007 write 0x008583 thru 0x009d50 (0x17ce bytes) 000305 1756675346.205910 trunc from 0x040000 to 0x03d919 (0x26e6 bytes) 000306 1756675346.212059 punch from 0x03c336 to 0x03d919 (0x15e4 bytes) 000308 1756675346.214197 read 0x027ea9 thru 0x02a3b6 (0x250e bytes) 000309 1756675346.219313 MIRROR_RESYNC 6 mirrors 000310 1756675346.626689 trunc from 0x0198d1 to 0x0206a8 (0x6dd8 bytes) 000311 1756675346.638775 MIRROR_EXTEND to 7 mirrors 000312 1756675346.818536 fallocate 0x02b35f thru 0x0357a4 (0xa446 bytes) 000313 1756675346.830754 mapwrite 0x016b7d thru 0x017910 (0xd94 bytes) 000314 1756675346.837794 mapread 0x033d4f thru 0x03d919 (0x9bcb bytes) 000315 1756675346.841790 trunc from 0x03d91a to 0x00f587 (0x2e392 bytes) 000316 1756675346.854366 write 0x00fd85 thru 0x01c9fa (0xcc76 bytes) 000317 1756675346.859168 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000318 1756675346.919944 read 0x0131b5 thru 0x01a748 (0x7594 bytes) 000319 1756675346.922713 read 0x00a2ce thru 0x016c87 (0xc9ba bytes) 000320 1756675346.925582 trunc from 0x00ca7a to 0x00f4a6 (0x2a2d bytes) 000321 1756675346.927688 punch from 0x01c5b5 to 0x01c9fa (0x446 bytes) 000322 1756675346.931120 MIRROR_EXTEND to 7 mirrors 000323 1756675347.097674 mapread 0x00ebb5 thru 0x0157f4 (0x6c40 bytes) 000324 1756675347.106757 mapread 0x014e46 thru 0x017224 (0x23df bytes) 000325 1756675347.110941 trunc from 0x01c9fb to 0x0141a1 (0x8859 bytes) 000326 1756675347.123683 mapwrite 0x0289e7 thru 0x03207f (0x9699 bytes) 000327 1756675347.138942 trunc from 0x032080 to 0x024646 (0xda39 bytes) 000328 1756675347.143131 fallocate 0x02920a thru 0x0355ae (0xc3a5 bytes) 000329 1756675347.144848 trunc from 0x0200ef to 0x02364a (0x355c bytes) 000330 1756675347.146498 trunc from 0x011f1f to 0x019c2f (0x7d11 bytes) 000332 1756675347.150102 MIRROR_EXTEND to 8 mirrors 000333 1756675347.353151 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000334 1756675347.489143 punch from 0x004b4a to 0x00c53e (0x79f5 bytes) 000336 1756675347.504927 trunc from 0x00cfec to 0x01cb96 (0xfbab bytes) 000337 1756675347.507796 write 0x00e1ec thru 0x017a0f (0x9824 bytes) 000338 1756675347.515969 trunc from 0x0216d2 to 0x024646 (0x2f75 bytes) 000339 1756675347.519088 punch from 0x008cc7 to 0x01765f (0xe999 bytes) 000340 1756675347.529667 MIRROR_RESYNC 7 mirrors 000341 1756675348.094881 mapwrite 0x0036fc thru 0x00623b (0x2b40 bytes) 000342 1756675348.150953 trunc from 0x024647 to 0x01be64 (0x87e2 bytes) 000343 1756675348.162468 read 0x016d7f thru 0x01be64 (0x50e6 bytes) 000344 1756675348.168148 fallocate 0x01aa31 thru 0x024772 (0x9d42 bytes) 000345 1756675348.171919 fallocate 0x01e738 thru 0x0238b5 (0x517e bytes) 000346 1756675348.175892 trunc from 0x00f43e to 0x016bd4 (0x7797 bytes) 000347 1756675348.179525 punch from 0x00139f to 0x00459e (0x3200 bytes) 000349 1756675348.183701 trunc from 0x0238b6 to 0x001b23 (0x21d92 bytes) 000351 1756675348.193889 read 0x0005fd thru 0x001b23 (0x1527 bytes) 000352 1756675348.197818 punch from 0x00030e to 0x001b23 (0x1816 bytes) 000353 1756675348.201298 mapwrite 0x036f97 thru 0x03dd87 (0x6df1 bytes) 000355 1756675348.228932 punch from 0x037661 to 0x03dd87 (0x6727 bytes) 000356 1756675348.232587 trunc from 0x03dd88 to 0x008bed (0x3519a bytes) 000357 1756675348.244626 MIRROR_EXTEND to 8 mirrors 000358 1756675348.447461 trunc from 0x0007c4 to 0x008bed (0x842a bytes) 000359 1756675348.467810 write 0x020025 thru 0x02029b (0x277 bytes) 000360 1756675348.471458 punch from 0x018b8c to 0x01ff15 (0x738a bytes) 000361 1756675348.474376 trunc from 0x02029c to 0x038e38 (0x18b9d bytes) 000362 1756675348.493176 read 0x02cf2a thru 0x038e38 (0xbf0f bytes) 000363 1756675348.498410 write 0x038486 thru 0x03ffff (0x7b7a bytes) 000364 1756675348.499557 trunc from 0x00aecd to 0x017683 (0xc7b7 bytes) 000366 1756675348.503072 mapread 0x02c9f4 thru 0x02ed78 (0x2385 bytes) 000368 1756675348.504771 write 0x035222 thru 0x03ffff (0xadde bytes) 000369 1756675348.508860 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000371 1756675348.617672 write 0x03a29d thru 0x03d0c2 (0x2e26 bytes) 000372 1756675348.627614 MIRROR_RESYNC 7 mirrors 000373 1756675349.204366 mapread 0x039ca8 thru 0x03ffff (0x6358 bytes) 000374 1756675349.215849 trunc from 0x025e99 to 0x035877 (0xf9df bytes) 000375 1756675349.243225 MIRROR_EXTEND to 8 mirrors 000376 1756675349.562523 mapread 0x03efcb thru 0x03ffff (0x1035 bytes) 000377 1756675349.577732 mapwrite 0x03b57f thru 0x03ffff (0x4a81 bytes) 000379 1756675349.644534 write 0x0004ca thru 0x007ae1 (0x7618 bytes) 000380 1756675349.653581 fallocate 0x00dde7 thru 0x0135fa (0x5814 bytes) 000381 1756675349.657620 mapwrite 0x020cae thru 0x02fc30 (0xef83 bytes) 000382 1756675349.681344 trunc from 0x040000 to 0x03c4e1 (0x3b1e bytes) 000383 1756675349.693773 punch from 0x0138ee to 0x01484f (0xf62 bytes) 000384 1756675349.697035 punch from 0x036384 to 0x03c4e1 (0x615e bytes) 000386 1756675349.703748 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000387 1756675349.808887 read 0x02b5f1 thru 0x032d85 (0x7795 bytes) 000388 1756675349.813644 read 0x02252f thru 0x02822f (0x5d01 bytes) 000389 1756675349.819337 read 0x01914a thru 0x0222e1 (0x9198 bytes) 000390 1756675349.820147 mapwrite 0x019e19 thru 0x01bef6 (0x20de bytes) 000391 1756675349.828827 write 0x00f7c1 thru 0x010470 (0xcb0 bytes) 000392 1756675349.832655 read 0x007f44 thru 0x0150c9 (0xd186 bytes) 000393 1756675349.833901 read 0x00941a thru 0x0138dc (0xa4c3 bytes) 000394 1756675349.835143 trunc from 0x03c4e2 to 0x03118c (0xb355 bytes) 000395 1756675349.844491 read 0x01efbe thru 0x025ded (0x6e30 bytes) 000396 1756675349.846492 trunc from 0x03118d to 0x02e510 (0x2c7c bytes) 000397 1756675349.854774 fallocate 0x02f54e thru 0x035585 (0x6038 bytes) 000398 1756675349.858311 mapwrite 0x03ab92 thru 0x03ffff (0x546e bytes) 000399 1756675349.880382 mapread 0x01a0fb thru 0x01b4bf (0x13c5 bytes) 000400 1756675349.881557 mapwrite 0x02aed7 thru 0x0330fe (0x8228 bytes) 000401 1756675349.892685 fallocate 0x032f46 thru 0x03ffff (0xd0ba bytes) 000402 1756675349.895800 trunc from 0x040000 to 0x00e1f8 (0x31e07 bytes) 000403 1756675349.906473 MIRROR_EXTEND to 8 mirrors 000404 1756675350.105554 trunc from 0x00e1f9 to 0x02a727 (0x1c52f bytes) 000407 1756675350.120789 mapread 0x00ce41 thru 0x00e7c3 (0x1983 bytes) 000408 1756675350.127556 trunc from 0x026511 to 0x027f7b (0x1a6b bytes) 000409 1756675350.129526 punch from 0x01216b to 0x01a267 (0x80fd bytes) 000410 1756675350.131760 write 0x02c211 thru 0x02c645 (0x435 bytes) 000411 1756675350.132685 trunc from 0x001068 to 0x00a837 (0x97d0 bytes) 000412 1756675350.135054 mapread 0x026aef thru 0x02c645 (0x5b57 bytes) 000414 1756675350.138962 write 0x00a6b7 thru 0x012ebc (0x8806 bytes) 000416 1756675350.142958 read 0x02b1d9 thru 0x02c645 (0x146d bytes) 000418 1756675350.143885 read 0x0085ef thru 0x017ef3 (0xf905 bytes) 000419 1756675350.148064 trunc from 0x02c646 to 0x027277 (0x53ce bytes) 000420 1756675350.156123 punch from 0x01dd8a to 0x026444 (0x86bb bytes) 000422 1756675350.158932 trunc from 0x017cd9 to 0x0188b4 (0xbdc bytes) 000423 1756675350.161336 trunc from 0x027278 to 0x0286cb (0x1454 bytes) 000425 1756675350.168546 fallocate 0x024897 thru 0x02a985 (0x60ef bytes) 000426 1756675350.171529 mapread 0x01e47b thru 0x02a985 (0xc50b bytes) 000427 1756675350.197930 mapread 0x002ff1 thru 0x00daac (0xaabc bytes) 000428 1756675350.199377 fallocate 0x013a4f thru 0x01f552 (0xbb04 bytes) 000429 1756675350.202178 read 0x009a0a thru 0x011852 (0x7e49 bytes) 000430 1756675350.202605 trunc from 0x003de9 to 0x0116a6 (0xd8be bytes) 000431 1756675350.210684 MIRROR_EXTEND to 9 mirrors 000432 1756675350.450369 trunc from 0x014265 to 0x018ee3 (0x4c7f bytes) 000433 1756675350.467801 fallocate 0x022384 thru 0x02fe70 (0xdaed bytes) 000434 1756675350.470175 write 0x006f33 thru 0x00fee1 (0x8faf bytes) 000435 1756675350.478894 MIRROR_RESYNC 9 mirrors 000437 1756675350.900632 punch from 0x00ba37 to 0x0175c0 (0xbb8a bytes) 000438 1756675350.914204 MIRROR_EXTEND to 10 mirrors 000439 1756675351.124591 punch from 0x01dfc4 to 0x02a985 (0xc9c2 bytes) 000441 1756675351.137553 mapwrite 0x020c58 thru 0x02b068 (0xa411 bytes) 000442 1756675351.160271 read 0x007712 thru 0x00c9c3 (0x52b2 bytes) 000443 1756675351.163172 fallocate 0x036b12 thru 0x03b9ad (0x4e9c bytes) 000444 1756675351.165257 mapwrite 0x030ad3 thru 0x03e3d7 (0xd905 bytes) 000445 1756675351.185691 read 0x012988 thru 0x01e2d4 (0xb94d bytes) 000446 1756675351.191629 MIRROR_EXTEND to 11 mirrors 000447 1756675351.370798 punch from 0x03d4bb to 0x03e3d7 (0xf1d bytes) 000448 1756675351.384252 fallocate 0x03dc6e thru 0x03ffff (0x2392 bytes) 000450 1756675351.386683 mapwrite 0x0131bd thru 0x01f69f (0xc4e3 bytes) 000451 1756675351.404604 trunc from 0x03e3d8 to 0x03bca4 (0x2733 bytes) 000452 1756675351.418420 mapwrite 0x03cb47 thru 0x03df6a (0x1424 bytes) 000454 1756675351.431904 trunc from 0x03df6b to 0x0050fe (0x38e6c bytes) 000455 1756675351.437725 trunc from 0x00457a to 0x0050fe (0xb85 bytes) 000456 1756675351.441165 MIRROR_EXTEND to 12 mirrors 000457 1756675351.623686 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000458 1756675351.734278 trunc from 0x0050ff to 0x031519 (0x2c41b bytes) 000459 1756675351.757720 read 0x025fa6 thru 0x031519 (0xb574 bytes) 000460 1756675351.761681 read 0x023013 thru 0x0309f0 (0xd9de bytes) 000462 1756675351.766155 punch from 0x0152ec to 0x01ea5c (0x9771 bytes) 000463 1756675351.768667 write 0x01aa97 thru 0x01ddb8 (0x3322 bytes) 000464 1756675351.774256 MIRROR_RESYNC 11 mirrors 000465 1756675352.528234 trunc from 0x00def4 to 0x0199d1 (0xbade bytes) 000466 1756675352.551739 mapread 0x020542 thru 0x027f20 (0x79df bytes) 000467 1756675352.563219 read 0x02c9a2 thru 0x031519 (0x4b78 bytes) 000468 1756675352.568583 write 0x002360 thru 0x00f827 (0xd4c8 bytes) 000469 1756675352.575706 write 0x00bea1 thru 0x00c4bc (0x61c bytes) 000470 1756675352.577081 trunc from 0x01d740 to 0x024e99 (0x775a bytes) 000471 1756675352.583920 MIRROR_EXTEND to 12 mirrors 000472 1756675352.806939 fallocate 0x029177 thru 0x032fc6 (0x9e50 bytes) 000473 1756675352.818771 punch from 0x01657c to 0x01a0c4 (0x3b49 bytes) 000474 1756675352.820616 trunc from 0x03151a to 0x017c45 (0x198d4 bytes) 000475 1756675352.832298 write 0x006390 thru 0x00a1e7 (0x3e58 bytes) 000476 1756675352.835407 fallocate 0x03c3d3 thru 0x03db23 (0x1751 bytes) 000477 1756675352.837281 mapread 0x00e499 thru 0x010269 (0x1dd1 bytes) 000478 1756675352.840188 trunc from 0x017c46 to 0x01b6b5 (0x3a70 bytes) 000479 1756675352.844802 trunc from 0x01b6b6 to 0x01f509 (0x3e54 bytes) 000480 1756675352.849191 mapwrite 0x031a17 thru 0x03d85f (0xbe49 bytes) 000481 1756675352.865890 write 0x000774 thru 0x00438e (0x3c1b bytes) 000482 1756675352.871064 MIRROR_EXTEND to 13 mirrors 000483 1756675353.052290 mapwrite 0x01a75b thru 0x01e879 (0x411f bytes) 000484 1756675353.090881 mapread 0x018927 thru 0x025463 (0xcb3d bytes) 000485 1756675353.095524 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000487 1756675353.165589 write 0x028eb7 thru 0x029cf9 (0xe43 bytes) 000489 1756675353.173036 MIRROR_RESYNC 12 mirrors 000490 1756675354.094334 write 0x017457 thru 0x024ce0 (0xd88a bytes) 000491 1756675354.108277 fallocate 0x03c40b thru 0x03ffff (0x3bf5 bytes) 000492 1756675354.112042 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000493 1756675354.174827 MIRROR_EXTEND to 12 mirrors 000494 1756675354.435857 write 0x0192d0 thru 0x01edeb (0x5b1c bytes) 000495 1756675354.461831 read 0x010f8d thru 0x01e82c (0xd8a0 bytes) 000496 1756675354.465829 trunc from 0x040000 to 0x03a994 (0x566b bytes) 000497 1756675354.474860 write 0x021eb4 thru 0x02d062 (0xb1af bytes) 000498 1756675354.481331 write 0x022ffc thru 0x032f36 (0xff3b bytes) 000499 1756675354.484226 mapwrite 0x02c1dc thru 0x03154b (0x5370 bytes) 000500 1756675354.492890 mapwrite 0x03f8cb thru 0x03ffff (0x735 bytes) 000501 1756675354.508461 mapread 0x021ae3 thru 0x0298fa (0x7e18 bytes) 000502 1756675354.509392 mapwrite 0x015ae8 thru 0x018956 (0x2e6f bytes) 000503 1756675354.516791 trunc from 0x040000 to 0x021085 (0x1ef7a bytes) 000504 1756675354.523771 punch from 0x002d67 to 0x005eaf (0x3149 bytes) 000505 1756675354.526358 trunc from 0x021086 to 0x00ce2e (0x14257 bytes) 000506 1756675354.533301 write 0x00b099 thru 0x012f07 (0x7e6f bytes) 000507 1756675354.538349 trunc from 0x0024e0 to 0x002aeb (0x60c bytes) 000508 1756675354.543478 MIRROR_EXTEND to 13 mirrors 000509 1756675354.776499 mapread 0x00e487 thru 0x012f07 (0x4a81 bytes) 000510 1756675354.787764 MIRROR_RESYNC 13 mirrors 000511 1756675355.259554 mapread 0x010ac3 thru 0x012f07 (0x2445 bytes) 000512 1756675355.268263 punch from 0x002146 to 0x00a6d4 (0x858f bytes) 000513 1756675355.285304 trunc from 0x006f2d to 0x00c23e (0x5312 bytes) 000514 1756675355.288244 trunc from 0x012f08 to 0x0074d9 (0xba2e bytes) 000515 1756675355.305540 mapwrite 0x030d8a thru 0x03970b (0x8982 bytes) 000516 1756675355.333206 MIRROR_EXTEND to 14 mirrors 000517 1756675355.548818 punch from 0x0157d2 to 0x018888 (0x30b7 bytes) 000518 1756675355.564492 read 0x005e25 thru 0x00eb1e (0x8cfa bytes) 000519 1756675355.567793 read 0x01803a thru 0x027c69 (0xfc30 bytes) 000520 1756675355.571444 read 0x01e03c thru 0x022f03 (0x4ec8 bytes) 000521 1756675355.571839 mapwrite 0x01b223 thru 0x01e7ca (0x35a8 bytes) 000522 1756675355.576971 mapwrite 0x038fd3 thru 0x03e74f (0x577d bytes) 000523 1756675355.595722 trunc from 0x03121c to 0x03523b (0x4020 bytes) 000524 1756675355.598961 MIRROR_RESYNC 14 mirrors 000525 1756675356.331609 trunc from 0x031744 to 0x03da65 (0xc322 bytes) 000526 1756675356.344665 trunc from 0x03e750 to 0x038222 (0x652d bytes) 000527 1756675356.353415 trunc from 0x038223 to 0x029922 (0xe900 bytes) 000528 1756675356.359871 read 0x00d76b thru 0x01d15b (0xf9f1 bytes) 000529 1756675356.364768 write 0x0030a4 thru 0x003624 (0x581 bytes) 000530 1756675356.367847 trunc from 0x029923 to 0x02fc53 (0x6331 bytes) 000531 1756675356.374871 trunc from 0x00f08b to 0x0141a7 (0x511d bytes) 000532 1756675356.377716 trunc from 0x02fc54 to 0x010f6c (0x1ece7 bytes) 000533 1756675356.384673 mapwrite 0x0380ed thru 0x038930 (0x844 bytes) 000535 1756675356.399497 punch from 0x002488 to 0x00d7a2 (0xb31b bytes) 000536 1756675356.401759 read 0x00a2ef thru 0x00b5c0 (0x12d2 bytes) 000538 1756675356.407137 mapwrite 0x016646 thru 0x016e17 (0x7d2 bytes) 000540 1756675356.414565 punch from 0x03260e to 0x038930 (0x6323 bytes) 000542 1756675356.417676 write 0x003a00 thru 0x00b5df (0x7be0 bytes) 000543 1756675356.418795 trunc from 0x038931 to 0x033633 (0x52fd bytes) 000544 1756675356.425775 trunc from 0x033634 to 0x01c3de (0x17255 bytes) 000545 1756675356.432663 mapread 0x00db6a thru 0x010a57 (0x2eee bytes) 000546 1756675356.433383 trunc from 0x01c3df to 0x0004fa (0x1bee4 bytes) 000547 1756675356.441209 write 0x003478 thru 0x005b33 (0x26bc bytes) 000548 1756675356.442602 read 0x00438a thru 0x005b33 (0x17aa bytes) 000549 1756675356.443407 trunc from 0x004867 to 0x005b33 (0x12cd bytes) 000552 1756675356.453478 MIRROR_EXTEND to 15 mirrors 000553 1756675356.721933 punch from 0x003cd8 to 0x005b33 (0x1e5c bytes) 000554 1756675356.749622 MIRROR_SPLIT mirror 8 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000555 1756675356.839341 write 0x02e990 thru 0x03b651 (0xccc2 bytes) 000556 1756675356.842212 trunc from 0x03b652 to 0x011eab (0x297a6 bytes) 000557 1756675356.853371 read 0x00db4c thru 0x011eab (0x4360 bytes) 000558 1756675356.856517 fallocate 0x012e49 thru 0x0156ea (0x28a2 bytes) 000559 1756675356.859226 trunc from 0x0156eb to 0x01b50c (0x5e22 bytes) 000560 1756675356.865661 write 0x02e741 thru 0x031413 (0x2cd3 bytes) 000562 1756675356.867529 fallocate 0x01f3cd thru 0x02c389 (0xcfbd bytes) 000563 1756675356.872047 MIRROR_RESYNC 14 mirrors 000564 1756675357.928548 trunc from 0x031414 to 0x019871 (0x17ba2 bytes) 000565 1756675357.954812 trunc from 0x019872 to 0x03b057 (0x217e6 bytes) 000566 1756675357.964871 trunc from 0x03b058 to 0x0203fe (0x1ac59 bytes) 000567 1756675357.972784 fallocate 0x039505 thru 0x03bf82 (0x2a7e bytes) 000568 1756675357.979295 MIRROR_RESYNC 14 mirrors 000569 1756675359.043411 write 0x00a9f1 thru 0x01a3d4 (0xf9e4 bytes) 000571 1756675359.062568 read 0x01d3ff thru 0x0203fe (0x3000 bytes) 000572 1756675359.064857 trunc from 0x008ec1 to 0x010b32 (0x7c72 bytes) 000573 1756675359.069620 write 0x03638a thru 0x03721c (0xe93 bytes) 000574 1756675359.070262 trunc from 0x03721d to 0x00309d (0x3417f bytes) 000575 1756675359.078130 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000576 1756675359.137496 mapread 0x001f9e thru 0x00309d (0x1100 bytes) 000577 1756675359.143252 MIRROR_EXTEND to 14 mirrors 000578 1756675359.292852 read 0x0001f5 thru 0x00309d (0x2ea9 bytes) 000579 1756675359.297807 MIRROR_EXTEND to 15 mirrors 000580 1756675359.442771 trunc from 0x00309e to 0x00239e (0xcff bytes) 000581 1756675359.464087 mapwrite 0x003b63 thru 0x00bf55 (0x83f3 bytes) 000582 1756675359.480085 mapwrite 0x006787 thru 0x012766 (0xbfe0 bytes) 000585 1756675359.493114 mapread 0x00f089 thru 0x012766 (0x36de bytes) 000586 1756675359.493601 write 0x0332c8 thru 0x037e5a (0x4b93 bytes) 000588 1756675359.494591 mapread 0x02497a thru 0x025c83 (0x130a bytes) 000590 1756675359.497390 read 0x01d1e5 thru 0x0236cc (0x64e8 bytes) 000591 1756675359.497759 mapwrite 0x025192 thru 0x02bea1 (0x6d10 bytes) 000592 1756675359.503501 mapread 0x0279a0 thru 0x0281d9 (0x83a bytes) 000593 1756675359.505400 MIRROR_EXTEND to 16 mirrors 000595 1756675359.675216 fallocate 0x0386e7 thru 0x03ffff (0x7919 bytes) 000596 1756675359.689874 trunc from 0x008f46 to 0x0148b7 (0xb972 bytes) 000597 1756675359.691451 trunc from 0x037e5b to 0x00b944 (0x2c516 bytes) 000598 1756675359.703379 trunc from 0x00728c to 0x00b944 (0x46b9 bytes) 000599 1756675359.705444 punch from 0x0015ec to 0x00b944 (0xa359 bytes) 000600 1756675359.707449 punch from 0x00b5dd to 0x00b944 (0x368 bytes) 000601 1756675359.709162 punch from 0x004983 to 0x00b944 (0x6fc2 bytes) 000602 1756675359.710851 fallocate 0x029122 thru 0x0350a4 (0xbf83 bytes) 000603 1756675359.712872 trunc from 0x0350a5 to 0x029089 (0xc01b bytes) 000604 1756675359.717242 mapread 0x012e22 thru 0x019253 (0x6432 bytes) 000605 1756675359.723405 mapwrite 0x021010 thru 0x025a99 (0x4a8a bytes) 000606 1756675359.731348 trunc from 0x02908a to 0x036cfc (0xdc73 bytes) 000607 1756675359.736118 write 0x029fa1 thru 0x02a372 (0x3d2 bytes) 000608 1756675359.739124 trunc from 0x036cfd to 0x00fe88 (0x26e74 bytes) 000611 1756675359.744860 mapwrite 0x02c585 thru 0x034ab6 (0x8532 bytes) 000613 1756675359.759307 punch from 0x00aba8 to 0x012d27 (0x8180 bytes) 000614 1756675359.761038 trunc from 0x034ab7 to 0x00cdff (0x27cb7 bytes) 000615 1756675359.766691 mapread 0x000815 thru 0x007a43 (0x722f bytes) 000616 1756675359.770448 punch from 0x00a389 to 0x00cdff (0x2a77 bytes) 000618 1756675359.772868 mapwrite 0x003f81 thru 0x0093e0 (0x5460 bytes) 000619 1756675359.779889 fallocate 0x00f32b thru 0x016020 (0x6cf6 bytes) 000620 1756675359.782289 fallocate 0x00d7d9 thru 0x017c83 (0xa4ab bytes) 000621 1756675359.784625 trunc from 0x017c84 to 0x03aff4 (0x23371 bytes) 000622 1756675359.792000 MIRROR_RESYNC 16 mirrors 000623 1756675360.518878 mapread 0x008347 thru 0x00907d (0xd37 bytes) 000624 1756675360.523494 write 0x02f60d thru 0x03c3d7 (0xcdcb bytes) 000626 1756675360.549170 trunc from 0x017e61 to 0x01af77 (0x3117 bytes) 000627 1756675360.552500 read 0x018507 thru 0x01beae (0x39a8 bytes) 000628 1756675360.556444 mapwrite 0x00559b thru 0x01501f (0xfa85 bytes) 000629 1756675360.582458 write 0x03fa37 thru 0x03ffff (0x5c9 bytes) 000631 1756675360.584139 trunc from 0x0044f0 to 0x00ba38 (0x7549 bytes) 000632 1756675360.587606 write 0x008f98 thru 0x00ac7f (0x1ce8 bytes) 000633 1756675360.593206 write 0x02dc13 thru 0x037cd2 (0xa0c0 bytes) 000634 1756675360.594620 mapwrite 0x03e654 thru 0x03ffff (0x19ac bytes) 000635 1756675360.608728 trunc from 0x040000 to 0x0372c9 (0x8d36 bytes) 000636 1756675360.620538 trunc from 0x0372ca to 0x011f70 (0x25359 bytes) 000637 1756675360.629638 trunc from 0x00fbe9 to 0x0116d0 (0x1ae8 bytes) 000638 1756675360.633143 mapread 0x00fd63 thru 0x011f70 (0x220e bytes) 000639 1756675360.637366 punch from 0x00bd64 to 0x011f70 (0x620d bytes) 000640 1756675360.640898 write 0x007ae6 thru 0x0099ad (0x1ec8 bytes) 000641 1756675360.642304 mapwrite 0x015465 thru 0x022ef4 (0xda90 bytes) 000642 1756675360.673704 punch from 0x013eac to 0x01f8f8 (0xba4d bytes) 000643 1756675360.679829 MIRROR_RESYNC 16 mirrors 000645 1756675361.738958 trunc from 0x01d06a to 0x022ef4 (0x5e8b bytes) 000646 1756675361.749088 punch from 0x01e152 to 0x022ef4 (0x4da3 bytes) 000647 1756675361.750578 mapwrite 0x03123e thru 0x03ed49 (0xdb0c bytes) 000649 1756675361.768683 mapwrite 0x035edc thru 0x0390cb (0x31f0 bytes) 000650 1756675361.774627 MIRROR_RESYNC 16 mirrors 000651 1756675362.871609 punch from 0x034cc8 to 0x03ed49 (0xa082 bytes) 000652 1756675362.883712 fallocate 0x00e952 thru 0x010cd6 (0x2385 bytes) 000653 1756675362.885663 mapwrite 0x02d9df thru 0x038d91 (0xb3b3 bytes) 000654 1756675362.900848 read 0x03eb0f thru 0x03ed49 (0x23b bytes) 000655 1756675362.901398 mapwrite 0x0017f5 thru 0x010679 (0xee85 bytes) 000656 1756675362.914100 mapwrite 0x01f6f6 thru 0x022793 (0x309e bytes) 000657 1756675362.921507 mapwrite 0x006bd2 thru 0x01524d (0xe67c bytes) 000658 1756675362.929548 trunc from 0x030c43 to 0x037d63 (0x7121 bytes) 000659 1756675362.931351 trunc from 0x024f44 to 0x02bd78 (0x6e35 bytes) 000661 1756675362.933649 mapread 0x0154eb thru 0x022bdf (0xd6f5 bytes) 000662 1756675362.934267 fallocate 0x02d04e thru 0x03137f (0x4332 bytes) 000663 1756675362.936109 write 0x00506b thru 0x014597 (0xf52d bytes) 000664 1756675362.936957 trunc from 0x03ed4a to 0x0027a8 (0x3c5a1 bytes) 000665 1756675362.943941 mapwrite 0x021c6d thru 0x022d76 (0x110a bytes) 000666 1756675362.955290 trunc from 0x022d77 to 0x007ccf (0x1b0a7 bytes) 000667 1756675362.960133 trunc from 0x007cd0 to 0x008f29 (0x125a bytes) 000668 1756675362.964610 fallocate 0x01903e thru 0x027b5e (0xeb21 bytes) 000669 1756675362.966488 write 0x03c487 thru 0x03ffff (0x3b79 bytes) 000671 1756675362.967759 mapwrite 0x00d552 thru 0x017767 (0xa216 bytes) 000672 1756675362.979567 mapread 0x017a09 thru 0x020167 (0x875f bytes) 000673 1756675362.982307 fallocate 0x015b0b thru 0x019dd9 (0x42cf bytes) 000674 1756675362.984146 punch from 0x027933 to 0x037809 (0xfed7 bytes) 000675 1756675362.985846 write 0x02bc93 thru 0x0340cb (0x8439 bytes) 000677 1756675362.990777 mapread 0x01fe5b thru 0x022644 (0x27ea bytes) 000678 1756675362.991338 write 0x0210cd thru 0x021ded (0xd21 bytes) 000679 1756675362.991839 write 0x011316 thru 0x01ec36 (0xd921 bytes) 000680 1756675362.992681 mapwrite 0x01455d thru 0x023d9c (0xf840 bytes) 000681 1756675362.999588 write 0x03564a thru 0x03f869 (0xa220 bytes) 000682 1756675363.002415 mapwrite 0x002d42 thru 0x009fce (0x728d bytes) 000683 1756675363.010087 write 0x03e95d thru 0x03ffff (0x16a3 bytes) 000684 1756675363.010664 trunc from 0x040000 to 0x0349db (0xb624 bytes) 000685 1756675363.015710 write 0x003a9f thru 0x00b075 (0x75d7 bytes) 000686 1756675363.016429 punch from 0x025453 to 0x026bfc (0x17aa bytes) 000687 1756675363.017968 trunc from 0x0349dc to 0x01d083 (0x17958 bytes) 000689 1756675363.023269 mapwrite 0x039c6c thru 0x03c9c7 (0x2d5c bytes) 000690 1756675363.036448 read 0x01d3bf thru 0x01ee53 (0x1a95 bytes) 000691 1756675363.041080 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000692 1756675363.105510 mapwrite 0x00ed70 thru 0x0133bd (0x464e bytes) 000693 1756675363.116301 mapread 0x02a782 thru 0x033107 (0x8986 bytes) 000694 1756675363.121308 fallocate 0x022f2c thru 0x029f9f (0x7074 bytes) 000695 1756675363.124782 MIRROR_EXTEND to 16 mirrors 000696 1756675363.291452 mapwrite 0x0032ea thru 0x008779 (0x5490 bytes) 000697 1756675363.333245 mapwrite 0x01fe43 thru 0x02d1d5 (0xd393 bytes) 000698 1756675363.345245 punch from 0x00b682 to 0x015783 (0xa102 bytes) 000699 1756675363.347177 trunc from 0x00695a to 0x00813a (0x17e1 bytes) 000700 1756675363.349011 mapread 0x034e34 thru 0x03c9c7 (0x7b94 bytes) 000701 1756675363.351948 mapwrite 0x01b002 thru 0x02187d (0x687c bytes) 000702 1756675363.358280 trunc from 0x002666 to 0x00b755 (0x90f0 bytes) 000704 1756675363.362075 write 0x00485a thru 0x012346 (0xdaed bytes) 000705 1756675363.368403 read 0x01aaa2 thru 0x0228bb (0x7e1a bytes) 000708 1756675363.369361 read 0x000442 thru 0x00eb29 (0xe6e8 bytes) 000710 1756675363.372216 punch from 0x008145 to 0x00c090 (0x3f4c bytes) 000711 1756675363.376652 trunc from 0x03c9c8 to 0x0358dd (0x70ea bytes) 000712 1756675363.387612 mapread 0x02cfe0 thru 0x032237 (0x5258 bytes) 000713 1756675363.388204 punch from 0x01373e to 0x016e2c (0x36ef bytes) 000715 1756675363.390286 trunc from 0x0358de to 0x012a17 (0x22ec6 bytes) 000716 1756675363.395050 mapwrite 0x02149b thru 0x02beee (0xaa54 bytes) 000718 1756675363.423576 read 0x01b162 thru 0x023b19 (0x89b8 bytes) 000719 1756675363.428225 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000720 1756675363.489817 mapread 0x01c8e0 thru 0x025e65 (0x9586 bytes) 000721 1756675363.495540 fallocate 0x000676 thru 0x004380 (0x3d0b bytes) 000722 1756675363.497396 write 0x024627 thru 0x030486 (0xbe60 bytes) 000725 1756675363.500229 trunc from 0x0119f3 to 0x0188f7 (0x6f05 bytes) 000726 1756675363.502246 mapwrite 0x025a6e thru 0x02a736 (0x4cc9 bytes) 000727 1756675363.507857 trunc from 0x00d764 to 0x0192f2 (0xbb8f bytes) 000728 1756675363.509619 trunc from 0x030487 to 0x006560 (0x29f26 bytes) 000729 1756675363.514494 write 0x03d9fb thru 0x03ffff (0x2605 bytes) 000730 1756675363.515188 punch from 0x00774e to 0x007754 (0x7 bytes) 000731 1756675363.518454 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000732 1756675363.577405 read 0x03224e thru 0x03ffff (0xddb2 bytes) 000733 1756675363.580425 mapwrite 0x0071f9 thru 0x015b64 (0xe96c bytes) 000735 1756675363.595227 read 0x02059e thru 0x02db48 (0xd5ab bytes) 000736 1756675363.598553 write 0x030d3f thru 0x0382e8 (0x75aa bytes) 000737 1756675363.601030 punch from 0x011eef to 0x0208e3 (0xe9f5 bytes) 000738 1756675363.604939 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000739 1756675363.682745 trunc from 0x00bf6f to 0x016153 (0xa1e5 bytes) 000740 1756675363.685350 write 0x019e44 thru 0x026a03 (0xcbc0 bytes) 000741 1756675363.692064 read 0x0133cc thru 0x01d187 (0x9dbc bytes) 000742 1756675363.695770 read 0x03c8b2 thru 0x03ffff (0x374e bytes) 000743 1756675363.702170 MIRROR_RESYNC 13 mirrors 000744 1756675364.932209 fallocate 0x005e8e thru 0x00cc8f (0x6e02 bytes) 000746 1756675364.946487 write 0x03d211 thru 0x03ffff (0x2def bytes) 000747 1756675364.948495 trunc from 0x0352a2 to 0x03732c (0x208b bytes) 000748 1756675364.950144 write 0x0201f3 thru 0x025296 (0x50a4 bytes) 000749 1756675364.953830 mapwrite 0x015f20 thru 0x023043 (0xd124 bytes) 000750 1756675364.964662 trunc from 0x040000 to 0x02ae21 (0x151de bytes) 000751 1756675364.970605 trunc from 0x020e6d to 0x029afa (0x8c8e bytes) 000752 1756675364.972237 trunc from 0x0199a2 to 0x022ac0 (0x911f bytes) 000754 1756675364.974660 mapread 0x029a09 thru 0x02ae21 (0x1419 bytes) 000755 1756675364.979451 MIRROR_RESYNC 13 mirrors 000756 1756675365.660289 punch from 0x029b39 to 0x02ae21 (0x12e9 bytes) 000757 1756675365.671596 mapread 0x00d713 thru 0x01cc98 (0xf586 bytes) 000758 1756675365.679627 mapread 0x0096c2 thru 0x013a06 (0xa345 bytes) 000759 1756675365.681706 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000760 1756675365.738049 mapwrite 0x00b67f thru 0x00d8c6 (0x2248 bytes) 000761 1756675365.747646 trunc from 0x02ae22 to 0x01917d (0x11ca4 bytes) 000762 1756675365.753335 mapread 0x00fcb2 thru 0x012806 (0x2b55 bytes) 000763 1756675365.753845 trunc from 0x01917e to 0x017be8 (0x1595 bytes) 000764 1756675365.758171 mapread 0x0145dc thru 0x017be8 (0x360d bytes) 000766 1756675365.760484 fallocate 0x024616 thru 0x0316eb (0xd0d6 bytes) 000767 1756675365.762224 mapread 0x011ca2 thru 0x017be8 (0x5f47 bytes) 000768 1756675365.762724 trunc from 0x017be9 to 0x03e25b (0x26673 bytes) 000769 1756675365.768412 MIRROR_RESYNC 12 mirrors 000770 1756675366.321243 read 0x01bd3c thru 0x01d421 (0x16e6 bytes) 000771 1756675366.323124 punch from 0x027926 to 0x037893 (0xff6e bytes) 000773 1756675366.334286 read 0x00e232 thru 0x0172c0 (0x908f bytes) 000774 1756675366.337549 mapwrite 0x011d2f thru 0x01e240 (0xc512 bytes) 000775 1756675366.344811 mapread 0x00ae1b thru 0x00c50d (0x16f3 bytes) 000776 1756675366.345450 mapread 0x01593b thru 0x01f8a4 (0x9f6a bytes) 000777 1756675366.346110 fallocate 0x01061c thru 0x019dfd (0x97e2 bytes) 000778 1756675366.348060 punch from 0x02a7df to 0x030607 (0x5e29 bytes) 000779 1756675366.351487 MIRROR_EXTEND to 13 mirrors 000780 1756675366.520747 mapwrite 0x03e671 thru 0x03ffff (0x198f bytes) 000781 1756675366.548652 mapread 0x00031e thru 0x00caed (0xc7d0 bytes) 000782 1756675366.551713 trunc from 0x040000 to 0x02c2ed (0x13d12 bytes) 000783 1756675366.557577 trunc from 0x015aab to 0x022f89 (0xd4df bytes) 000784 1756675366.559296 read 0x003219 thru 0x00ceec (0x9cd4 bytes) 000785 1756675366.559666 read 0x01e193 thru 0x02c2ed (0xe15b bytes) 000786 1756675366.562719 mapwrite 0x027034 thru 0x035727 (0xe6f4 bytes) 000788 1756675366.576517 trunc from 0x02cd4a to 0x03399c (0x6c53 bytes) 000789 1756675366.578179 mapread 0x01f3e4 thru 0x02157d (0x219a bytes) 000790 1756675366.580160 MIRROR_RESYNC 13 mirrors 000791 1756675367.309742 trunc from 0x013f4d to 0x01e0aa (0xa15e bytes) 000792 1756675367.320846 mapwrite 0x00cce2 thru 0x01a4e5 (0xd804 bytes) 000793 1756675367.335786 trunc from 0x018894 to 0x01a536 (0x1ca3 bytes) 000794 1756675367.337633 trunc from 0x01514d to 0x01b6c3 (0x6577 bytes) 000795 1756675367.339611 punch from 0x0243ce to 0x032f59 (0xeb8c bytes) 000796 1756675367.341480 mapwrite 0x01b2e5 thru 0x0273f1 (0xc10d bytes) 000797 1756675367.352998 read 0x0182e1 thru 0x01920c (0xf2c bytes) 000798 1756675367.353519 trunc from 0x035728 to 0x0090a9 (0x2c67e bytes) 000799 1756675367.359688 trunc from 0x0090aa to 0x00caf1 (0x3a48 bytes) 000800 1756675367.364361 read 0x00c669 thru 0x00caf1 (0x489 bytes) 000801 1756675367.368686 MIRROR_EXTEND to 14 mirrors 000802 1756675367.527818 write 0x00bc03 thru 0x00f3a2 (0x37a0 bytes) 000804 1756675367.541951 write 0x03bb9e thru 0x03ffff (0x4462 bytes) 000805 1756675367.542728 write 0x0125f6 thru 0x020d76 (0xe781 bytes) 000806 1756675367.546251 punch from 0x00007b to 0x00f319 (0xf29f bytes) 000807 1756675367.550760 trunc from 0x040000 to 0x018014 (0x27feb bytes) 000808 1756675367.561225 write 0x03ecd9 thru 0x03ffff (0x1327 bytes) 000809 1756675367.561957 mapread 0x031fe7 thru 0x03bbb4 (0x9bce bytes) 000810 1756675367.566136 write 0x00ba7a thru 0x01a165 (0xe6ec bytes) 000812 1756675367.570513 mapread 0x02ac43 thru 0x030b8e (0x5f4c bytes) 000813 1756675367.571106 trunc from 0x040000 to 0x014420 (0x2bbdf bytes) 000814 1756675367.576814 mapread 0x003e16 thru 0x00d286 (0x9471 bytes) 000815 1756675367.579534 fallocate 0x00eaf4 thru 0x013e32 (0x533f bytes) 000816 1756675367.581678 mapwrite 0x03faf0 thru 0x03ffff (0x510 bytes) 000818 1756675367.593605 trunc from 0x012ddf to 0x0196e9 (0x690b bytes) 000819 1756675367.595287 trunc from 0x019ff0 to 0x0286b1 (0xe6c2 bytes) 000820 1756675367.597186 mapread 0x01edd6 thru 0x028661 (0x988c bytes) 000821 1756675367.602470 mapwrite 0x023563 thru 0x02fff2 (0xca90 bytes) 000822 1756675367.610789 MIRROR_EXTEND to 15 mirrors 000823 1756675367.779815 write 0x02740f thru 0x02eeec (0x7ade bytes) 000826 1756675367.796184 MIRROR_RESYNC 15 mirrors 000827 1756675368.512967 trunc from 0x00d8ce to 0x01a79d (0xced0 bytes) 000828 1756675368.525871 trunc from 0x040000 to 0x0061e3 (0x39e1c bytes) 000829 1756675368.531666 mapwrite 0x02a3d3 thru 0x0306c2 (0x62f0 bytes) 000830 1756675368.547332 mapwrite 0x03da8b thru 0x03efa2 (0x1518 bytes) 000831 1756675368.558417 read 0x01e146 thru 0x024754 (0x660f bytes) 000832 1756675368.561153 read 0x00b66d thru 0x019fb0 (0xe944 bytes) 000833 1756675368.564157 trunc from 0x03efa3 to 0x02959c (0x15a06 bytes) 000835 1756675368.569078 read 0x024b5f thru 0x02959c (0x4a3e bytes) 000836 1756675368.569427 write 0x002cd5 thru 0x005acf (0x2dfb bytes) 000837 1756675368.570099 mapread 0x01db0b thru 0x02609f (0x8595 bytes) 000838 1756675368.570676 mapread 0x026dc0 thru 0x028a1e (0x1c5f bytes) 000839 1756675368.571243 mapread 0x004750 thru 0x00c623 (0x7ed4 bytes) 000840 1756675368.572152 trunc from 0x022f5e to 0x02855c (0x55ff bytes) 000841 1756675368.575659 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000842 1756675368.636958 mapwrite 0x02398d thru 0x025641 (0x1cb5 bytes) 000843 1756675368.649081 MIRROR_EXTEND to 15 mirrors 000844 1756675368.829741 MIRROR_RESYNC 15 mirrors 000845 1756675369.657328 mapread 0x0165ce thru 0x023fcc (0xd9ff bytes) 000846 1756675369.665495 mapread 0x027a08 thru 0x02959c (0x1b95 bytes) 000847 1756675369.666169 punch from 0x01561d to 0x02056f (0xaf53 bytes) 000848 1756675369.685114 MIRROR_RESYNC 15 mirrors 000849 1756675370.503569 punch from 0x0017b8 to 0x006edb (0x5724 bytes) 000850 1756675370.513595 punch from 0x0086eb to 0x016db3 (0xe6c9 bytes) 000852 1756675370.515287 trunc from 0x02959d to 0x01bfe7 (0xd5b5 bytes) 000853 1756675370.520442 write 0x00bf95 thru 0x0185a8 (0xc614 bytes) 000854 1756675370.523645 mapwrite 0x02b5fe thru 0x02fb9a (0x459d bytes) 000855 1756675370.535799 mapwrite 0x03cd29 thru 0x03ffff (0x32d7 bytes) 000856 1756675370.547225 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000857 1756675370.604207 mapread 0x00db10 thru 0x00f412 (0x1903 bytes) 000858 1756675370.609285 punch from 0x03136f to 0x033859 (0x24eb bytes) 000859 1756675370.610823 read 0x018ae1 thru 0x01ec7a (0x619a bytes) 000860 1756675370.613229 mapread 0x00c523 thru 0x017961 (0xb43f bytes) 000861 1756675370.613880 mapwrite 0x03bdf7 thru 0x03ffff (0x4209 bytes) 000862 1756675370.621561 trunc from 0x040000 to 0x016b0f (0x294f0 bytes) 000863 1756675370.627493 trunc from 0x016b10 to 0x017c6b (0x115c bytes) 000864 1756675370.631619 write 0x00a1f2 thru 0x00bc06 (0x1a15 bytes) 000865 1756675370.632238 trunc from 0x017c6c to 0x0153f2 (0x2879 bytes) 000866 1756675370.636850 mapread 0x005b3f thru 0x0151f6 (0xf6b8 bytes) 000867 1756675370.640168 mapwrite 0x035c7f thru 0x0385cb (0x294d bytes) 000868 1756675370.652103 mapread 0x02aac7 thru 0x02dba0 (0x30da bytes) 000869 1756675370.654614 write 0x0294ae thru 0x03501a (0xbb6d bytes) 000870 1756675370.655511 mapread 0x0370ce thru 0x0385cb (0x14fe bytes) 000871 1756675370.657523 MIRROR_RESYNC 14 mirrors 000872 1756675371.164447 write 0x017611 thru 0x01b0eb (0x3adb bytes) 000873 1756675371.177291 write 0x035b80 thru 0x039b95 (0x4016 bytes) 000874 1756675371.180186 mapread 0x038a50 thru 0x039b95 (0x1146 bytes) 000875 1756675371.180772 mapwrite 0x00062b thru 0x0008db (0x2b1 bytes) 000877 1756675371.188465 read 0x004df3 thru 0x008f62 (0x4170 bytes) 000878 1756675371.188998 fallocate 0x01567b thru 0x02513b (0xfac1 bytes) 000879 1756675371.190771 write 0x02b55c thru 0x02ecb7 (0x375c bytes) 000880 1756675371.193300 write 0x00e9a7 thru 0x016d20 (0x837a bytes) 000881 1756675371.195149 punch from 0x02448c to 0x028ace (0x4643 bytes) 000882 1756675371.198233 MIRROR_EXTEND to 15 mirrors 000883 1756675371.357503 punch from 0x003652 to 0x01297f (0xf32e bytes) 000885 1756675371.371181 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000887 1756675371.425401 punch from 0x019ed1 to 0x01f973 (0x5aa3 bytes) 000888 1756675371.427198 trunc from 0x00d66e to 0x01c646 (0xefd9 bytes) 000889 1756675371.429024 mapwrite 0x0317ee thru 0x03d6b9 (0xbecc bytes) 000890 1756675371.451372 punch from 0x036747 to 0x03d530 (0x6dea bytes) 000891 1756675371.453191 fallocate 0x03e1f4 thru 0x03f013 (0xe20 bytes) 000892 1756675371.455010 fallocate 0x02f663 thru 0x038ae9 (0x9487 bytes) 000893 1756675371.456812 mapwrite 0x02f5ed thru 0x03500f (0x5a23 bytes) 000894 1756675371.462281 mapwrite 0x00d57c thru 0x00ed4f (0x17d4 bytes) 000895 1756675371.468364 trunc from 0x00bbc9 to 0x016d38 (0xb170 bytes) 000896 1756675371.470051 mapread 0x02186f thru 0x0261fc (0x498e bytes) 000897 1756675371.472810 mapwrite 0x03b968 thru 0x03ffff (0x4698 bytes) 000898 1756675371.484466 read 0x0046a7 thru 0x009db7 (0x5711 bytes) 000899 1756675371.486389 mapwrite 0x013d29 thru 0x01bb5b (0x7e33 bytes) 000900 1756675371.494606 trunc from 0x040000 to 0x031e74 (0xe18b bytes) 000901 1756675371.500394 MIRROR_RESYNC 14 mirrors 000902 1756675372.115409 trunc from 0x031e75 to 0x017e23 (0x1a051 bytes) 000903 1756675372.141028 write 0x02f6b8 thru 0x03ccaa (0xd5f3 bytes) 000904 1756675372.142786 mapwrite 0x00ec65 thru 0x014d93 (0x612f bytes) 000905 1756675372.156754 fallocate 0x02fa94 thru 0x0378fc (0x7e69 bytes) 000906 1756675372.159320 read 0x026000 thru 0x0260a8 (0xa9 bytes) 000907 1756675372.161675 mapwrite 0x01a6f2 thru 0x01d34d (0x2c5c bytes) 000908 1756675372.169996 trunc from 0x01740e to 0x02587c (0xe46f bytes) 000909 1756675372.172026 mapwrite 0x006874 thru 0x009650 (0x2ddd bytes) 000910 1756675372.179369 mapread 0x030c29 thru 0x03ccaa (0xc082 bytes) 000911 1756675372.181646 MIRROR_RESYNC 14 mirrors 000912 1756675372.971035 punch from 0x01b510 to 0x02236c (0x6e5d bytes) 000913 1756675372.984236 punch from 0x03593d to 0x03ccaa (0x736e bytes) 000914 1756675372.985973 punch from 0x03b5b5 to 0x03ccaa (0x16f6 bytes) 000916 1756675372.987798 read 0x035104 thru 0x03ccaa (0x7ba7 bytes) 000917 1756675372.990919 punch from 0x02874e to 0x0308f2 (0x81a5 bytes) 000918 1756675372.994141 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000919 1756675373.052414 mapwrite 0x03600e thru 0x0387f4 (0x27e7 bytes) 000920 1756675373.062601 mapread 0x00c539 thru 0x012deb (0x68b3 bytes) 000922 1756675373.065737 mapread 0x005ab3 thru 0x0091c6 (0x3714 bytes) 000923 1756675373.067778 write 0x00fee4 thru 0x01d000 (0xd11d bytes) 000924 1756675373.073101 trunc from 0x03ccab to 0x01ba2b (0x2127f bytes) 000925 1756675373.082543 write 0x0106a5 thru 0x017e52 (0x77ae bytes) 000926 1756675373.083232 trunc from 0x00346e to 0x003908 (0x49b bytes) 000927 1756675373.085071 write 0x00e949 thru 0x01a85f (0xbf17 bytes) 000928 1756675373.085900 mapwrite 0x02ec98 thru 0x032e6d (0x41d6 bytes) 000929 1756675373.098790 trunc from 0x032e6e to 0x020c68 (0x12205 bytes) 000930 1756675373.103628 mapread 0x013d49 thru 0x01c758 (0x8a10 bytes) 000931 1756675373.106635 mapread 0x01e40b thru 0x020c68 (0x285e bytes) 000932 1756675373.107202 trunc from 0x008dbb to 0x00ec95 (0x5edb bytes) 000933 1756675373.110801 MIRROR_EXTEND to 14 mirrors 000934 1756675373.358396 write 0x0154e4 thru 0x0182bd (0x2dda bytes) 000935 1756675373.387859 write 0x01e4c9 thru 0x01fbed (0x1725 bytes) 000936 1756675373.391805 trunc from 0x01f26f to 0x020c68 (0x19fa bytes) 000937 1756675373.400223 MIRROR_RESYNC 14 mirrors 000938 1756675374.272680 mapwrite 0x03b0e5 thru 0x03ffff (0x4f1b bytes) 000939 1756675374.305191 MIRROR_EXTEND to 15 mirrors 000940 1756675374.584936 MIRROR_EXTEND to 16 mirrors 000942 1756675374.824983 MIRROR_RESYNC 16 mirrors 000943 1756675375.922245 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x55f:0x0]: Invalid argument (22) 000944 1756675376.080067 write 0x024724 thru 0x030fe8 (0xc8c5 bytes) 000945 1756675376.094288 mapwrite 0x020fb9 thru 0x02717b (0x61c3 bytes) 000946 1756675376.101881 write 0x0200fc thru 0x024198 (0x409d bytes) 000947 1756675376.102483 read 0x00bc0b thru 0x016894 (0xac8a bytes) 000948 1756675376.104919 trunc from 0x02fcaa to 0x038118 (0x846f bytes) 000950 1756675376.108424 MIRROR_EXTEND to 16 mirrors 000951 1756675376.296195 punch from 0x021b83 to 0x02dbf4 (0xc072 bytes) 000952 1756675376.309371 mapread 0x02752c thru 0x031bc4 (0xa699 bytes) 000953 1756675376.315152 mapread 0x01c36f thru 0x02a639 (0xe2cb bytes) 000955 1756675376.319246 punch from 0x014bfd to 0x01f6c0 (0xaac4 bytes) 000956 1756675376.321287 trunc from 0x040000 to 0x030a5f (0xf5a0 bytes) 000957 1756675376.332430 punch from 0x02ac3a to 0x030a5f (0x5e26 bytes) 000958 1756675376.335661 MIRROR_RESYNC 16 mirrors 000960 1756675377.924539 trunc from 0x030a60 to 0x009933 (0x2712c bytes) 000961 1756675377.947780 punch from 0x009819 to 0x009933 (0x11b bytes) 000962 1756675377.951106 punch from 0x0043ef to 0x00653f (0x2151 bytes) 000965 1756675377.956424 write 0x02bde3 thru 0x03304f (0x726d bytes) 000966 1756675377.958473 trunc from 0x000f0a to 0x00ba74 (0xab6b bytes) 000969 1756675377.963554 fallocate 0x01b1d4 thru 0x02b16e (0xff9b bytes) 000971 1756675377.968086 fallocate 0x030822 thru 0x03abaf (0xa38e bytes) 000972 1756675377.972035 fallocate 0x009fec thru 0x017e4b (0xde60 bytes) 000973 1756675377.975860 trunc from 0x03abb0 to 0x00bac5 (0x2f0ea bytes) 000974 1756675377.987967 punch from 0x003274 to 0x0073c6 (0x4153 bytes) 000975 1756675377.991340 mapwrite 0x0397cd thru 0x03ffff (0x6833 bytes) 000976 1756675378.021463 read 0x011cdb thru 0x01c227 (0xa54d bytes) 000977 1756675378.027096 write 0x00a124 thru 0x0151f1 (0xb0ce bytes) 000979 1756675378.033815 mapread 0x008f2a thru 0x018093 (0xf16a bytes) 000980 1756675378.035130 punch from 0x01a335 to 0x025a01 (0xb6cd bytes) 000981 1756675378.038469 trunc from 0x040000 to 0x0125c9 (0x2da36 bytes) 000982 1756675378.047585 trunc from 0x0125ca to 0x0124c5 (0x104 bytes) 000984 1756675378.056775 write 0x022d51 thru 0x027cb6 (0x4f66 bytes) 000985 1756675378.058460 read 0x01840f thru 0x026090 (0xdc82 bytes) 000986 1756675378.064142 punch from 0x0224b3 to 0x027cb6 (0x5804 bytes) 000988 1756675378.072659 trunc from 0x001aab to 0x005ea8 (0x43fe bytes) 000989 1756675378.076109 trunc from 0x022a8a to 0x023f83 (0x14fa bytes) 000990 1756675378.079668 fallocate 0x015d9b thru 0x023e9c (0xe102 bytes) 000991 1756675378.083506 write 0x00e0f2 thru 0x017c79 (0x9b88 bytes) 000992 1756675378.085223 mapwrite 0x03a6b1 thru 0x03cda1 (0x26f1 bytes) 000993 1756675378.109121 mapwrite 0x00e2b0 thru 0x01d0e2 (0xee33 bytes) 000994 1756675378.123372 trunc from 0x0104fa to 0x01e210 (0xdd17 bytes) 000995 1756675378.126883 fallocate 0x0074e9 thru 0x008ec5 (0x19dd bytes) 000996 1756675378.130255 fallocate 0x0017c1 thru 0x00c98f (0xb1cf bytes) 000997 1756675378.133834 trunc from 0x03cda2 to 0x024dfb (0x17fa6 bytes) 000998 1756675378.142446 mapread 0x00bc09 thru 0x01a8b8 (0xecb0 bytes) 000999 1756675378.171802 read 0x00d1e2 thru 0x017f1d (0xad3c bytes) 001000 1756675378.172553 mapread 0x0107d8 thru 0x010cd4 (0x4fd bytes) All operations completed A-OK! PASS 70a (40s) == sanity-flr test 71: check mirror extend parent fid ==== 17:23:00 (1756675380) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x5a9:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 17:23:03 (1756675383) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2744576 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3572064 .. done Extending file size to 5440288 .. Extending file size to 3764000 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8822368 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5843712 .. Extending file size to 7555552 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7595776 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9104224 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9219584 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9231488 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9255744 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8981280 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 61791 61792 61794 61796 61799 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) 192.168.201.147@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:24:10 (1756675450) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1076288 .. Extending file size to 3576928 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8703936 .. Extending file size to 6054016 .. Extending file size to 8110112 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8952640 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8997952 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9159168 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9258080 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 65493 65494 65496 65498 65500 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) 192.168.201.147@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:25:14 (1756675514) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2704 1284984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2976 1284712 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14628 3592392 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23060 3583960 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37688 7176352 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2704 1284984 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2976 1284712 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14628 3592392 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23060 3583960 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37688 7176352 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0420573 s, 49.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0587439 s, 35.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:25:20 (1756675520) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0349403 s, 60.0 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 17:25:22 (1756675522) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:25:24 (1756675524) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0309448 s, 67.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:25:27 (1756675527) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0261531 s, 80.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x806:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x807:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x806:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x806:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x807:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x806:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x807:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:25:29 (1756675529) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0343808 s, 61.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204c (23s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:25:52 (1756675552) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153121 s, 68.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x808:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x809:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x808:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x808:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x809:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x808:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x809:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:26:25 (1756675585) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.045601 s, 46.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (34s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:26:59 (1756675619) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg147-server: Pool lustre.test_204f created Adding targets to pool oleg147-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg147-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0452609 s, 46.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg147-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg147-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg147-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (50s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:27:49 (1756675669) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:27:52 (1756675672) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:27:54 (1756675674) oleg147-server: Pool lustre.test_206 created Adding targets to pool oleg147-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg147-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg147-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg147-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg147-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:28:09 (1756675689) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255326 s, 41.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105258 s, 99.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x832e8af Mirror 3: 0x832e8af PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:28:12 (1756675692) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:28:13 (1756675693) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:28:14 (1756675694) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0197396 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x7b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x23f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x23f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x23f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1630000 Read this (4 bytes): BBBB 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:28:30 (1756675710) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00395845 s, 2.1 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:28:33 (1756675713) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262114 s, 40.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7d lmm_fid: [0x200000405:0x7d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7d lmm_fid: [0x200000405:0x7d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611261 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-08-31 17:28:34.000000000 -0400 Modify: 2025-08-31 17:28:34.000000000 -0400 Change: 2025-08-31 17:28:34.000000000 -0400 Birth: 2025-08-31 17:28:33.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:28:36 (1756675716) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138362 s, 75.8 MB/s oleg147-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg147-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg147-server: O/280000401/d26/ba: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:28:50 (1756675730) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0252975 s, 1.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00512878 s, 799 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 31 17:28 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x82:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 31 17:28 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 828 sec ============ 17:28:53 (1756675733) === sanity-flr: start cleanup 17:28:53 (1756675733) === === sanity-flr: finish cleanup 17:28:55 (1756675735) ===