-----============= acceptance-small: sanity-flr ============----- Thu Sep 25 14:08:58 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg230-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:09:05 (1758823745) === oleg230-client.virtnet: executing check_config_client /mnt/lustre oleg230-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg230-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e5c11cad000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5c11cad000.idle_timeout=debug disable quota as required oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:09:17 (1758823757) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8189] == sanity-flr test 0a: lfs mirror create with -N option == 14:09:17 (1758823757) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:09:22 (1758823762) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:09:23 (1758823763) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:09:24 (1758823764) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:09:29 (1758823769) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:09:31 (1758823771) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 14:09:32 (1758823772) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e5c11cad000.stats=clear osc.lustre-OST0001-osc-ffff9e5c11cad000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:09:34 (1758823774) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:09:37 (1758823777) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:09:39 (1758823779) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:09:40 (1758823780) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 14:09:43 (1758823783) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:09:45 (1758823785) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:09:48 (1758823788) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:09:50 (1758823790) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:09:53 (1758823793) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:09:55 (1758823795) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249922 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202064 s, 51.9 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:09:58 (1758823798) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0431661 s, 97.2 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:10:00 (1758823800) Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5c11cad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5c11cad000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5c11cad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5c11cad000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:10:26 (1758823826) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.188477 s, 61.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5c11cad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5c11cad000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:10:39 (1758823839) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5c11cad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5c11cad000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5c11cad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5c11cad000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (28s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:11:07 (1758823867) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.30961 s, 40.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 25 14:11 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.419s user 0m0.000s sys 0m0.147s Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5c11cad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5c11cad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.497s user 0m0.000s sys 0m0.141s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5c11cad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5c11cad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.165s user 0m0.002s sys 0m0.177s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e5c11cad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5c11cad000.ost_server_uuid in FULL state after 0 sec PASS 33b (56s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:12:03 (1758823923) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:12:05 (1758823925) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:12:06 (1758823926) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 14:12:08 (1758823928) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 14:12:10 (1758823930) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.214941 s, 58.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:12:15 (1758823935) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109642 s, 95.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110573 s, 94.8 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:12:18 (1758823938) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:12:21 (1758823941) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000971215 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00141775 s, 722 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000582923 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000704307 s, 1.5 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 14:12:25 (1758823945) create test files with size 15745774 .. 96decb0adaa334e4e3cbae9ca20bebf7 - c7f483639f37391512432c5d2bb7d403 - 7703838293a8f9ad0c7fbae04495dbd0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 14:12:33 (1758823953) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:12:38 (1758823958) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:12:41 (1758823961) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398661 s, 26.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155494 s, 67.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 14:12:45 (1758823965) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.064383 s, 32.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.184503 s, 22.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0609877 s, 34.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 14:12:51 (1758823971) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 14:12:52 (1758823972) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:12:54 (1758823974) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156856 s, 66.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 14:12:57 (1758823977) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.103923 s, 30.3 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:13:01 (1758823981) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x48:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:13:05 (1758823985) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.114971 s, 91.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 ** after mirror split, file size=10485760 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:13:20 (1758824000) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9e5c10c9e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.123603 s, 84.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9e5c18b64800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:13:25 (1758824005) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (15s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:13:40 (1758824020) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:13:55 (1758824035) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 14:13:57 (1758824037) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.112217 s, 28.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0790265 s, 39.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 29012f8a77d3d67e1e997317a2b9ee06 - mirror 1 checksum: c22beb46ba0bf3ce96b01073dbf8d6a0 - mirror 2 checksum: 29012f8a77d3d67e1e997317a2b9ee06 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Sep 25 14:13:58 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Sep 25 14:13:58 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 29012f8a77d3d67e1e997317a2b9ee06 - mirror 1 checksum: c22beb46ba0bf3ce96b01073dbf8d6a0 - mirror 2 checksum: c22beb46ba0bf3ce96b01073dbf8d6a0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Sep 25 14:13:58 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x50:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Sep 25 14:13:58 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:14:01 (1758824041) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148123 s, 70.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:14:05 (1758824045) Failing mds1 on oleg230-server Stopping /mnt/lustre-mds1 (opts:) on oleg230-server 14:14:13 (1758824053) shut down facet: mds1 facet_host: oleg230-server facet_failover_host: oleg230-server Failover mds1 to oleg230-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 14:14:29 (1758824069) targets are mounted 14:14:29 (1758824069) facet_failover done oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181841 s, 57.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 50A (33s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:14:38 (1758824078) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0272795 s, 751 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00826208 s, 2.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:14:43 (1758824083) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0206558 s, 991 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00560073 s, 3.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:14:48 (1758824088) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00529996 s, 3.1 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:14:51 (1758824091) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0276505 s, 741 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:14:55 (1758824095) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00313925 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109773 s, 373 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:14:58 (1758824098) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:15:08 (1758824108) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:15:29 (1758824129) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:15:47 (1758824147) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:16:05 (1758824165) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 14:16:08 (1758824168) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 55826 mirror split pid 55827 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 55826 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 55827 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 14:17:12 (1758824232) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 4232 000004 1758824232.923417 write 0x030b70 thru 0x03c5f7 (0xba88 bytes) 000005 1758824232.927869 mapread 0x0172f0 thru 0x01ed74 (0x7a85 bytes) 000006 1758824232.937402 write 0x028983 thru 0x02bf1a (0x3598 bytes) 000007 1758824232.943144 read 0x034c28 thru 0x03aac1 (0x5e9a bytes) truncating to largest ever: 0x2b028 000008 1758824232.943754 trunc from 0x03c5f8 to 0x02b027 (0x115d0 bytes) 000009 1758824232.953326 mapwrite 0x001fb6 thru 0x00c80e (0xa859 bytes) 000010 1758824232.977135 mapread 0x01c308 thru 0x01d72c (0x1425 bytes) 000012 1758824232.978398 mapread 0x01ab1d thru 0x02694e (0xbe32 bytes) 000013 1758824232.982812 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x40000 000014 1758824233.249104 fallocate 0x0346b9 thru 0x03ffff (0xb947 bytes) 000015 1758824233.266223 mapwrite 0x030e60 thru 0x036747 (0x58e8 bytes) 000017 1758824233.297137 write 0x0365ab thru 0x03dc66 (0x76bc bytes) 000018 1758824233.298381 trunc from 0x02afe3 to 0x02b0bc (0xda bytes) 000019 1758824233.302648 punch from 0x00965c to 0x00b978 (0x231d bytes) 000020 1758824233.305976 trunc from 0x03dc67 to 0x03191f (0xc347 bytes) 000021 1758824233.315576 write 0x0124b2 thru 0x01f38f (0xcede bytes) 000022 1758824233.323527 write 0x011321 thru 0x01a6c9 (0x93a9 bytes) 000023 1758824233.324718 trunc from 0x031920 to 0x03364d (0x1d2e bytes) 000024 1758824233.336102 mapread 0x01f867 thru 0x022369 (0x2b03 bytes) 000027 1758824233.344315 MIRROR_SPLIT mirror 1 to 1 mirrors 000029 1758824233.401049 read 0x006f5e thru 0x010360 (0x9403 bytes) 000030 1758824233.406424 punch from 0x009c8c to 0x011048 (0x73bd bytes) 000031 1758824233.409604 write 0x00670a thru 0x015239 (0xeb30 bytes) 000032 1758824233.415549 mapread 0x022af1 thru 0x03051a (0xda2a bytes) 000033 1758824233.428340 fallocate 0x0375be thru 0x03775b (0x19e bytes) 000034 1758824233.433370 trunc from 0x01574c to 0x01dad5 (0x838a bytes) 000035 1758824233.441047 mapread 0x0237f3 thru 0x02fcdb (0xc4e9 bytes) 000036 1758824233.441920 write 0x023e02 thru 0x02f138 (0xb337 bytes) 000037 1758824233.442944 write 0x0282ff thru 0x02bdf3 (0x3af5 bytes) 000038 1758824233.443713 mapread 0x032b62 thru 0x03775b (0x4bfa bytes) 000042 1758824233.449303 punch from 0x02baa7 to 0x036de3 (0xb33d bytes) 000043 1758824233.457158 trunc from 0x0239c3 to 0x028734 (0x4d72 bytes) 000044 1758824233.461426 trunc from 0x03775c to 0x03ceef (0x5794 bytes) 000045 1758824233.471275 MIRROR_EXTEND to 2 mirrors 000046 1758824233.796730 mapwrite 0x00d0f2 thru 0x016348 (0x9257 bytes) 000048 1758824233.844223 mapread 0x013922 thru 0x0195c7 (0x5ca6 bytes) 000049 1758824233.851522 MIRROR_EXTEND to 3 mirrors 000050 1758824234.185995 fallocate 0x038d6f thru 0x03ffff (0x7291 bytes) 000052 1758824234.214349 write 0x002b5d thru 0x01114f (0xe5f3 bytes) 000053 1758824234.229479 write 0x00a58e thru 0x011a5f (0x74d2 bytes) 000054 1758824234.230624 mapread 0x02d213 thru 0x034f6b (0x7d59 bytes) 000055 1758824234.237524 trunc from 0x040000 to 0x020c7e (0x1f381 bytes) 000056 1758824234.254966 write 0x01f5dd thru 0x02a4dd (0xaf01 bytes) 000057 1758824234.266546 trunc from 0x00aaa5 to 0x012645 (0x7ba1 bytes) 000058 1758824234.282378 fallocate 0x02513d thru 0x0347b4 (0xf678 bytes) 000059 1758824234.286625 punch from 0x01ba07 to 0x01fcb1 (0x42ab bytes) 000060 1758824234.291661 mapwrite 0x005fdf thru 0x0127e3 (0xc805 bytes) 000061 1758824234.319144 trunc from 0x0347b5 to 0x005680 (0x2f134 bytes) 000062 1758824234.328723 punch from 0x005593 to 0x005680 (0xee bytes) 000063 1758824234.332067 trunc from 0x005681 to 0x02929b (0x23c1b bytes) 000064 1758824234.342996 trunc from 0x02929c to 0x00538e (0x23f0d bytes) 000065 1758824234.358354 MIRROR_RESYNC 3 mirrors 000066 1758824234.649412 read 0x00447f thru 0x00538e (0xf10 bytes) 000067 1758824234.653103 read 0x003931 thru 0x00538e (0x1a5e bytes) 000068 1758824234.656485 mapwrite 0x000444 thru 0x005efb (0x5ab8 bytes) 000069 1758824234.712722 mapread 0x002c79 thru 0x005efb (0x3283 bytes) 000070 1758824234.716506 MIRROR_EXTEND to 4 mirrors 000071 1758824234.957527 punch from 0x0000b7 to 0x001391 (0x12db bytes) 000072 1758824234.978895 fallocate 0x01f48b thru 0x02b6b0 (0xc226 bytes) 000073 1758824234.982076 trunc from 0x000816 to 0x005efb (0x56e6 bytes) 000074 1758824234.985518 read 0x0047b1 thru 0x005efb (0x174b bytes) 000075 1758824234.989664 trunc from 0x005efc to 0x0096ea (0x37ef bytes) 000076 1758824235.013747 punch from 0x0002e5 to 0x0096ea (0x9406 bytes) 000077 1758824235.017619 trunc from 0x0070c4 to 0x0096ea (0x2627 bytes) 000078 1758824235.022914 write 0x02f95c thru 0x039b19 (0xa1be bytes) 000079 1758824235.024316 trunc from 0x0267a0 to 0x02af1f (0x4780 bytes) 000080 1758824235.032579 mapread 0x02cc47 thru 0x032223 (0x55dd bytes) 000081 1758824235.049322 write 0x010045 thru 0x01138c (0x1348 bytes) 000082 1758824235.055553 fallocate 0x015149 thru 0x020990 (0xb848 bytes) 000083 1758824235.059693 read 0x03561c thru 0x039b19 (0x44fe bytes) 000084 1758824235.060274 fallocate 0x0154c4 thru 0x01faed (0xa62a bytes) 000085 1758824235.064121 mapread 0x00f080 thru 0x013f74 (0x4ef5 bytes) 000086 1758824235.064950 trunc from 0x01dd70 to 0x022fc3 (0x5254 bytes) 000087 1758824235.068048 trunc from 0x039b1a to 0x02a375 (0xf7a4 bytes) 000088 1758824235.077007 write 0x031af1 thru 0x03f606 (0xdb16 bytes) 000089 1758824235.078725 write 0x036ac0 thru 0x03c598 (0x5ad9 bytes) 000090 1758824235.079722 fallocate 0x01fe94 thru 0x02acf3 (0xae60 bytes) 000091 1758824235.084078 trunc from 0x03f607 to 0x029d9f (0x15867 bytes) 000092 1758824235.096156 read 0x008dfe thru 0x00b9a9 (0x2bac bytes) 000093 1758824235.106337 MIRROR_RESYNC 4 mirrors 000095 1758824235.524548 MIRROR_EXTEND to 5 mirrors 000096 1758824235.909226 trunc from 0x015bef to 0x019531 (0x3943 bytes) 000097 1758824235.936706 fallocate 0x031254 thru 0x038c85 (0x7a32 bytes) 000098 1758824235.943330 trunc from 0x00f9b4 to 0x01bd27 (0xc374 bytes) 000099 1758824235.949071 read 0x03086b thru 0x035747 (0x4edd bytes) 000100 1758824235.962064 fallocate 0x030770 thru 0x03b0ae (0xa93f bytes) 000101 1758824235.970228 read 0x034167 thru 0x038c85 (0x4b1f bytes) 000102 1758824235.978707 trunc from 0x038c86 to 0x01df9a (0x1aceb bytes) 000103 1758824236.010677 write 0x0049be thru 0x010b8f (0xc1d2 bytes) 000105 1758824236.012430 mapread 0x01588e thru 0x01df9a (0x870d bytes) 000107 1758824236.016943 MIRROR_EXTEND to 6 mirrors 000108 1758824236.290638 mapread 0x01ab04 thru 0x01df9a (0x3497 bytes) 000110 1758824236.305981 MIRROR_EXTEND to 7 mirrors 000111 1758824236.653175 punch from 0x008d55 to 0x017978 (0xec24 bytes) 000112 1758824236.687071 mapwrite 0x021d86 thru 0x0314d8 (0xf753 bytes) 000114 1758824236.774920 write 0x0100ef thru 0x012534 (0x2446 bytes) 000115 1758824236.783774 fallocate 0x02c754 thru 0x0392a2 (0xcb4f bytes) 000116 1758824236.789084 read 0x03587d thru 0x037604 (0x1d88 bytes) 000117 1758824236.794314 punch from 0x02c7be to 0x033c91 (0x74d4 bytes) 000118 1758824236.800996 write 0x01cdd5 thru 0x024ec1 (0x80ed bytes) 000120 1758824236.802796 write 0x03d044 thru 0x03ffff (0x2fbc bytes) 000122 1758824236.805538 trunc from 0x040000 to 0x01bae1 (0x2451e bytes) 000123 1758824236.820780 read 0x00cdfc thru 0x017de3 (0xafe8 bytes) 000124 1758824236.824570 MIRROR_RESYNC 7 mirrors 000125 1758824237.637247 trunc from 0x007fe2 to 0x00b24c (0x326b bytes) 000126 1758824237.656802 trunc from 0x01bae2 to 0x00a48b (0x11656 bytes) 000127 1758824237.665912 mapread 0x006ea7 thru 0x00a13b (0x3295 bytes) 000128 1758824237.676902 read 0x004d5b thru 0x009ff4 (0x529a bytes) 000129 1758824237.677555 write 0x03b5ee thru 0x03c86e (0x1281 bytes) 000131 1758824237.679009 mapwrite 0x006180 thru 0x00b3d3 (0x5254 bytes) 000132 1758824237.694439 punch from 0x001af5 to 0x009291 (0x779d bytes) 000133 1758824237.700916 trunc from 0x01a552 to 0x01fbfd (0x56ac bytes) 000134 1758824237.704521 trunc from 0x008bc7 to 0x0182df (0xf719 bytes) 000135 1758824237.714150 punch from 0x012fc5 to 0x01519a (0x21d6 bytes) 000136 1758824237.717591 write 0x00d1f7 thru 0x011684 (0x448e bytes) 000137 1758824237.724116 trunc from 0x03c86f to 0x003320 (0x3954e bytes) 000138 1758824237.733137 trunc from 0x000852 to 0x003320 (0x2acf bytes) 000139 1758824237.737577 punch from 0x002d56 to 0x003320 (0x5cb bytes) 000140 1758824237.743048 write 0x02f859 thru 0x033f12 (0x46ba bytes) 000141 1758824237.744336 write 0x01da15 thru 0x0238fe (0x5eea bytes) 000142 1758824237.750383 mapwrite 0x03c05c thru 0x03ffff (0x3fa4 bytes) 000143 1758824237.780426 fallocate 0x03bf65 thru 0x03ffff (0x409b bytes) 000144 1758824237.783905 trunc from 0x01f306 to 0x02cf03 (0xdbfe bytes) 000146 1758824237.787637 mapread 0x00e6c9 thru 0x011636 (0x2f6e bytes) 000148 1758824237.794065 mapread 0x0107af thru 0x015ea1 (0x56f3 bytes) 000149 1758824237.799322 MIRROR_EXTEND to 8 mirrors 000150 1758824238.148157 trunc from 0x017932 to 0x01af2f (0x35fe bytes) 000151 1758824238.164610 punch from 0x01f136 to 0x02388f (0x475a bytes) 000152 1758824238.167672 read 0x02823d thru 0x0351f0 (0xcfb4 bytes) 000153 1758824238.173156 read 0x00915a thru 0x01005b (0x6f02 bytes) 000154 1758824238.182513 MIRROR_EXTEND to 9 mirrors 000155 1758824238.542619 mapwrite 0x033087 thru 0x0374c3 (0x443d bytes) 000156 1758824238.601465 mapwrite 0x00c81a thru 0x0160bd (0x98a4 bytes) 000157 1758824238.625722 fallocate 0x023ebc thru 0x027c02 (0x3d47 bytes) 000158 1758824238.630186 mapread 0x005f11 thru 0x013178 (0xd268 bytes) 000160 1758824238.635645 MIRROR_RESYNC 9 mirrors 000161 1758824239.835551 read 0x01bb35 thru 0x01c551 (0xa1d bytes) 000162 1758824239.839592 mapwrite 0x038c47 thru 0x03ffff (0x73b9 bytes) 000163 1758824239.895267 trunc from 0x036eef to 0x03e3c3 (0x74d5 bytes) 000164 1758824239.898715 punch from 0x033366 to 0x03eac5 (0xb760 bytes) 000165 1758824239.902197 write 0x004af4 thru 0x00ade9 (0x62f6 bytes) 000166 1758824239.911315 MIRROR_EXTEND to 10 mirrors 000167 1758824240.365603 write 0x035af4 thru 0x038f1d (0x342a bytes) 000169 1758824240.401765 trunc from 0x040000 to 0x02ec15 (0x113ea bytes) 000170 1758824240.417593 write 0x0262f9 thru 0x0343dd (0xe0e5 bytes) 000171 1758824240.426067 write 0x0014f9 thru 0x00fc8d (0xe795 bytes) 000172 1758824240.435901 trunc from 0x00483e to 0x008ce7 (0x44aa bytes) 000173 1758824240.447227 trunc from 0x0343de to 0x025c1c (0xe7c1 bytes) 000174 1758824240.457277 read 0x00943e thru 0x016bda (0xd79d bytes) 000175 1758824240.462634 trunc from 0x01b5f0 to 0x025c1c (0xa62d bytes) 000176 1758824240.466432 punch from 0x00e49b to 0x00ec95 (0x7fb bytes) 000177 1758824240.470202 mapwrite 0x0399a7 thru 0x03d8cb (0x3f25 bytes) 000178 1758824240.499299 fallocate 0x00f84b thru 0x015315 (0x5acb bytes) 000179 1758824240.502404 read 0x03924a thru 0x03d8cb (0x4682 bytes) 000180 1758824240.502974 write 0x01bff5 thru 0x023f62 (0x7f6e bytes) 000181 1758824240.510235 mapwrite 0x026b79 thru 0x0336b7 (0xcb3f bytes) 000182 1758824240.523977 punch from 0x035751 to 0x03b8b2 (0x6162 bytes) 000183 1758824240.527377 read 0x01b68e thru 0x0280ad (0xca20 bytes) 000185 1758824240.528524 punch from 0x0024b8 to 0x00ab95 (0x86de bytes) 000186 1758824240.534026 fallocate 0x035cb8 thru 0x03bd92 (0x60db bytes) 000187 1758824240.538046 trunc from 0x01a9f9 to 0x029baa (0xf1b2 bytes) 000188 1758824240.548809 MIRROR_EXTEND to 11 mirrors 000189 1758824240.932420 trunc from 0x03d8cc to 0x0230a7 (0x1a824 bytes) 000192 1758824240.977897 trunc from 0x000c0a to 0x00b8c4 (0xacbb bytes) 000193 1758824240.982110 mapwrite 0x025c9f thru 0x02603d (0x39f bytes) 000194 1758824241.005571 fallocate 0x001d13 thru 0x00ddd6 (0xc0c4 bytes) 000195 1758824241.011532 write 0x0305b0 thru 0x036bdc (0x662d bytes) 000196 1758824241.012668 mapread 0x02f11c thru 0x036bdc (0x7ac1 bytes) 000197 1758824241.021563 trunc from 0x036bdd to 0x024385 (0x12857 bytes) 000198 1758824241.034412 mapread 0x021f02 thru 0x023387 (0x1486 bytes) 000199 1758824241.040522 punch from 0x009906 to 0x019602 (0xfcfd bytes) 000200 1758824241.044586 fallocate 0x006582 thru 0x010fce (0xaa4d bytes) 000201 1758824241.048188 fallocate 0x02d2d5 thru 0x02ee86 (0x1bb2 bytes) 000202 1758824241.056933 mapread 0x016fb2 thru 0x01a1f5 (0x3244 bytes) 000204 1758824241.064120 read 0x00665e thru 0x0143bd (0xdd60 bytes) 000205 1758824241.071206 trunc from 0x02ee87 to 0x03931f (0xa499 bytes) 000206 1758824241.093895 MIRROR_RESYNC 11 mirrors 000207 1758824241.922008 read 0x01cafa thru 0x02c89e (0xfda5 bytes) 000208 1758824241.927587 read 0x002f06 thru 0x00d514 (0xa60f bytes) 000210 1758824241.944399 MIRROR_SPLIT mirror 5 to 10 mirrors 000211 1758824242.221682 mapwrite 0x0296dc thru 0x037686 (0xdfab bytes) 000212 1758824242.300447 punch from 0x01031b to 0x015499 (0x517f bytes) 000213 1758824242.303288 mapread 0x02ec46 thru 0x03931f (0xa6da bytes) 000214 1758824242.304319 mapwrite 0x02c667 thru 0x03918f (0xcb29 bytes) 000215 1758824242.320825 mapwrite 0x02aa6e thru 0x02f6b5 (0x4c48 bytes) 000216 1758824242.335742 write 0x003bf5 thru 0x0083f5 (0x4801 bytes) 000217 1758824242.348480 MIRROR_RESYNC 10 mirrors 000218 1758824243.421671 punch from 0x019f6c to 0x0243fc (0xa491 bytes) 000219 1758824243.448159 read 0x02f362 thru 0x038bab (0x984a bytes) 000220 1758824243.453493 punch from 0x030ac9 to 0x032c50 (0x2188 bytes) 000221 1758824243.459572 fallocate 0x0379e4 thru 0x03ffff (0x861c bytes) 000222 1758824243.463800 trunc from 0x00d0bd to 0x01ce6e (0xfdb2 bytes) 000223 1758824243.467982 fallocate 0x0049f2 thru 0x014908 (0xff17 bytes) 000224 1758824243.472051 fallocate 0x01a679 thru 0x023640 (0x8fc8 bytes) 000226 1758824243.477790 trunc from 0x018b08 to 0x021f33 (0x942c bytes) 000227 1758824243.481588 trunc from 0x02fa7d to 0x03931f (0x98a3 bytes) 000228 1758824243.485238 trunc from 0x01a1b2 to 0x01b94f (0x179e bytes) 000229 1758824243.489013 mapwrite 0x020ada thru 0x024253 (0x377a bytes) 000230 1758824243.514177 trunc from 0x0327e3 to 0x03407b (0x1899 bytes) 000231 1758824243.518474 trunc from 0x039320 to 0x005e62 (0x334bd bytes) 000232 1758824243.531155 punch from 0x001c49 to 0x005e62 (0x421a bytes) 000233 1758824243.541504 punch from 0x002fda to 0x005e62 (0x2e89 bytes) 000234 1758824243.546422 read 0x0041df thru 0x005e62 (0x1c84 bytes) 000235 1758824243.562569 write 0x009723 thru 0x00c8ff (0x31dd bytes) 000236 1758824243.563758 punch from 0x00913c to 0x00bdde (0x2ca3 bytes) 000237 1758824243.575389 read 0x00a2b8 thru 0x00c8ff (0x2648 bytes) 000238 1758824243.582522 punch from 0x00a78e to 0x00c8ff (0x2172 bytes) 000239 1758824243.586456 read 0x002146 thru 0x00888a (0x6745 bytes) 000241 1758824243.587522 trunc from 0x00687c to 0x00c8ff (0x6084 bytes) 000242 1758824243.594010 read 0x003ab7 thru 0x00c8ff (0x8e49 bytes) 000243 1758824243.601207 write 0x01a962 thru 0x027bc3 (0xd262 bytes) 000244 1758824243.605814 MIRROR_RESYNC 10 mirrors 000245 1758824244.984232 read 0x0133ec thru 0x01e1f7 (0xae0c bytes) 000246 1758824244.993157 punch from 0x0031bd to 0x0059e4 (0x2828 bytes) 000247 1758824245.015633 read 0x018975 thru 0x023de6 (0xb472 bytes) 000248 1758824245.026419 write 0x017400 thru 0x026cd2 (0xf8d3 bytes) 000249 1758824245.028434 write 0x020ef1 thru 0x029330 (0x8440 bytes) 000250 1758824245.029763 mapread 0x0277ce thru 0x02792a (0x15d bytes) 000251 1758824245.031155 punch from 0x01160b to 0x018e26 (0x781c bytes) 000252 1758824245.044967 trunc from 0x029331 to 0x00139f (0x27f91 bytes) 000253 1758824245.058702 read 0x000648 thru 0x00139f (0xd58 bytes) 000254 1758824245.059374 punch from 0x000686 to 0x00139f (0xd1a bytes) 000255 1758824245.063727 read 0x0002d1 thru 0x00139f (0x10cf bytes) 000256 1758824245.068301 mapwrite 0x010c27 thru 0x014534 (0x390e bytes) 000258 1758824245.103354 mapwrite 0x032901 thru 0x035cdf (0x33df bytes) 000259 1758824245.135166 mapwrite 0x000075 thru 0x006a4b (0x69d7 bytes) 000260 1758824245.146296 fallocate 0x00edbf thru 0x01d4d4 (0xe716 bytes) 000261 1758824245.151812 trunc from 0x012bfe to 0x015e20 (0x3223 bytes) 000262 1758824245.163259 MIRROR_SPLIT mirror 5 to 9 mirrors 000263 1758824245.301732 trunc from 0x035ce0 to 0x00cdef (0x28ef0 bytes) 000264 1758824245.318206 trunc from 0x00cdf0 to 0x03c31b (0x2f52c bytes) 000265 1758824245.328536 fallocate 0x03ad7e thru 0x03ffff (0x5282 bytes) 000266 1758824245.336780 MIRROR_RESYNC 9 mirrors 000267 1758824246.015951 mapread 0x00338b thru 0x00f081 (0xbcf7 bytes) 000268 1758824246.027803 write 0x0075da thru 0x014a21 (0xd448 bytes) 000269 1758824246.057986 trunc from 0x040000 to 0x03d81e (0x27e1 bytes) 000270 1758824246.082376 trunc from 0x00993e to 0x014901 (0xafc4 bytes) 000271 1758824246.091112 mapread 0x03868d thru 0x03d81e (0x5192 bytes) 000272 1758824246.099439 fallocate 0x01bda5 thru 0x02b14d (0xf3a9 bytes) 000273 1758824246.103618 read 0x00cd17 thru 0x016b5d (0x9e47 bytes) 000274 1758824246.110560 read 0x02779e thru 0x031c5a (0xa4bd bytes) 000275 1758824246.116293 fallocate 0x03510a thru 0x0385b3 (0x34aa bytes) 000276 1758824246.119368 mapwrite 0x03427d thru 0x03bba9 (0x792d bytes) 000277 1758824246.135702 mapwrite 0x0371e8 thru 0x03b169 (0x3f82 bytes) 000278 1758824246.146473 trunc from 0x03d81f to 0x02f041 (0xe7dd bytes) 000281 1758824246.160966 fallocate 0x013e8e thru 0x0144d9 (0x64c bytes) 000283 1758824246.167746 MIRROR_RESYNC 9 mirrors 000284 1758824247.066135 fallocate 0x022ae2 thru 0x0329ef (0xff0e bytes) 000285 1758824247.088491 punch from 0x02c394 to 0x02fa15 (0x3682 bytes) 000286 1758824247.094978 read 0x01e193 thru 0x026399 (0x8207 bytes) 000287 1758824247.101412 punch from 0x0326e4 to 0x0329ef (0x30c bytes) 000288 1758824247.105975 mapwrite 0x036f4e thru 0x03c84e (0x5901 bytes) 000289 1758824247.168722 trunc from 0x03c84f to 0x030d9d (0xbab1 bytes) 000290 1758824247.187213 MIRROR_SPLIT mirror 2 to 8 mirrors 000291 1758824247.276013 MIRROR_SPLIT mirror 5 to 7 mirrors 000292 1758824247.362552 read 0x02f976 thru 0x030d9d (0x1428 bytes) 000293 1758824247.369131 fallocate 0x03747b thru 0x03daa3 (0x6629 bytes) 000295 1758824247.374112 read 0x016a37 thru 0x01bb79 (0x5143 bytes) 000296 1758824247.381755 mapread 0x00a478 thru 0x016cb1 (0xc83a bytes) 000297 1758824247.382791 write 0x038bff thru 0x03ffff (0x7401 bytes) 000298 1758824247.386784 trunc from 0x033870 to 0x034d08 (0x1499 bytes) 000299 1758824247.394527 MIRROR_RESYNC 7 mirrors 000300 1758824248.023538 MIRROR_EXTEND to 8 mirrors 000301 1758824248.350016 mapwrite 0x027ff0 thru 0x037931 (0xf942 bytes) 000302 1758824248.423337 mapread 0x0247b5 thru 0x02e764 (0x9fb0 bytes) 000303 1758824248.424377 trunc from 0x03fda0 to 0x03ffff (0x260 bytes) 000306 1758824248.429042 punch from 0x03b24a to 0x03ffff (0x4db6 bytes) 000308 1758824248.433046 trunc from 0x040000 to 0x02791e (0x186e1 bytes) 000309 1758824248.456618 mapwrite 0x0220f4 thru 0x024c39 (0x2b46 bytes) 000310 1758824248.470389 MIRROR_RESYNC 8 mirrors 000311 1758824249.388478 MIRROR_SPLIT mirror 1 to 7 mirrors 000312 1758824249.565669 punch from 0x008fcb to 0x013bdf (0xac15 bytes) 000313 1758824249.590074 fallocate 0x00e672 thru 0x01059f (0x1f2e bytes) 000314 1758824249.594355 punch from 0x0068df to 0x015765 (0xee87 bytes) 000315 1758824249.598491 punch from 0x0182a8 to 0x01aebd (0x2c16 bytes) 000316 1758824249.602018 trunc from 0x00e345 to 0x0171c9 (0x8e85 bytes) 000317 1758824249.606939 fallocate 0x038dab thru 0x039cc0 (0xf16 bytes) 000318 1758824249.611426 trunc from 0x02791f to 0x02d89f (0x5f81 bytes) 000321 1758824249.629139 write 0x00cf79 thru 0x01b317 (0xe39f bytes) 000322 1758824249.637484 mapread 0x00bd7a thru 0x016e3c (0xb0c3 bytes) 000323 1758824249.644617 punch from 0x015c86 to 0x017324 (0x169f bytes) 000326 1758824249.658904 punch from 0x004538 to 0x006a92 (0x255b bytes) 000327 1758824249.679058 MIRROR_EXTEND to 8 mirrors 000328 1758824250.077124 mapread 0x008565 thru 0x00a83c (0x22d8 bytes) 000330 1758824250.104732 punch from 0x01eb8a to 0x029366 (0xa7dd bytes) 000331 1758824250.133101 punch from 0x0149ec to 0x018c48 (0x425d bytes) 000332 1758824250.136970 mapwrite 0x022e06 thru 0x02cfcb (0xa1c6 bytes) 000333 1758824250.157066 mapwrite 0x00659f thru 0x00cea8 (0x690a bytes) 000334 1758824250.184873 MIRROR_SPLIT mirror 4 to 7 mirrors 000335 1758824250.287365 MIRROR_RESYNC 7 mirrors 000336 1758824251.305386 trunc from 0x007766 to 0x00e998 (0x7233 bytes) 000338 1758824251.323080 punch from 0x004faa to 0x0058d0 (0x927 bytes) 000339 1758824251.326408 write 0x008a43 thru 0x015cc5 (0xd283 bytes) 000340 1758824251.333089 trunc from 0x02d8a0 to 0x01e299 (0xf606 bytes) 000341 1758824251.344188 read 0x010dac thru 0x017efc (0x7151 bytes) 000343 1758824251.349063 fallocate 0x0086d0 thru 0x014400 (0xbd31 bytes) 000344 1758824251.355700 punch from 0x016f97 to 0x01e299 (0x7303 bytes) 000345 1758824251.359714 mapwrite 0x00daab thru 0x01c6eb (0xec41 bytes) 000346 1758824251.382226 trunc from 0x00761f to 0x0082f3 (0xcd5 bytes) 000347 1758824251.386394 fallocate 0x008a94 thru 0x01366a (0xabd7 bytes) 000348 1758824251.393151 MIRROR_SPLIT mirror 4 to 6 mirrors 000349 1758824251.479139 write 0x01bbbe thru 0x022b70 (0x6fb3 bytes) 000350 1758824251.485009 trunc from 0x022b71 to 0x0107da (0x12396 bytes) 000351 1758824251.495395 mapwrite 0x0284ac thru 0x029a26 (0x157b bytes) 000352 1758824251.526363 trunc from 0x029a27 to 0x0011cd (0x28859 bytes) 000353 1758824251.537661 write 0x023b25 thru 0x03261f (0xeafb bytes) 000354 1758824251.539398 read 0x007b39 thru 0x00bd49 (0x4211 bytes) 000355 1758824251.544128 trunc from 0x032620 to 0x03f95a (0xd33b bytes) 000356 1758824251.554472 fallocate 0x010662 thru 0x0118c9 (0x1268 bytes) 000357 1758824251.558046 mapwrite 0x037169 thru 0x03b883 (0x471b bytes) 000359 1758824251.580621 read 0x0058ca thru 0x00ff95 (0xa6cc bytes) 000360 1758824251.588510 punch from 0x0091f2 to 0x016f2a (0xdd39 bytes) 000361 1758824251.594329 mapwrite 0x00c55f thru 0x013025 (0x6ac7 bytes) 000362 1758824251.632578 trunc from 0x006755 to 0x0112ae (0xab5a bytes) 000363 1758824251.644344 trunc from 0x03f95b to 0x02ee36 (0x10b24 bytes) 000366 1758824251.656927 mapread 0x00d109 thru 0x014a4b (0x7943 bytes) 000367 1758824251.682042 mapread 0x01dfd6 thru 0x01f21b (0x1246 bytes) 000368 1758824251.683024 read 0x00dde8 thru 0x01a005 (0xc21e bytes) 000369 1758824251.693757 mapread 0x017add thru 0x0262e1 (0xe805 bytes) 000371 1758824251.695179 trunc from 0x013e4b to 0x021328 (0xd4de bytes) 000372 1758824251.700830 mapwrite 0x03f17f thru 0x03f8a4 (0x726 bytes) 000374 1758824251.722450 trunc from 0x03f8a5 to 0x004f03 (0x3a9a1 bytes) 000375 1758824251.737724 trunc from 0x00193b to 0x004f03 (0x35c9 bytes) 000376 1758824251.741097 mapread 0x003677 thru 0x004f03 (0x188d bytes) 000377 1758824251.747167 mapwrite 0x00d411 thru 0x0153c8 (0x7fb8 bytes) 000378 1758824251.798061 fallocate 0x013f67 thru 0x01a3df (0x6479 bytes) 000379 1758824251.802543 punch from 0x009f0a to 0x0199bb (0xfab2 bytes) 000380 1758824251.807020 trunc from 0x008612 to 0x015153 (0xcb42 bytes) 000381 1758824251.810697 trunc from 0x004151 to 0x00750b (0x33bb bytes) 000382 1758824251.814427 read 0x003d8d thru 0x006d22 (0x2f96 bytes) 000383 1758824251.820712 trunc from 0x00143b to 0x004f97 (0x3b5d bytes) 000384 1758824251.824101 trunc from 0x01a3e0 to 0x004cb9 (0x15726 bytes) 000385 1758824251.835662 write 0x017f01 thru 0x027bc2 (0xfcc2 bytes) 000386 1758824251.837533 read 0x011c2b thru 0x014ec6 (0x329c bytes) 000387 1758824251.843683 trunc from 0x027bc3 to 0x03e841 (0x16c7f bytes) 000388 1758824251.858113 MIRROR_SPLIT mirror 2 to 5 mirrors 000389 1758824251.984166 punch from 0x03c293 to 0x03e841 (0x25af bytes) 000391 1758824251.990115 trunc from 0x013f12 to 0x018b93 (0x4c82 bytes) 000392 1758824251.995099 fallocate 0x03a67e thru 0x03ffff (0x5982 bytes) 000393 1758824252.006512 MIRROR_RESYNC 5 mirrors 000394 1758824252.554284 mapread 0x00c4dd thru 0x01277a (0x629e bytes) 000395 1758824252.568214 punch from 0x03f3ab to 0x03ffff (0xc55 bytes) 000396 1758824252.591971 mapwrite 0x00c913 thru 0x015969 (0x9057 bytes) 000397 1758824252.618873 trunc from 0x02160d to 0x02716f (0x5b63 bytes) 000398 1758824252.625578 MIRROR_SPLIT mirror 1 to 4 mirrors 000399 1758824252.749463 write 0x0268a0 thru 0x02e7b7 (0x7f18 bytes) 000400 1758824252.765690 mapwrite 0x0008dc thru 0x007e86 (0x75ab bytes) 000401 1758824252.793459 read 0x017a79 thru 0x0218ff (0x9e87 bytes) 000402 1758824252.798678 mapwrite 0x025cd7 thru 0x02dbe9 (0x7f13 bytes) 000403 1758824252.817704 read 0x0115a0 thru 0x012583 (0xfe4 bytes) 000405 1758824252.824889 read 0x01b887 thru 0x025d16 (0xa490 bytes) 000406 1758824252.825515 trunc from 0x02792a to 0x03283d (0xaf14 bytes) 000407 1758824252.829162 fallocate 0x00e278 thru 0x01c915 (0xe69e bytes) 000408 1758824252.832165 write 0x031d6d thru 0x033a54 (0x1ce8 bytes) 000409 1758824252.838456 punch from 0x02e7d3 to 0x03db62 (0xf390 bytes) 000410 1758824252.845401 trunc from 0x040000 to 0x01808f (0x27f70 bytes) 000411 1758824252.853817 trunc from 0x00ae12 to 0x017624 (0xc813 bytes) 000412 1758824252.858035 punch from 0x00755b to 0x007ea6 (0x94c bytes) 000413 1758824252.860821 trunc from 0x0030a2 to 0x003ed0 (0xe2f bytes) 000414 1758824252.866264 mapwrite 0x0147d2 thru 0x0228cd (0xe0fc bytes) 000415 1758824252.933043 read 0x010c32 thru 0x0153d1 (0x47a0 bytes) 000416 1758824252.938187 trunc from 0x0228ce to 0x00b0ad (0x17820 bytes) 000417 1758824252.946775 read 0x006acd thru 0x00b0ad (0x45e1 bytes) 000418 1758824252.947540 read 0x002070 thru 0x003354 (0x12e5 bytes) 000419 1758824252.948116 mapread 0x0045cb thru 0x004ef3 (0x929 bytes) 000420 1758824252.948882 punch from 0x001469 to 0x00b0ad (0x9c45 bytes) 000421 1758824252.952265 fallocate 0x01f4c3 thru 0x02725a (0x7d98 bytes) 000422 1758824252.955892 write 0x036189 thru 0x03ffff (0x9e77 bytes) 000423 1758824252.957325 write 0x03a56d thru 0x03ffff (0x5a93 bytes) 000424 1758824252.958482 write 0x031bc1 thru 0x0383c5 (0x6805 bytes) 000425 1758824252.967150 punch from 0x002913 to 0x00c579 (0x9c67 bytes) 000426 1758824252.979697 MIRROR_EXTEND to 5 mirrors 000427 1758824253.306439 mapwrite 0x000e16 thru 0x00892f (0x7b1a bytes) 000428 1758824253.403998 write 0x0216be thru 0x03083a (0xf17d bytes) 000430 1758824253.422144 MIRROR_EXTEND to 6 mirrors 000431 1758824253.779441 write 0x007b69 thru 0x00d809 (0x5ca1 bytes) 000432 1758824253.815453 punch from 0x036c1d to 0x03ffff (0x93e3 bytes) 000433 1758824253.820098 fallocate 0x019aae thru 0x022471 (0x89c4 bytes) 000434 1758824253.825174 mapwrite 0x000b2a thru 0x002c34 (0x210b bytes) 000435 1758824253.849816 trunc from 0x02456f to 0x02affb (0x6a8d bytes) 000436 1758824253.854322 fallocate 0x0137e9 thru 0x01f02f (0xb847 bytes) 000437 1758824253.858731 trunc from 0x012d4f to 0x016d15 (0x3fc7 bytes) 000438 1758824253.862799 mapread 0x03ff48 thru 0x03ffff (0xb8 bytes) 000439 1758824253.869129 punch from 0x0102e9 to 0x017f28 (0x7c40 bytes) 000440 1758824253.876544 MIRROR_SPLIT mirror 1 to 5 mirrors 000441 1758824253.961780 trunc from 0x02e414 to 0x035edb (0x7ac8 bytes) 000442 1758824253.965176 trunc from 0x00c0a2 to 0x00cfcc (0xf2b bytes) 000443 1758824253.968761 fallocate 0x00991d thru 0x01785d (0xdf41 bytes) 000445 1758824253.973988 write 0x037864 thru 0x03a61d (0x2dba bytes) 000446 1758824253.982793 punch from 0x03199c to 0x036849 (0x4eae bytes) 000447 1758824253.987356 trunc from 0x040000 to 0x0303bd (0xfc42 bytes) 000448 1758824254.001884 MIRROR_RESYNC 5 mirrors 000449 1758824254.660212 mapread 0x012016 thru 0x015ca3 (0x3c8e bytes) 000450 1758824254.673825 mapread 0x00146b thru 0x00a8f7 (0x948d bytes) 000452 1758824254.679690 mapread 0x0270f0 thru 0x0303bd (0x92ce bytes) 000453 1758824254.688264 trunc from 0x014e6f to 0x01ff57 (0xb0e9 bytes) 000454 1758824254.707005 punch from 0x018b13 to 0x01c105 (0x35f3 bytes) 000455 1758824254.710945 mapread 0x029d56 thru 0x0303bd (0x6668 bytes) 000456 1758824254.716585 fallocate 0x024313 thru 0x0332f5 (0xefe3 bytes) 000457 1758824254.721798 write 0x01ea2e thru 0x0241d2 (0x57a5 bytes) 000458 1758824254.729395 trunc from 0x0303be to 0x0052d6 (0x2b0e7 bytes) 000459 1758824254.741693 mapwrite 0x0234a2 thru 0x02ebec (0xb74b bytes) 000460 1758824254.775957 fallocate 0x00dd27 thru 0x012428 (0x4702 bytes) 000461 1758824254.779895 fallocate 0x005e91 thru 0x013db8 (0xdf28 bytes) 000462 1758824254.783270 write 0x0103a3 thru 0x018194 (0x7df2 bytes) 000463 1758824254.788930 fallocate 0x023a72 thru 0x02c485 (0x8a14 bytes) 000465 1758824254.795682 trunc from 0x02ebed to 0x02460c (0xa5e0 bytes) 000466 1758824254.811102 MIRROR_SPLIT mirror 1 to 4 mirrors 000467 1758824254.929302 MIRROR_EXTEND to 5 mirrors 000468 1758824255.221991 MIRROR_EXTEND to 6 mirrors 000469 1758824255.517542 fallocate 0x01bfe5 thru 0x021b15 (0x5b31 bytes) 000470 1758824255.537776 write 0x03408d thru 0x038cf6 (0x4c6a bytes) 000471 1758824255.541264 read 0x00bfc3 thru 0x00f093 (0x30d1 bytes) 000472 1758824255.548055 MIRROR_RESYNC 6 mirrors 000473 1758824256.237115 MIRROR_EXTEND to 7 mirrors 000474 1758824256.554090 MIRROR_RESYNC 7 mirrors 000475 1758824256.716016 read 0x0164c2 thru 0x02410d (0xdc4c bytes) 000476 1758824256.722145 mapread 0x036a7d thru 0x038cf6 (0x227a bytes) 000477 1758824256.740235 punch from 0x00f8f4 to 0x0140c0 (0x47cd bytes) 000478 1758824256.763426 trunc from 0x038cf7 to 0x014ba9 (0x2414d bytes) 000479 1758824256.779498 mapwrite 0x01296d thru 0x01459e (0x1c32 bytes) 000480 1758824256.802112 read 0x0089a7 thru 0x00b06a (0x26c4 bytes) 000481 1758824256.807305 trunc from 0x00cf63 to 0x00d173 (0x211 bytes) 000482 1758824256.810656 trunc from 0x009ff1 to 0x00d9c7 (0x39d7 bytes) 000483 1758824256.813714 fallocate 0x00f184 thru 0x017bfb (0x8a78 bytes) 000484 1758824256.816988 mapread 0x00e869 thru 0x014ec6 (0x665e bytes) 000485 1758824256.820117 MIRROR_EXTEND to 8 mirrors 000486 1758824257.115463 trunc from 0x017bfc to 0x02afe9 (0x133ee bytes) 000487 1758824257.167161 fallocate 0x014030 thru 0x022a2f (0xea00 bytes) 000488 1758824257.176281 MIRROR_EXTEND to 9 mirrors 000489 1758824257.500347 mapwrite 0x00087a thru 0x00aad6 (0xa25d bytes) 000490 1758824257.557048 write 0x01c14b thru 0x022d2e (0x6be4 bytes) 000491 1758824257.564072 fallocate 0x015222 thru 0x01dacb (0x88aa bytes) 000492 1758824257.567771 mapread 0x018027 thru 0x01f4b5 (0x748f bytes) 000493 1758824257.568649 mapread 0x021034 thru 0x02afe9 (0x9fb6 bytes) 000494 1758824257.575055 write 0x0300e1 thru 0x03721f (0x713f bytes) 000495 1758824257.576320 mapwrite 0x006aef thru 0x016666 (0xfb78 bytes) 000496 1758824257.595113 punch from 0x01e0d0 to 0x01e166 (0x97 bytes) 000497 1758824257.597968 fallocate 0x03722c thru 0x039964 (0x2739 bytes) 000498 1758824257.601629 mapread 0x01b020 thru 0x021db5 (0x6d96 bytes) 000499 1758824257.605550 fallocate 0x02337a thru 0x02568f (0x2316 bytes) 000500 1758824257.611258 MIRROR_RESYNC 9 mirrors 000501 1758824258.599168 mapread 0x00ff24 thru 0x0179a9 (0x7a86 bytes) 000502 1758824258.609013 read 0x01ec73 thru 0x020d01 (0x208f bytes) 000503 1758824258.614025 MIRROR_SPLIT mirror 8 to 8 mirrors 000505 1758824258.792118 read 0x0172dd thru 0x01b80a (0x452e bytes) 000506 1758824258.797000 fallocate 0x01e2be thru 0x01f7b2 (0x14f5 bytes) 000507 1758824258.823714 punch from 0x0024cf to 0x00e11f (0xbc51 bytes) 000508 1758824258.827309 mapread 0x02103c thru 0x022f8b (0x1f50 bytes) 000510 1758824258.838214 trunc from 0x039965 to 0x0264f0 (0x13474 bytes) 000511 1758824258.856041 read 0x01b67a thru 0x021f92 (0x6919 bytes) 000512 1758824258.863563 fallocate 0x022ace thru 0x02740a (0x493d bytes) 000513 1758824258.869587 mapwrite 0x026ee7 thru 0x02f19a (0x82b4 bytes) 000514 1758824258.899721 trunc from 0x02f19b to 0x0149be (0x1a7dc bytes) 000517 1758824258.912077 MIRROR_SPLIT mirror 3 to 7 mirrors 000518 1758824259.017679 punch from 0x013dbc to 0x0149be (0xc03 bytes) 000519 1758824259.020931 trunc from 0x0149bf to 0x0142ef (0x6cf bytes) 000520 1758824259.034133 trunc from 0x0142f0 to 0x02a0b9 (0x15dca bytes) 000521 1758824259.045449 mapread 0x014c21 thru 0x020e7f (0xc25f bytes) 000522 1758824259.060465 MIRROR_SPLIT mirror 2 to 6 mirrors 000523 1758824259.149285 trunc from 0x02a0ba to 0x024e39 (0x5280 bytes) 000524 1758824259.162112 write 0x0173e1 thru 0x018244 (0xe64 bytes) 000525 1758824259.171320 MIRROR_RESYNC 6 mirrors 000526 1758824259.979968 write 0x0226ad thru 0x02e8c1 (0xc215 bytes) 000527 1758824260.000455 read 0x00120e thru 0x002429 (0x121c bytes) 000528 1758824260.005827 read 0x001099 thru 0x002db2 (0x1d1a bytes) 000530 1758824260.006737 punch from 0x0010e9 to 0x003fb7 (0x2ecf bytes) 000531 1758824260.010751 fallocate 0x035d14 thru 0x0379c7 (0x1cb4 bytes) 000532 1758824260.014448 mapwrite 0x03b40e thru 0x03ffff (0x4bf2 bytes) 000533 1758824260.056538 trunc from 0x040000 to 0x01fc69 (0x20396 bytes) 000534 1758824260.069406 write 0x01381f thru 0x01f99c (0xc17e bytes) 000535 1758824260.070859 write 0x028022 thru 0x02e3b0 (0x638f bytes) 000536 1758824260.072002 mapread 0x010b24 thru 0x01464e (0x3b2b bytes) 000537 1758824260.075622 MIRROR_RESYNC 6 mirrors 000538 1758824260.779914 fallocate 0x0261bb thru 0x02d3e8 (0x722e bytes) 000539 1758824260.813026 trunc from 0x02e3b1 to 0x01e76b (0xfc45 bytes) 000540 1758824260.852728 mapwrite 0x037e14 thru 0x038f02 (0x10ef bytes) 000541 1758824260.895702 write 0x02044b thru 0x0284e0 (0x8096 bytes) 000542 1758824260.903813 mapread 0x033c76 thru 0x0346f5 (0xa80 bytes) 000543 1758824260.911492 fallocate 0x00a4c9 thru 0x017569 (0xd0a1 bytes) 000544 1758824260.916808 read 0x008272 thru 0x00b89c (0x362b bytes) 000545 1758824260.922661 fallocate 0x037434 thru 0x03f684 (0x8251 bytes) 000546 1758824260.927368 write 0x00678e thru 0x00b5ce (0x4e41 bytes) 000547 1758824260.934234 punch from 0x025da2 to 0x0277bf (0x1a1e bytes) 000548 1758824260.949623 fallocate 0x0132d9 thru 0x0196f4 (0x641c bytes) 000549 1758824260.953751 write 0x00928e thru 0x010e54 (0x7bc7 bytes) 000550 1758824260.954750 read 0x02a57c thru 0x032348 (0x7dcd bytes) 000551 1758824260.960701 write 0x009745 thru 0x016b81 (0xd43d bytes) 000552 1758824260.962130 trunc from 0x01dcf0 to 0x024e44 (0x7155 bytes) 000553 1758824260.967234 fallocate 0x01a96a thru 0x01c0f0 (0x1787 bytes) 000554 1758824260.971884 trunc from 0x03f685 to 0x039109 (0x657b bytes) 000555 1758824260.988073 MIRROR_RESYNC 6 mirrors 000557 1758824261.943896 trunc from 0x03543e to 0x037dfa (0x29bd bytes) 000558 1758824261.959272 mapread 0x0057da thru 0x011f2b (0xc752 bytes) 000559 1758824261.975035 punch from 0x033f20 to 0x039109 (0x51ea bytes) 000560 1758824261.977913 fallocate 0x032967 thru 0x03ffff (0xd699 bytes) 000561 1758824261.981897 mapread 0x01ca21 thru 0x01f740 (0x2d20 bytes) 000562 1758824261.987728 trunc from 0x01f13b to 0x028b6f (0x9a35 bytes) 000563 1758824261.991147 write 0x034a01 thru 0x03ffff (0xb5ff bytes) 000565 1758824262.000053 MIRROR_RESYNC 6 mirrors 000566 1758824262.693680 mapread 0x03d429 thru 0x03ffff (0x2bd7 bytes) 000567 1758824262.704307 trunc from 0x040000 to 0x00cdea (0x33215 bytes) 000568 1758824262.732106 trunc from 0x003ea3 to 0x00cdea (0x8f48 bytes) 000569 1758824262.737400 MIRROR_SPLIT mirror 1 to 5 mirrors 000570 1758824262.830769 read 0x00a94f thru 0x00cdea (0x249c bytes) 000571 1758824262.834520 mapwrite 0x01bb1e thru 0x020c4a (0x512d bytes) 000572 1758824262.866716 mapread 0x01030b thru 0x013c19 (0x390f bytes) 000573 1758824262.874404 MIRROR_EXTEND to 6 mirrors 000574 1758824263.169641 punch from 0x00957e to 0x00b48a (0x1f0d bytes) 000575 1758824263.191632 mapread 0x013fdb thru 0x01e0d5 (0xa0fb bytes) 000576 1758824263.206936 MIRROR_EXTEND to 7 mirrors 000578 1758824263.531208 mapwrite 0x0342c6 thru 0x038fe0 (0x4d1b bytes) 000579 1758824263.571492 mapread 0x005726 thru 0x011fec (0xc8c7 bytes) 000581 1758824263.582733 MIRROR_SPLIT mirror 1 to 6 mirrors 000582 1758824263.660999 mapread 0x001019 thru 0x00e469 (0xd451 bytes) 000583 1758824263.671766 MIRROR_RESYNC 6 mirrors 000584 1758824264.258512 read 0x0358e6 thru 0x038fe0 (0x36fb bytes) 000585 1758824264.261368 mapread 0x02c4be thru 0x030926 (0x4469 bytes) 000586 1758824264.268953 trunc from 0x038fe1 to 0x00e8fe (0x2a6e2 bytes) 000587 1758824264.289319 MIRROR_SPLIT mirror 2 to 5 mirrors 000588 1758824264.379792 trunc from 0x00e8ff to 0x00a0f0 (0x480e bytes) 000589 1758824264.398177 mapread 0x009b90 thru 0x00a0f0 (0x561 bytes) 000590 1758824264.410627 trunc from 0x00a0f1 to 0x03f3ce (0x352de bytes) 000591 1758824264.421312 read 0x0389e8 thru 0x03c1fc (0x3815 bytes) 000592 1758824264.429268 mapwrite 0x0232bf thru 0x025814 (0x2556 bytes) 000593 1758824264.445971 read 0x01f866 thru 0x02c0e0 (0xc87b bytes) 000594 1758824264.459266 trunc from 0x01f5b6 to 0x0270d3 (0x7b1e bytes) 000595 1758824264.468299 punch from 0x039134 to 0x03f3ce (0x629b bytes) 000597 1758824264.473669 trunc from 0x0147c7 to 0x015242 (0xa7c bytes) 000598 1758824264.477652 mapwrite 0x022c58 thru 0x02a353 (0x76fc bytes) 000599 1758824264.513036 write 0x02f280 thru 0x03f0cd (0xfe4e bytes) 000600 1758824264.519182 trunc from 0x03f3cf to 0x03d901 (0x1acd bytes) 000601 1758824264.531432 write 0x03fa2c thru 0x03ffff (0x5d4 bytes) 000602 1758824264.532368 mapwrite 0x00807d thru 0x00a244 (0x21c8 bytes) 000603 1758824264.544573 mapwrite 0x01aadb thru 0x028d76 (0xe29c bytes) 000604 1758824264.571358 MIRROR_SPLIT mirror 3 to 4 mirrors 000605 1758824264.664606 mapread 0x01129c thru 0x01a7fa (0x955f bytes) 000606 1758824264.683338 write 0x02582f thru 0x02f666 (0x9e38 bytes) 000607 1758824264.698218 write 0x016d60 thru 0x020cf3 (0x9f94 bytes) 000608 1758824264.704622 MIRROR_RESYNC 4 mirrors 000609 1758824265.247566 mapwrite 0x002745 thru 0x00c8e5 (0xa1a1 bytes) 000610 1758824265.327766 punch from 0x0294ba to 0x032452 (0x8f99 bytes) 000611 1758824265.331884 mapwrite 0x00a2f4 thru 0x013b6d (0x987a bytes) 000612 1758824265.350996 trunc from 0x022a4e to 0x02e304 (0xb8b7 bytes) 000613 1758824265.357104 fallocate 0x011dcb thru 0x01fc48 (0xde7e bytes) 000614 1758824265.363732 write 0x02afea thru 0x02da42 (0x2a59 bytes) 000615 1758824265.369759 mapread 0x025fe9 thru 0x031974 (0xb98c bytes) 000616 1758824265.374294 MIRROR_RESYNC 4 mirrors 000617 1758824266.022762 MIRROR_RESYNC 4 mirrors 000618 1758824266.140027 punch from 0x004528 to 0x008936 (0x440f bytes) 000619 1758824266.160556 write 0x02781e thru 0x03316c (0xb94f bytes) 000620 1758824266.171944 MIRROR_RESYNC 4 mirrors 000621 1758824266.783613 mapwrite 0x016364 thru 0x016bf4 (0x891 bytes) 000623 1758824266.835001 trunc from 0x040000 to 0x023dfe (0x1c201 bytes) 000624 1758824266.854770 write 0x004746 thru 0x00d4ae (0x8d69 bytes) 000625 1758824266.865551 trunc from 0x003a60 to 0x00713e (0x36df bytes) 000626 1758824266.885163 fallocate 0x02b9f8 thru 0x03543d (0x9a46 bytes) 000627 1758824266.890303 write 0x024484 thru 0x028426 (0x3fa3 bytes) 000628 1758824266.899857 write 0x004153 thru 0x00c9ed (0x889b bytes) 000629 1758824266.907222 mapwrite 0x034a45 thru 0x03c276 (0x7832 bytes) 000630 1758824266.939912 read 0x02c83a thru 0x0391dd (0xc9a4 bytes) 000631 1758824266.940585 write 0x037dc0 thru 0x03ffff (0x8240 bytes) 000632 1758824266.941706 write 0x01267a thru 0x01aaa1 (0x8428 bytes) 000633 1758824266.947124 fallocate 0x03553b thru 0x03ffff (0xaac5 bytes) 000635 1758824266.950985 write 0x00d94e thru 0x0162fc (0x89af bytes) 000636 1758824266.952170 mapwrite 0x022a44 thru 0x02b31d (0x88da bytes) 000637 1758824266.966218 MIRROR_EXTEND to 5 mirrors 000638 1758824267.249479 mapwrite 0x003659 thru 0x013174 (0xfb1c bytes) 000640 1758824267.317623 trunc from 0x01dee0 to 0x020ac1 (0x2be2 bytes) 000641 1758824267.323118 MIRROR_RESYNC 5 mirrors 000642 1758824267.790347 write 0x02390a thru 0x029162 (0x5859 bytes) 000643 1758824267.808330 read 0x000eeb thru 0x00916a (0x8280 bytes) 000644 1758824267.815709 MIRROR_SPLIT mirror 2 to 4 mirrors 000645 1758824267.896814 mapwrite 0x002568 thru 0x009f3c (0x79d5 bytes) 000646 1758824267.918937 trunc from 0x01acf3 to 0x02a5c7 (0xf8d5 bytes) 000647 1758824267.922127 read 0x01b6f9 thru 0x01e2f3 (0x2bfb bytes) 000649 1758824267.926038 trunc from 0x040000 to 0x0044a3 (0x3bb5c bytes) 000651 1758824267.937540 punch from 0x001f35 to 0x0044a3 (0x256f bytes) 000652 1758824267.942286 mapread 0x001e40 thru 0x0044a3 (0x2664 bytes) 000653 1758824267.947636 fallocate 0x004647 thru 0x00ff93 (0xb94d bytes) 000654 1758824267.951742 trunc from 0x00ff94 to 0x00d7c6 (0x27cd bytes) 000655 1758824267.964331 mapwrite 0x037e91 thru 0x03ffff (0x816f bytes) 000656 1758824267.989945 fallocate 0x02de34 thru 0x03a276 (0xc443 bytes) 000657 1758824267.993360 write 0x030623 thru 0x03ea84 (0xe462 bytes) 000659 1758824267.995173 write 0x032e20 thru 0x03ffff (0xd1e0 bytes) 000660 1758824267.996378 trunc from 0x040000 to 0x025dd1 (0x1a22e bytes) 000661 1758824268.008229 MIRROR_SPLIT mirror 2 to 3 mirrors 000662 1758824268.095045 mapwrite 0x0150a5 thru 0x0201c3 (0xb11f bytes) 000663 1758824268.120162 mapwrite 0x03f303 thru 0x03fc09 (0x907 bytes) 000665 1758824268.144610 mapwrite 0x038ebd thru 0x039897 (0x9db bytes) 000666 1758824268.154303 fallocate 0x02b7ac thru 0x033251 (0x7aa6 bytes) 000667 1758824268.158917 mapwrite 0x019c9d thru 0x026a9c (0xce00 bytes) 000668 1758824268.180182 MIRROR_EXTEND to 4 mirrors 000669 1758824268.483269 read 0x00bdde thru 0x00ced4 (0x10f7 bytes) 000670 1758824268.494460 trunc from 0x03dd4b to 0x03fc09 (0x1ebf bytes) 000671 1758824268.525410 MIRROR_EXTEND to 5 mirrors 000672 1758824268.892542 trunc from 0x03fc0a to 0x020fef (0x1ec1a bytes) 000673 1758824268.944015 mapwrite 0x02cff2 thru 0x033bb9 (0x6bc8 bytes) 000674 1758824268.990466 MIRROR_RESYNC 5 mirrors 000675 1758824269.774390 trunc from 0x03087a to 0x033bb9 (0x3340 bytes) 000676 1758824269.806036 MIRROR_RESYNC 5 mirrors 000678 1758824270.550194 read 0x02d79e thru 0x033bb9 (0x641c bytes) 000679 1758824270.554616 write 0x01ecad thru 0x0247fb (0x5b4f bytes) 000680 1758824270.578270 mapwrite 0x02cfba thru 0x02e324 (0x136b bytes) 000681 1758824270.598427 MIRROR_EXTEND to 6 mirrors 000682 1758824270.913879 mapwrite 0x02951f thru 0x02e5dd (0x50bf bytes) 000683 1758824270.970916 MIRROR_EXTEND to 7 mirrors 000684 1758824271.333264 trunc from 0x03259d to 0x03293e (0x3a2 bytes) 000685 1758824271.354504 write 0x010c2e thru 0x01d9e9 (0xcdbc bytes) 000686 1758824271.367240 MIRROR_RESYNC 7 mirrors 000687 1758824272.184809 fallocate 0x0200cd thru 0x024216 (0x414a bytes) 000688 1758824272.201927 write 0x02e22b thru 0x033cc6 (0x5a9c bytes) 000689 1758824272.208138 read 0x00e8f8 thru 0x01bf36 (0xd63f bytes) 000690 1758824272.213753 mapwrite 0x0379e3 thru 0x03add2 (0x33f0 bytes) 000691 1758824272.238411 fallocate 0x02135e thru 0x02c822 (0xb4c5 bytes) 000692 1758824272.241366 mapread 0x000417 thru 0x005857 (0x5441 bytes) 000693 1758824272.245772 trunc from 0x03add3 to 0x00563a (0x35798 bytes) 000694 1758824272.257758 MIRROR_RESYNC 7 mirrors 000695 1758824272.766611 mapwrite 0x03f09f thru 0x03ffff (0xf61 bytes) 000696 1758824272.807301 trunc from 0x03b256 to 0x03ffff (0x4daa bytes) 000697 1758824272.813599 MIRROR_EXTEND to 8 mirrors 000698 1758824273.051083 trunc from 0x040000 to 0x008b6b (0x37494 bytes) 000699 1758824273.079065 punch from 0x008227 to 0x008b6b (0x945 bytes) 000700 1758824273.081687 read 0x005d8c thru 0x008b6b (0x2de0 bytes) 000701 1758824273.084905 punch from 0x0036ac to 0x003bf8 (0x54d bytes) 000702 1758824273.087625 trunc from 0x008b6c to 0x02a7d8 (0x21c6d bytes) 000703 1758824273.096948 trunc from 0x02a7d9 to 0x01f35c (0xb47c bytes) 000705 1758824273.106756 write 0x03a837 thru 0x03b62f (0xdf9 bytes) 000706 1758824273.108049 fallocate 0x008593 thru 0x00ea7d (0x64eb bytes) 000707 1758824273.111583 fallocate 0x001474 thru 0x009dda (0x8967 bytes) 000708 1758824273.116754 MIRROR_EXTEND to 9 mirrors 000709 1758824273.382175 mapread 0x019d57 thru 0x01fbf7 (0x5ea1 bytes) 000710 1758824273.399914 trunc from 0x03b630 to 0x01c3f6 (0x1f239 bytes) 000711 1758824273.431076 mapread 0x01669d thru 0x01c3f6 (0x5d5a bytes) 000712 1758824273.436398 mapread 0x00b321 thru 0x017c13 (0xc8f3 bytes) 000713 1758824273.442213 trunc from 0x010c1a to 0x0155b4 (0x499b bytes) 000714 1758824273.446677 mapwrite 0x01f463 thru 0x02eaf4 (0xf692 bytes) 000715 1758824273.479403 read 0x00f7a3 thru 0x01643d (0x6c9b bytes) 000716 1758824273.483878 trunc from 0x02eaf5 to 0x022489 (0xc66b bytes) 000717 1758824273.492603 read 0x01da68 thru 0x022489 (0x4a22 bytes) 000718 1758824273.495670 MIRROR_SPLIT mirror 7 to 8 mirrors 000719 1758824273.592065 trunc from 0x02248a to 0x02cecb (0xaa42 bytes) 000720 1758824273.604303 read 0x0115b3 thru 0x016f0f (0x595d bytes) 000721 1758824273.608669 fallocate 0x0175b4 thru 0x01d361 (0x5dae bytes) 000722 1758824273.612412 mapwrite 0x022253 thru 0x02c6fb (0xa4a9 bytes) 000723 1758824273.633555 write 0x039aa5 thru 0x03ffff (0x655b bytes) 000724 1758824273.634756 fallocate 0x02fc18 thru 0x03acd4 (0xb0bd bytes) 000725 1758824273.638141 punch from 0x025251 to 0x02b24b (0x5ffb bytes) 000726 1758824273.643769 MIRROR_SPLIT mirror 2 to 7 mirrors 000727 1758824273.734754 MIRROR_EXTEND to 8 mirrors 000728 1758824274.058026 trunc from 0x01505a to 0x024263 (0xf20a bytes) 000729 1758824274.079599 mapread 0x00ba31 thru 0x018d06 (0xd2d6 bytes) 000730 1758824274.094598 fallocate 0x00c568 thru 0x01b037 (0xead0 bytes) 000731 1758824274.098656 trunc from 0x02038a to 0x02b940 (0xb5b7 bytes) 000732 1758824274.102831 read 0x026f9b thru 0x034ac6 (0xdb2c bytes) 000733 1758824274.107255 write 0x000bd5 thru 0x000c2f (0x5b bytes) 000734 1758824274.113360 fallocate 0x0218ad thru 0x02e51c (0xcc70 bytes) 000735 1758824274.116383 trunc from 0x00d926 to 0x0179bd (0xa098 bytes) 000736 1758824274.119780 read 0x0368ae thru 0x03ffff (0x9752 bytes) 000737 1758824274.129504 MIRROR_SPLIT mirror 6 to 7 mirrors 000738 1758824274.235116 write 0x01987f thru 0x028b23 (0xf2a5 bytes) 000739 1758824274.243367 mapwrite 0x00b48a thru 0x015324 (0x9e9b bytes) 000740 1758824274.263046 trunc from 0x02128d to 0x024679 (0x33ed bytes) 000741 1758824274.266897 read 0x034c6a thru 0x034e4c (0x1e3 bytes) 000742 1758824274.270216 mapwrite 0x00efd8 thru 0x01c72a (0xd753 bytes) 000743 1758824274.281995 fallocate 0x0355e7 thru 0x03ffff (0xaa19 bytes) 000744 1758824274.284999 mapread 0x03a6c6 thru 0x03ffff (0x593a bytes) 000746 1758824274.291174 mapwrite 0x001611 thru 0x00b640 (0xa030 bytes) 000748 1758824274.306949 read 0x000b43 thru 0x005cb5 (0x5173 bytes) 000749 1758824274.312680 mapread 0x01f449 thru 0x0211df (0x1d97 bytes) 000750 1758824274.316403 MIRROR_RESYNC 7 mirrors 000751 1758824274.893074 mapwrite 0x03b4ac thru 0x03ffff (0x4b54 bytes) 000752 1758824274.942594 mapread 0x02b7c7 thru 0x039fe7 (0xe821 bytes) 000753 1758824274.949895 fallocate 0x011f77 thru 0x014753 (0x27dd bytes) 000754 1758824274.953304 punch from 0x005bc3 to 0x015a44 (0xfe82 bytes) 000755 1758824274.956579 write 0x038379 thru 0x03ffff (0x7c87 bytes) 000756 1758824274.957676 mapwrite 0x015c03 thru 0x017580 (0x197e bytes) 000758 1758824274.971920 punch from 0x002463 to 0x0072f1 (0x4e8f bytes) 000759 1758824274.975157 punch from 0x030a21 to 0x034e5a (0x443a bytes) 000760 1758824274.977994 trunc from 0x040000 to 0x0276c3 (0x1893c bytes) 000761 1758824274.993571 punch from 0x01418d to 0x0218bb (0xd72f bytes) 000762 1758824274.996554 write 0x010d38 thru 0x01ebdf (0xdea8 bytes) 000763 1758824275.000116 read 0x00beba thru 0x019d28 (0xde6f bytes) 000764 1758824275.002944 read 0x02577c thru 0x0276c3 (0x1f48 bytes) 000765 1758824275.003506 trunc from 0x001a2c to 0x00e56c (0xcb41 bytes) 000766 1758824275.008949 MIRROR_EXTEND to 8 mirrors 000767 1758824275.277905 MIRROR_SPLIT mirror 4 to 7 mirrors 000768 1758824275.416224 trunc from 0x012f84 to 0x01e51c (0xb599 bytes) 000769 1758824275.433138 trunc from 0x0276c4 to 0x006b0e (0x20bb5 bytes) 000770 1758824275.444893 punch from 0x001e9b to 0x006b0e (0x4c74 bytes) 000771 1758824275.448563 mapread 0x000748 thru 0x004a41 (0x42fa bytes) 000772 1758824275.459325 mapread 0x000356 thru 0x006b0e (0x67b9 bytes) 000773 1758824275.460198 write 0x035f14 thru 0x03937b (0x3468 bytes) 000774 1758824275.461329 trunc from 0x03937c to 0x00ee69 (0x2a512 bytes) 000775 1758824275.472008 mapwrite 0x005f55 thru 0x00ce7c (0x6f28 bytes) 000776 1758824275.490400 punch from 0x0054be to 0x00b255 (0x5d98 bytes) 000777 1758824275.494822 punch from 0x001483 to 0x0023fb (0xf79 bytes) 000778 1758824275.500073 fallocate 0x02bc89 thru 0x03b5d0 (0xf948 bytes) 000780 1758824275.503865 fallocate 0x03626c thru 0x036feb (0xd80 bytes) 000781 1758824275.507215 read 0x01b284 thru 0x01c3ae (0x112b bytes) 000782 1758824275.511059 write 0x03b735 thru 0x03f59f (0x3e6b bytes) 000783 1758824275.515950 trunc from 0x03843d to 0x03f59f (0x7163 bytes) 000784 1758824275.522536 fallocate 0x025b72 thru 0x02cc17 (0x70a6 bytes) 000785 1758824275.525338 fallocate 0x02bca7 thru 0x033759 (0x7ab3 bytes) 000786 1758824275.528218 read 0x01fa11 thru 0x02ebca (0xf1ba bytes) 000787 1758824275.533519 trunc from 0x03f5a0 to 0x0288a3 (0x16cfc bytes) 000788 1758824275.540419 mapread 0x012195 thru 0x016b61 (0x49cd bytes) 000789 1758824275.543289 MIRROR_EXTEND to 8 mirrors 000790 1758824275.772681 fallocate 0x02c282 thru 0x0307e6 (0x4565 bytes) 000791 1758824275.786800 fallocate 0x03d1e5 thru 0x03f5b5 (0x23d1 bytes) 000792 1758824275.789251 fallocate 0x01b7d4 thru 0x025507 (0x9d34 bytes) 000794 1758824275.792156 trunc from 0x02bee7 to 0x02d17f (0x1299 bytes) 000795 1758824275.794374 write 0x021781 thru 0x02e84e (0xd0ce bytes) 000796 1758824275.800712 punch from 0x02ef94 to 0x0307e6 (0x1853 bytes) 000797 1758824275.808282 MIRROR_SPLIT mirror 4 to 7 mirrors 000798 1758824275.881739 trunc from 0x0307e7 to 0x01382a (0x1cfbc bytes) 000799 1758824275.900498 trunc from 0x01382b to 0x018beb (0x53c1 bytes) 000800 1758824275.906999 fallocate 0x018edf thru 0x01d4d0 (0x45f2 bytes) 000802 1758824275.910423 fallocate 0x016cdc thru 0x022b22 (0xbe47 bytes) 000803 1758824275.913781 trunc from 0x01058f to 0x018beb (0x865d bytes) 000804 1758824275.917146 mapwrite 0x01d942 thru 0x020607 (0x2cc6 bytes) 000805 1758824275.937513 fallocate 0x034b02 thru 0x03c3c5 (0x78c4 bytes) 000807 1758824275.940571 trunc from 0x00f92d to 0x015f20 (0x65f4 bytes) 000809 1758824275.943743 mapwrite 0x01f4d0 thru 0x0243c0 (0x4ef1 bytes) 000810 1758824275.961457 read 0x00fc09 thru 0x013e7d (0x4275 bytes) 000811 1758824275.965137 trunc from 0x0243c1 to 0x01f56f (0x4e51 bytes) 000812 1758824275.971760 read 0x00273d thru 0x00f9d0 (0xd294 bytes) 000813 1758824275.976035 punch from 0x0019c8 to 0x010b84 (0xf1bd bytes) 000814 1758824275.979291 punch from 0x001709 to 0x0113b1 (0xfca9 bytes) 000816 1758824275.982392 trunc from 0x01f570 to 0x003ed9 (0x1b696 bytes) 000817 1758824275.990594 MIRROR_SPLIT mirror 4 to 6 mirrors 000818 1758824276.056920 trunc from 0x003eda to 0x009ac2 (0x5be9 bytes) 000819 1758824276.065794 write 0x03ae59 thru 0x03ed04 (0x3eac bytes) 000820 1758824276.066874 read 0x01b8ff thru 0x0201c6 (0x48c8 bytes) 000822 1758824276.071620 trunc from 0x03c590 to 0x03ed04 (0x2775 bytes) 000823 1758824276.081118 MIRROR_EXTEND to 7 mirrors 000824 1758824276.314771 trunc from 0x03ed05 to 0x019835 (0x254cf bytes) 000825 1758824276.347188 read 0x0062bb thru 0x007193 (0xed9 bytes) 000826 1758824276.351554 mapwrite 0x03711b thru 0x03af66 (0x3e4c bytes) 000827 1758824276.382576 trunc from 0x03af67 to 0x00a2bb (0x30cab bytes) 000828 1758824276.394290 mapwrite 0x0026ea thru 0x004361 (0x1c78 bytes) 000829 1758824276.406612 trunc from 0x00a2bc to 0x025bb4 (0x1b8f9 bytes) 000830 1758824276.414481 trunc from 0x025bb5 to 0x01a9f1 (0xb1c3 bytes) 000831 1758824276.423445 trunc from 0x011ec5 to 0x0151ba (0x32f6 bytes) 000832 1758824276.427148 mapwrite 0x0397d4 thru 0x03ffff (0x682c bytes) 000833 1758824276.452926 read 0x01ddee thru 0x02c2e7 (0xe4fa bytes) 000834 1758824276.457918 mapwrite 0x03ccc1 thru 0x03ffff (0x333f bytes) 000835 1758824276.466645 trunc from 0x03fb0e to 0x03ffff (0x4f2 bytes) 000837 1758824276.470141 trunc from 0x040000 to 0x028e1c (0x171e3 bytes) 000838 1758824276.482057 MIRROR_RESYNC 7 mirrors 000840 1758824276.914103 MIRROR_EXTEND to 8 mirrors 000841 1758824277.214649 punch from 0x0048d3 to 0x00bd1e (0x744c bytes) 000842 1758824277.235359 punch from 0x00d8c2 to 0x018c8c (0xb3cb bytes) 000843 1758824277.238688 punch from 0x01ccf7 to 0x024f80 (0x828a bytes) 000844 1758824277.241521 punch from 0x00d4f3 to 0x00e7f2 (0x1300 bytes) 000846 1758824277.244782 fallocate 0x039c50 thru 0x03ffff (0x63b0 bytes) 000847 1758824277.249320 write 0x015866 thru 0x0210ab (0xb846 bytes) 000848 1758824277.257523 write 0x03ed68 thru 0x03ffff (0x1298 bytes) 000849 1758824277.258298 punch from 0x03fc8e to 0x03ffff (0x372 bytes) 000850 1758824277.264704 mapread 0x01984d thru 0x029282 (0xfa36 bytes) 000851 1758824277.269729 trunc from 0x040000 to 0x03e4e5 (0x1b1a bytes) 000852 1758824277.289154 punch from 0x02de20 to 0x03597b (0x7b5c bytes) 000853 1758824277.291825 mapwrite 0x01f62b thru 0x02eeb9 (0xf88f bytes) 000854 1758824277.310983 fallocate 0x034b23 thru 0x03ffff (0xb4dd bytes) 000855 1758824277.316191 trunc from 0x03e4e6 to 0x005939 (0x38bac bytes) 000856 1758824277.324481 write 0x02d285 thru 0x030819 (0x3595 bytes) 000857 1758824277.326046 punch from 0x028eb0 to 0x02a969 (0x1aba bytes) 000858 1758824277.329399 read 0x02e66a thru 0x030819 (0x21b0 bytes) 000859 1758824277.330006 trunc from 0x028ab4 to 0x029de9 (0x1336 bytes) 000860 1758824277.336584 MIRROR_EXTEND to 9 mirrors 000861 1758824277.596879 trunc from 0x00096f to 0x00c499 (0xbb2b bytes) 000863 1758824277.625571 write 0x016603 thru 0x0191d8 (0x2bd6 bytes) 000864 1758824277.633162 write 0x023c03 thru 0x0305f0 (0xc9ee bytes) 000865 1758824277.642437 mapread 0x0225fd thru 0x02d66a (0xb06e bytes) 000866 1758824277.643422 read 0x02406c thru 0x02b3c4 (0x7359 bytes) 000867 1758824277.644021 trunc from 0x005d6c to 0x013e29 (0xe0be bytes) 000868 1758824277.647505 write 0x017e38 thru 0x020da2 (0x8f6b bytes) 000869 1758824277.648865 mapwrite 0x03c0fb thru 0x03ffff (0x3f05 bytes) 000870 1758824277.686430 mapwrite 0x005395 thru 0x014a0c (0xf678 bytes) 000871 1758824277.713422 fallocate 0x02eef6 thru 0x0340a4 (0x51af bytes) 000872 1758824277.717237 read 0x00e3fe thru 0x01c3c1 (0xdfc4 bytes) 000873 1758824277.717912 mapread 0x034fc5 thru 0x03f841 (0xa87d bytes) 000874 1758824277.724615 mapread 0x0214e8 thru 0x0270e1 (0x5bfa bytes) 000875 1758824277.725528 trunc from 0x00f0b8 to 0x013af2 (0x4a3b bytes) 000877 1758824277.729869 write 0x0143c3 thru 0x01aa40 (0x667e bytes) 000878 1758824277.731116 mapwrite 0x016140 thru 0x018698 (0x2559 bytes) 000879 1758824277.744738 punch from 0x0133f8 to 0x01a690 (0x7299 bytes) 000880 1758824277.749637 write 0x003160 thru 0x008236 (0x50d7 bytes) 000881 1758824277.750615 read 0x0062cd thru 0x0082a7 (0x1fdb bytes) 000882 1758824277.751167 trunc from 0x03c5a7 to 0x03ffff (0x3a59 bytes) 000883 1758824277.756168 fallocate 0x023109 thru 0x023fa6 (0xe9e bytes) 000884 1758824277.760002 mapread 0x02ad10 thru 0x037250 (0xc541 bytes) 000885 1758824277.760900 write 0x019911 thru 0x01ee40 (0x5530 bytes) 000886 1758824277.766554 read 0x01cd48 thru 0x02b97d (0xec36 bytes) 000887 1758824277.767191 mapread 0x003d2b thru 0x003f30 (0x206 bytes) 000888 1758824277.767748 read 0x03980e thru 0x03c5c5 (0x2db8 bytes) 000889 1758824277.768155 mapread 0x02ed04 thru 0x0311b5 (0x24b2 bytes) 000890 1758824277.769128 trunc from 0x03d252 to 0x03ed71 (0x1b20 bytes) 000891 1758824277.773186 punch from 0x00a463 to 0x00a9c4 (0x562 bytes) 000892 1758824277.776313 mapread 0x036681 thru 0x03ffff (0x997f bytes) 000893 1758824277.783141 trunc from 0x040000 to 0x025208 (0x1adf7 bytes) 000894 1758824277.800380 MIRROR_SPLIT mirror 7 to 8 mirrors 000895 1758824277.903628 read 0x00b080 thru 0x00cb2a (0x1aab bytes) 000896 1758824277.906565 trunc from 0x01fcd1 to 0x0228e2 (0x2c12 bytes) 000897 1758824277.910008 fallocate 0x034ead thru 0x03d7c4 (0x8918 bytes) 000898 1758824277.917027 MIRROR_EXTEND to 9 mirrors 000899 1758824278.260960 punch from 0x00e0d2 to 0x01acbb (0xcbea bytes) 000900 1758824278.278172 read 0x0119b8 thru 0x01c2e1 (0xa92a bytes) 000901 1758824278.281979 mapread 0x0133de thru 0x01ce88 (0x9aab bytes) 000902 1758824278.282778 write 0x00b9e4 thru 0x013b47 (0x8164 bytes) 000903 1758824278.290342 MIRROR_EXTEND to 10 mirrors 000904 1758824278.579598 write 0x01c2e4 thru 0x024104 (0x7e21 bytes) 000905 1758824278.603582 mapread 0x0017dd thru 0x00499e (0x31c2 bytes) 000906 1758824278.608112 write 0x011309 thru 0x016583 (0x527b bytes) 000907 1758824278.614129 MIRROR_SPLIT mirror 2 to 9 mirrors 000908 1758824278.695732 read 0x01f5f4 thru 0x0235d9 (0x3fe6 bytes) 000910 1758824278.702047 punch from 0x0010f3 to 0x004763 (0x3671 bytes) 000911 1758824278.704928 mapread 0x0001d5 thru 0x00be96 (0xbcc2 bytes) 000912 1758824278.715865 read 0x003e1a thru 0x0115aa (0xd791 bytes) 000913 1758824278.720526 mapwrite 0x005172 thru 0x00ca75 (0x7904 bytes) 000914 1758824278.731353 punch from 0x0038ec to 0x0055fc (0x1d11 bytes) 000915 1758824278.734922 trunc from 0x025209 to 0x0068ce (0x1e93a bytes) 000916 1758824278.755543 trunc from 0x001318 to 0x0023c4 (0x10ad bytes) 000917 1758824278.758611 punch from 0x003437 to 0x0068ce (0x3498 bytes) 000918 1758824278.762622 write 0x00c44c thru 0x01a936 (0xe4eb bytes) 000919 1758824278.764103 trunc from 0x008f29 to 0x00d25d (0x4335 bytes) 000920 1758824278.772307 mapread 0x013e03 thru 0x01a936 (0x6b34 bytes) 000921 1758824278.775371 MIRROR_EXTEND to 10 mirrors 000922 1758824278.985713 trunc from 0x01a937 to 0x020758 (0x5e22 bytes) 000923 1758824279.008865 fallocate 0x035a69 thru 0x03ffff (0xa597 bytes) 000924 1758824279.011189 trunc from 0x001fbb to 0x0104b0 (0xe4f6 bytes) 000926 1758824279.013811 mapread 0x01dcaf thru 0x020758 (0x2aaa bytes) 000927 1758824279.020627 mapwrite 0x01dc70 thru 0x01f1e0 (0x1571 bytes) 000928 1758824279.027131 punch from 0x00421d to 0x008316 (0x40fa bytes) 000929 1758824279.029511 fallocate 0x012ca7 thru 0x01d2ef (0xa649 bytes) 000930 1758824279.031950 fallocate 0x03785e thru 0x03ffff (0x87a2 bytes) 000932 1758824279.034863 trunc from 0x040000 to 0x00f395 (0x30c6a bytes) 000933 1758824279.043194 trunc from 0x00f396 to 0x03f134 (0x2fd9f bytes) 000934 1758824279.050398 trunc from 0x03ba7e to 0x03be55 (0x3d8 bytes) 000935 1758824279.054805 MIRROR_EXTEND to 11 mirrors 000936 1758824279.305779 mapread 0x01b3c8 thru 0x027904 (0xc53d bytes) 000938 1758824279.316345 write 0x03309b thru 0x034733 (0x1699 bytes) 000940 1758824279.334386 trunc from 0x0086ef to 0x013d8b (0xb69d bytes) 000943 1758824279.339110 MIRROR_RESYNC 11 mirrors 000944 1758824280.024890 MIRROR_RESYNC 11 mirrors 000946 1758824280.184564 MIRROR_SPLIT mirror 7 to 10 mirrors 000947 1758824280.360273 write 0x02789d thru 0x029829 (0x1f8d bytes) 000949 1758824280.387884 mapwrite 0x03e443 thru 0x03ffff (0x1bbd bytes) 000951 1758824280.415531 write 0x02c959 thru 0x03aafa (0xe1a2 bytes) 000952 1758824280.422884 punch from 0x03aebb to 0x03ffff (0x5145 bytes) 000953 1758824280.434355 punch from 0x024d80 to 0x02af81 (0x6202 bytes) 000954 1758824280.439418 write 0x03bb59 thru 0x03d915 (0x1dbd bytes) 000955 1758824280.444741 mapwrite 0x02cff7 thru 0x02eeba (0x1ec4 bytes) 000956 1758824280.457012 read 0x02177a thru 0x02f8b8 (0xe13f bytes) 000957 1758824280.461575 punch from 0x002efe to 0x0057ef (0x28f2 bytes) 000959 1758824280.466455 write 0x0240bc thru 0x02d791 (0x96d6 bytes) 000960 1758824280.470113 MIRROR_RESYNC 10 mirrors 000961 1758824281.172544 MIRROR_EXTEND to 11 mirrors 000962 1758824281.525513 mapwrite 0x026bc6 thru 0x035359 (0xe794 bytes) 000963 1758824281.579931 mapwrite 0x02f942 thru 0x03560d (0x5ccc bytes) 000964 1758824281.591968 mapread 0x009931 thru 0x00ae65 (0x1535 bytes) 000965 1758824281.598135 mapread 0x030b9b thru 0x03871a (0x7b80 bytes) 000966 1758824281.599114 read 0x03d98f thru 0x03dd64 (0x3d6 bytes) 000967 1758824281.603455 mapread 0x010d35 thru 0x01330f (0x25db bytes) 000968 1758824281.609297 mapwrite 0x020959 thru 0x02debf (0xd567 bytes) 000969 1758824281.624677 MIRROR_RESYNC 11 mirrors 000970 1758824282.446658 read 0x0398ec thru 0x03ffff (0x6714 bytes) 000971 1758824282.451412 write 0x005655 thru 0x00b7e2 (0x618e bytes) 000972 1758824282.471985 fallocate 0x02d19e thru 0x03beeb (0xed4e bytes) 000974 1758824282.475347 write 00000000 thru 0x006fb0 (0x6fb1 bytes) 000975 1758824282.476544 trunc from 0x0077f2 to 0x0146a3 (0xceb2 bytes) 000976 1758824282.483145 mapwrite 0x001562 thru 0x004da5 (0x3844 bytes) 000977 1758824282.491501 mapwrite 0x02468b thru 0x031b95 (0xd50b bytes) 000978 1758824282.511987 MIRROR_RESYNC 11 mirrors 000979 1758824283.132005 MIRROR_RESYNC 11 mirrors 000980 1758824283.252288 punch from 0x01999d to 0x01b16c (0x17d0 bytes) 000981 1758824283.266653 read 0x01ffe1 thru 0x020eb0 (0xed0 bytes) 000982 1758824283.269246 read 0x02a437 thru 0x02e137 (0x3d01 bytes) 000983 1758824283.273726 mapread 0x006e14 thru 0x01234c (0xb539 bytes) 000986 1758824283.275467 mapread 0x0235b1 thru 0x024767 (0x11b7 bytes) 000987 1758824283.276100 fallocate 0x01d420 thru 0x023705 (0x62e6 bytes) 000988 1758824283.279022 punch from 0x018c12 to 0x01fd24 (0x7113 bytes) 000989 1758824283.281507 write 0x00a2f7 thru 0x018e3a (0xeb44 bytes) 000990 1758824283.284411 mapread 0x01e207 thru 0x02db25 (0xf91f bytes) 000991 1758824283.288653 trunc from 0x00c09c to 0x012992 (0x68f7 bytes) 000992 1758824283.294596 write 0x028a9d thru 0x02d585 (0x4ae9 bytes) 000993 1758824283.295580 trunc from 0x035e14 to 0x03b32d (0x551a bytes) 000994 1758824283.298319 read 0x0192f2 thru 0x01dd0c (0x4a1b bytes) 000995 1758824283.302715 trunc from 0x040000 to 0x00e5be (0x31a41 bytes) 000996 1758824283.317442 mapwrite 0x02d044 thru 0x03394c (0x6909 bytes) 000997 1758824283.348569 MIRROR_SPLIT mirror 7 to 10 mirrors 000998 1758824283.417773 trunc from 0x0309c4 to 0x03394c (0x2f89 bytes) 000999 1758824283.420785 trunc from 0x03394d to 0x0117fd (0x2214f bytes) All operations completed A-OK! PASS 70a (53s) == sanity-flr test 71: check mirror extend parent fid ==== 14:18:05 (1758824285) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x35:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 200a: stress test ===================== 14:18:09 (1758824289) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1936448 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4843136 .. failed Extending file size to 6924832 .. Extending file size to 8210688 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7898688 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8908960 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8885280 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 60761 60762 60763 60767 60768 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:19:15 (1758824355) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6821600 .. Extending file size to 5423872 .. done Extending file size to 6065344 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7642784 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8750464 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8623808 .. Extending file size to 9248000 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8813408 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8908352 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9273632 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 64160 64161 64162 64167 64168 done 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:20:21 (1758824421) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3508 1284180 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1928 1285760 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21724 3585296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15872 3591148 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37596 7176444 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3508 1284180 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1928 1285760 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 21724 3585296 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 15872 3591148 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37596 7176444 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0487761 s, 43.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0610257 s, 34.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:20:28 (1758824428) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0422173 s, 49.7 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 14:20:30 (1758824430) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x69c:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:20:33 (1758824433) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0352797 s, 59.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x69d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x69e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:20:36 (1758824436) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0233828 s, 89.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x38:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x39:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x37:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x38:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x39:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3a:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:20:39 (1758824439) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0354438 s, 59.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x69f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x69f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a0:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:21:04 (1758824464) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252447 s, 41.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a5:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:21:39 (1758824499) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0305677 s, 68.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6a4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (37s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:22:16 (1758824536) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg230-server: Pool lustre.test_204f created Adding targets to pool oleg230-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg230-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.026437 s, 79.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x39:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x39:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg230-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg230-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg230-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:23:05 (1758824585) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:23:08 (1758824588) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:23:11 (1758824591) oleg230-server: Pool lustre.test_206 created Adding targets to pool oleg230-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg230-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg230-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg230-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg230-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:23:28 (1758824608) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030736 s, 34.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149854 s, 70.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb4821eda Mirror 3: 0xb4821eda PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:23:31 (1758824611) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:23:32 (1758824612) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:23:33 (1758824613) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.01153 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x910000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1470000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1470000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1470000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xab0000 Read this (4 bytes): BBBB 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:23:51 (1758824631) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00454232 s, 1.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:23:53 (1758824633) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178768 s, 58.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x4d7 lmm_fid: [0x200000405:0x4d7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x4d7 lmm_fid: [0x200000405:0x4d7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x6af:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612375 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 14:23:54.000000000 -0400 Modify: 2025-09-25 14:23:54.000000000 -0400 Change: 2025-09-25 14:23:54.000000000 -0400 Birth: 2025-09-25 14:23:54.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:23:56 (1758824636) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139129 s, 75.4 MB/s oleg230-server: debugfs 1.47.1-wc2 (08-Nov-2024) fail_loc=0x1428 Waiting for MDT destroys to complete oleg230-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg230-server: O/280000401/d16/6b0: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:24:09 (1758824649) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0665569 s, 615 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00951004 s, 431 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 25 14:24 /mnt/lustre/f211.sanity-flr size after mirror delete -rw-r--r-- 1 root root 4096 Sep 25 14:24 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 913 sec ============ 14:24:12 (1758824652) === sanity-flr: start cleanup 14:24:12 (1758824652) === === sanity-flr: finish cleanup 14:24:15 (1758824655) ===