-----============= acceptance-small: sanity-flr ============----- Thu Jun 5 07:07:31 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg201-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:07:40 (1749121660) === oleg201-client.virtnet: executing check_config_client /mnt/lustre oleg201-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg201-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b38c5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b38c5800.idle_timeout=debug disable quota as required oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:07:55 (1749121675) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7950] == sanity-flr test 0a: lfs mirror create with -N option == 07:07:56 (1749121676) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:08:04 (1749121684) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:08:06 (1749121686) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:08:07 (1749121687) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:08:16 (1749121696) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:08:18 (1749121698) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 07:08:19 (1749121699) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b38c5800.stats=clear osc.lustre-OST0001-osc-ffff8800b38c5800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:08:22 (1749121702) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:08:27 (1749121707) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:08:30 (1749121710) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:08:32 (1749121712) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 07:08:36 (1749121716) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:08:39 (1749121719) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:08:42 (1749121722) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:08:46 (1749121726) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:08:50 (1749121730) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:08:53 (1749121733) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.232976 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0548025 s, 19.1 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:08:56 (1749121736) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB) copied, 0.0547247 s, 115 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:08:59 (1749121739) Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b38c5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b38c5800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b38c5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b38c5800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (36s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:09:35 (1749121775) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB) copied, 0.27831 s, 33.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b38c5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b38c5800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:09:53 (1749121793) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b38c5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b38c5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b38c5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b38c5800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (33s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:10:26 (1749121826) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.25573 s, 41.8 MB/s -rw-r--r-- 1 root root 52428800 Jun 5 07:10 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.662s user 0m0.000s sys 0m0.133s Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b38c5800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b38c5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.435s user 0m0.000s sys 0m0.115s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b38c5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b38c5800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.555s user 0m0.002s sys 0m0.129s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg201-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b38c5800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b38c5800.ost_server_uuid in FULL state after 0 sec PASS 33b (39s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:11:05 (1749121865) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:11:07 (1749121867) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:11:09 (1749121869) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 07:11:11 (1749121871) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 07:11:14 (1749121874) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.263403 s, 47.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:11:21 (1749121881) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139397 s, 75.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200721 s, 52.2 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:11:25 (1749121885) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:11:29 (1749121889) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.002028 s, 505 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00228561 s, 448 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00327103 s, 313 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0014464 s, 708 kB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 07:11:33 (1749121893) create test files with size 15732601 .. ba854729a3673b0bb5283af6667cb5c7 - b63d28130a632f0f45859e7fc6cafe03 - dbe84efc55a301a669d966485ca66b86 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 07:11:44 (1749121904) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:11:50 (1749121910) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:11:53 (1749121913) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0383319 s, 27.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0242298 s, 43.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 07:11:57 (1749121917) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0605018 s, 34.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.160855 s, 26.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0889166 s, 23.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 07:12:03 (1749121923) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 07:12:05 (1749121925) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:12:07 (1749121927) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131446 s, 79.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 07:12:10 (1749121930) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.128568 s, 24.5 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:12:14 (1749121934) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:12:18 (1749121938) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.161708 s, 64.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:12:32 (1749121952) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff88012c436800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.158556 s, 66.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a92cb800.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:12:38 (1749121958) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:12:49 (1749121969) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:13:01 (1749121981) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 07:13:02 (1749121982) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.101643 s, 30.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0625105 s, 50.3 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: a90ed15db5bb4e9a33fe9b380dc83d30 - mirror 1 checksum: f1aa7be883b8414f1843f1966b3e9173 - mirror 2 checksum: a90ed15db5bb4e9a33fe9b380dc83d30 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 5 07:13:04 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 5 07:13:04 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a90ed15db5bb4e9a33fe9b380dc83d30 - mirror 1 checksum: f1aa7be883b8414f1843f1966b3e9173 - mirror 2 checksum: f1aa7be883b8414f1843f1966b3e9173 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 5 07:13:04 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 5 07:13:04 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:13:06 (1749121986) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136305 s, 76.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:13:08 (1749121988) Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server 07:13:16 (1749121996) shut down facet: mds1 facet_host: oleg201-server facet_failover_host: oleg201-server Failover mds1 to oleg201-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 07:13:31 (1749122011) targets are mounted 07:13:31 (1749122011) facet_failover done oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107546 s, 97.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:13:39 (1749122019) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0199763 s, 1.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0138045 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:13:43 (1749122023) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0189888 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00389095 s, 5.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:13:47 (1749122027) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00328109 s, 5.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:13:49 (1749122029) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0189983 s, 1.1 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:13:52 (1749122032) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00557398 s, 735 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.018592 s, 220 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:13:54 (1749122034) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:14:01 (1749122041) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:14:21 (1749122061) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:14:38 (1749122078) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:14:56 (1749122096) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 07:14:59 (1749122099) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26775 mirror split pid 26776 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26775 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4948: 26776 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 07:16:01 (1749122161) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 2161 skipping zero size read fallocating to largest ever: 0x40000 000003 1749122161.936414 fallocate 0x03ed42 thru 0x03ffff (0x12be bytes) 000005 1749122161.944332 trunc from 00000000 to 0x02e258 (0x2e259 bytes) 000006 1749122161.955351 write 0x02ab6f thru 0x035803 (0xac95 bytes) 000007 1749122161.959096 mapread 0x027638 thru 0x0356f7 (0xe0c0 bytes) 000008 1749122161.965466 mapwrite 0x02877e thru 0x030e07 (0x868a bytes) 000009 1749122161.981814 mapread 0x00139c thru 0x00fb9b (0xe800 bytes) 000010 1749122161.988713 trunc from 0x035804 to 0x02889a (0xcf69 bytes) 000011 1749122161.996905 trunc from 0x010cb5 to 0x01a4c0 (0x980c bytes) 000013 1749122161.999617 fallocate 0x01b6d7 thru 0x02b62b (0xff55 bytes) 000014 1749122162.003342 trunc from 0x02b62c to 0x026772 (0x4eb9 bytes) 000015 1749122162.013760 trunc from 0x009900 to 0x00daba (0x41bb bytes) 000016 1749122162.017652 mapread 0x00c2a9 thru 0x0191d5 (0xcf2d bytes) 000017 1749122162.032583 write 0x031271 thru 0x03f3e0 (0xe170 bytes) 000018 1749122162.033825 punch from 0x03a950 to 0x03f25e (0x490f bytes) 000019 1749122162.042186 trunc from 0x0365c6 to 0x037619 (0x1054 bytes) 000020 1749122162.045412 mapread 0x01ed54 thru 0x021023 (0x22d0 bytes) 000021 1749122162.048896 trunc from 0x021a5f to 0x024c58 (0x31fa bytes) 000022 1749122162.052359 fallocate 0x022a6b thru 0x02e6fd (0xbc93 bytes) 000024 1749122162.062624 mapread 0x0385f7 thru 0x03f3e0 (0x6dea bytes) 000026 1749122162.070992 mapwrite 0x02a35f thru 0x032061 (0x7d03 bytes) 000027 1749122162.095543 read 0x01a857 thru 0x01bac8 (0x1272 bytes) 000028 1749122162.101614 MIRROR_EXTEND to 2 mirrors 000029 1749122162.399531 punch from 0x00e6cc to 0x01829a (0x9bcf bytes) 000031 1749122162.418194 write 0x037e4c thru 0x038891 (0xa46 bytes) 000032 1749122162.424793 trunc from 0x02e1f6 to 0x03654a (0x8355 bytes) 000033 1749122162.428165 read 0x0327a3 thru 0x035d51 (0x35af bytes) 000034 1749122162.431436 trunc from 0x03f3e1 to 0x00bcb2 (0x3372e bytes) 000035 1749122162.454579 trunc from 0x006826 to 0x00b04f (0x482a bytes) 000036 1749122162.459463 MIRROR_RESYNC 2 mirrors 000037 1749122162.651606 write 0x03ca54 thru 0x03f1af (0x275c bytes) 000038 1749122162.671170 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000040 1749122162.739747 fallocate 0x030095 thru 0x03368c (0x35f8 bytes) 000041 1749122162.744195 trunc from 0x004ac2 to 0x012016 (0xd555 bytes) 000042 1749122162.748552 mapwrite 0x03a22a thru 0x03ffff (0x5dd6 bytes) 000043 1749122162.785207 punch from 0x0151c3 to 0x018cf0 (0x3b2e bytes) 000044 1749122162.788127 trunc from 0x034be3 to 0x03ffff (0xb41d bytes) 000045 1749122162.791048 read 0x01f678 thru 0x0251aa (0x5b33 bytes) 000047 1749122162.796631 trunc from 0x02dd4d to 0x031951 (0x3c05 bytes) 000049 1749122162.801517 mapwrite 0x02bfea thru 0x03aa74 (0xea8b bytes) 000050 1749122162.829517 trunc from 0x040000 to 0x020d45 (0x1f2ba bytes) 000051 1749122162.840457 read 0x01c0ef thru 0x020d45 (0x4c57 bytes) 000052 1749122162.848302 mapwrite 0x02759c thru 0x02a104 (0x2b69 bytes) 000053 1749122162.868117 MIRROR_EXTEND to 2 mirrors 000054 1749122163.078648 read 0x007759 thru 0x00f7f8 (0x80a0 bytes) 000055 1749122163.090227 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000056 1749122163.151338 mapwrite 0x02a91e thru 0x03230b (0x79ee bytes) 000057 1749122163.196017 mapwrite 0x01069a thru 0x01c1e3 (0xbb4a bytes) 000058 1749122163.214679 mapread 0x01be17 thru 0x01cef8 (0x10e2 bytes) 000059 1749122163.215217 trunc from 0x0235b0 to 0x0272a4 (0x3cf5 bytes) 000060 1749122163.219178 read 0x00cb2b thru 0x00cf49 (0x41f bytes) 000062 1749122163.220927 trunc from 0x03230c to 0x02aa30 (0x78db bytes) 000063 1749122163.228497 mapread 0x01c2ca thru 0x029970 (0xd6a7 bytes) 000064 1749122163.233355 read 0x01e375 thru 0x020627 (0x22b3 bytes) 000065 1749122163.233991 read 0x029620 thru 0x02aa30 (0x1411 bytes) 000066 1749122163.234355 mapread 0x029e75 thru 0x02aa30 (0xbbc bytes) 000067 1749122163.235344 fallocate 0x0001ff thru 0x00f65b (0xf45d bytes) 000068 1749122163.240473 mapread 0x00d2db thru 0x018387 (0xb0ad bytes) 000069 1749122163.242439 mapwrite 0x014665 thru 0x015ad8 (0x1474 bytes) 000070 1749122163.254123 MIRROR_EXTEND to 2 mirrors 000071 1749122163.456647 trunc from 0x02635c to 0x02aa30 (0x46d5 bytes) 000072 1749122163.475477 read 0x023718 thru 0x027b79 (0x4462 bytes) 000073 1749122163.479045 read 0x002980 thru 0x00b8cd (0x8f4e bytes) 000074 1749122163.488526 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000075 1749122163.564009 fallocate 0x002089 thru 0x002a64 (0x9dc bytes) 000076 1749122163.570281 mapread 0x019432 thru 0x01f49c (0x606b bytes) 000077 1749122163.585068 punch from 0x025b46 to 0x02aa30 (0x4eeb bytes) 000080 1749122163.588324 punch from 0x0192ec to 0x02870a (0xf41f bytes) 000081 1749122163.593483 fallocate 0x0058ee thru 0x010e02 (0xb515 bytes) 000083 1749122163.600118 write 0x000d35 thru 0x0026d0 (0x199c bytes) 000084 1749122163.607443 trunc from 0x02aa31 to 0x0259f2 (0x503e bytes) 000086 1749122163.626181 punch from 0x01858b to 0x019f79 (0x19ef bytes) 000087 1749122163.628668 write 0x01d579 thru 0x0204bf (0x2f47 bytes) 000088 1749122163.634308 mapwrite 0x020430 thru 0x02daf7 (0xd6c8 bytes) 000089 1749122163.662594 read 0x016c7c thru 0x022d57 (0xc0dc bytes) 000090 1749122163.663416 punch from 0x02a950 to 0x02daf7 (0x31a8 bytes) 000091 1749122163.667140 trunc from 0x02daf8 to 0x02f4a5 (0x19ae bytes) 000092 1749122163.676973 punch from 0x018f7b to 0x0270ce (0xe154 bytes) 000093 1749122163.685696 MIRROR_EXTEND to 2 mirrors 000094 1749122163.911730 write 0x0274d4 thru 0x02feea (0x8a17 bytes) 000095 1749122163.934242 write 0x001a1c thru 0x008cd1 (0x72b6 bytes) 000096 1749122163.939363 fallocate 0x0108ab thru 0x019605 (0x8d5b bytes) 000098 1749122163.942487 read 0x019a93 thru 0x02386f (0x9ddd bytes) 000099 1749122163.948680 MIRROR_EXTEND to 3 mirrors 000100 1749122164.180704 trunc from 0x02feeb to 0x022400 (0xdaea bytes) 000101 1749122164.202252 mapwrite 0x00ad96 thru 0x017442 (0xc6ad bytes) 000102 1749122164.227845 trunc from 0x022401 to 0x00b541 (0x16ebf bytes) 000104 1749122164.237676 mapread 0x0017aa thru 0x00b541 (0x9d98 bytes) 000105 1749122164.242234 read 0x000973 thru 0x0023b2 (0x1a40 bytes) 000106 1749122164.244743 mapread 0x0094eb thru 0x00b541 (0x2057 bytes) 000107 1749122164.245674 mapwrite 0x013477 thru 0x016ae8 (0x3672 bytes) 000108 1749122164.271565 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000109 1749122164.362274 MIRROR_RESYNC 2 mirrors 000110 1749122164.528717 punch from 0x00f96c to 0x016ae8 (0x717d bytes) 000111 1749122164.547638 mapwrite 0x01007a thru 0x01589e (0x5825 bytes) 000113 1749122164.569055 write 0x00d2fc thru 0x00fa95 (0x279a bytes) 000114 1749122164.569927 write 0x024b32 thru 0x0286d5 (0x3ba4 bytes) 000115 1749122164.570698 punch from 0x00c99f to 0x00f42c (0x2a8e bytes) 000116 1749122164.580983 MIRROR_EXTEND to 3 mirrors 000117 1749122164.805835 mapread 0x026d85 thru 0x0286d5 (0x1951 bytes) 000118 1749122164.823761 punch from 0x014070 to 0x01e483 (0xa414 bytes) 000119 1749122164.839546 fallocate 0x02947f thru 0x0360cd (0xcc4f bytes) 000120 1749122164.842303 write 0x014e8b thru 0x019e64 (0x4fda bytes) 000121 1749122164.848639 punch from 0x01515b to 0x021d1c (0xcbc2 bytes) 000122 1749122164.858723 fallocate 0x00a8b6 thru 0x011c2c (0x7377 bytes) 000123 1749122164.866714 write 0x03bee4 thru 0x03ffff (0x411c bytes) 000125 1749122164.867742 punch from 0x033599 to 0x03964c (0x60b4 bytes) 000126 1749122164.872195 punch from 0x001a19 to 0x00240b (0x9f3 bytes) 000127 1749122164.876634 mapwrite 0x00ea94 thru 0x019c15 (0xb182 bytes) 000128 1749122164.901338 write 0x02e1af thru 0x0314f4 (0x3346 bytes) 000129 1749122164.908746 trunc from 0x016701 to 0x0200aa (0x99aa bytes) 000130 1749122164.915132 mapwrite 0x022392 thru 0x027bdf (0x584e bytes) 000131 1749122164.933716 trunc from 0x0270b5 to 0x02aaf6 (0x3a42 bytes) 000132 1749122164.938350 trunc from 0x040000 to 0x02ebee (0x11411 bytes) 000133 1749122164.958357 mapread 0x00d0d9 thru 0x01216d (0x5095 bytes) 000134 1749122164.964305 fallocate 0x028fd5 thru 0x02c7b4 (0x37e0 bytes) 000136 1749122164.967485 punch from 0x0054a2 to 0x00b34d (0x5eac bytes) 000137 1749122164.973868 trunc from 0x02ebef to 0x03bbe9 (0xcffb bytes) 000138 1749122164.982395 write 0x0276d4 thru 0x02e9bf (0x72ec bytes) 000139 1749122164.988507 trunc from 0x00d6c1 to 0x017a01 (0xa341 bytes) 000140 1749122164.991939 mapwrite 0x023663 thru 0x032756 (0xf0f4 bytes) 000142 1749122165.016067 punch from 0x02b852 to 0x030b95 (0x5344 bytes) 000143 1749122165.019347 trunc from 0x03bbea to 0x00d281 (0x2e968 bytes) 000144 1749122165.029791 trunc from 0x00d282 to 0x02bd31 (0x1eab0 bytes) 000145 1749122165.036770 fallocate 0x00c118 thru 0x010a33 (0x491c bytes) 000147 1749122165.039760 mapread 0x01d347 thru 0x022895 (0x554f bytes) 000148 1749122165.044540 trunc from 0x02bd32 to 0x0250ce (0x6c63 bytes) 000149 1749122165.050039 punch from 0x00aa18 to 0x017578 (0xcb61 bytes) 000150 1749122165.053198 write 0x039ec1 thru 0x03ffff (0x613f bytes) 000152 1749122165.054714 punch from 0x00672c to 0x008c31 (0x2506 bytes) 000153 1749122165.059610 punch from 0x004044 to 0x005066 (0x1023 bytes) 000154 1749122165.064043 trunc from 0x040000 to 0x014ef1 (0x2b10e bytes) 000155 1749122165.070922 write 0x01f2c3 thru 0x02eabb (0xf7f9 bytes) 000156 1749122165.075900 MIRROR_EXTEND to 4 mirrors 000157 1749122165.302153 fallocate 0x022591 thru 0x023bcf (0x163f bytes) 000158 1749122165.316084 trunc from 0x00062d to 0x00cf66 (0xc93a bytes) 000159 1749122165.318088 read 0x021390 thru 0x02c4d2 (0xb143 bytes) 000160 1749122165.322029 punch from 0x024cae to 0x02eabb (0x9e0e bytes) 000162 1749122165.324781 trunc from 0x02eabc to 0x014c14 (0x19ea7 bytes) 000163 1749122165.334909 punch from 0x002eb6 to 0x00cd2b (0x9e76 bytes) 000164 1749122165.337739 trunc from 0x014c15 to 0x016618 (0x1a04 bytes) 000165 1749122165.344636 fallocate 0x014012 thru 0x018404 (0x43f3 bytes) 000166 1749122165.348524 write 0x034863 thru 0x03b4f1 (0x6c8f bytes) 000167 1749122165.349574 trunc from 0x03b4f2 to 0x012a84 (0x28a6d bytes) 000168 1749122165.359187 mapwrite 0x01951a thru 0x0252dc (0xbdc3 bytes) 000169 1749122165.390528 read 0x00aa2e thru 0x017f67 (0xd53a bytes) 000170 1749122165.394468 trunc from 0x0252dd to 0x038db5 (0x13ad9 bytes) 000171 1749122165.401139 mapwrite 0x01d5a1 thru 0x02bcb8 (0xe718 bytes) 000172 1749122165.417712 fallocate 0x01ac98 thru 0x01e2ea (0x3653 bytes) 000174 1749122165.420714 trunc from 0x038db6 to 0x039462 (0x6ad bytes) 000175 1749122165.427262 mapread 0x0354d8 thru 0x039462 (0x3f8b bytes) 000176 1749122165.430285 trunc from 0x006906 to 0x012500 (0xbbfb bytes) 000177 1749122165.432994 mapwrite 0x03bfb3 thru 0x03ffff (0x404d bytes) 000179 1749122165.460699 write 0x02e8dc thru 0x03aa6f (0xc194 bytes) 000180 1749122165.462312 read 0x01cd0e thru 0x02194f (0x4c42 bytes) 000181 1749122165.463124 trunc from 0x022e45 to 0x023731 (0x8ed bytes) 000182 1749122165.467439 punch from 0x034b71 to 0x038e8a (0x431a bytes) 000183 1749122165.477393 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000184 1749122165.581691 write 0x0358df thru 0x03a7be (0x4ee0 bytes) 000185 1749122165.588041 write 0x00953d thru 0x014804 (0xb2c8 bytes) 000186 1749122165.595591 fallocate 0x01f773 thru 0x02af29 (0xb7b7 bytes) 000187 1749122165.599586 fallocate 0x022ece thru 0x02bf24 (0x9057 bytes) 000191 1749122165.604429 fallocate 0x008e52 thru 0x00a145 (0x12f4 bytes) 000192 1749122165.607489 fallocate 0x013445 thru 0x013587 (0x143 bytes) 000193 1749122165.610361 mapread 0x027216 thru 0x028fcc (0x1db7 bytes) 000194 1749122165.616174 trunc from 0x040000 to 0x006e58 (0x391a7 bytes) 000195 1749122165.623707 write 0x001bc7 thru 0x00db3d (0xbf77 bytes) 000196 1749122165.626980 punch from 0x008c50 to 0x00db3d (0x4eee bytes) 000197 1749122165.633660 mapread 0x00152d thru 0x00a273 (0x8d47 bytes) 000198 1749122165.639979 trunc from 0x00db3e to 0x01f567 (0x11a2a bytes) 000199 1749122165.651386 trunc from 0x010654 to 0x010f90 (0x93d bytes) 000200 1749122165.655473 MIRROR_EXTEND to 4 mirrors 000201 1749122165.908012 trunc from 0x009139 to 0x01770a (0xe5d2 bytes) 000202 1749122165.930753 trunc from 0x01cf36 to 0x01dce9 (0xdb4 bytes) 000203 1749122165.933270 mapwrite 0x011f77 thru 0x019539 (0x75c3 bytes) 000204 1749122165.956392 write 0x02058b thru 0x02adc2 (0xa838 bytes) 000205 1749122165.960020 write 0x00d5d1 thru 0x0130d3 (0x5b03 bytes) 000206 1749122165.968289 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000207 1749122166.071164 trunc from 0x02adc3 to 0x03b193 (0x103d1 bytes) 000208 1749122166.091950 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000209 1749122166.196842 punch from 0x029b45 to 0x037089 (0xd545 bytes) 000210 1749122166.201919 fallocate 0x0054a9 thru 0x010b02 (0xb65a bytes) 000211 1749122166.205585 trunc from 0x03b194 to 0x015bda (0x255b9 bytes) 000212 1749122166.214367 punch from 0x014b3c to 0x015bda (0x109f bytes) 000213 1749122166.216891 trunc from 0x015bdb to 0x019493 (0x38b9 bytes) 000214 1749122166.227030 write 0x03f3fb thru 0x03ffff (0xc05 bytes) 000215 1749122166.228582 write 0x032654 thru 0x03f59f (0xcf4c bytes) 000216 1749122166.237433 trunc from 0x040000 to 0x0009a4 (0x3f65b bytes) 000217 1749122166.245413 trunc from 0x0009a5 to 0x0237e7 (0x22e43 bytes) 000218 1749122166.253350 write 0x009f27 thru 0x00b56e (0x1648 bytes) 000219 1749122166.260516 trunc from 0x023313 to 0x0237e7 (0x4d5 bytes) 000220 1749122166.263292 mapread 0x00870b thru 0x0146c7 (0xbfbd bytes) 000221 1749122166.270448 trunc from 0x017d66 to 0x01fd02 (0x7f9d bytes) 000222 1749122166.274734 mapread 0x011e2f thru 0x0178f9 (0x5acb bytes) 000223 1749122166.282667 write 0x02a038 thru 0x02cb5e (0x2b27 bytes) 000224 1749122166.283437 trunc from 0x01f0cc to 0x02af62 (0xbe97 bytes) 000226 1749122166.298685 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000227 1749122166.372289 write 0x038846 thru 0x03a830 (0x1feb bytes) 000228 1749122166.377595 write 0x017ef7 thru 0x023578 (0xb682 bytes) 000229 1749122166.386900 mapwrite 0x000616 thru 0x008065 (0x7a50 bytes) 000230 1749122166.409954 mapwrite 0x024d1d thru 0x029f79 (0x525d bytes) 000231 1749122166.428249 write 0x03d941 thru 0x03ffff (0x26bf bytes) 000232 1749122166.429250 trunc from 0x040000 to 0x02a67c (0x15983 bytes) 000233 1749122166.435247 mapread 0x007065 thru 0x0159b3 (0xe94f bytes) 000234 1749122166.441745 trunc from 0x02a67d to 0x0025c4 (0x280b8 bytes) 000236 1749122166.454201 mapwrite 0x008a6f thru 0x015b77 (0xd109 bytes) 000237 1749122166.492798 read 0x0088db thru 0x009e5d (0x1583 bytes) 000238 1749122166.493634 trunc from 0x015b78 to 0x0171f1 (0x167a bytes) 000239 1749122166.501972 trunc from 0x0171f2 to 0x0040e5 (0x1310c bytes) 000240 1749122166.511523 mapread 0x002d67 thru 0x0040e5 (0x137f bytes) 000241 1749122166.513119 trunc from 0x001e13 to 0x0040e5 (0x22d3 bytes) 000242 1749122166.521432 trunc from 0x002f09 to 0x0040e5 (0x11dd bytes) 000243 1749122166.526497 write 0x02c7a5 thru 0x03891e (0xc17a bytes) 000244 1749122166.527602 trunc from 0x01448f to 0x019192 (0x4d04 bytes) 000245 1749122166.530546 read 0x026d5d thru 0x02b986 (0x4c2a bytes) 000246 1749122166.535423 mapwrite 0x014ef8 thru 0x01fd8b (0xae94 bytes) 000247 1749122166.552481 mapread 0x02fcd3 thru 0x037669 (0x7997 bytes) 000250 1749122166.557777 MIRROR_EXTEND to 2 mirrors 000251 1749122166.828978 MIRROR_RESYNC 2 mirrors 000252 1749122166.942018 mapwrite 0x028930 thru 0x02aac6 (0x2197 bytes) 000253 1749122167.010128 mapread 0x036292 thru 0x03891e (0x268d bytes) 000254 1749122167.014493 read 0x0235dd thru 0x030279 (0xcc9d bytes) 000255 1749122167.015140 mapwrite 0x02c75d thru 0x037e46 (0xb6ea bytes) 000257 1749122167.033381 MIRROR_EXTEND to 3 mirrors 000258 1749122167.280807 punch from 0x01720e to 0x02443f (0xd232 bytes) 000259 1749122167.300171 write 0x03103f thru 0x031cd5 (0xc97 bytes) 000260 1749122167.307641 read 0x02ac4a thru 0x03891e (0xdcd5 bytes) 000261 1749122167.316116 punch from 0x034df6 to 0x03891e (0x3b29 bytes) 000262 1749122167.324101 fallocate 0x039b46 thru 0x03de9a (0x4355 bytes) 000263 1749122167.328504 mapread 0x039b69 thru 0x03b43f (0x18d7 bytes) 000264 1749122167.339570 trunc from 0x034069 to 0x03de9a (0x9e32 bytes) 000265 1749122167.342121 trunc from 0x027d28 to 0x02ab2e (0x2e07 bytes) 000266 1749122167.345532 mapwrite 0x033948 thru 0x03fb96 (0xc24f bytes) 000267 1749122167.389308 read 0x0033a8 thru 0x011aa7 (0xe700 bytes) 000268 1749122167.398841 punch from 0x005a6d to 0x00f0cd (0x9661 bytes) 000270 1749122167.405454 fallocate 0x026d2e thru 0x02f1ca (0x849d bytes) 000271 1749122167.408025 punch from 0x00f20e to 0x015652 (0x6445 bytes) 000272 1749122167.410462 punch from 0x035db7 to 0x03d560 (0x77aa bytes) 000273 1749122167.414874 MIRROR_EXTEND to 4 mirrors 000275 1749122167.661554 write 0x031a59 thru 0x03a692 (0x8c3a bytes) 000276 1749122167.685783 trunc from 0x03fb97 to 0x0369e7 (0x91af bytes) 000277 1749122167.707598 mapread 0x01a1fe thru 0x01f0ff (0x4f02 bytes) 000280 1749122167.712029 fallocate 0x03c350 thru 0x03d23b (0xeec bytes) 000281 1749122167.714472 fallocate 0x02a230 thru 0x02c233 (0x2004 bytes) 000282 1749122167.717969 mapwrite 0x00e2ed thru 0x01cb9e (0xe8b2 bytes) 000283 1749122167.740309 punch from 0x00790f to 0x008d9e (0x1490 bytes) 000284 1749122167.743166 fallocate 0x0046bb thru 0x0105bd (0xbf03 bytes) 000285 1749122167.747417 read 0x032630 thru 0x0369e7 (0x43b8 bytes) 000286 1749122167.747830 trunc from 0x0144ed to 0x023049 (0xeb5d bytes) 000287 1749122167.752795 punch from 0x01a08d to 0x027dd7 (0xdd4b bytes) 000288 1749122167.757616 write 0x0107fb thru 0x014f38 (0x473e bytes) 000289 1749122167.761420 trunc from 0x02cf79 to 0x0369e7 (0x9a6f bytes) 000290 1749122167.764640 mapwrite 0x025ae5 thru 0x02a02a (0x4546 bytes) 000291 1749122167.781649 read 0x00b490 thru 0x019daa (0xe91b bytes) 000292 1749122167.786726 trunc from 0x02d350 to 0x02edec (0x1a9d bytes) 000293 1749122167.789318 fallocate 0x02e584 thru 0x02f733 (0x11b0 bytes) 000294 1749122167.792678 fallocate 0x0242f5 thru 0x032551 (0xe25d bytes) 000295 1749122167.796631 trunc from 0x0109e6 to 0x01c849 (0xbe64 bytes) 000296 1749122167.799427 fallocate 0x027930 thru 0x029649 (0x1d1a bytes) 000297 1749122167.802225 mapread 0x01f8cf thru 0x021aa1 (0x21d3 bytes) 000299 1749122167.803935 trunc from 0x01e52c to 0x0281fd (0x9cd2 bytes) 000300 1749122167.806815 mapwrite 0x02b326 thru 0x030547 (0x5222 bytes) 000301 1749122167.826771 fallocate 0x010e4e thru 0x011b0e (0xcc1 bytes) 000302 1749122167.834864 MIRROR_RESYNC 4 mirrors 000303 1749122168.415729 read 0x01a56e thru 0x02281a (0x82ad bytes) 000304 1749122168.420255 trunc from 0x0180fe to 0x024097 (0xbf9a bytes) 000305 1749122168.443242 read 0x01d319 thru 0x01f817 (0x24ff bytes) 000306 1749122168.448339 trunc from 0x0369e8 to 0x003bdf (0x32e08 bytes) 000307 1749122168.458473 mapread 0x001714 thru 0x0036ab (0x1f98 bytes) 000308 1749122168.459245 mapwrite 0x006258 thru 0x00d28d (0x7036 bytes) 000309 1749122168.497639 write 0x003fd1 thru 0x010f81 (0xcfb1 bytes) 000310 1749122168.498802 fallocate 0x03d276 thru 0x03ffff (0x2d8a bytes) 000311 1749122168.503673 trunc from 0x0073f3 to 0x00d7ff (0x640d bytes) 000312 1749122168.516712 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000313 1749122168.623694 write 0x01555e thru 0x01fe80 (0xa923 bytes) 000315 1749122168.628016 trunc from 0x01fe81 to 0x00315c (0x1cd24 bytes) 000316 1749122168.639191 punch from 0x001de7 to 0x00271c (0x936 bytes) 000317 1749122168.647232 write 0x01fc42 thru 0x021a9c (0x1e5b bytes) 000318 1749122168.650082 read 0x002570 thru 0x00a56b (0x7ffc bytes) 000319 1749122168.663688 mapread 0x000d1c thru 0x001013 (0x2f8 bytes) 000320 1749122168.673469 MIRROR_RESYNC 3 mirrors 000321 1749122168.887295 fallocate 0x02f8db thru 0x03c77f (0xcea5 bytes) 000323 1749122168.900693 punch from 0x016c5d to 0x01fa4a (0x8dee bytes) 000324 1749122168.903455 fallocate 0x00b98f thru 0x013b9c (0x820e bytes) 000325 1749122168.907083 fallocate 0x0294a2 thru 0x02c672 (0x31d1 bytes) 000326 1749122168.910590 write 0x00f5a0 thru 0x01ba88 (0xc4e9 bytes) 000327 1749122168.916821 punch from 0x00df97 to 0x01c4e0 (0xe54a bytes) 000328 1749122168.927115 punch from 0x00eeec to 0x014c26 (0x5d3b bytes) 000329 1749122168.929731 read 0x002083 thru 0x00320b (0x1189 bytes) 000330 1749122168.933454 mapread 0x0122ec thru 0x016c4d (0x4962 bytes) 000331 1749122168.940401 mapwrite 0x02d2fe thru 0x02d95b (0x65e bytes) 000332 1749122168.974766 read 0x017d40 thru 0x01c9ab (0x4c6c bytes) 000334 1749122168.976138 write 0x005706 thru 0x007546 (0x1e41 bytes) 000335 1749122168.984797 MIRROR_RESYNC 3 mirrors 000336 1749122169.337797 punch from 0x02036f to 0x029f6d (0x9bff bytes) 000337 1749122169.355645 mapread 0x01766c thru 0x01a30a (0x2c9f bytes) 000338 1749122169.368548 punch from 0x01f719 to 0x0217f0 (0x20d8 bytes) 000339 1749122169.376307 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000340 1749122169.478900 trunc from 0x00d6cf to 0x0152d7 (0x7c09 bytes) 000341 1749122169.484824 MIRROR_EXTEND to 3 mirrors 000342 1749122169.759169 mapread 0x01db38 thru 0x029622 (0xbaeb bytes) 000343 1749122169.780202 punch from 0x02b24a to 0x02d95b (0x2712 bytes) 000344 1749122169.801595 punch from 0x00637a to 0x010f61 (0xabe8 bytes) 000345 1749122169.808021 mapwrite 0x022e3d thru 0x0230aa (0x26e bytes) 000346 1749122169.832383 read 0x00f5f0 thru 0x014b91 (0x55a2 bytes) 000347 1749122169.839754 fallocate 0x0073f9 thru 0x010452 (0x905a bytes) 000348 1749122169.843327 mapwrite 0x0184d7 thru 0x0229a9 (0xa4d3 bytes) 000349 1749122169.860899 fallocate 0x03a03a thru 0x03ffff (0x5fc6 bytes) 000350 1749122169.864548 read 0x0166b7 thru 0x02144d (0xad97 bytes) 000351 1749122169.865777 punch from 0x002d80 to 0x005f13 (0x3194 bytes) 000353 1749122169.879433 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000354 1749122170.003871 trunc from 0x02ce5b to 0x02d95b (0xb01 bytes) 000355 1749122170.006795 mapwrite 0x03a6d9 thru 0x03ffff (0x5927 bytes) 000356 1749122170.055965 MIRROR_EXTEND to 3 mirrors 000357 1749122170.363737 punch from 0x00d3be to 0x00f700 (0x2343 bytes) 000358 1749122170.388588 mapread 0x009ac0 thru 0x01867c (0xebbd bytes) 000359 1749122170.406287 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000360 1749122170.517085 MIRROR_RESYNC 2 mirrors 000362 1749122170.833764 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000363 1749122170.896836 read 0x00f776 thru 0x019e81 (0xa70c bytes) 000364 1749122170.900820 mapwrite 0x03985b thru 0x03ffff (0x67a5 bytes) 000365 1749122170.935448 mapread 0x027415 thru 0x034b53 (0xd73f bytes) 000366 1749122170.944736 mapread 0x03561e thru 0x037fe0 (0x29c3 bytes) 000367 1749122170.945447 write 0x022e54 thru 0x02e3a7 (0xb554 bytes) 000368 1749122170.956033 fallocate 0x014195 thru 0x01ef4f (0xadbb bytes) 000370 1749122170.960209 fallocate 0x038afe thru 0x039bdb (0x10de bytes) 000371 1749122170.965632 punch from 0x03b80a to 0x03ffff (0x47f6 bytes) 000372 1749122170.970589 punch from 0x01eed9 to 0x02e8c0 (0xf9e8 bytes) 000373 1749122170.980143 mapread 0x00f200 thru 0x019cef (0xaaf0 bytes) 000374 1749122170.990422 write 0x01d063 thru 0x028043 (0xafe1 bytes) 000375 1749122170.994022 write 0x029c5d thru 0x02c8a5 (0x2c49 bytes) 000376 1749122170.999180 read 0x03b60e thru 0x03e6b3 (0x30a6 bytes) 000377 1749122171.004180 trunc from 0x01ff25 to 0x02b86b (0xb947 bytes) 000379 1749122171.010491 read 0x02db48 thru 0x037a8a (0x9f43 bytes) 000380 1749122171.011123 write 0x003bdd thru 0x00cc51 (0x9075 bytes) 000382 1749122171.012285 mapwrite 0x015c5b thru 0x019def (0x4195 bytes) 000383 1749122171.022998 trunc from 0x040000 to 0x01c796 (0x23869 bytes) 000384 1749122171.045418 trunc from 0x01c797 to 0x02199a (0x5204 bytes) 000385 1749122171.054679 trunc from 0x003771 to 0x003c24 (0x4b4 bytes) 000386 1749122171.058922 write 0x03616e thru 0x036dd1 (0xc64 bytes) 000387 1749122171.059704 mapwrite 0x02c100 thru 0x031894 (0x5795 bytes) 000388 1749122171.076784 mapread 0x03560a thru 0x036dd1 (0x17c8 bytes) 000389 1749122171.078360 write 0x018ffa thru 0x0278a9 (0xe8b0 bytes) 000390 1749122171.079720 read 0x002e1e thru 0x003ab8 (0xc9b bytes) 000391 1749122171.080152 fallocate 0x01f8f4 thru 0x02725f (0x796c bytes) 000392 1749122171.086150 fallocate 0x00c314 thru 0x013726 (0x7413 bytes) 000394 1749122171.089834 read 0x01e5a7 thru 0x01f1ab (0xc05 bytes) 000396 1749122171.090809 write 0x003465 thru 0x00a0d3 (0x6c6f bytes) 000397 1749122171.091618 trunc from 0x036dd2 to 0x013c18 (0x231b9 bytes) 000398 1749122171.100586 write 0x012389 thru 0x0139e8 (0x1660 bytes) 000399 1749122171.101281 trunc from 0x013c19 to 0x03eebd (0x2b2a5 bytes) 000400 1749122171.111232 fallocate 0x036577 thru 0x03ffff (0x9a89 bytes) 000401 1749122171.114552 fallocate 0x01c828 thru 0x02816d (0xb946 bytes) 000402 1749122171.118159 punch from 0x01c8da to 0x021f99 (0x56c0 bytes) 000403 1749122171.120634 write 0x0127d4 thru 0x020628 (0xde55 bytes) 000404 1749122171.124143 punch from 0x028c7f to 0x029225 (0x5a7 bytes) 000406 1749122171.126489 write 0x002fb6 thru 0x0105e1 (0xd62c bytes) 000407 1749122171.127724 write 0x01fc76 thru 0x025aa6 (0x5e31 bytes) 000408 1749122171.133988 MIRROR_EXTEND to 2 mirrors 000409 1749122171.382631 trunc from 0x010d0a to 0x01e5d0 (0xd8c7 bytes) 000410 1749122171.403146 fallocate 0x02ae15 thru 0x031d16 (0x6f02 bytes) 000411 1749122171.405657 mapwrite 0x018227 thru 0x02817b (0xff55 bytes) 000412 1749122171.437615 punch from 0x033eb8 to 0x037e58 (0x3fa1 bytes) 000413 1749122171.443817 fallocate 0x03ee5a thru 0x03ffff (0x11a6 bytes) 000414 1749122171.447137 trunc from 0x031aa0 to 0x032086 (0x5e7 bytes) 000415 1749122171.450695 read 0x02bd4d thru 0x0333aa (0x765e bytes) 000416 1749122171.459800 trunc from 0x03eebe to 0x023798 (0x1b725 bytes) 000417 1749122171.501142 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000418 1749122171.565172 fallocate 0x02e4c9 thru 0x03ba2e (0xd566 bytes) 000419 1749122171.568762 trunc from 0x01d33a to 0x02295a (0x5621 bytes) 000420 1749122171.572528 mapwrite 0x0245e5 thru 0x02e075 (0x9a91 bytes) 000421 1749122171.602735 trunc from 0x03ba2f to 0x00435f (0x376cf bytes) 000423 1749122171.623486 read 0x0033d2 thru 0x00435f (0xf8e bytes) 000424 1749122171.632211 fallocate 0x01216b thru 0x01fd11 (0xdba7 bytes) 000425 1749122171.637677 trunc from 0x004360 to 0x02aa55 (0x266f6 bytes) 000426 1749122171.651272 trunc from 0x02713b to 0x029e0a (0x2cd0 bytes) 000428 1749122171.658068 write 0x000895 thru 0x00cc7e (0xc3ea bytes) 000430 1749122171.666390 read 0x0219c7 thru 0x0242b5 (0x28ef bytes) 000431 1749122171.675929 read 0x0032bd thru 0x010b5c (0xd8a0 bytes) 000432 1749122171.676730 read 0x025615 thru 0x026516 (0xf02 bytes) 000434 1749122171.677693 mapwrite 0x02fdf1 thru 0x0324ff (0x270f bytes) 000435 1749122171.714765 trunc from 0x032500 to 0x011075 (0x2148a bytes) 000436 1749122171.724268 punch from 0x009761 to 0x011075 (0x7915 bytes) 000440 1749122171.730346 read 0x010a34 thru 0x011075 (0x642 bytes) 000441 1749122171.736785 read 0x00f6b0 thru 0x011075 (0x19c6 bytes) 000442 1749122171.737150 fallocate 0x0139f0 thru 0x0216b9 (0xdcca bytes) 000443 1749122171.742750 trunc from 0x011076 to 0x026de9 (0x15d74 bytes) 000444 1749122171.757619 write 0x00ed66 thru 0x01b005 (0xc2a0 bytes) 000445 1749122171.763625 trunc from 0x026dea to 0x0150ed (0x11cfc bytes) 000446 1749122171.775222 trunc from 0x0059f3 to 0x012d4e (0xd35c bytes) 000447 1749122171.790085 punch from 0x00a466 to 0x0150ed (0xac88 bytes) 000448 1749122171.793600 trunc from 0x00dd9c to 0x00ddbf (0x24 bytes) 000449 1749122171.797216 fallocate 0x03cd56 thru 0x03ffff (0x32aa bytes) 000450 1749122171.800805 fallocate 0x004e3f thru 0x00945f (0x4621 bytes) 000452 1749122171.804753 mapwrite 0x020aca thru 0x028327 (0x785e bytes) 000454 1749122171.822140 fallocate 0x02b0b2 thru 0x039f62 (0xeeb1 bytes) 000455 1749122171.825181 trunc from 0x026b0b to 0x0323c6 (0xb8bc bytes) 000457 1749122171.828890 mapread 0x037bb6 thru 0x03ffff (0x844a bytes) 000458 1749122171.830077 mapwrite 0x006e96 thru 0x00d2a3 (0x640e bytes) 000459 1749122171.877758 trunc from 0x029ce1 to 0x02b858 (0x1b78 bytes) 000460 1749122171.880676 trunc from 0x040000 to 0x024e23 (0x1b1dc bytes) 000461 1749122171.890241 mapread 0x01024b thru 0x012bef (0x29a5 bytes) 000463 1749122171.892738 read 0x018d0c thru 0x024e23 (0xc118 bytes) 000464 1749122171.893333 punch from 0x00a172 to 0x011ba1 (0x7a30 bytes) 000465 1749122171.897676 fallocate 0x031087 thru 0x037bcf (0x6b49 bytes) 000466 1749122171.901488 trunc from 0x024e24 to 0x00bba1 (0x19282 bytes) 000468 1749122171.910396 fallocate 0x01ae94 thru 0x01bb8e (0xcfb bytes) 000469 1749122171.912844 mapwrite 0x00498c thru 0x00dece (0x9543 bytes) 000470 1749122171.942643 read 0x00a306 thru 0x01249f (0x819a bytes) 000471 1749122171.944745 trunc from 0x01bb8f to 0x02e9ba (0x12e2c bytes) 000472 1749122171.954833 write 0x003c01 thru 0x004d41 (0x1141 bytes) 000473 1749122171.955443 mapwrite 0x02e896 thru 0x03c4aa (0xdc15 bytes) 000475 1749122171.994050 trunc from 0x03c4ab to 0x010b02 (0x2b9a8 bytes) 000476 1749122172.004568 punch from 0x00a07f to 0x00f5b1 (0x5533 bytes) 000477 1749122172.010634 write 0x032724 thru 0x039f93 (0x7870 bytes) 000481 1749122172.013562 fallocate 0x00e0f3 thru 0x0118c0 (0x37ce bytes) 000482 1749122172.019431 trunc from 0x02927f to 0x038a20 (0xf7a2 bytes) 000483 1749122172.028845 trunc from 0x017c6e to 0x01d99d (0x5d30 bytes) 000484 1749122172.032038 write 0x00332f thru 0x010fe1 (0xdcb3 bytes) 000485 1749122172.034345 write 0x027510 thru 0x032afe (0xb5ef bytes) 000487 1749122172.040413 fallocate 0x012054 thru 0x01cb36 (0xaae3 bytes) 000488 1749122172.044571 mapwrite 0x010721 thru 0x015614 (0x4ef4 bytes) 000489 1749122172.076745 trunc from 0x039f94 to 0x005800 (0x34793 bytes) 000492 1749122172.089041 trunc from 0x000578 to 0x005800 (0x5289 bytes) 000493 1749122172.095116 MIRROR_EXTEND to 2 mirrors 000494 1749122172.295600 mapwrite 0x026441 thru 0x026dc5 (0x985 bytes) 000495 1749122172.322457 mapread 0x00acc2 thru 0x00fdc8 (0x5107 bytes) 000497 1749122172.332660 mapwrite 0x026582 thru 0x02a232 (0x3cb1 bytes) 000498 1749122172.361243 trunc from 0x00f0b9 to 0x01a042 (0xaf8a bytes) 000499 1749122172.363968 mapwrite 0x0176d2 thru 0x01ec05 (0x7534 bytes) 000501 1749122172.382337 mapread 0x0071e5 thru 0x015fdf (0xedfb bytes) 000502 1749122172.386207 mapread 0x01628c thru 0x0166c0 (0x435 bytes) 000503 1749122172.387226 mapwrite 0x01f0c7 thru 0x02a916 (0xb850 bytes) 000504 1749122172.408693 write 0x03341f thru 0x03feb5 (0xca97 bytes) 000505 1749122172.409827 write 0x00af1e thru 0x00c3b9 (0x149c bytes) 000506 1749122172.410435 punch from 0x01b4ef to 0x01ced9 (0x19eb bytes) 000508 1749122172.414025 read 0x03a356 thru 0x03cc1c (0x28c7 bytes) 000509 1749122172.416410 punch from 0x01c4c9 to 0x02358e (0x70c6 bytes) 000510 1749122172.419589 write 0x00d312 thru 0x0163df (0x90ce bytes) 000511 1749122172.420406 trunc from 0x0168c7 to 0x01cbe7 (0x6321 bytes) 000512 1749122172.431649 trunc from 0x00d151 to 0x01b323 (0xe1d3 bytes) 000513 1749122172.436606 read 0x036c5f thru 0x03feb5 (0x9257 bytes) 000514 1749122172.437090 mapwrite 0x00ac16 thru 0x014b20 (0x9f0b bytes) 000515 1749122172.457755 trunc from 0x03feb6 to 0x034347 (0xbb6e bytes) 000516 1749122172.467903 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000517 1749122172.523591 trunc from 0x02a345 to 0x02dba6 (0x3862 bytes) 000518 1749122172.530802 MIRROR_EXTEND to 2 mirrors 000519 1749122172.790525 trunc from 0x017fc7 to 0x01813a (0x174 bytes) 000521 1749122172.809551 write 0x00bfea thru 0x010b1c (0x4b33 bytes) 000522 1749122172.816394 fallocate 0x0269da thru 0x02b621 (0x4c48 bytes) 000523 1749122172.819186 mapwrite 0x01c6ae thru 0x01cbad (0x500 bytes) 000524 1749122172.835485 write 0x007a14 thru 0x01329f (0xb88c bytes) 000525 1749122172.848282 read 0x00c0c0 thru 0x00fb2f (0x3a70 bytes) 000526 1749122172.850142 mapwrite 0x00f939 thru 0x013deb (0x44b3 bytes) 000527 1749122172.863087 write 0x0161a3 thru 0x025cd2 (0xfb30 bytes) 000528 1749122172.867342 mapread 0x023e5f thru 0x02f038 (0xb1da bytes) 000529 1749122172.874110 write 0x026a54 thru 0x02d7df (0x6d8c bytes) 000530 1749122172.874763 trunc from 0x034348 to 0x002419 (0x31f2e bytes) 000531 1749122172.882451 punch from 0x001c59 to 0x002419 (0x7c1 bytes) 000532 1749122172.885583 punch from 0x001583 to 0x002419 (0xe97 bytes) 000533 1749122172.889290 mapwrite 0x00298d thru 0x009419 (0x6a8d bytes) 000534 1749122172.913476 trunc from 0x00941a to 0x03235d (0x28f44 bytes) 000535 1749122172.922764 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000538 1749122172.993983 write 0x02283e thru 0x024cbe (0x2481 bytes) 000539 1749122173.002701 trunc from 0x03235e to 0x02c224 (0x6139 bytes) 000540 1749122173.009778 punch from 0x0289cf to 0x02c224 (0x3856 bytes) 000541 1749122173.012512 fallocate 0x0149ba thru 0x021fc6 (0xd60d bytes) 000542 1749122173.017103 write 0x018a8b thru 0x0222ec (0x9862 bytes) 000543 1749122173.023724 fallocate 0x003fe5 thru 0x006a30 (0x2a4c bytes) 000544 1749122173.026560 punch from 0x01e424 to 0x0200ce (0x1cab bytes) 000545 1749122173.032927 mapread 0x024094 thru 0x02c224 (0x8191 bytes) 000547 1749122173.038326 write 0x014fd5 thru 0x0183ba (0x33e6 bytes) 000548 1749122173.040119 fallocate 0x00a310 thru 0x0187b9 (0xe4aa bytes) 000550 1749122173.044441 read 0x00a870 thru 0x00de75 (0x3606 bytes) 000551 1749122173.048836 mapread 0x000f90 thru 0x005cf6 (0x4d67 bytes) 000553 1749122173.056065 trunc from 0x02c225 to 0x03ac96 (0xea72 bytes) 000554 1749122173.064315 punch from 0x01a50f to 0x02393b (0x942d bytes) 000555 1749122173.067535 mapwrite 0x010377 thru 0x01b2cf (0xaf59 bytes) 000556 1749122173.086712 mapread 0x03500b thru 0x038536 (0x352c bytes) 000557 1749122173.091418 trunc from 0x03ac97 to 0x035854 (0x5442 bytes) 000558 1749122173.104961 MIRROR_EXTEND to 2 mirrors 000559 1749122173.324834 mapwrite 0x03b42f thru 0x03ffff (0x4bd1 bytes) 000560 1749122173.381929 write 0x0112dd thru 0x0207f5 (0xf519 bytes) 000561 1749122173.390563 trunc from 0x040000 to 0x00b21b (0x34de4 bytes) 000562 1749122173.404209 fallocate 0x00b7d2 thru 0x0157c4 (0x9ff3 bytes) 000563 1749122173.407422 trunc from 0x0157c5 to 0x031dee (0x1c62a bytes) 000564 1749122173.418053 MIRROR_EXTEND to 3 mirrors 000565 1749122173.688640 fallocate 0x0004ac thru 0x004c6b (0x47c0 bytes) 000566 1749122173.712398 trunc from 0x01bd82 to 0x0207fc (0x4a7b bytes) 000567 1749122173.715725 read 0x021378 thru 0x021960 (0x5e9 bytes) 000568 1749122173.718645 mapwrite 0x00b9ae thru 0x010e98 (0x54eb bytes) 000569 1749122173.739700 fallocate 0x031cb9 thru 0x038de5 (0x712d bytes) 000570 1749122173.747110 punch from 0x01ce7d to 0x0247cb (0x794f bytes) 000571 1749122173.751463 mapwrite 0x03143f thru 0x03a653 (0x9215 bytes) 000572 1749122173.792081 mapwrite 0x007127 thru 0x008ccd (0x1ba7 bytes) 000573 1749122173.803050 punch from 0x0109f2 to 0x011e31 (0x1440 bytes) 000574 1749122173.805814 mapread 0x035800 thru 0x03a653 (0x4e54 bytes) 000575 1749122173.807218 mapread 0x01049f thru 0x016be7 (0x6749 bytes) 000576 1749122173.816031 trunc from 0x03a654 to 0x0205b8 (0x1a09b bytes) 000578 1749122173.824380 punch from 0x00cc59 to 0x01b26f (0xe617 bytes) 000579 1749122173.827597 read 0x00aa3a thru 0x016c6c (0xc233 bytes) 000580 1749122173.838182 trunc from 0x01a087 to 0x01afc2 (0xf3c bytes) 000581 1749122173.841302 trunc from 0x011295 to 0x01651d (0x5289 bytes) 000582 1749122173.847218 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000583 1749122173.958411 mapwrite 0x03e5cd thru 0x03ffff (0x1a33 bytes) 000584 1749122173.992288 write 0x02f03b thru 0x03e246 (0xf20c bytes) 000585 1749122174.003865 write 0x00f7e8 thru 0x01ca65 (0xd27e bytes) 000586 1749122174.012413 mapread 0x008892 thru 0x011f51 (0x96c0 bytes) 000587 1749122174.013620 mapwrite 0x01b1db thru 0x025ddb (0xac01 bytes) 000588 1749122174.036141 trunc from 0x040000 to 0x02a825 (0x157da bytes) 000589 1749122174.043455 read 0x029f3a thru 0x02a825 (0x8ec bytes) 000590 1749122174.043780 write 0x03e348 thru 0x03ffff (0x1cb8 bytes) 000591 1749122174.047514 MIRROR_EXTEND to 3 mirrors 000592 1749122174.347093 mapwrite 0x02eb21 thru 0x02eec4 (0x3a4 bytes) 000593 1749122174.410062 MIRROR_EXTEND to 4 mirrors 000594 1749122174.710207 MIRROR_RESYNC 4 mirrors 000595 1749122175.100659 trunc from 0x040000 to 0x01974b (0x268b4 bytes) 000596 1749122175.130616 trunc from 0x01974c to 0x000c36 (0x18b15 bytes) 000597 1749122175.140843 mapread 0x0003ae thru 0x000bd9 (0x82c bytes) 000598 1749122175.146202 mapread 0x000b58 thru 0x000c36 (0xdf bytes) 000599 1749122175.148082 write 0x01baa6 thru 0x01c60e (0xb69 bytes) 000600 1749122175.148827 write 0x02b7f8 thru 0x02c414 (0xc1d bytes) 000601 1749122175.149460 write 0x03b40f thru 0x03ffff (0x4bf1 bytes) 000602 1749122175.150177 mapwrite 0x022b26 thru 0x02e95f (0xbe3a bytes) 000603 1749122175.174923 read 0x0244d5 thru 0x02e99a (0xa4c6 bytes) 000605 1749122175.176290 mapwrite 0x033f7b thru 0x03f3a2 (0xb428 bytes) 000606 1749122175.194181 mapwrite 0x008a16 thru 0x00f984 (0x6f6f bytes) 000607 1749122175.207768 mapwrite 0x0130d2 thru 0x01cb16 (0x9a45 bytes) 000608 1749122175.221604 trunc from 0x02aebf to 0x03a8a7 (0xf9e9 bytes) 000609 1749122175.226181 punch from 0x0368c6 to 0x03ffff (0x973a bytes) 000610 1749122175.231747 write 0x030e0f thru 0x03a63d (0x982f bytes) 000611 1749122175.242437 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000612 1749122175.348781 trunc from 0x03e99e to 0x03ffff (0x1662 bytes) 000613 1749122175.352184 write 0x00ce63 thru 0x019fb0 (0xd14e bytes) 000614 1749122175.358534 write 0x01cac2 thru 0x02af44 (0xe483 bytes) 000615 1749122175.365701 mapwrite 0x0379e9 thru 0x03ffff (0x8617 bytes) 000616 1749122175.382726 trunc from 0x035a85 to 0x03ffff (0xa57b bytes) 000617 1749122175.388879 MIRROR_RESYNC 3 mirrors 000619 1749122175.662815 trunc from 0x040000 to 0x03b14e (0x4eb1 bytes) 000620 1749122175.694321 mapwrite 0x0154e8 thru 0x01a459 (0x4f72 bytes) 000621 1749122175.722002 MIRROR_EXTEND to 4 mirrors 000622 1749122175.942903 trunc from 0x03b14f to 0x00da66 (0x2d6e8 bytes) 000624 1749122175.976438 trunc from 0x00da67 to 0x030f11 (0x234ab bytes) 000625 1749122175.984373 trunc from 0x003c51 to 0x00edc3 (0xb173 bytes) 000626 1749122175.991160 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000627 1749122176.097473 mapread 0x02d8eb thru 0x02e5d0 (0xce6 bytes) 000628 1749122176.102251 mapread 0x01e6c9 thru 0x01f8ca (0x1202 bytes) 000630 1749122176.108233 mapwrite 0x027cc0 thru 0x029e36 (0x2177 bytes) 000631 1749122176.126782 write 0x02fe1c thru 0x0377f6 (0x79db bytes) 000632 1749122176.129993 fallocate 0x03eb3d thru 0x03ffff (0x14c3 bytes) 000633 1749122176.132408 mapwrite 0x026d6e thru 0x02ff8f (0x9222 bytes) 000635 1749122176.149666 MIRROR_RESYNC 3 mirrors 000636 1749122176.483472 MIRROR_EXTEND to 4 mirrors 000637 1749122176.784604 write 0x016c6d thru 0x020ae0 (0x9e74 bytes) 000638 1749122176.812470 trunc from 0x01d804 to 0x0245a3 (0x6da0 bytes) 000640 1749122176.819712 fallocate 0x002bd6 thru 0x00747f (0x48aa bytes) 000641 1749122176.822761 write 0x03e4c4 thru 0x03ffff (0x1b3c bytes) 000642 1749122176.826779 mapwrite 0x0031dc thru 0x00e20c (0xb031 bytes) 000643 1749122176.847657 trunc from 0x040000 to 0x003744 (0x3c8bb bytes) 000644 1749122176.872212 mapwrite 0x012638 thru 0x02100d (0xe9d6 bytes) 000645 1749122176.911597 trunc from 0x006e9c to 0x011103 (0xa268 bytes) 000647 1749122176.917257 fallocate 0x02e1d3 thru 0x0341a7 (0x5fd5 bytes) 000648 1749122176.923978 fallocate 0x01cb32 thru 0x02bcec (0xf1bb bytes) 000649 1749122176.929398 MIRROR_EXTEND to 5 mirrors 000650 1749122177.200295 MIRROR_EXTEND to 6 mirrors 000651 1749122177.465717 fallocate 0x017c7f thru 0x01a11f (0x24a1 bytes) 000652 1749122177.479551 trunc from 0x00ea94 to 0x01b8fc (0xce69 bytes) 000653 1749122177.482582 mapread 0x0271d9 thru 0x02a4be (0x32e6 bytes) 000654 1749122177.491143 mapwrite 0x01b684 thru 0x0264cd (0xae4a bytes) 000655 1749122177.506975 read 0x0168b7 thru 0x01ef32 (0x867c bytes) 000656 1749122177.507609 mapread 0x02b125 thru 0x02bcec (0xbc8 bytes) 000657 1749122177.509340 punch from 0x01dd87 to 0x02b843 (0xdabd bytes) 000658 1749122177.512102 fallocate 0x015a67 thru 0x01df26 (0x84c0 bytes) 000659 1749122177.517894 write 0x007e10 thru 0x011db8 (0x9fa9 bytes) 000660 1749122177.528042 MIRROR_RESYNC 6 mirrors 000661 1749122178.264353 trunc from 0x0072f9 to 0x00b85f (0x4567 bytes) 000662 1749122178.288799 punch from 0x01c6ec to 0x01ed37 (0x264c bytes) 000663 1749122178.292436 trunc from 0x005157 to 0x010f7a (0xbe24 bytes) 000664 1749122178.295387 trunc from 0x02bced to 0x01d754 (0xe598 bytes) 000665 1749122178.307959 fallocate 0x01eece thru 0x0274e6 (0x8619 bytes) 000667 1749122178.311663 punch from 0x0266a3 to 0x0274e6 (0xe44 bytes) 000668 1749122178.316737 trunc from 0x0274e7 to 0x03b8f2 (0x1440c bytes) 000669 1749122178.334410 trunc from 0x0242d0 to 0x028269 (0x3f9a bytes) 000670 1749122178.339484 MIRROR_RESYNC 6 mirrors 000672 1749122179.358060 mapread 0x00db53 thru 0x01cacf (0xef7d bytes) 000673 1749122179.373530 trunc from 0x037962 to 0x038846 (0xee5 bytes) 000674 1749122179.396542 write 0x0127c0 thru 0x017d3a (0x557b bytes) 000676 1749122179.402661 punch from 0x007d5d to 0x00d11c (0x53c0 bytes) 000677 1749122179.407120 trunc from 0x030247 to 0x03a431 (0xa1eb bytes) 000678 1749122179.409685 fallocate 0x0396a7 thru 0x03ffff (0x6959 bytes) 000679 1749122179.414054 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000680 1749122179.526513 trunc from 0x0364e3 to 0x03b8f2 (0x5410 bytes) 000681 1749122179.529306 trunc from 0x038f5b to 0x03b8f2 (0x2998 bytes) 000682 1749122179.534029 trunc from 0x03b8f3 to 0x037ab7 (0x3e3b bytes) 000683 1749122179.545367 mapread 0x00d843 thru 0x01759a (0x9d58 bytes) 000684 1749122179.559532 read 0x031865 thru 0x036d9c (0x5538 bytes) 000685 1749122179.574277 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000687 1749122179.682730 trunc from 0x037ab8 to 0x008110 (0x2f9a7 bytes) 000688 1749122179.695945 punch from 0x005459 to 0x008110 (0x2cb8 bytes) 000689 1749122179.699248 trunc from 0x004d8c to 0x008110 (0x3385 bytes) 000690 1749122179.702360 write 0x00c6d5 thru 0x01a842 (0xe16e bytes) 000691 1749122179.703593 write 0x039ae7 thru 0x03ffff (0x6519 bytes) 000692 1749122179.704399 mapwrite 0x026ad6 thru 0x034c74 (0xe19f bytes) 000693 1749122179.731219 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000694 1749122179.837755 trunc from 0x040000 to 0x03ce25 (0x31da bytes) 000695 1749122179.850463 mapwrite 0x013b49 thru 0x015c0a (0x20c2 bytes) 000696 1749122179.877837 trunc from 0x03ce26 to 0x03976b (0x36ba bytes) 000697 1749122179.889959 MIRROR_RESYNC 3 mirrors 000698 1749122180.391542 trunc from 0x03976c to 0x034769 (0x5002 bytes) 000699 1749122180.415185 mapread 0x0324d4 thru 0x034769 (0x2296 bytes) 000701 1749122180.428994 read 0x024376 thru 0x02ee3d (0xaac8 bytes) 000702 1749122180.434147 mapread 0x030ee3 thru 0x034769 (0x3887 bytes) 000703 1749122180.435920 mapread 0x0083eb thru 0x011fd8 (0x9bee bytes) 000704 1749122180.446905 trunc from 0x03476a to 0x008f66 (0x2b803 bytes) 000705 1749122180.465911 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000706 1749122180.568632 punch from 0x007dc5 to 0x008f66 (0x11a2 bytes) 000707 1749122180.570977 trunc from 0x008f67 to 0x01140f (0x84a9 bytes) 000708 1749122180.581013 mapread 0x0012bd thru 0x003f9a (0x2cde bytes) 000709 1749122180.591481 read 0x00ba17 thru 0x01140f (0x59f9 bytes) 000710 1749122180.594317 mapread 0x00f713 thru 0x01140f (0x1cfd bytes) 000711 1749122180.594746 mapread 0x005a57 thru 0x01140f (0xb9b9 bytes) 000712 1749122180.595369 mapwrite 0x0373e3 thru 0x03ffff (0x8c1d bytes) 000713 1749122180.625073 mapwrite 0x025e15 thru 0x02f721 (0x990d bytes) 000714 1749122180.655395 read 0x028501 thru 0x0304d3 (0x7fd3 bytes) 000715 1749122180.655920 mapread 0x02182a thru 0x02c686 (0xae5d bytes) 000716 1749122180.656508 trunc from 0x03b7cc to 0x03e9ba (0x31ef bytes) 000717 1749122180.672519 MIRROR_RESYNC 2 mirrors 000718 1749122180.940073 read 0x00226b thru 0x00aca7 (0x8a3d bytes) 000719 1749122180.944251 trunc from 0x036a96 to 0x03ffff (0x956a bytes) 000721 1749122180.969349 punch from 0x03e725 to 0x03ffff (0x18db bytes) 000722 1749122180.974506 read 0x018d9d thru 0x01f08d (0x62f1 bytes) 000723 1749122180.987068 fallocate 0x03ca8d thru 0x03ffff (0x3573 bytes) 000724 1749122180.995002 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000725 1749122181.061300 mapwrite 0x0284c0 thru 0x02e676 (0x61b7 bytes) 000726 1749122181.082099 mapwrite 0x02bac8 thru 0x03569d (0x9bd6 bytes) 000728 1749122181.098226 punch from 0x023c33 to 0x02c7c7 (0x8b95 bytes) 000729 1749122181.102192 trunc from 0x025e90 to 0x033e02 (0xdf73 bytes) 000730 1749122181.106672 read 0x03025f thru 0x034272 (0x4014 bytes) 000731 1749122181.110327 punch from 0x031423 to 0x031a99 (0x677 bytes) 000732 1749122181.115832 trunc from 0x023347 to 0x02881e (0x54d8 bytes) 000734 1749122181.121675 mapread 0x006134 thru 0x010e92 (0xad5f bytes) 000736 1749122181.134223 read 0x02b596 thru 0x02e5b9 (0x3024 bytes) 000738 1749122181.140885 trunc from 0x040000 to 0x03416f (0xbe90 bytes) 000739 1749122181.166693 trunc from 0x034170 to 0x025243 (0xef2c bytes) 000740 1749122181.180177 MIRROR_EXTEND to 2 mirrors 000741 1749122181.399913 mapwrite 0x00b802 thru 0x01b6ee (0xfeed bytes) 000742 1749122181.469369 read 0x023578 thru 0x0245fe (0x1087 bytes) 000743 1749122181.474083 trunc from 0x025244 to 0x0033dd (0x21e66 bytes) 000744 1749122181.499779 punch from 0x0029ce to 0x0033dd (0xa10 bytes) 000745 1749122181.505236 MIRROR_EXTEND to 3 mirrors 000747 1749122181.703634 mapwrite 0x00224c thru 0x002349 (0xfe bytes) 000748 1749122181.756377 read 0x000c48 thru 0x0033dd (0x2796 bytes) 000750 1749122181.762796 mapwrite 0x029583 thru 0x031b63 (0x85e1 bytes) 000751 1749122181.805362 write 0x00b62d thru 0x00cf5b (0x192f bytes) 000752 1749122181.809536 trunc from 0x00778d to 0x0175a5 (0xfe19 bytes) 000753 1749122181.815808 write 0x0376d1 thru 0x03ffff (0x892f bytes) 000754 1749122181.819398 fallocate 0x01df7b thru 0x02c339 (0xe3bf bytes) 000755 1749122181.823130 fallocate 0x026c43 thru 0x0277b7 (0xb75 bytes) 000756 1749122181.827220 mapwrite 0x0168b9 thru 0x0213fc (0xab44 bytes) 000757 1749122181.844821 write 0x009d93 thru 0x00c31d (0x258b bytes) 000758 1749122181.850632 punch from 0x028ff9 to 0x030e02 (0x7e0a bytes) 000759 1749122181.855141 write 0x026745 thru 0x02c6e1 (0x5f9d bytes) 000760 1749122181.861290 write 0x03861c thru 0x03ffff (0x79e4 bytes) 000761 1749122181.862175 trunc from 0x040000 to 0x009807 (0x367f8 bytes) 000762 1749122181.871363 read 0x002cbb thru 0x0082a8 (0x55ee bytes) 000763 1749122181.872581 write 0x03b1c7 thru 0x03ffff (0x4e39 bytes) 000764 1749122181.873781 mapread 0x0091b7 thru 0x00bf30 (0x2d7a bytes) 000765 1749122181.883922 MIRROR_EXTEND to 4 mirrors 000766 1749122182.198120 write 0x02e750 thru 0x03beed (0xd79e bytes) 000768 1749122182.224870 trunc from 0x040000 to 0x036404 (0x9bfb bytes) 000769 1749122182.241079 punch from 0x012c69 to 0x01ea76 (0xbe0e bytes) 000771 1749122182.247757 write 0x03cdd8 thru 0x03ffff (0x3228 bytes) 000772 1749122182.250146 mapwrite 0x019bbd thru 0x023ede (0xa322 bytes) 000773 1749122182.275627 trunc from 0x012eae to 0x01391f (0xa72 bytes) 000774 1749122182.280067 fallocate 0x01be79 thru 0x02bc99 (0xfe21 bytes) 000775 1749122182.287057 MIRROR_EXTEND to 5 mirrors 000777 1749122182.637209 trunc from 0x02a4f9 to 0x02d54c (0x3054 bytes) 000778 1749122182.659299 write 0x0160ff thru 0x01e5c4 (0x84c6 bytes) 000780 1749122182.667232 trunc from 0x00140f to 0x005c26 (0x4818 bytes) 000781 1749122182.672208 MIRROR_EXTEND to 6 mirrors 000785 1749122182.983635 trunc from 0x040000 to 0x0395fb (0x6a04 bytes) 000786 1749122183.023950 write 0x00ee20 thru 0x01915a (0xa33b bytes) 000787 1749122183.029531 mapwrite 0x0170ee thru 0x020e17 (0x9d2a bytes) 000788 1749122183.046945 trunc from 0x01fbc9 to 0x02bce7 (0xc11f bytes) 000789 1749122183.050158 write 0x0303ef thru 0x035727 (0x5339 bytes) 000790 1749122183.057100 write 0x0330c1 thru 0x0394b6 (0x63f6 bytes) 000791 1749122183.062925 mapread 0x0065f7 thru 0x011ccc (0xb6d6 bytes) 000792 1749122183.066961 fallocate 0x024622 thru 0x03336f (0xed4e bytes) 000793 1749122183.069540 trunc from 0x0395fc to 0x010b77 (0x28a84 bytes) 000794 1749122183.082497 mapread 0x005d5b thru 0x007578 (0x181e bytes) 000795 1749122183.087609 trunc from 0x010b78 to 0x03ef19 (0x2e3a2 bytes) 000796 1749122183.095139 mapwrite 0x03db7b thru 0x03ffff (0x2485 bytes) 000797 1749122183.116927 write 0x0091f7 thru 0x00c5ab (0x33b5 bytes) 000798 1749122183.117680 trunc from 0x019aac to 0x028891 (0xede6 bytes) 000799 1749122183.121636 punch from 0x035c15 to 0x03b07d (0x5469 bytes) 000800 1749122183.125386 fallocate 0x03db6c thru 0x03ffff (0x2494 bytes) 000802 1749122183.128944 mapwrite 0x02e96e thru 0x033df0 (0x5483 bytes) 000803 1749122183.144900 trunc from 0x040000 to 0x01e245 (0x21dba bytes) 000804 1749122183.154453 trunc from 0x0072dd to 0x0082b3 (0xfd7 bytes) 000805 1749122183.161684 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000807 1749122183.273804 mapread 0x000bb3 thru 0x00f825 (0xec73 bytes) 000808 1749122183.295965 trunc from 0x01e246 to 0x02e736 (0x104f1 bytes) 000809 1749122183.308633 trunc from 0x00d5c7 to 0x01059a (0x2fd4 bytes) 000811 1749122183.312224 read 0x0046ad thru 0x00db05 (0x9459 bytes) 000812 1749122183.316491 mapwrite 0x03fa85 thru 0x03ffff (0x57b bytes) 000813 1749122183.338749 fallocate 0x021f10 thru 0x031caa (0xfd9b bytes) 000814 1749122183.341348 read 0x02aa17 thru 0x02e923 (0x3f0d bytes) 000815 1749122183.346489 mapwrite 0x01ece3 thru 0x02655c (0x787a bytes) 000817 1749122183.362231 mapwrite 0x0261ec thru 0x02b48b (0x52a0 bytes) 000818 1749122183.375807 write 0x016dc2 thru 0x026c14 (0xfe53 bytes) 000819 1749122183.377182 mapread 0x038a8e thru 0x03ffff (0x7572 bytes) 000820 1749122183.378680 mapwrite 0x026a96 thru 0x030c91 (0xa1fc bytes) 000821 1749122183.393619 mapwrite 0x028b17 thru 0x037df1 (0xf2db bytes) 000822 1749122183.407141 write 0x000678 thru 0x00b788 (0xb111 bytes) 000823 1749122183.410897 MIRROR_EXTEND to 6 mirrors 000824 1749122183.680585 mapwrite 0x03bdad thru 0x03ffff (0x4253 bytes) 000825 1749122183.740241 trunc from 0x040000 to 0x03be9f (0x4160 bytes) 000826 1749122183.764194 write 0x03c6e9 thru 0x03ffff (0x3917 bytes) 000827 1749122183.765008 trunc from 0x040000 to 0x015e94 (0x2a16b bytes) 000828 1749122183.773974 trunc from 0x011e23 to 0x013d4c (0x1f2a bytes) 000829 1749122183.778252 punch from 0x009f03 to 0x015e94 (0xbf92 bytes) 000830 1749122183.783259 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000831 1749122183.889506 MIRROR_EXTEND to 6 mirrors 000832 1749122184.123274 trunc from 0x00af2b to 0x00bb8e (0xc64 bytes) 000834 1749122184.146936 write 0x008471 thru 0x011be6 (0x9776 bytes) 000835 1749122184.154777 trunc from 0x015e95 to 0x007456 (0xea3e bytes) 000836 1749122184.164929 read 0x001a8c thru 0x007456 (0x59cb bytes) 000838 1749122184.170818 mapwrite 0x00dff7 thru 0x015a11 (0x7a1b bytes) 000839 1749122184.197307 read 0x0114a4 thru 0x015a11 (0x456e bytes) 000840 1749122184.197754 read 0x004c19 thru 0x0051d0 (0x5b8 bytes) 000841 1749122184.198153 fallocate 0x013acf thru 0x01cd54 (0x9286 bytes) 000842 1749122184.202707 trunc from 0x0005fb to 0x002a00 (0x2406 bytes) 000843 1749122184.206831 MIRROR_RESYNC 6 mirrors 000844 1749122184.795505 mapwrite 0x036f60 thru 0x03d3b9 (0x645a bytes) 000845 1749122184.842476 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000846 1749122184.930544 trunc from 0x03d3ba to 0x033cf5 (0x96c4 bytes) 000848 1749122184.944332 write 0x00b03b thru 0x0145a9 (0x956f bytes) 000850 1749122184.949901 mapwrite 0x022d94 thru 0x0282af (0x551c bytes) 000851 1749122184.965336 fallocate 0x03ab54 thru 0x03ffff (0x54ac bytes) 000852 1749122184.970506 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000853 1749122185.065471 read 0x02f18e thru 0x033e7b (0x4cee bytes) 000854 1749122185.070753 mapread 0x00cef1 thru 0x011492 (0x45a2 bytes) 000855 1749122185.087514 write 0x0011f6 thru 0x00ce19 (0xbc24 bytes) 000856 1749122185.098710 trunc from 0x040000 to 0x005798 (0x3a867 bytes) 000857 1749122185.109761 MIRROR_EXTEND to 5 mirrors 000858 1749122185.307280 MIRROR_EXTEND to 6 mirrors 000859 1749122185.475150 mapread 0x002f11 thru 0x005798 (0x2888 bytes) 000860 1749122185.484615 mapwrite 0x002ef6 thru 0x00d3b1 (0xa4bc bytes) 000861 1749122185.530167 mapread 0x001cef thru 0x003ded (0x20ff bytes) 000862 1749122185.530674 fallocate 0x030cc1 thru 0x031dcd (0x110d bytes) 000863 1749122185.533127 mapread 0x004994 thru 0x00d3b1 (0x8a1e bytes) 000864 1749122185.534066 trunc from 0x00cb10 to 0x00d3b1 (0x8a2 bytes) 000866 1749122185.537507 mapread 0x00054d thru 0x008cbf (0x8773 bytes) 000867 1749122185.541766 punch from 0x0077a3 to 0x00d3b1 (0x5c0f bytes) 000868 1749122185.545420 write 0x03ba5a thru 0x03ffff (0x45a6 bytes) 000870 1749122185.546744 punch from 0x003b9b to 0x0129cf (0xee35 bytes) 000871 1749122185.552413 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000872 1749122185.644343 mapwrite 0x00040d thru 0x004b32 (0x4726 bytes) 000873 1749122185.665189 trunc from 0x040000 to 0x02d17c (0x12e83 bytes) 000874 1749122185.677612 MIRROR_EXTEND to 6 mirrors 000875 1749122185.890212 trunc from 0x02d17d to 0x038588 (0xb40c bytes) 000876 1749122185.927357 fallocate 0x009704 thru 0x00d113 (0x3a10 bytes) 000877 1749122185.934226 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000878 1749122186.008601 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000879 1749122186.105250 MIRROR_RESYNC 4 mirrors 000880 1749122186.368042 write 0x0311e2 thru 0x03fc79 (0xea98 bytes) 000881 1749122186.389986 fallocate 0x017657 thru 0x0218bb (0xa265 bytes) 000882 1749122186.396303 trunc from 0x00a3a8 to 0x00feee (0x5b47 bytes) 000884 1749122186.400350 write 0x032660 thru 0x03bf33 (0x98d4 bytes) 000885 1749122186.401212 fallocate 0x03649d thru 0x03e333 (0x7e97 bytes) 000886 1749122186.404482 punch from 0x006610 to 0x006c41 (0x632 bytes) 000887 1749122186.408658 punch from 0x00d22f to 0x019ad7 (0xc8a9 bytes) 000888 1749122186.419717 mapwrite 0x03d0c4 thru 0x03e42b (0x1368 bytes) 000889 1749122186.438209 punch from 0x018b93 to 0x01cebb (0x4329 bytes) 000890 1749122186.441745 punch from 0x01376f to 0x014b6c (0x13fe bytes) 000891 1749122186.444555 write 0x02a314 thru 0x02f5de (0x52cb bytes) 000892 1749122186.452435 write 0x0041da thru 0x00a0c2 (0x5ee9 bytes) 000893 1749122186.458560 mapread 0x03019e thru 0x039b40 (0x99a3 bytes) 000894 1749122186.462925 trunc from 0x03fc7a to 0x0150fe (0x2ab7b bytes) 000895 1749122186.483358 punch from 0x000cbb to 0x0066fe (0x5a44 bytes) 000896 1749122186.494155 trunc from 0x0050af to 0x00d28c (0x81de bytes) 000897 1749122186.498486 read 0x005609 thru 0x00722a (0x1c22 bytes) 000898 1749122186.503034 trunc from 0x0150ff to 0x013819 (0x18e5 bytes) 000899 1749122186.511639 punch from 0x00423e to 0x012df1 (0xebb4 bytes) 000900 1749122186.514057 fallocate 0x033d49 thru 0x03b0df (0x7397 bytes) 000901 1749122186.522608 MIRROR_EXTEND to 5 mirrors 000902 1749122186.790271 trunc from 0x0198fd to 0x01c9f2 (0x30f6 bytes) 000904 1749122186.806569 fallocate 0x014f2d thru 0x01c122 (0x71f6 bytes) 000905 1749122186.809106 fallocate 0x01ae0f thru 0x02369b (0x888d bytes) 000906 1749122186.811384 punch from 0x00dd45 to 0x010a25 (0x2ce1 bytes) 000908 1749122186.813841 mapwrite 0x035135 thru 0x03a0d5 (0x4fa1 bytes) 000909 1749122186.838252 MIRROR_RESYNC 5 mirrors 000910 1749122187.409605 fallocate 0x00dbda thru 0x00eca3 (0x10ca bytes) 000912 1749122187.428118 fallocate 0x022e95 thru 0x02a91d (0x7a89 bytes) 000913 1749122187.430235 mapwrite 0x002ac7 thru 0x00ee33 (0xc36d bytes) 000914 1749122187.458210 mapread 0x029ebd thru 0x038c7d (0xedc1 bytes) 000915 1749122187.470788 write 0x018e66 thru 0x0238a0 (0xaa3b bytes) 000916 1749122187.477272 trunc from 0x02d912 to 0x02e59f (0xc8e bytes) 000917 1749122187.482744 write 0x0334f3 thru 0x035009 (0x1b17 bytes) 000918 1749122187.483795 write 0x033ba8 thru 0x03fdd7 (0xc230 bytes) 000919 1749122187.485702 mapread 0x013a2e thru 0x023064 (0xf637 bytes) 000920 1749122187.490509 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000921 1749122187.627599 read 0x00020a thru 0x00f0c0 (0xeeb7 bytes) 000922 1749122187.634863 punch from 0x011a91 to 0x01f5ec (0xdb5c bytes) 000923 1749122187.639393 write 0x021e8a thru 0x0265e8 (0x475f bytes) 000925 1749122187.648053 trunc from 0x03fdd8 to 0x0363f8 (0x99df bytes) 000926 1749122187.657218 trunc from 0x0363f9 to 0x0241da (0x1221e bytes) 000928 1749122187.668919 punch from 0x0141bb to 0x015269 (0x10af bytes) 000929 1749122187.673462 trunc from 0x0241db to 0x0144e9 (0xfcf1 bytes) 000930 1749122187.684095 mapread 0x00cbf8 thru 0x00de13 (0x121c bytes) 000931 1749122187.684900 mapread 0x006c58 thru 0x013cfd (0xd0a6 bytes) 000932 1749122187.690838 fallocate 0x0319ba thru 0x03ffff (0xe646 bytes) 000933 1749122187.694559 write 0x03a322 thru 0x03ffff (0x5cde bytes) 000934 1749122187.696026 trunc from 0x040000 to 0x015768 (0x2a897 bytes) 000935 1749122187.704570 trunc from 0x015769 to 0x034533 (0x1edcb bytes) 000937 1749122187.719505 MIRROR_RESYNC 4 mirrors 000938 1749122188.067511 mapwrite 0x020b67 thru 0x02dce3 (0xd17d bytes) 000939 1749122188.146628 mapread 0x0195d2 thru 0x0230c5 (0x9af4 bytes) 000940 1749122188.153146 write 0x00e503 thru 0x01d293 (0xed91 bytes) 000941 1749122188.160839 read 0x01e938 thru 0x026f8a (0x8653 bytes) 000942 1749122188.161460 write 0x015173 thru 0x01b414 (0x62a2 bytes) 000943 1749122188.162111 write 0x018327 thru 0x018aa3 (0x77d bytes) 000945 1749122188.165792 MIRROR_RESYNC 4 mirrors 000947 1749122188.556496 trunc from 0x013f80 to 0x01bb23 (0x7ba4 bytes) 000948 1749122188.575924 write 0x028143 thru 0x034281 (0xc13f bytes) 000949 1749122188.583820 trunc from 0x034534 to 0x00fe0a (0x24729 bytes) 000950 1749122188.605837 fallocate 0x035ba9 thru 0x039057 (0x34af bytes) 000951 1749122188.608104 read 0x002430 thru 0x0042b0 (0x1e81 bytes) 000952 1749122188.610080 write 0x01bcf4 thru 0x02a38f (0xe69c bytes) 000953 1749122188.612140 read 0x000b25 thru 0x01033d (0xf819 bytes) 000954 1749122188.616991 trunc from 0x02a390 to 0x009b78 (0x20817 bytes) 000955 1749122188.625767 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000958 1749122188.743166 MIRROR_EXTEND to 4 mirrors 000959 1749122188.953633 write 0x028f1a thru 0x030057 (0x713e bytes) 000960 1749122188.972716 punch from 0x016103 to 0x021f70 (0xbe6e bytes) 000961 1749122188.975589 mapread 0x00e9a7 thru 0x01b6d6 (0xcd30 bytes) 000962 1749122188.987757 trunc from 0x00f0ed to 0x01e008 (0xef1c bytes) 000963 1749122188.991580 trunc from 0x021d5a to 0x025b88 (0x3e2f bytes) 000964 1749122188.993842 trunc from 0x030023 to 0x030057 (0x35 bytes) 000965 1749122189.001759 read 0x0042f4 thru 0x00626e (0x1f7b bytes) 000966 1749122189.005255 mapwrite 0x024c9e thru 0x034890 (0xfbf3 bytes) 000967 1749122189.061226 punch from 0x00a811 to 0x00ca59 (0x2249 bytes) 000968 1749122189.064022 trunc from 0x034891 to 0x013939 (0x20f57 bytes) 000969 1749122189.075328 trunc from 0x012e18 to 0x013939 (0xb22 bytes) 000970 1749122189.077928 mapread 0x006288 thru 0x00ce29 (0x6ba2 bytes) 000971 1749122189.085628 trunc from 0x005834 to 0x013939 (0xe106 bytes) 000972 1749122189.090466 write 0x013ff4 thru 0x01c657 (0x8664 bytes) 000973 1749122189.097572 punch from 0x0070a4 to 0x0139cd (0xc92a bytes) 000974 1749122189.107433 trunc from 0x014e84 to 0x01c657 (0x77d4 bytes) 000975 1749122189.110735 write 0x039a9c thru 0x03ffff (0x6564 bytes) 000976 1749122189.113027 mapread 0x033067 thru 0x03dc90 (0xac2a bytes) 000977 1749122189.116784 fallocate 0x01e8b7 thru 0x025f54 (0x769e bytes) 000978 1749122189.119951 fallocate 0x01cc65 thru 0x021b5c (0x4ef8 bytes) 000979 1749122189.126184 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000980 1749122189.241277 trunc from 0x040000 to 0x030314 (0xfceb bytes) 000981 1749122189.253404 trunc from 0x030315 to 0x03f4bb (0xf1a7 bytes) 000982 1749122189.274906 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c:0x0]: Invalid argument (22) 000983 1749122189.373097 trunc from 0x03f4bc to 0x0277ae (0x17d0d bytes) 000984 1749122189.383013 punch from 0x018c68 to 0x01b7da (0x2b73 bytes) 000985 1749122189.387666 MIRROR_EXTEND to 3 mirrors 000986 1749122189.704380 read 0x016725 thru 0x022d51 (0xc62d bytes) 000987 1749122189.718394 MIRROR_RESYNC 3 mirrors 000988 1749122190.074372 write 0x03d0d1 thru 0x03ffff (0x2f2f bytes) 000989 1749122190.100067 trunc from 0x039e21 to 0x03d789 (0x3969 bytes) 000991 1749122190.109081 read 0x0103a3 thru 0x01588a (0x54e8 bytes) 000992 1749122190.117078 write 0x022751 thru 0x02436a (0x1c1a bytes) 000993 1749122190.129616 mapwrite 0x039ea9 thru 0x03ffff (0x6157 bytes) 000994 1749122190.163635 trunc from 0x032049 to 0x038738 (0x66f0 bytes) 000995 1749122190.170734 trunc from 0x040000 to 0x038b61 (0x749e bytes) 000998 1749122190.209416 write 0x03b60d thru 0x03f4e8 (0x3edc bytes) 000999 1749122190.213826 read 0x026d86 thru 0x030247 (0x94c2 bytes) 001000 1749122190.228076 trunc from 0x02ddaa to 0x039946 (0xbb9d bytes) All operations completed A-OK! PASS 70a (30s) == sanity-flr test 71: check mirror extend parent fid ==== 07:16:32 (1749122192) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x200000405:0x56e:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 200a: stress test ===================== 07:16:36 (1749122196) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2943520 .. done Extending file size to 7727712 .. Extending file size to 3499072 .. Extending file size to 6712768 .. Extending file size to 8023008 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8241056 .. Extending file size to 8070976 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8203552 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8261568 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8909984 .. Extending file size to 8708128 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8749152 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9306016 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9160544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 31990 31991 31994 31995 31996 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) 192.168.202.101@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:17:42 (1749122262) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3287552 .. Extending file size to 7988928 .. failed Extending file size to 4360480 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8275680 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8313280 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8440832 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9118528 .. Extending file size to 9132128 .. Extending file size to 8995136 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 2948 2950 2951 2952 2953 done 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) 192.168.202.101@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:18:47 (1749122327) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3172 1284516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2044 1285644 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14680 3592340 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22656 3584364 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37336 7176704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3172 1284516 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2044 1285644 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14680 3592340 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22656 3584364 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37336 7176704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0438855 s, 47.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0728846 s, 28.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:18:53 (1749122333) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0471914 s, 44.4 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 07:18:56 (1749122336) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x63d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x63d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x63e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x63d:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:18:58 (1749122338) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0341923 s, 61.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:19:01 (1749122341) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0423148 s, 49.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x63f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x640:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x63e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x63f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x640:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x63e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x63f:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:19:03 (1749122343) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0280989 s, 74.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x641:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x640:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x641:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x640:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x641:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:19:27 (1749122367) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0284835 s, 36.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x642:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x643:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x642:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x642:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x643:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x642:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x643:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:20:02 (1749122402) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x644:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x645:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0512744 s, 40.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x644:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x645:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x644:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x644:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x645:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x644:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:20:37 (1749122437) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg201-server: Pool lustre.test_204f created Adding targets to pool oleg201-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg201-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.034349 s, 61.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg201-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg201-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg201-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:21:26 (1749122486) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:21:28 (1749122488) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:21:30 (1749122490) oleg201-server: Pool lustre.test_206 created Adding targets to pool oleg201-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg201-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg201-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg201-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg201-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:21:43 (1749122503) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.021659 s, 48.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108263 s, 96.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x7cdfe47d Mirror 3: 0x7cdfe47d PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:21:46 (1749122506) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:21:47 (1749122507) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:21:48 (1749122508) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0187015 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2580000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1800000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1800000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1800000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17e0000 Read this (4 bytes): BBBB 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:22:04 (1749122524) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00443868 s, 1.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:22:06 (1749122526) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016127 s, 65.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59b lmm_fid: [0x200000405:0x59b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59b lmm_fid: [0x200000405:0x59b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x650:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339612571 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-06-05 07:22:06.000000000 -0400 Modify: 2025-06-05 07:22:06.000000000 -0400 Change: 2025-06-05 07:22:07.000000000 -0400 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:22:08 (1749122528) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114998 s, 91.2 MB/s oleg201-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg201-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg201-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg201-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg201-server: O/280000401/d17/651: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:22:22 (1749122542) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0569671 s, 719 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0143437 s, 286 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 5 07:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5a0:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 5 07:22 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 894 sec ============ 07:22:25 (1749122545) === sanity-flr: start cleanup 07:22:25 (1749122545) === === sanity-flr: finish cleanup 07:22:28 (1749122548) ===