-----============= acceptance-small: sanity-flr ============----- Wed May 21 13:13:09 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg451-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:13:14 (1747847594) === oleg451-client.virtnet: executing check_config_client /mnt/lustre oleg451-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg451-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012a410000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a410000.idle_timeout=debug disable quota as required oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:13:24 (1747847604) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7950] == sanity-flr test 0a: lfs mirror create with -N option == 13:13:24 (1747847604) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:13:29 (1747847609) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:13:30 (1747847610) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:13:31 (1747847611) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:13:38 (1747847618) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (0s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:13:38 (1747847618) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 13:13:39 (1747847619) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88012a410000.stats=clear osc.lustre-OST0001-osc-ffff88012a410000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:13:42 (1747847622) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:13:45 (1747847625) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:13:47 (1747847627) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:13:48 (1747847628) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 13:13:51 (1747847631) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:13:53 (1747847633) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:13:56 (1747847636) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:13:58 (1747847638) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:14:02 (1747847642) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:14:04 (1747847644) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0624783 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0377826 s, 27.8 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:14:07 (1747847647) 19+0 records in 19+0 records out 19922944 bytes (20 MB) copied, 0.210343 s, 94.7 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:14:10 (1747847650) Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a410000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a410000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a410000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a410000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:14:37 (1747847677) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.112168 s, 46.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a410000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a410000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:14:53 (1747847693) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a410000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a410000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a410000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a410000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:15:19 (1747847719) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.02178 s, 51.3 MB/s -rw-r--r-- 1 root root 52428800 May 21 13:15 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.347s user 0m0.000s sys 0m0.121s Stopping /mnt/lustre-ost1 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88012a410000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a410000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.362s user 0m0.000s sys 0m0.129s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg451-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg451-server oleg451-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88012a410000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a410000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.386s user 0m0.002s sys 0m0.124s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg451-server: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg451-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg451-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012a410000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012a410000.ost_server_uuid in FULL state after 0 sec PASS 33b (25s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:15:44 (1747847744) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:15:45 (1747847745) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:15:46 (1747847746) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (0s) == sanity-flr test 35: allow to write to mirrored files == 13:15:46 (1747847746) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 13:15:48 (1747847748) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.154415 s, 81.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:15:51 (1747847751) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111315 s, 94.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110486 s, 94.9 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:15:53 (1747847753) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:15:56 (1747847756) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000681446 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000906714 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000684254 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000827086 s, 1.2 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 13:15:58 (1747847758) create test files with size 15752833 .. 68d0e350bee191a1cb2dfe1765a53ffd - bfda5781e168d638ffbc3d311381f7ea - 91217a141d66724539cadb28f927ed80 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 13:16:03 (1747847763) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:16:06 (1747847766) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:16:08 (1747847768) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0232391 s, 45.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105429 s, 99.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 13:16:10 (1747847770) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0245225 s, 85.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0911121 s, 46.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0350995 s, 59.7 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 13:16:12 (1747847772) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 13:16:13 (1747847773) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:16:14 (1747847774) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107991 s, 97.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 13:16:16 (1747847776) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.092833 s, 33.9 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:16:18 (1747847778) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4a:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x48:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:16:20 (1747847780) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0768622 s, 136 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:16:43 (1747847803) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff880130910800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.11677 s, 89.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg451-client.virtnet /mnt/lustre (opts:) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a3def000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:16:48 (1747847808) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (9s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:16:57 (1747847817) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:17:07 (1747847827) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 13:17:08 (1747847828) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.107347 s, 29.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0749791 s, 42.0 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: e898522d2542d5f0844c4d5e515d1d3f - mirror 1 checksum: 334c7b46ac5cd6c8cfb26e95f9b9bbf3 - mirror 2 checksum: e898522d2542d5f0844c4d5e515d1d3f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 21 13:17:08 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 21 13:17:08 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e898522d2542d5f0844c4d5e515d1d3f - mirror 1 checksum: 334c7b46ac5cd6c8cfb26e95f9b9bbf3 - mirror 2 checksum: 334c7b46ac5cd6c8cfb26e95f9b9bbf3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 21 13:17:08 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 21 13:17:08 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:17:11 (1747847831) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125777 s, 83.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 529408.. 530431: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:17:13 (1747847833) Failing mds1 on oleg451-server Stopping /mnt/lustre-mds1 (opts:) on oleg451-server 13:17:21 (1747847841) shut down facet: mds1 facet_host: oleg451-server facet_failover_host: oleg451-server Failover mds1 to oleg451-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 13:17:36 (1747847856) targets are mounted 13:17:36 (1747847856) facet_failover done oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013603 s, 77.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:17:44 (1747847864) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0122134 s, 1.7 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00534559 s, 3.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:17:48 (1747847868) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0186672 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00571932 s, 3.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:17:52 (1747847872) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00555144 s, 3.0 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:17:55 (1747847875) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0194302 s, 1.1 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:17:58 (1747847878) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00466689 s, 878 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0255315 s, 160 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:18:02 (1747847882) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:18:11 (1747847891) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:18:32 (1747847912) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:18:50 (1747847930) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:19:09 (1747847949) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 13:19:12 (1747847952) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 26703 mirror split pid 26705 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4942: 26703 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4942: 26705 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 13:20:14 (1747848014) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8015 000001 1747848015.623049 mapwrite 0x02adfa thru 0x035a50 (0xac57 bytes) truncating to largest ever: 0x32d38 000002 1747848015.666567 trunc from 0x035a51 to 0x032d37 (0x2d19 bytes) 000003 1747848015.674083 fallocate 0x014496 thru 0x020da3 (0xc90e bytes) 000004 1747848015.677755 trunc from 0x01830a to 0x024492 (0xc189 bytes) 000005 1747848015.680845 punch from 0x032ac1 to 0x032d37 (0x277 bytes) 000006 1747848015.684705 read 0x018660 thru 0x02611a (0xdabb bytes) 000007 1747848015.690706 write 0x00a569 thru 0x00e5aa (0x4042 bytes) fallocating to largest ever: 0x3ab1f 000008 1747848015.696429 fallocate 0x02eb7d thru 0x03ab1e (0xbfa2 bytes) 000009 1747848015.700166 write 0x00d762 thru 0x01c5ce (0xee6d bytes) 000010 1747848015.701879 trunc from 0x032d38 to 0x03742f (0x46f8 bytes) 000011 1747848015.722275 MIRROR_EXTEND to 2 mirrors 000012 1747848016.033751 MIRROR_EXTEND to 3 mirrors 000013 1747848016.340266 trunc from 0x0182c5 to 0x01f147 (0x6e83 bytes) 000014 1747848016.363407 read 0x026073 thru 0x0283b2 (0x2340 bytes) 000015 1747848016.367665 mapread 0x012166 thru 0x01ef08 (0xcda3 bytes) 000016 1747848016.380983 trunc from 0x010b0b to 0x01ebb0 (0xe0a6 bytes) 000017 1747848016.384401 punch from 0x007208 to 0x01334d (0xc146 bytes) 000018 1747848016.390889 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000019 1747848016.502164 mapwrite 0x00daf6 thru 0x01d697 (0xfba2 bytes) 000020 1747848016.549502 punch from 0x024672 to 0x02c140 (0x7acf bytes) 000021 1747848016.552437 punch from 0x02cbb2 to 0x02e360 (0x17af bytes) 000022 1747848016.555546 trunc from 0x037430 to 0x02ef6a (0x84c5 bytes) 000023 1747848016.581041 write 0x035bef thru 0x0362a5 (0x6b7 bytes) 000024 1747848016.584469 MIRROR_EXTEND to 3 mirrors 000025 1747848016.913761 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000027 1747848017.120889 trunc from 0x0362a6 to 0x01209b (0x2420a bytes) 000028 1747848017.158045 mapwrite 0x037d4a thru 0x03ffff (0x82b6 bytes) 000029 1747848017.199663 trunc from 0x038e07 to 0x03ffff (0x71f9 bytes) 000030 1747848017.205976 write 0x03ec15 thru 0x03eeef (0x2db bytes) 000031 1747848017.216408 trunc from 0x040000 to 0x01ece7 (0x21318 bytes) 000032 1747848017.234337 read 0x0002c3 thru 0x00efbb (0xecf9 bytes) 000033 1747848017.247602 read 0x01930d thru 0x01ece7 (0x59db bytes) 000034 1747848017.253445 trunc from 0x01ece8 to 0x0202ea (0x1603 bytes) 000035 1747848017.264926 read 0x000333 thru 0x006a36 (0x6704 bytes) 000036 1747848017.265489 trunc from 0x0202eb to 0x02eb38 (0xe84e bytes) 000037 1747848017.279602 write 0x009121 thru 0x015f46 (0xce26 bytes) 000039 1747848017.281041 punch from 0x02dc21 to 0x02eb38 (0xf18 bytes) 000040 1747848017.285548 trunc from 0x01e1f4 to 0x026656 (0x8463 bytes) 000041 1747848017.290194 read 0x0096aa thru 0x009f7f (0x8d6 bytes) 000042 1747848017.290618 trunc from 0x0233a9 to 0x02eb38 (0xb790 bytes) 000044 1747848017.295287 mapread 0x009ae9 thru 0x00c46f (0x2987 bytes) 000045 1747848017.296363 read 0x020e64 thru 0x02a87e (0x9a1b bytes) 000046 1747848017.304671 read 0x0084cf thru 0x0100b8 (0x7bea bytes) 000048 1747848017.305723 fallocate 0x0236f3 thru 0x024451 (0xd5f bytes) 000049 1747848017.308321 mapread 0x01fef4 thru 0x027fcb (0x80d8 bytes) 000050 1747848017.310484 write 0x0086ba thru 0x00c7f0 (0x4137 bytes) 000051 1747848017.313567 punch from 0x01fb15 to 0x026fcc (0x74b8 bytes) 000052 1747848017.320969 write 0x032e8f thru 0x032f9a (0x10c bytes) 000053 1747848017.323411 fallocate 0x00f895 thru 0x01476c (0x4ed8 bytes) 000055 1747848017.329053 trunc from 0x032f9b to 0x00aca3 (0x282f7 bytes) 000056 1747848017.343219 mapwrite 0x034e08 thru 0x03f1e1 (0xa3da bytes) 000057 1747848017.388033 read 0x00b4aa thru 0x017e32 (0xc989 bytes) 000058 1747848017.388851 punch from 0x02251c to 0x029358 (0x6e3d bytes) 000059 1747848017.394274 read 0x001c81 thru 0x00791e (0x5c9e bytes) 000060 1747848017.394771 read 0x02e02b thru 0x03d192 (0xf168 bytes) 000061 1747848017.395331 write 0x007da3 thru 0x00e5c9 (0x6827 bytes) 000062 1747848017.397413 write 0x03f222 thru 0x03ffff (0xdde bytes) 000063 1747848017.398056 punch from 0x004fab to 0x00aca5 (0x5cfb bytes) 000064 1747848017.409614 mapwrite 0x0188a6 thru 0x01b320 (0x2a7b bytes) 000065 1747848017.426148 trunc from 0x01bdbc to 0x026179 (0xa3be bytes) 000066 1747848017.429791 trunc from 0x00bed0 to 0x00d258 (0x1389 bytes) 000067 1747848017.433008 trunc from 0x018310 to 0x021e6f (0x9b60 bytes) 000069 1747848017.437199 trunc from 0x037d95 to 0x03ffff (0x826b bytes) 000070 1747848017.441222 trunc from 0x03c42a to 0x03ffff (0x3bd6 bytes) 000071 1747848017.443895 trunc from 0x010a4a to 0x01d068 (0xc61f bytes) 000072 1747848017.446381 read 0x02144f thru 0x02cd70 (0xb922 bytes) 000073 1747848017.452250 mapread 0x032152 thru 0x03ba93 (0x9942 bytes) 000074 1747848017.454469 mapread 0x004ffa thru 0x00dec3 (0x8eca bytes) 000075 1747848017.455094 mapread 0x03e284 thru 0x03ffff (0x1d7c bytes) 000077 1747848017.455806 mapread 0x00ee6c thru 0x014768 (0x58fd bytes) 000078 1747848017.456450 trunc from 0x040000 to 0x005271 (0x3ad8e bytes) 000079 1747848017.466006 trunc from 0x005272 to 0x0024f8 (0x2d79 bytes) 000080 1747848017.477547 MIRROR_RESYNC 2 mirrors 000081 1747848017.694217 MIRROR_RESYNC 2 mirrors 000082 1747848017.762189 fallocate 0x014165 thru 0x01ce12 (0x8cae bytes) 000083 1747848017.782258 mapwrite 0x014223 thru 0x01fccb (0xbaa9 bytes) 000084 1747848017.843549 trunc from 0x01fccc to 0x000627 (0x1f6a4 bytes) 000085 1747848017.856340 read 0x0004cc thru 0x000627 (0x15c bytes) 000086 1747848017.860406 fallocate 0x024c43 thru 0x02ad5d (0x611b bytes) 000087 1747848017.864538 punch from 0x000331 to 0x000627 (0x2f7 bytes) 000088 1747848017.870739 mapread 0x000040 thru 0x000627 (0x5e8 bytes) 000089 1747848017.874841 mapwrite 0x02274e thru 0x024a0e (0x22c1 bytes) 000090 1747848017.904538 punch from 0x012f6f to 0x01c1dd (0x926f bytes) 000091 1747848017.908172 write 0x01fa03 thru 0x022730 (0x2d2e bytes) 000092 1747848017.908891 trunc from 0x024a0f to 0x034de6 (0x103d8 bytes) 000093 1747848017.920882 punch from 0x01e839 to 0x0289a6 (0xa16e bytes) 000094 1747848017.930473 punch from 0x01fde6 to 0x02cbef (0xce0a bytes) 000095 1747848017.934115 mapwrite 0x00600f thru 0x009700 (0x36f2 bytes) 000096 1747848017.954484 fallocate 0x01892b thru 0x0197a8 (0xe7e bytes) 000097 1747848017.959487 write 0x02a733 thru 0x038108 (0xd9d6 bytes) 000098 1747848017.965839 mapread 0x0058f0 thru 0x00d7d3 (0x7ee4 bytes) 000099 1747848017.967191 fallocate 0x02b604 thru 0x02ee40 (0x383d bytes) 000100 1747848017.973279 mapwrite 0x008593 thru 0x011875 (0x92e3 bytes) 000101 1747848018.010460 fallocate 0x02622c thru 0x02e490 (0x8265 bytes) 000102 1747848018.018741 fallocate 0x0197e2 thru 0x0239cb (0xa1ea bytes) 000103 1747848018.022131 read 0x006c17 thru 0x011f76 (0xb360 bytes) 000104 1747848018.022860 fallocate 0x02b7ab thru 0x032eb5 (0x770b bytes) 000105 1747848018.027396 fallocate 0x006283 thru 0x013348 (0xd0c6 bytes) 000106 1747848018.030845 read 0x020368 thru 0x022c3d (0x28d6 bytes) 000107 1747848018.039479 MIRROR_EXTEND to 3 mirrors 000108 1747848018.305456 trunc from 0x029c4c to 0x02bf25 (0x22da bytes) 000109 1747848018.327827 trunc from 0x02698d to 0x02c06c (0x56e0 bytes) 000110 1747848018.331035 read 0x036e36 thru 0x038108 (0x12d3 bytes) 000111 1747848018.333955 mapwrite 0x019aab thru 0x0290ca (0xf620 bytes) truncating to largest ever: 0x3f996 000113 1747848018.365343 trunc from 0x038109 to 0x03f995 (0x788d bytes) 000114 1747848018.376237 read 0x015302 thru 0x0208c1 (0xb5c0 bytes) 000115 1747848018.376763 trunc from 0x03f996 to 0x00d2cd (0x326c8 bytes) 000116 1747848018.385590 trunc from 0x00d2ce to 0x03a824 (0x2d557 bytes) 000117 1747848018.394420 trunc from 0x03a825 to 0x0312f9 (0x952b bytes) 000118 1747848018.403133 read 0x0228ef thru 0x02661c (0x3d2e bytes) 000119 1747848018.411338 MIRROR_RESYNC 3 mirrors 000120 1747848018.675512 MIRROR_RESYNC 3 mirrors 000121 1747848018.743749 MIRROR_EXTEND to 4 mirrors 000122 1747848018.929781 punch from 0x02719d to 0x02ef7f (0x7de3 bytes) 000123 1747848018.942041 punch from 0x01ada7 to 0x02995f (0xebb9 bytes) 000124 1747848018.944374 mapwrite 0x02606e thru 0x02ca80 (0x6a13 bytes) 000125 1747848018.963831 punch from 0x006c92 to 0x00c7b0 (0x5b1f bytes) 000126 1747848018.969795 MIRROR_EXTEND to 5 mirrors 000127 1747848019.197022 read 0x02a226 thru 0x0312f9 (0x70d4 bytes) 000128 1747848019.205008 trunc from 0x015377 to 0x01bc4c (0x68d6 bytes) 000129 1747848019.225289 MIRROR_EXTEND to 6 mirrors 000130 1747848019.496071 mapread 0x000ee5 thru 0x00e63e (0xd75a bytes) 000131 1747848019.512501 mapwrite 0x01e4e8 thru 0x026e65 (0x897e bytes) 000132 1747848019.558591 trunc from 0x002f65 to 0x006ea8 (0x3f44 bytes) 000133 1747848019.560829 mapwrite 0x00c297 thru 0x010983 (0x46ed bytes) 000134 1747848019.573161 trunc from 0x0312fa to 0x01996f (0x1798a bytes) 000135 1747848019.583702 trunc from 0x00e2be to 0x014e40 (0x6b83 bytes) 000136 1747848019.586760 fallocate 0x01471f thru 0x01d536 (0x8e18 bytes) 000137 1747848019.590079 mapwrite 0x018701 thru 0x026a42 (0xe342 bytes) 000138 1747848019.616665 mapwrite 0x03f9c4 thru 0x03ffff (0x63c bytes) 000139 1747848019.633081 mapwrite 0x0157b6 thru 0x01727b (0x1ac6 bytes) 000140 1747848019.640237 mapread 0x020a9f thru 0x0255a3 (0x4b05 bytes) 000143 1747848019.641535 fallocate 0x02fe34 thru 0x038e19 (0x8fe6 bytes) 000144 1747848019.644107 read 0x004b47 thru 0x009ac5 (0x4f7f bytes) 000145 1747848019.646621 mapread 0x03baf7 thru 0x03ffff (0x4509 bytes) 000146 1747848019.647373 trunc from 0x019453 to 0x01a32a (0xed8 bytes) 000147 1747848019.650076 punch from 0x00a95c to 0x010a57 (0x60fc bytes) 000148 1747848019.652705 trunc from 0x030d09 to 0x03df7a (0xd272 bytes) 000149 1747848019.655159 trunc from 0x040000 to 0x019993 (0x2666c bytes) 000150 1747848019.661592 mapwrite 0x02d954 thru 0x0384b9 (0xab66 bytes) 000151 1747848019.684219 read 0x006332 thru 0x010805 (0xa4d4 bytes) fallocating to largest ever: 0x40000 000152 1747848019.688334 fallocate 0x0379c1 thru 0x03ffff (0x863f bytes) 000153 1747848019.691083 punch from 0x00604a to 0x0119bd (0xb974 bytes) 000154 1747848019.693665 read 0x03473f thru 0x03fae5 (0xb3a7 bytes) 000157 1747848019.698966 punch from 0x01e143 to 0x0286ec (0xa5aa bytes) 000159 1747848019.702382 fallocate 0x0391c6 thru 0x03d469 (0x42a4 bytes) 000160 1747848019.704859 punch from 0x02804a to 0x0364dc (0xe493 bytes) 000161 1747848019.707389 mapread 0x023bce thru 0x0289e4 (0x4e17 bytes) 000162 1747848019.719067 trunc from 0x040000 to 0x02a72c (0x158d3 bytes) 000163 1747848019.724792 write 0x03a7da thru 0x03d222 (0x2a49 bytes) 000164 1747848019.725502 mapread 0x0301c9 thru 0x03635a (0x6192 bytes) 000165 1747848019.736939 read 0x007eaa thru 0x00d6a8 (0x57ff bytes) 000166 1747848019.737433 write 0x03bbdd thru 0x03d5e6 (0x1a0a bytes) 000167 1747848019.737952 trunc from 0x002f49 to 0x0100ae (0xd166 bytes) 000168 1747848019.740736 fallocate 0x029813 thru 0x0304f8 (0x6ce6 bytes) 000169 1747848019.743237 trunc from 0x03d5e7 to 0x00f912 (0x2dcd4 bytes) 000170 1747848019.749470 mapwrite 0x02da1d thru 0x0362fa (0x88de bytes) 000171 1747848019.772192 trunc from 0x022bc4 to 0x025924 (0x2d61 bytes) 000172 1747848019.775160 mapread 0x00b634 thru 0x01059e (0x4f6b bytes) 000173 1747848019.775856 read 0x003ebd thru 0x005b83 (0x1cc7 bytes) 000174 1747848019.776277 write 0x03ef0d thru 0x03f306 (0x3fa bytes) 000175 1747848019.776971 trunc from 0x03f307 to 0x034a53 (0xa8b3 bytes) 000176 1747848019.784726 write 0x03069e thru 0x03bb57 (0xb4ba bytes) 000177 1747848019.789112 MIRROR_EXTEND to 7 mirrors 000178 1747848020.016634 mapread 0x031eee thru 0x03546b (0x357e bytes) 000182 1747848020.043473 mapwrite 0x0225b5 thru 0x027590 (0x4fdc bytes) 000184 1747848020.103253 MIRROR_RESYNC 7 mirrors 000185 1747848020.731933 fallocate 0x011ab3 thru 0x01eb16 (0xd064 bytes) 000186 1747848020.745087 trunc from 0x029336 to 0x036028 (0xccf3 bytes) 000187 1747848020.749059 MIRROR_RESYNC 7 mirrors 000188 1747848021.441038 MIRROR_EXTEND to 8 mirrors 000189 1747848021.716058 mapread 0x033017 thru 0x033024 (0xe bytes) 000190 1747848021.719024 trunc from 0x028c00 to 0x036e57 (0xe258 bytes) 000191 1747848021.744740 trunc from 0x03bb58 to 0x0319b4 (0xa1a3 bytes) 000192 1747848021.761433 MIRROR_RESYNC 8 mirrors 000193 1747848022.536116 fallocate 0x00b431 thru 0x01994a (0xe51a bytes) 000194 1747848022.560231 read 0x0202a5 thru 0x02dcd5 (0xda31 bytes) 000195 1747848022.566976 fallocate 0x03d9a8 thru 0x03e88a (0xee3 bytes) 000196 1747848022.570406 trunc from 0x02edd0 to 0x03353c (0x476d bytes) 000198 1747848022.578835 read 0x01a6ad thru 0x01aaa4 (0x3f8 bytes) 000199 1747848022.593825 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000200 1747848022.697339 trunc from 0x03e88b to 0x03140f (0xd47b bytes) 000201 1747848022.717835 trunc from 0x031410 to 0x018fe3 (0x1842c bytes) 000202 1747848022.728379 punch from 0x002978 to 0x006102 (0x378b bytes) 000203 1747848022.731551 fallocate 0x01d880 thru 0x01dbb4 (0x335 bytes) 000204 1747848022.734758 trunc from 0x006ec9 to 0x00e96b (0x7aa3 bytes) 000205 1747848022.738731 fallocate 0x0195d8 thru 0x02086b (0x7294 bytes) 000206 1747848022.744568 read 0x004a1f thru 0x012653 (0xdc35 bytes) 000207 1747848022.751489 mapwrite 0x00fdc9 thru 0x012648 (0x2880 bytes) 000208 1747848022.764337 write 0x02f07b thru 0x03a653 (0xb5d9 bytes) 000209 1747848022.768881 MIRROR_EXTEND to 8 mirrors 000210 1747848023.073627 write 0x0043f4 thru 0x013b80 (0xf78d bytes) 000211 1747848023.097963 punch from 0x01e6ca to 0x022b57 (0x448e bytes) 000212 1747848023.101707 punch from 0x011103 to 0x01c9dd (0xb8db bytes) 000213 1747848023.116114 mapwrite 0x0168c7 thru 0x0210ec (0xa826 bytes) 000214 1747848023.141004 mapread 0x037bae thru 0x038738 (0xb8b bytes) 000215 1747848023.145413 fallocate 0x02479d thru 0x02e3f8 (0x9c5c bytes) 000216 1747848023.148060 mapread 0x033836 thru 0x03a653 (0x6e1e bytes) 000217 1747848023.148717 write 0x000a34 thru 0x00d9d0 (0xcf9d bytes) 000218 1747848023.152818 trunc from 0x020e80 to 0x02b82d (0xa9ae bytes) 000219 1747848023.157178 fallocate 0x032d6d thru 0x03cb60 (0x9df4 bytes) 000220 1747848023.161398 mapread 0x014a3a thru 0x014afa (0xc1 bytes) 000221 1747848023.162594 mapwrite 0x02a41a thru 0x02ef52 (0x4b39 bytes) 000222 1747848023.181810 punch from 0x009c8f to 0x00de4b (0x41bd bytes) 000223 1747848023.184142 fallocate 0x00d13e thru 0x011bbe (0x4a81 bytes) 000225 1747848023.186723 punch from 0x0373e0 to 0x03a653 (0x3274 bytes) 000226 1747848023.189715 trunc from 0x020a82 to 0x020d65 (0x2e4 bytes) 000227 1747848023.192939 trunc from 0x02bd5d to 0x03a653 (0xe8f7 bytes) 000228 1747848023.195202 write 0x019911 thru 0x01c0b7 (0x27a7 bytes) 000229 1747848023.198593 MIRROR_EXTEND to 9 mirrors 000230 1747848023.528472 read 0x005502 thru 0x005716 (0x215 bytes) 000231 1747848023.534987 mapread 0x024ec9 thru 0x0281ac (0x32e4 bytes) 000232 1747848023.560516 MIRROR_RESYNC 9 mirrors 000233 1747848024.455067 mapwrite 0x00d002 thru 0x0177fa (0xa7f9 bytes) 000234 1747848024.538101 write 0x00bf09 thru 0x00ea7d (0x2b75 bytes) 000235 1747848024.538902 read 0x015244 thru 0x01f762 (0xa51f bytes) 000236 1747848024.539372 write 0x020985 thru 0x02ed54 (0xe3d0 bytes) 000237 1747848024.549958 punch from 0x01a2ff to 0x0266f5 (0xc3f7 bytes) 000238 1747848024.558407 trunc from 0x03a654 to 0x01c429 (0x1e22a bytes) 000240 1747848024.579837 MIRROR_EXTEND to 10 mirrors 000241 1747848024.901602 mapwrite 0x03e6c9 thru 0x03ffff (0x1937 bytes) 000242 1747848024.957163 trunc from 0x01f04c to 0x02e80e (0xf7c3 bytes) 000243 1747848024.961769 mapwrite 0x018b09 thru 0x024964 (0xbe5c bytes) 000244 1747848025.000954 read 0x037bfc thru 0x03e1bb (0x65c0 bytes) 000245 1747848025.006166 write 0x00d344 thru 0x01ba00 (0xe6bd bytes) 000246 1747848025.013613 read 0x03a377 thru 0x03ffff (0x5c89 bytes) 000247 1747848025.015409 write 0x02f2cc thru 0x030ae4 (0x1819 bytes) 000248 1747848025.025608 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000249 1747848025.167853 punch from 0x0262d2 to 0x02ac12 (0x4941 bytes) 000251 1747848025.171735 read 0x015a00 thru 0x025318 (0xf919 bytes) 000252 1747848025.178878 mapread 0x03a74d thru 0x03c70f (0x1fc3 bytes) 000253 1747848025.191804 mapwrite 0x03b7d2 thru 0x03e859 (0x3088 bytes) 000254 1747848025.199925 write 0x027f95 thru 0x037034 (0xf0a0 bytes) 000255 1747848025.208122 write 0x03b9e8 thru 0x03ffff (0x4618 bytes) 000257 1747848025.209465 punch from 0x014370 to 0x019b17 (0x57a8 bytes) 000258 1747848025.213425 read 0x0135e2 thru 0x02271f (0xf13e bytes) 000259 1747848025.221154 trunc from 0x02ce9f to 0x03c064 (0xf1c6 bytes) 000260 1747848025.230637 fallocate 0x0036c3 thru 0x006972 (0x32b0 bytes) 000261 1747848025.233486 fallocate 0x002fd4 thru 0x01010f (0xd13c bytes) 000262 1747848025.236417 trunc from 0x0324ba to 0x0374b4 (0x4ffb bytes) 000263 1747848025.242526 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000264 1747848025.375022 mapread 0x0381e7 thru 0x03ffff (0x7e19 bytes) 000265 1747848025.392008 mapwrite 0x0076b6 thru 0x00a824 (0x316f bytes) 000266 1747848025.409583 mapwrite 0x02db5e thru 0x039b74 (0xc017 bytes) 000267 1747848025.428447 read 0x0222c8 thru 0x024f90 (0x2cc9 bytes) 000268 1747848025.431783 punch from 0x0344e9 to 0x03c5f3 (0x810b bytes) 000269 1747848025.435106 punch from 0x03a035 to 0x03ffff (0x5fcb bytes) 000270 1747848025.438413 trunc from 0x040000 to 0x03b308 (0x4cf7 bytes) 000271 1747848025.456536 punch from 0x01519a to 0x01524b (0xb2 bytes) 000272 1747848025.460572 fallocate 0x038f43 thru 0x03ffff (0x70bd bytes) 000273 1747848025.463250 read 0x005c6d thru 0x014e3a (0xf1ce bytes) 000274 1747848025.468801 trunc from 0x02a447 to 0x02f3f0 (0x4faa bytes) 000275 1747848025.472883 write 0x027ea2 thru 0x034bad (0xcd0c bytes) 000276 1747848025.473990 trunc from 0x02870b to 0x02ceb9 (0x47af bytes) 000277 1747848025.482975 write 0x02604b thru 0x028d21 (0x2cd7 bytes) 000278 1747848025.487637 fallocate 0x012ea2 thru 0x020853 (0xd9b2 bytes) 000279 1747848025.492338 trunc from 0x040000 to 0x033e9a (0xc165 bytes) 000281 1747848025.499947 trunc from 0x02324f to 0x02ef31 (0xbce3 bytes) 000282 1747848025.511548 trunc from 0x033e9b to 0x030e2e (0x306c bytes) 000283 1747848025.518694 punch from 0x01a0f4 to 0x02562f (0xb53c bytes) 000284 1747848025.523219 trunc from 0x02d083 to 0x030e2e (0x3dac bytes) 000285 1747848025.530054 MIRROR_RESYNC 8 mirrors 000288 1747848026.452273 read 0x019ae6 thru 0x01a30e (0x829 bytes) 000289 1747848026.455416 punch from 0x01adf0 to 0x0288f3 (0xdb04 bytes) 000290 1747848026.482160 fallocate 0x02ac06 thru 0x02ea55 (0x3e50 bytes) 000291 1747848026.485727 mapread 0x0137c9 thru 0x01fac5 (0xc2fd bytes) 000292 1747848026.502034 mapwrite 0x02ba34 thru 0x038ebc (0xd489 bytes) 000293 1747848026.537783 mapwrite 0x0247e2 thru 0x02bead (0x76cc bytes) 000294 1747848026.552470 trunc from 0x038ebd to 0x0227b4 (0x16708 bytes) 000295 1747848026.559953 trunc from 0x0227b5 to 0x00afbd (0x177f7 bytes) 000296 1747848026.570286 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000297 1747848026.658945 read 0x00000e thru 0x00afbd (0xafb0 bytes) 000298 1747848026.662875 trunc from 0x00afbe to 0x00767e (0x393f bytes) 000299 1747848026.671426 mapwrite 0x03258b thru 0x032a54 (0x4ca bytes) 000300 1747848026.691625 mapwrite 0x035299 thru 0x03f073 (0x9ddb bytes) 000301 1747848026.725057 fallocate 0x00209f thru 0x008643 (0x65a5 bytes) 000302 1747848026.728557 read 0x01575c thru 0x01c28f (0x6b34 bytes) 000303 1747848026.736579 trunc from 0x00fa5c to 0x013ec5 (0x446a bytes) 000304 1747848026.739568 mapread 0x035492 thru 0x039d61 (0x48d0 bytes) 000305 1747848026.740508 fallocate 0x03e636 thru 0x03ffff (0x19ca bytes) 000306 1747848026.744357 mapwrite 0x02d722 thru 0x030959 (0x3238 bytes) 000308 1747848026.754049 fallocate 0x0391f8 thru 0x03ffff (0x6e08 bytes) 000309 1747848026.761132 MIRROR_RESYNC 7 mirrors 000310 1747848027.522125 write 0x02638f thru 0x02c89f (0x6511 bytes) 000312 1747848027.539506 punch from 0x0399e7 to 0x03ffff (0x6619 bytes) 000313 1747848027.542389 trunc from 0x023298 to 0x02605c (0x2dc5 bytes) 000314 1747848027.551007 fallocate 0x0398cb thru 0x03ffff (0x6735 bytes) 000315 1747848027.553657 punch from 0x007867 to 0x00e4b3 (0x6c4d bytes) 000317 1747848027.556834 write 0x01954e thru 0x0217af (0x8262 bytes) 000318 1747848027.561101 trunc from 0x040000 to 0x03401a (0xbfe5 bytes) 000319 1747848027.571018 read 0x011942 thru 0x013ced (0x23ac bytes) 000320 1747848027.573699 trunc from 0x03401b to 0x037fb3 (0x3f99 bytes) 000321 1747848027.582596 MIRROR_EXTEND to 8 mirrors 000322 1747848027.800112 read 0x01329e thru 0x0218f1 (0xe654 bytes) 000323 1747848027.808482 trunc from 0x02c18e to 0x031d73 (0x5be6 bytes) 000324 1747848027.827077 trunc from 0x037fb4 to 0x00e9ac (0x29607 bytes) 000325 1747848027.845469 mapwrite 0x03cf8c thru 0x03ffff (0x3074 bytes) 000326 1747848027.873107 mapread 0x0210a4 thru 0x02a02f (0x8f8c bytes) 000327 1747848027.881493 write 0x033be9 thru 0x03729b (0x36b3 bytes) 000328 1747848027.885539 read 0x0160f9 thru 0x01d66d (0x7575 bytes) 000329 1747848027.896079 write 0x037359 thru 0x03ffff (0x8ca7 bytes) 000330 1747848027.897043 trunc from 0x040000 to 0x018462 (0x27b9d bytes) 000331 1747848027.905766 punch from 0x0057ca to 0x011021 (0xb858 bytes) 000332 1747848027.909759 mapwrite 0x035903 thru 0x035fc4 (0x6c2 bytes) 000333 1747848027.930731 mapread 0x01e56d thru 0x021318 (0x2dac bytes) 000334 1747848027.941830 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000336 1747848028.017168 read 0x004355 thru 0x011f99 (0xdc45 bytes) 000337 1747848028.021505 read 0x01e61d thru 0x022e24 (0x4808 bytes) 000338 1747848028.027697 mapwrite 0x03d8c9 thru 0x03e7df (0xf17 bytes) 000339 1747848028.051539 fallocate 0x003d45 thru 0x00edba (0xb076 bytes) 000340 1747848028.054132 mapwrite 0x02e5be thru 0x0394f5 (0xaf38 bytes) 000341 1747848028.062051 punch from 0x00ca36 to 0x0138a4 (0x6e6f bytes) 000342 1747848028.064281 trunc from 0x03e7e0 to 0x039129 (0x56b6 bytes) 000344 1747848028.070657 mapwrite 0x01952f thru 0x01b0ac (0x1b7e bytes) 000346 1747848028.080953 write 0x00a8b3 thru 0x011daf (0x74fd bytes) 000347 1747848028.084213 mapwrite 0x022f66 thru 0x024f6d (0x2008 bytes) 000348 1747848028.093079 fallocate 0x013302 thru 0x0193d0 (0x60cf bytes) 000349 1747848028.096221 write 0x024ab8 thru 0x032418 (0xd961 bytes) 000350 1747848028.097315 trunc from 0x03912a to 0x0272c9 (0x11e60 bytes) 000351 1747848028.105950 write 0x0209e2 thru 0x021870 (0xe8f bytes) 000352 1747848028.106895 fallocate 0x03de51 thru 0x03ffff (0x21af bytes) 000353 1747848028.109356 mapread 0x01f460 thru 0x02455f (0x5100 bytes) 000354 1747848028.109954 trunc from 0x040000 to 0x02b02b (0x14fd4 bytes) 000355 1747848028.119767 write 0x00f3be thru 0x01bbcf (0xc812 bytes) 000356 1747848028.120892 mapread 0x00f4c3 thru 0x0165f6 (0x7134 bytes) 000357 1747848028.123645 MIRROR_EXTEND to 8 mirrors 000358 1747848028.298424 fallocate 0x00446e thru 0x005c71 (0x1804 bytes) 000359 1747848028.310506 mapread 0x01c433 thru 0x02b02b (0xebf9 bytes) 000360 1747848028.320843 trunc from 0x02b02c to 0x020faa (0xa081 bytes) 000362 1747848028.336211 write 0x03c907 thru 0x03f238 (0x2932 bytes) 000363 1747848028.337063 mapread 0x030c3e thru 0x0310d2 (0x495 bytes) 000364 1747848028.340445 trunc from 0x021c05 to 0x028a5a (0x6e56 bytes) 000365 1747848028.344840 MIRROR_RESYNC 8 mirrors 000367 1747848028.950770 trunc from 0x03f239 to 0x0243e1 (0x1ae57 bytes) 000368 1747848028.970358 mapread 0x00a7d2 thru 0x00e103 (0x3932 bytes) 000369 1747848028.977087 trunc from 0x01f451 to 0x0243e1 (0x4f91 bytes) 000370 1747848028.979721 fallocate 0x00a811 thru 0x017f72 (0xd762 bytes) 000371 1747848028.982467 punch from 0x00a2b6 to 0x011a15 (0x7760 bytes) 000373 1747848028.985732 trunc from 0x017db8 to 0x018aae (0xcf7 bytes) 000374 1747848028.988619 trunc from 0x0243e2 to 0x03f97e (0x1b59d bytes) 000375 1747848028.998608 mapread 0x003745 thru 0x005163 (0x1a1f bytes) 000376 1747848029.002225 mapwrite 0x00f44f thru 0x013fc8 (0x4b7a bytes) 000377 1747848029.022609 read 0x03ea18 thru 0x03f97e (0xf67 bytes) 000378 1747848029.026911 fallocate 0x0167cd thru 0x01dd66 (0x759a bytes) 000379 1747848029.032824 MIRROR_EXTEND to 9 mirrors 000380 1747848029.200220 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000381 1747848029.300141 MIRROR_EXTEND to 9 mirrors 000382 1747848029.459026 trunc from 0x011e21 to 0x01b3e9 (0x95c9 bytes) 000383 1747848029.470972 mapwrite 0x028f45 thru 0x037623 (0xe6df bytes) 000384 1747848029.488611 fallocate 0x011d31 thru 0x01c942 (0xac12 bytes) 000385 1747848029.491402 trunc from 0x03f97f to 0x03cb38 (0x2e46 bytes) 000386 1747848029.505944 read 0x013482 thru 0x016895 (0x3414 bytes) 000387 1747848029.510399 MIRROR_RESYNC 9 mirrors 000388 1747848030.353143 trunc from 0x0328a9 to 0x03751a (0x4c72 bytes) 000389 1747848030.366196 write 0x03c03a thru 0x03ffff (0x3fc6 bytes) 000391 1747848030.370710 read 0x00939a thru 0x01114d (0x7db4 bytes) 000392 1747848030.375051 punch from 0x012bf1 to 0x0167c2 (0x3bd2 bytes) 000393 1747848030.377689 mapread 0x035a4c thru 0x03ffff (0xa5b4 bytes) 000395 1747848030.383491 punch from 0x023da4 to 0x03357a (0xf7d7 bytes) 000396 1747848030.388089 fallocate 0x032037 thru 0x03ffff (0xdfc9 bytes) 000397 1747848030.391897 write 0x0344e8 thru 0x03c839 (0x8352 bytes) 000398 1747848030.392842 write 0x03d9e1 thru 0x03ffff (0x261f bytes) 000400 1747848030.393851 mapread 0x03f0bd thru 0x03ffff (0xf43 bytes) 000401 1747848030.394489 mapwrite 0x02ba6d thru 0x02d75b (0x1cef bytes) 000402 1747848030.411213 mapread 0x01dba5 thru 0x020b9e (0x2ffa bytes) 000403 1747848030.415782 punch from 0x019f30 to 0x01ee16 (0x4ee7 bytes) 000404 1747848030.418496 read 0x028d5c thru 0x02feaa (0x714f bytes) 000406 1747848030.419372 write 0x016253 thru 0x01d268 (0x7016 bytes) 000407 1747848030.425119 mapwrite 0x023e58 thru 0x02c9d2 (0x8b7b bytes) 000408 1747848030.438691 punch from 0x02d207 to 0x039c09 (0xca03 bytes) 000409 1747848030.442009 write 0x0262ae thru 0x0300a9 (0x9dfc bytes) 000410 1747848030.445557 trunc from 0x040000 to 0x02b9ca (0x14635 bytes) 000411 1747848030.454827 mapwrite 0x019499 thru 0x01b979 (0x24e1 bytes) 000412 1747848030.466071 read 0x020a13 thru 0x02b9ca (0xafb8 bytes) 000413 1747848030.470033 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000414 1747848030.547124 trunc from 0x02b9cb to 0x00bd17 (0x1fcb3 bytes) 000415 1747848030.554584 trunc from 0x0009bc to 0x00bd17 (0xb35c bytes) 000416 1747848030.559557 MIRROR_RESYNC 8 mirrors 000417 1747848030.878638 write 0x001797 thru 0x009acb (0x8335 bytes) 000418 1747848030.895391 mapread 0x0045c4 thru 0x00bd17 (0x7754 bytes) 000419 1747848030.898452 trunc from 0x005194 to 0x00735d (0x21ca bytes) 000421 1747848030.906244 write 0x037208 thru 0x03ffff (0x8df8 bytes) 000422 1747848030.907856 mapread 0x029794 thru 0x029fcd (0x83a bytes) 000423 1747848030.912190 mapread 0x00838a thru 0x00b214 (0x2e8b bytes) 000424 1747848030.912794 mapread 0x03820d thru 0x03aba6 (0x299a bytes) 000425 1747848030.913375 write 0x007389 thru 0x00fa64 (0x86dc bytes) 000426 1747848030.920224 mapwrite 0x03d6a4 thru 0x03ffff (0x295c bytes) 000427 1747848030.932944 write 0x02ad5d thru 0x02b26e (0x512 bytes) 000428 1747848030.933639 mapread 0x02e51f thru 0x0386c1 (0xa1a3 bytes) 000429 1747848030.938319 read 0x01024f thru 0x016db5 (0x6b67 bytes) 000430 1747848030.947888 write 0x024338 thru 0x025ec2 (0x1b8b bytes) 000431 1747848030.948888 trunc from 0x010b02 to 0x020349 (0xf848 bytes) 000432 1747848030.952460 trunc from 0x0117d7 to 0x01fa44 (0xe26e bytes) 000433 1747848030.955224 punch from 0x025f92 to 0x02ce33 (0x6ea2 bytes) 000434 1747848030.966142 read 0x0352b6 thru 0x0392a8 (0x3ff3 bytes) 000436 1747848030.970369 MIRROR_RESYNC 8 mirrors 000437 1747848031.557494 punch from 0x02b4ee to 0x036572 (0xb085 bytes) 000438 1747848031.572207 mapwrite 0x00b98b thru 0x018833 (0xcea9 bytes) 000439 1747848031.595489 read 0x02c124 thru 0x033f5c (0x7e39 bytes) 000441 1747848031.600275 write 0x02ab31 thru 0x03409b (0x956b bytes) 000442 1747848031.605682 punch from 0x0359fa to 0x03c33a (0x6941 bytes) 000443 1747848031.609003 mapread 0x02d518 thru 0x03bff0 (0xead9 bytes) 000444 1747848031.613855 trunc from 0x040000 to 0x02571c (0x1a8e3 bytes) 000445 1747848031.622522 trunc from 0x02571d to 0x0003c0 (0x2535c bytes) 000446 1747848031.629491 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000447 1747848031.705246 trunc from 0x0003c1 to 0x003b2b (0x376b bytes) 000448 1747848031.712251 punch from 0x000eaa to 0x003b2b (0x2c82 bytes) 000449 1747848031.714103 punch from 0x00223b to 0x0025b5 (0x37b bytes) 000450 1747848031.716057 mapwrite 0x002a15 thru 0x01138a (0xe976 bytes) 000452 1747848031.744068 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000453 1747848031.827076 trunc from 0x00f6da to 0x01138a (0x1cb1 bytes) 000454 1747848031.829876 mapread 0x00b3c8 thru 0x01138a (0x5fc3 bytes) 000455 1747848031.842568 MIRROR_RESYNC 6 mirrors 000456 1747848032.105326 punch from 0x00b4f7 to 0x01138a (0x5e94 bytes) 000457 1747848032.118334 trunc from 0x0102f3 to 0x01138a (0x1098 bytes) 000458 1747848032.120698 write 0x01a5a0 thru 0x023a14 (0x9475 bytes) 000459 1747848032.121717 punch from 0x01b926 to 0x023a14 (0x80ef bytes) 000460 1747848032.128181 mapread 0x018a1e thru 0x01ef37 (0x651a bytes) 000461 1747848032.136551 punch from 0x012ff8 to 0x0182e1 (0x52ea bytes) 000462 1747848032.139614 write 0x037ea8 thru 0x03ffff (0x8158 bytes) 000463 1747848032.140700 mapread 0x02fd21 thru 0x03d4a8 (0xd788 bytes) 000464 1747848032.145729 trunc from 0x02e364 to 0x02fc66 (0x1903 bytes) 000467 1747848032.148779 trunc from 0x00de53 to 0x012848 (0x49f6 bytes) 000468 1747848032.151558 write 0x03140c thru 0x0394fb (0x80f0 bytes) 000469 1747848032.152547 mapwrite 0x022cc7 thru 0x0264c3 (0x37fd bytes) 000471 1747848032.166096 mapwrite 0x013879 thru 0x022ac0 (0xf248 bytes) 000472 1747848032.177846 mapread 0x01d561 thru 0x0284d1 (0xaf71 bytes) 000473 1747848032.178562 trunc from 0x03b08d to 0x03ffff (0x4f73 bytes) 000474 1747848032.183268 MIRROR_EXTEND to 7 mirrors 000475 1747848032.413356 write 0x0346c5 thru 0x03ffff (0xb93b bytes) 000476 1747848032.427269 read 0x0284b3 thru 0x0355c0 (0xd10e bytes) 000477 1747848032.433222 MIRROR_EXTEND to 8 mirrors 000478 1747848032.676432 punch from 0x01dfb1 to 0x02270d (0x475d bytes) 000479 1747848032.701275 fallocate 0x001201 thru 0x00c6c4 (0xb4c4 bytes) 000480 1747848032.705170 MIRROR_EXTEND to 9 mirrors 000482 1747848032.922925 trunc from 0x02cc8e to 0x037bc5 (0xaf38 bytes) 000483 1747848032.951554 read 0x03fd48 thru 0x03ffff (0x2b8 bytes) 000484 1747848032.956262 read 0x021b52 thru 0x02879b (0x6c4a bytes) 000486 1747848032.967151 fallocate 0x000dc2 thru 0x001cfc (0xf3b bytes) 000487 1747848032.971183 mapwrite 0x0185db thru 0x027d69 (0xf78f bytes) 000488 1747848032.993165 punch from 0x012409 to 0x0125fa (0x1f2 bytes) 000489 1747848032.997227 trunc from 0x040000 to 0x019f12 (0x260ed bytes) 000490 1747848033.021887 mapwrite 0x01ef9d thru 0x02788b (0x88ef bytes) 000491 1747848033.069836 mapread 0x00f65a thru 0x019212 (0x9bb9 bytes) 000492 1747848033.073659 trunc from 0x02788c to 0x007809 (0x20082 bytes) 000493 1747848033.080753 fallocate 0x024809 thru 0x0298ed (0x50e5 bytes) 000494 1747848033.083458 write 0x035879 thru 0x03ffff (0xa787 bytes) 000495 1747848033.084460 read 0x0315d7 thru 0x037ca4 (0x66ce bytes) 000496 1747848033.088009 fallocate 0x02bbc8 thru 0x037586 (0xb9bf bytes) 000497 1747848033.090040 write 0x00c682 thru 0x0153e4 (0x8d63 bytes) 000498 1747848033.090707 trunc from 0x040000 to 0x00f26e (0x30d91 bytes) 000499 1747848033.096834 trunc from 0x00f26f to 0x03b431 (0x2c1c3 bytes) 000500 1747848033.102376 trunc from 0x03b432 to 0x00c2ca (0x2f167 bytes) 000501 1747848033.110129 write 0x039b24 thru 0x03b748 (0x1c25 bytes) 000503 1747848033.111467 trunc from 0x03b749 to 0x00b473 (0x302d5 bytes) 000504 1747848033.119971 trunc from 0x002c17 to 0x00352f (0x919 bytes) 000505 1747848033.122786 trunc from 0x00b474 to 0x033c73 (0x28800 bytes) 000506 1747848033.129715 trunc from 0x0210f3 to 0x023e6f (0x2d7d bytes) 000507 1747848033.134588 MIRROR_EXTEND to 10 mirrors 000510 1747848033.364224 trunc from 0x033c74 to 0x0366d8 (0x2a65 bytes) 000511 1747848033.392556 write 0x021486 thru 0x02f2d6 (0xde51 bytes) 000512 1747848033.396675 read 0x01e14b thru 0x02b232 (0xd0e8 bytes) 000513 1747848033.398978 mapread 0x001cd3 thru 0x00b30e (0x963c bytes) 000514 1747848033.404671 trunc from 0x01d7cc to 0x0272c4 (0x9af9 bytes) 000515 1747848033.410386 fallocate 0x02b300 thru 0x035ca2 (0xa9a3 bytes) 000516 1747848033.413931 fallocate 0x03d4b6 thru 0x03ffff (0x2b4a bytes) 000518 1747848033.416612 trunc from 0x02a63b to 0x02c586 (0x1f4c bytes) 000519 1747848033.419896 mapread 0x003d54 thru 0x0109c6 (0xcc73 bytes) 000520 1747848033.420586 mapwrite 0x01b0d8 thru 0x023885 (0x87ae bytes) 000521 1747848033.434238 write 0x0048ad thru 0x0125bd (0xdd11 bytes) 000522 1747848033.435205 punch from 0x03006e to 0x0311dd (0x1170 bytes) 000524 1747848033.437990 fallocate 0x020b5f thru 0x0266e1 (0x5b83 bytes) 000525 1747848033.444625 MIRROR_RESYNC 10 mirrors 000526 1747848034.378981 punch from 0x00291f to 0x0065f1 (0x3cd3 bytes) 000527 1747848034.400473 trunc from 0x040000 to 0x02ae45 (0x151ba bytes) 000528 1747848034.409697 mapread 0x0005ba thru 0x0008a4 (0x2eb bytes) 000529 1747848034.414901 MIRROR_RESYNC 10 mirrors 000530 1747848034.952037 mapwrite 0x030133 thru 0x03ee9c (0xed6a bytes) 000532 1747848034.989086 trunc from 0x03ee9d to 0x022572 (0x1c92a bytes) 000533 1747848034.996443 punch from 0x00b21b to 0x016286 (0xb06c bytes) 000534 1747848034.998420 trunc from 0x01b729 to 0x022572 (0x6e4a bytes) 000535 1747848035.002090 MIRROR_EXTEND to 11 mirrors 000536 1747848035.150551 trunc from 0x022573 to 0x038d8b (0x16819 bytes) 000538 1747848035.163878 trunc from 0x038d8c to 0x00b499 (0x2d8f2 bytes) 000539 1747848035.169630 read 0x0093f7 thru 0x00b499 (0x20a3 bytes) 000540 1747848035.172373 trunc from 0x002eed to 0x00b499 (0x85ad bytes) 000541 1747848035.174312 write 0x0047ae thru 0x0140fd (0xf950 bytes) 000542 1747848035.179171 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000543 1747848035.249423 trunc from 0x010a40 to 0x01347e (0x2a3f bytes) 000544 1747848035.251729 write 0x0356b6 thru 0x03ffff (0xa94a bytes) 000545 1747848035.254658 trunc from 0x038dcf to 0x03c942 (0x3b74 bytes) 000546 1747848035.259426 fallocate 0x00953a thru 0x01934f (0xfe16 bytes) 000548 1747848035.262182 mapwrite 0x0014c7 thru 0x00c2aa (0xade4 bytes) 000549 1747848035.278664 fallocate 0x0309ef thru 0x030b24 (0x136 bytes) 000550 1747848035.280911 mapread 0x002f22 thru 0x003dc1 (0xea0 bytes) 000552 1747848035.281844 trunc from 0x040000 to 0x035d60 (0xa29f bytes) 000553 1747848035.288952 fallocate 0x035071 thru 0x03ffff (0xaf8f bytes) 000555 1747848035.291557 write 0x016e2f thru 0x018a75 (0x1c47 bytes) 000556 1747848035.295949 fallocate 0x002189 thru 0x0092e0 (0x7158 bytes) 000557 1747848035.298769 fallocate 0x017716 thru 0x01a99b (0x3286 bytes) 000559 1747848035.301759 mapwrite 0x013a59 thru 0x023221 (0xf7c9 bytes) 000560 1747848035.318228 mapwrite 0x0375ef thru 0x03c48e (0x4ea0 bytes) 000561 1747848035.341019 trunc from 0x03c48f to 0x02f53a (0xcf54 bytes) 000562 1747848035.348451 write 0x006596 thru 0x011f52 (0xb9bd bytes) 000563 1747848035.349467 mapread 0x017d2e thru 0x0243a5 (0xc678 bytes) 000564 1747848035.352112 MIRROR_RESYNC 10 mirrors 000565 1747848035.740177 read 0x0116bf thru 0x01dc26 (0xc568 bytes) 000566 1747848035.744117 trunc from 0x020358 to 0x02208e (0x1d37 bytes) 000567 1747848035.759434 write 0x00e04a thru 0x017fb4 (0x9f6b bytes) 000568 1747848035.764685 read 0x0034ab thru 0x010ab9 (0xd60f bytes) 000569 1747848035.769488 fallocate 0x019f1a thru 0x020886 (0x696d bytes) 000571 1747848035.772752 read 0x00e4a5 thru 0x012803 (0x435f bytes) 000572 1747848035.773217 trunc from 0x02f53b to 0x01bcc8 (0x13872 bytes) 000573 1747848035.787956 mapwrite 0x029649 thru 0x02d401 (0x3db9 bytes) 000576 1747848035.810358 read 0x00ea5a thru 0x01cbe6 (0xe18d bytes) 000577 1747848035.810852 trunc from 0x02d402 to 0x01a8f9 (0x12b08 bytes) 000578 1747848035.818503 fallocate 0x010712 thru 0x014edf (0x47ce bytes) 000579 1747848035.821586 mapwrite 0x02245e thru 0x026f2d (0x4ad0 bytes) 000580 1747848035.844636 read 0x0161d6 thru 0x02517b (0xefa6 bytes) 000581 1747848035.845245 trunc from 0x026f2e to 0x007858 (0x1f6d5 bytes) 000582 1747848035.851639 read 0x006950 thru 0x007858 (0xf09 bytes) 000583 1747848035.855168 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000584 1747848035.935474 trunc from 0x007859 to 0x0243ae (0x1cb56 bytes) 000585 1747848035.944572 punch from 0x0076a9 to 0x00a2e3 (0x2c3b bytes) 000586 1747848035.946902 trunc from 0x0243af to 0x00f6d2 (0x14cdc bytes) 000587 1747848035.953453 read 0x00eefc thru 0x00f6d2 (0x7d7 bytes) 000588 1747848035.957290 mapread 0x00e570 thru 0x00f6d2 (0x1163 bytes) 000590 1747848035.958374 trunc from 0x00f6d3 to 0x00599a (0x9d38 bytes) 000592 1747848035.964438 mapread 0x004c9f thru 0x00599a (0xcfc bytes) 000593 1747848035.968396 read 0x0058c7 thru 0x00599a (0xd4 bytes) 000594 1747848035.968832 read 0x004d32 thru 0x00599a (0xc69 bytes) 000595 1747848035.969226 write 0x02b239 thru 0x030f79 (0x5d41 bytes) 000596 1747848035.970126 mapwrite 0x003be8 thru 0x00e091 (0xa4aa bytes) 000597 1747848035.984445 mapread 0x004a29 thru 0x00fcca (0xb2a2 bytes) 000598 1747848035.985188 punch from 0x0041d1 to 0x00726a (0x309a bytes) 000599 1747848035.988237 trunc from 0x011893 to 0x02162a (0xfd98 bytes) 000600 1747848035.992880 MIRROR_EXTEND to 10 mirrors 000601 1747848036.200515 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000602 1747848036.320357 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000605 1747848036.404994 write 0x036362 thru 0x03c467 (0x6106 bytes) 000606 1747848036.414203 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000607 1747848036.500228 trunc from 0x0160a0 to 0x01ef17 (0x8e78 bytes) 000609 1747848036.503769 mapread 0x00bfee thru 0x01b086 (0xf099 bytes) 000610 1747848036.521535 mapwrite 0x03627b thru 0x037998 (0x171e bytes) 000611 1747848036.537066 fallocate 0x025a0a thru 0x02f9e1 (0x9fd8 bytes) 000612 1747848036.541888 MIRROR_RESYNC 7 mirrors 000613 1747848037.210669 trunc from 0x01ea2f to 0x0251ec (0x67be bytes) 000614 1747848037.223399 trunc from 0x03c468 to 0x034650 (0x7e17 bytes) 000615 1747848037.230064 read 0x01b0ef thru 0x029ecb (0xeddd bytes) 000616 1747848037.234220 trunc from 0x034651 to 0x0029f1 (0x31c5f bytes) 000617 1747848037.240369 mapread 0x000121 thru 0x0029f1 (0x28d1 bytes) 000618 1747848037.247003 fallocate 0x024e58 thru 0x025026 (0x1cf bytes) 000619 1747848037.249519 trunc from 0x00109e to 0x00a741 (0x96a4 bytes) 000620 1747848037.251880 fallocate 0x02fa80 thru 0x032418 (0x2999 bytes) 000622 1747848037.254472 punch from 0x005e9d to 0x00dc4c (0x7db0 bytes) 000623 1747848037.256669 trunc from 0x025027 to 0x0253d7 (0x3b1 bytes) 000624 1747848037.264174 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000625 1747848037.335474 fallocate 0x0294a9 thru 0x035e13 (0xc96b bytes) 000626 1747848037.338103 read 0x00392c thru 0x00f8dd (0xbfb2 bytes) 000627 1747848037.341776 mapread 0x0071f3 thru 0x00eac3 (0x78d1 bytes) 000629 1747848037.342653 write 0x03afad thru 0x03ffff (0x5053 bytes) 000630 1747848037.345123 mapread 0x03da4a thru 0x03ffff (0x25b6 bytes) 000632 1747848037.346486 trunc from 0x040000 to 0x0048f8 (0x3b707 bytes) 000633 1747848037.353348 write 0x036b5f thru 0x03ffff (0x94a1 bytes) 000634 1747848037.354433 mapread 0x035d1f thru 0x03d6df (0x79c1 bytes) 000635 1747848037.357760 trunc from 0x01903e to 0x01febc (0x6e7f bytes) 000636 1747848037.359854 mapwrite 0x035d13 thru 0x03e37a (0x8668 bytes) 000637 1747848037.370233 trunc from 0x031eb1 to 0x03ffff (0xe14f bytes) 000640 1747848037.374044 mapread 0x00cadd thru 0x0108c8 (0x3dec bytes) 000641 1747848037.381729 mapread 0x017650 thru 0x020a8c (0x943d bytes) 000642 1747848037.389835 read 0x020ba4 thru 0x02f6bd (0xeb1a bytes) 000643 1747848037.394774 mapread 0x02befc thru 0x033b33 (0x7c38 bytes) 000644 1747848037.396603 mapwrite 0x036299 thru 0x03de3a (0x7ba2 bytes) 000645 1747848037.404745 mapwrite 0x0050f2 thru 0x011c9e (0xcbad bytes) 000646 1747848037.415358 punch from 0x012c62 to 0x0130ff (0x49e bytes) 000647 1747848037.417220 fallocate 0x01f4b6 thru 0x020d7e (0x18c9 bytes) 000648 1747848037.419800 punch from 0x013ddc to 0x01db5e (0x9d83 bytes) 000649 1747848037.424345 MIRROR_RESYNC 6 mirrors 000650 1747848037.746609 write 0x022930 thru 0x030a72 (0xe143 bytes) 000651 1747848037.761347 write 0x034a82 thru 0x0390cc (0x464b bytes) 000652 1747848037.765828 trunc from 0x040000 to 0x0079cb (0x38634 bytes) 000653 1747848037.778716 punch from 0x003907 to 0x0079cb (0x40c5 bytes) 000654 1747848037.781578 mapwrite 0x02b777 thru 0x03905c (0xd8e6 bytes) 000656 1747848037.810469 trunc from 0x000565 to 0x00e705 (0xe1a1 bytes) 000657 1747848037.813736 trunc from 0x000660 to 0x00ab25 (0xa4c6 bytes) 000658 1747848037.819220 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000659 1747848037.889915 write 0x013ef0 thru 0x023ebe (0xffcf bytes) 000660 1747848037.897037 trunc from 0x027c26 to 0x032152 (0xa52d bytes) 000661 1747848037.899697 punch from 0x017c35 to 0x01faf3 (0x7ebf bytes) 000662 1747848037.906518 trunc from 0x021580 to 0x02ea6d (0xd4ee bytes) 000663 1747848037.910811 MIRROR_RESYNC 5 mirrors 000664 1747848038.329978 fallocate 0x00769a thru 0x00bbd0 (0x4537 bytes) 000665 1747848038.343623 trunc from 0x009c74 to 0x0188ad (0xec3a bytes) 000666 1747848038.346056 mapread 0x0204ed thru 0x028f4b (0x8a5f bytes) 000667 1747848038.355460 MIRROR_RESYNC 5 mirrors 000668 1747848038.597886 mapread 0x019238 thru 0x01fc8f (0x6a58 bytes) 000669 1747848038.603907 trunc from 0x01b05f to 0x02748e (0xc430 bytes) 000670 1747848038.614905 punch from 0x02e1be to 0x0328f6 (0x4739 bytes) 000672 1747848038.616848 mapwrite 0x02dc7c thru 0x034c51 (0x6fd6 bytes) 000673 1747848038.627238 trunc from 0x03905d to 0x02352b (0x15b31 bytes) 000674 1747848038.633813 trunc from 0x018c02 to 0x01ea17 (0x5e16 bytes) 000675 1747848038.636670 mapwrite 0x02e274 thru 0x03bdb2 (0xdb3f bytes) 000677 1747848038.655510 write 0x0108d1 thru 0x01c76f (0xbe9f bytes) 000678 1747848038.660346 trunc from 0x03bdb3 to 0x035307 (0x6aab bytes) 000679 1747848038.665020 trunc from 0x035308 to 0x016772 (0x1eb95 bytes) 000680 1747848038.669226 trunc from 0x016773 to 0x021d6a (0xb5f8 bytes) 000681 1747848038.673628 mapwrite 0x01a512 thru 0x02230e (0x7dfd bytes) 000682 1747848038.688097 trunc from 0x01a135 to 0x01edae (0x4c7a bytes) 000685 1747848038.690594 read 0x0181f5 thru 0x019ff6 (0x1e02 bytes) 000686 1747848038.691033 fallocate 0x02cec9 thru 0x0322ba (0x53f2 bytes) 000687 1747848038.693784 punch from 0x021b0c to 0x02230e (0x803 bytes) 000688 1747848038.696086 read 0x0210fc thru 0x02230e (0x1213 bytes) 000689 1747848038.700031 read 0x00ef7d thru 0x01dfaf (0xf033 bytes) 000690 1747848038.700530 fallocate 0x03c8bf thru 0x03ffff (0x3741 bytes) 000691 1747848038.702951 write 0x02e420 thru 0x03b319 (0xcefa bytes) 000693 1747848038.704187 trunc from 0x018313 to 0x0267df (0xe4cd bytes) 000694 1747848038.708421 MIRROR_EXTEND to 6 mirrors 000696 1747848038.861163 punch from 0x025cf6 to 0x02c538 (0x6843 bytes) 000698 1747848038.874826 punch from 0x02fe5c to 0x03b319 (0xb4be bytes) 000700 1747848038.877590 read 0x00151b thru 0x00fe92 (0xe978 bytes) 000702 1747848038.881620 write 0x01dc38 thru 0x027903 (0x9ccc bytes) 000703 1747848038.887112 fallocate 0x00cec9 thru 0x013565 (0x669d bytes) 000704 1747848038.889569 punch from 0x00e7a2 to 0x00fbf2 (0x1451 bytes) 000706 1747848038.892156 mapread 0x0164c9 thru 0x020b07 (0xa63f bytes) 000707 1747848038.896773 mapread 0x01c090 thru 0x01d443 (0x13b4 bytes) 000708 1747848038.898415 write 0x00eb8f thru 0x0164a7 (0x7919 bytes) 000709 1747848038.902860 mapread 0x028b8a thru 0x02973e (0xbb5 bytes) 000711 1747848038.908464 read 0x02dc74 thru 0x032b16 (0x4ea3 bytes) 000713 1747848038.912224 fallocate 0x0206f2 thru 0x022329 (0x1c38 bytes) 000714 1747848038.915532 punch from 0x031b17 to 0x039049 (0x7533 bytes) 000715 1747848038.917655 read 0x003f74 thru 0x010636 (0xc6c3 bytes) 000716 1747848038.918172 trunc from 0x03b31a to 0x018885 (0x22a94 bytes) 000717 1747848038.932445 punch from 0x00c060 to 0x01449a (0x843b bytes) 000718 1747848038.938244 trunc from 0x018886 to 0x037ece (0x1f649 bytes) 000719 1747848038.943726 trunc from 0x037ecf to 0x01d65e (0x1a870 bytes) 000720 1747848038.948052 write 0x002537 thru 0x004489 (0x1f53 bytes) 000722 1747848038.949133 write 0x0021e2 thru 0x00f0f0 (0xcf0f bytes) 000723 1747848038.952635 write 0x005cb2 thru 0x00ea63 (0x8db2 bytes) 000726 1747848038.954181 mapwrite 0x01b8bd thru 0x02438f (0x8ad3 bytes) 000727 1747848038.972330 trunc from 0x0218f1 to 0x02438f (0x2a9f bytes) 000728 1747848038.974451 write 0x0277e1 thru 0x02dbbd (0x63dd bytes) 000730 1747848038.975892 mapwrite 0x033d2b thru 0x03bed0 (0x81a6 bytes) 000731 1747848038.996970 fallocate 0x0102f3 thru 0x01dd46 (0xda54 bytes) 000732 1747848038.999606 read 0x004b1a thru 0x00a029 (0x5510 bytes) 000733 1747848039.000001 trunc from 0x03bed1 to 0x02cdb8 (0xf118 bytes) 000734 1747848039.006449 fallocate 0x03421c thru 0x03f9ef (0xb7d4 bytes) 000735 1747848039.008798 trunc from 0x02bdee to 0x03538a (0x959d bytes) 000736 1747848039.011344 mapread 0x02fc9e thru 0x03a6c2 (0xaa25 bytes) 000738 1747848039.015851 trunc from 0x037702 to 0x03e530 (0x6e2f bytes) 000739 1747848039.017775 trunc from 0x019688 to 0x021259 (0x7bd2 bytes) 000740 1747848039.020229 trunc from 0x03f9f0 to 0x007bb5 (0x37e3a bytes) 000741 1747848039.026287 write 0x00082f thru 0x00f338 (0xeb0a bytes) 000742 1747848039.028938 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000743 1747848039.102046 fallocate 0x037197 thru 0x03a763 (0x35cd bytes) 000744 1747848039.104254 write 0x010da3 thru 0x0151f2 (0x4450 bytes) 000745 1747848039.109562 fallocate 0x01ea49 thru 0x021d03 (0x32bb bytes) 000746 1747848039.111612 mapwrite 0x03f724 thru 0x03ffff (0x8dc bytes) 000747 1747848039.129230 fallocate 0x008598 thru 0x00c8cf (0x4338 bytes) 000748 1747848039.131208 trunc from 0x016c0d to 0x0170f7 (0x4eb bytes) 000749 1747848039.133083 write 0x02ed4b thru 0x032d7a (0x4030 bytes) 000750 1747848039.136562 trunc from 0x040000 to 0x02dc4c (0x123b3 bytes) 000752 1747848039.142376 mapwrite 0x004388 thru 0x00cc63 (0x88dc bytes) 000754 1747848039.153873 read 0x015280 thru 0x022d48 (0xdac9 bytes) 000755 1747848039.157092 write 0x008cd2 thru 0x013514 (0xa843 bytes) 000756 1747848039.157902 write 0x02a51f thru 0x02c0ce (0x1bb0 bytes) 000757 1747848039.158399 fallocate 0x0150a7 thru 0x01e996 (0x98f0 bytes) 000758 1747848039.162694 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000759 1747848039.223061 trunc from 0x018953 to 0x0265ba (0xdc68 bytes) 000760 1747848039.225213 punch from 0x028c69 to 0x02dc4c (0x4fe4 bytes) 000762 1747848039.227965 fallocate 0x02c1af thru 0x0348b1 (0x8703 bytes) 000766 1747848039.231133 trunc from 0x0348b2 to 0x036858 (0x1fa7 bytes) 000767 1747848039.239259 mapwrite 0x03bac0 thru 0x03f418 (0x3959 bytes) 000768 1747848039.258703 punch from 0x02e3ae to 0x02fd27 (0x197a bytes) 000769 1747848039.260752 read 0x02d6af thru 0x03bfaa (0xe8fc bytes) 000770 1747848039.264502 fallocate 0x00f9b3 thru 0x014c34 (0x5282 bytes) 000771 1747848039.266768 trunc from 0x01945c to 0x0219a2 (0x8547 bytes) 000772 1747848039.268953 write 0x02e05b thru 0x037560 (0x9506 bytes) 000773 1747848039.269889 mapwrite 0x028a9c thru 0x033e70 (0xb3d5 bytes) 000775 1747848039.281459 mapread 0x02089b thru 0x027da9 (0x750f bytes) 000776 1747848039.285090 trunc from 0x0101bf to 0x01a9f8 (0xa83a bytes) 000777 1747848039.287518 mapwrite 0x032c7a thru 0x038cda (0x6061 bytes) 000778 1747848039.297765 fallocate 0x018d83 thru 0x02269b (0x9919 bytes) 000779 1747848039.299944 write 0x02d67c thru 0x0319d2 (0x4357 bytes) 000780 1747848039.300695 mapwrite 0x0364bc thru 0x03689c (0x3e1 bytes) 000781 1747848039.307129 trunc from 0x02fca0 to 0x03014b (0x4ac bytes) 000782 1747848039.309364 mapwrite 0x038c12 thru 0x03d915 (0x4d04 bytes) 000783 1747848039.317543 mapread 0x02efa8 thru 0x02f892 (0x8eb bytes) 000784 1747848039.320807 mapread 0x0229df thru 0x027845 (0x4e67 bytes) 000785 1747848039.321398 read 0x035f52 thru 0x03b2a7 (0x5356 bytes) 000787 1747848039.322149 trunc from 0x03f419 to 0x03d21d (0x21fb bytes) 000788 1747848039.329514 trunc from 0x03d21e to 0x03d2fe (0xe1 bytes) 000789 1747848039.336983 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000790 1747848039.398064 fallocate 0x002496 thru 0x00f0df (0xcc4a bytes) 000791 1747848039.400642 mapwrite 0x008ca0 thru 0x0151dd (0xc53e bytes) 000792 1747848039.424231 MIRROR_RESYNC 3 mirrors 000793 1747848039.568665 write 0x01d88f thru 0x02c26b (0xe9dd bytes) 000794 1747848039.581538 trunc from 0x007331 to 0x01445d (0xd12d bytes) 000795 1747848039.583648 mapwrite 0x032f89 thru 0x03ef4f (0xbfc7 bytes) 000796 1747848039.609104 mapread 0x039519 thru 0x03ef4f (0x5a37 bytes) 000798 1747848039.611609 MIRROR_EXTEND to 4 mirrors 000800 1747848039.745187 mapread 0x025283 thru 0x032bc8 (0xd946 bytes) 000802 1747848039.760288 read 0x0191a6 thru 0x0287eb (0xf646 bytes) 000803 1747848039.765546 trunc from 0x001b6b to 0x0027d0 (0xc66 bytes) 000804 1747848039.777227 fallocate 0x0208d4 thru 0x0209e5 (0x112 bytes) 000805 1747848039.779616 fallocate 0x007d8b thru 0x00bbc0 (0x3e36 bytes) 000806 1747848039.781799 mapread 0x037231 thru 0x03ef4f (0x7d1f bytes) 000807 1747848039.785327 write 0x0008c2 thru 0x000a8f (0x1ce bytes) 000808 1747848039.790528 fallocate 0x001a93 thru 0x01054e (0xeabc bytes) 000809 1747848039.794480 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000810 1747848039.856883 read 0x03ac0a thru 0x03ef4f (0x4346 bytes) 000811 1747848039.860128 write 0x00bf35 thru 0x0126b6 (0x6782 bytes) 000812 1747848039.865498 punch from 0x00fca9 to 0x016ed2 (0x722a bytes) 000813 1747848039.870348 trunc from 0x00b69d to 0x00fc83 (0x45e7 bytes) 000814 1747848039.872555 write 0x033b0e thru 0x03ffff (0xc4f2 bytes) 000815 1747848039.876415 trunc from 0x013ea3 to 0x01e6ab (0xa809 bytes) 000816 1747848039.878662 mapread 0x000fc1 thru 0x004ebc (0x3efc bytes) 000817 1747848039.883431 fallocate 0x0319d3 thru 0x03fc8e (0xe2bc bytes) 000818 1747848039.885253 trunc from 0x040000 to 0x019ea3 (0x2615c bytes) 000819 1747848039.895461 write 0x023cce thru 0x0277aa (0x3add bytes) 000820 1747848039.896350 trunc from 0x0221c4 to 0x0277aa (0x55e7 bytes) 000821 1747848039.901155 write 0x0023fa thru 0x009b5f (0x7766 bytes) 000822 1747848039.901974 punch from 0x023f02 to 0x0272f5 (0x33f4 bytes) 000823 1747848039.904207 trunc from 0x00020b to 0x00f096 (0xee8c bytes) 000824 1747848039.912988 MIRROR_RESYNC 3 mirrors 000825 1747848040.130918 fallocate 0x01383b thru 0x01470e (0xed4 bytes) 000826 1747848040.141252 write 0x018a69 thru 0x02488f (0xbe27 bytes) 000827 1747848040.144673 punch from 0x013455 to 0x013598 (0x144 bytes) 000828 1747848040.146331 read 0x011c24 thru 0x014d25 (0x3102 bytes) 000830 1747848040.148497 mapread 0x0095a4 thru 0x00e192 (0x4bef bytes) 000831 1747848040.151262 mapread 0x002a78 thru 0x006acc (0x4055 bytes) 000832 1747848040.154024 mapread 0x005e20 thru 0x006ebc (0x109d bytes) 000833 1747848040.154584 punch from 0x01dd01 to 0x0277aa (0x9aaa bytes) 000834 1747848040.159820 fallocate 0x032f37 thru 0x03957b (0x6645 bytes) 000835 1747848040.161897 trunc from 0x0277ab to 0x02fd3a (0x8590 bytes) 000836 1747848040.168180 mapwrite 0x019c67 thru 0x01e51c (0x48b6 bytes) 000837 1747848040.179211 read 0x01de20 thru 0x027ba3 (0x9d84 bytes) 000838 1747848040.182828 read 0x000307 thru 0x00bc1e (0xb918 bytes) 000839 1747848040.183250 mapwrite 0x033a83 thru 0x03e96c (0xaeea bytes) 000840 1747848040.201170 trunc from 0x03e96d to 0x02df8d (0x109df bytes) 000842 1747848040.206858 mapwrite 0x002aa5 thru 0x006f39 (0x4495 bytes) 000843 1747848040.214735 write 0x03272a thru 0x0350a7 (0x297e bytes) 000844 1747848040.215506 punch from 0x022516 to 0x02c304 (0x9def bytes) 000845 1747848040.218253 mapwrite 0x018e52 thru 0x02718d (0xe33c bytes) 000846 1747848040.235527 fallocate 0x0322c5 thru 0x03e661 (0xc39d bytes) 000847 1747848040.237542 mapwrite 0x02059b thru 0x02c4d6 (0xbf3c bytes) 000848 1747848040.250986 read 0x022718 thru 0x028345 (0x5c2e bytes) 000849 1747848040.251404 fallocate 0x01379c thru 0x016620 (0x2e85 bytes) 000850 1747848040.253156 trunc from 0x033dff to 0x037aae (0x3cb0 bytes) 000851 1747848040.254899 mapread 0x024f51 thru 0x028eb5 (0x3f65 bytes) 000852 1747848040.255456 fallocate 0x0027ba thru 0x00fb37 (0xd37e bytes) 000853 1747848040.257415 trunc from 0x03e662 to 0x009bc6 (0x34a9b bytes) 000854 1747848040.264788 MIRROR_RESYNC 3 mirrors 000855 1747848040.366244 fallocate 0x03d8fd thru 0x03db04 (0x208 bytes) 000856 1747848040.379930 MIRROR_EXTEND to 4 mirrors 000857 1747848040.476925 trunc from 0x009bc7 to 0x004c95 (0x4f31 bytes) 000858 1747848040.495541 MIRROR_EXTEND to 5 mirrors 000859 1747848040.605606 trunc from 0x004c96 to 0x00cb88 (0x7ef3 bytes) 000860 1747848040.627723 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000862 1747848040.692917 mapread 0x0027eb thru 0x0083da (0x5bf0 bytes) 000863 1747848040.698829 fallocate 0x038145 thru 0x039c78 (0x1b34 bytes) 000865 1747848040.701253 trunc from 0x00cb89 to 0x02e187 (0x215ff bytes) 000866 1747848040.707357 punch from 0x007ef7 to 0x015218 (0xd322 bytes) 000867 1747848040.710874 MIRROR_RESYNC 4 mirrors 000868 1747848040.874539 fallocate 0x00e784 thru 0x00eee6 (0x763 bytes) 000869 1747848040.892140 read 0x01e401 thru 0x027331 (0x8f31 bytes) 000870 1747848040.895848 read 0x00bfcc thru 0x0129e5 (0x6a1a bytes) 000871 1747848040.901646 write 0x02ba42 thru 0x03aa5c (0xf01b bytes) 000872 1747848040.902972 mapwrite 0x033d4a thru 0x03d946 (0x9bfd bytes) 000873 1747848040.929055 punch from 0x0371c0 to 0x03d946 (0x6787 bytes) 000874 1747848040.931198 punch from 0x0148ec to 0x02213b (0xd850 bytes) 000875 1747848040.933401 punch from 0x015d6a to 0x01ae91 (0x5128 bytes) 000876 1747848040.937838 MIRROR_EXTEND to 5 mirrors 000877 1747848041.085664 trunc from 0x03d947 to 0x00b633 (0x32313 bytes) 000878 1747848041.100551 trunc from 0x001e66 to 0x007d3d (0x5ed8 bytes) 000879 1747848041.102733 read 0x007c14 thru 0x00b633 (0x3a20 bytes) 000880 1747848041.106374 mapwrite 0x018812 thru 0x026499 (0xdc88 bytes) 000881 1747848041.131139 fallocate 0x02aef0 thru 0x02fb4e (0x4c5f bytes) 000882 1747848041.133130 trunc from 0x00d276 to 0x00e45c (0x11e7 bytes) 000883 1747848041.136763 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000884 1747848041.201875 fallocate 0x01a968 thru 0x021b1c (0x71b5 bytes) 000886 1747848041.203982 read 0x00188a thru 0x003226 (0x199d bytes) 000887 1747848041.205821 mapwrite 0x012e01 thru 0x0223e8 (0xf5e8 bytes) 000888 1747848041.220017 fallocate 0x01bbdf thru 0x01f4e0 (0x3902 bytes) 000889 1747848041.221488 read 0x00fbd0 thru 0x0107d8 (0xc09 bytes) 000890 1747848041.221893 punch from 0x00bd7e to 0x01b637 (0xf8ba bytes) 000891 1747848041.223975 read 0x01d63e thru 0x026499 (0x8e5c bytes) 000892 1747848041.229016 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000894 1747848041.285640 fallocate 0x008723 thru 0x00b509 (0x2de7 bytes) 000896 1747848041.289799 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000897 1747848041.347753 write 0x00b4fa thru 0x01a62c (0xf133 bytes) 000898 1747848041.352791 mapwrite 0x03bdf6 thru 0x03ffff (0x420a bytes) 000899 1747848041.370511 mapread 0x028a77 thru 0x031c88 (0x9212 bytes) 000901 1747848041.375735 mapread 0x00499e thru 0x007989 (0x2fec bytes) 000902 1747848041.378567 mapwrite 0x008943 thru 0x00d359 (0x4a17 bytes) 000903 1747848041.385120 mapwrite 0x01cd2c thru 0x02a7a2 (0xda77 bytes) 000904 1747848041.395022 trunc from 0x040000 to 0x037d2d (0x82d2 bytes) 000907 1747848041.400365 read 0x01622c thru 0x01f075 (0x8e4a bytes) 000908 1747848041.400851 trunc from 0x037d2e to 0x02de52 (0x9edb bytes) 000909 1747848041.405701 trunc from 0x02726e to 0x02bd1b (0x4aae bytes) 000910 1747848041.407877 fallocate 0x0324a0 thru 0x03ffff (0xdb60 bytes) 000911 1747848041.409830 trunc from 0x024ee9 to 0x02924d (0x4365 bytes) 000912 1747848041.412957 MIRROR_RESYNC 2 mirrors 000914 1747848041.545821 punch from 0x013071 to 0x0153e1 (0x2371 bytes) 000915 1747848041.555819 fallocate 0x030e8c thru 0x033f5b (0x30d0 bytes) 000917 1747848041.558056 read 0x014f45 thru 0x01f9c0 (0xaa7c bytes) 000918 1747848041.561238 punch from 0x014b9e to 0x018c0c (0x406f bytes) 000920 1747848041.563901 mapread 0x022c71 thru 0x03195d (0xeced bytes) 000921 1747848041.572274 fallocate 0x0213cc thru 0x02b4f4 (0xa129 bytes) 000922 1747848041.574385 write 0x00967a thru 0x00cafa (0x3481 bytes) 000923 1747848041.578619 fallocate 0x02bee9 thru 0x02fd84 (0x3e9c bytes) 000924 1747848041.580838 write 0x031977 thru 0x0366da (0x4d64 bytes) 000925 1747848041.583388 fallocate 0x01590d thru 0x0206d7 (0xadcb bytes) 000926 1747848041.585433 trunc from 0x040000 to 0x01cc14 (0x233eb bytes) 000927 1747848041.598575 punch from 0x01092c to 0x01cc14 (0xc2e9 bytes) 000928 1747848041.600901 punch from 0x006524 to 0x0142ed (0xddca bytes) 000930 1747848041.605422 fallocate 0x03a91a thru 0x03ffff (0x56e6 bytes) 000931 1747848041.607838 fallocate 0x02d53d thru 0x03d292 (0xfd56 bytes) 000932 1747848041.610186 punch from 0x00db28 to 0x016a94 (0x8f6d bytes) 000933 1747848041.612440 trunc from 0x01cc15 to 0x010e0f (0xbe05 bytes) 000934 1747848041.617687 fallocate 0x02f6f5 thru 0x033543 (0x3e4f bytes) 000935 1747848041.619971 trunc from 0x033544 to 0x014c26 (0x1e91d bytes) 000936 1747848041.625909 mapwrite 0x032c8d thru 0x0342d1 (0x1645 bytes) 000937 1747848041.642143 fallocate 0x034e0a thru 0x03f0e5 (0xa2dc bytes) 000939 1747848041.644381 mapwrite 0x00fb4a thru 0x01bd5d (0xc214 bytes) 000940 1747848041.657838 write 0x01c6a7 thru 0x028ed5 (0xc82f bytes) 000941 1747848041.661049 read 0x00d8ec thru 0x0167e0 (0x8ef5 bytes) 000942 1747848041.661568 trunc from 0x0342d2 to 0x01a19e (0x1a133 bytes) 000943 1747848041.666637 trunc from 0x01a19f to 0x00fc51 (0xa54d bytes) 000944 1747848041.671079 fallocate 0x00b644 thru 0x00f7ae (0x416b bytes) 000945 1747848041.672958 mapwrite 0x0220b7 thru 0x030576 (0xe4c0 bytes) 000946 1747848041.691498 fallocate 0x03c5fc thru 0x03d251 (0xc56 bytes) 000947 1747848041.693769 read 0x010472 thru 0x0125e8 (0x2177 bytes) 000948 1747848041.694174 trunc from 0x0076f9 to 0x016c55 (0xf55d bytes) 000949 1747848041.696379 read 0x01da2c thru 0x022bed (0x51c2 bytes) 000950 1747848041.698634 MIRROR_EXTEND to 3 mirrors 000951 1747848041.847357 trunc from 0x03d252 to 0x02103a (0x1c217 bytes) 000953 1747848041.868597 trunc from 0x02103b to 0x00d693 (0x139a7 bytes) 000954 1747848041.874190 mapwrite 0x01e410 thru 0x01fb83 (0x1774 bytes) 000955 1747848041.888250 fallocate 0x01ae81 thru 0x020e90 (0x6010 bytes) 000956 1747848041.890517 fallocate 0x015bb5 thru 0x023b04 (0xdf50 bytes) 000957 1747848041.893139 punch from 0x016d0e to 0x021274 (0xa567 bytes) 000958 1747848041.894892 punch from 0x01d3ef to 0x01e959 (0x156b bytes) 000959 1747848041.896492 fallocate 0x017295 thru 0x01d0e5 (0x5e51 bytes) 000960 1747848041.898407 mapwrite 0x02fb68 thru 0x037735 (0x7bce bytes) 000961 1747848041.914077 MIRROR_EXTEND to 4 mirrors 000962 1747848042.070021 read 0x012cd8 thru 0x014782 (0x1aab bytes) 000963 1747848042.073980 mapwrite 0x030df4 thru 0x03c4b1 (0xb6be bytes) 000964 1747848042.105626 fallocate 0x019168 thru 0x01d228 (0x40c1 bytes) 000965 1747848042.108216 mapread 0x034298 thru 0x035557 (0x12c0 bytes) 000966 1747848042.110711 MIRROR_EXTEND to 5 mirrors 000967 1747848042.276119 punch from 0x01c07a to 0x01f4df (0x3466 bytes) 000968 1747848042.289070 write 0x0105dc thru 0x019219 (0x8c3e bytes) 000969 1747848042.294583 trunc from 0x019023 to 0x01ed94 (0x5d72 bytes) 000970 1747848042.300524 punch from 0x001212 to 0x00e3d4 (0xd1c3 bytes) 000971 1747848042.302715 mapread 0x00e08a thru 0x018710 (0xa687 bytes) 000972 1747848042.306336 mapwrite 0x002014 thru 0x00bd34 (0x9d21 bytes) 000973 1747848042.319947 mapread 0x004110 thru 0x007547 (0x3438 bytes) 000974 1747848042.320576 fallocate 0x01b201 thru 0x0265ce (0xb3ce bytes) 000976 1747848042.323800 write 0x00a0bd thru 0x011cdb (0x7c1f bytes) 000977 1747848042.324664 read 0x0033e5 thru 0x00ffc2 (0xcbde bytes) 000978 1747848042.325147 read 0x01ed29 thru 0x0271d3 (0x84ab bytes) 000979 1747848042.329594 write 0x025f82 thru 0x02a0c2 (0x4141 bytes) 000980 1747848042.331424 trunc from 0x03c4b2 to 0x035ee1 (0x65d0 bytes) 000981 1747848042.344550 mapread 0x01da1d thru 0x01dea3 (0x487 bytes) 000982 1747848042.345172 fallocate 0x01e134 thru 0x02893f (0xa80c bytes) 000983 1747848042.347622 trunc from 0x035ee2 to 0x0366ed (0x80c bytes) 000985 1747848042.353826 fallocate 0x02081f thru 0x02f7af (0xef91 bytes) 000986 1747848042.356372 trunc from 0x024895 to 0x02ecf3 (0xa45f bytes) 000987 1747848042.365856 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000988 1747848042.436403 mapread 0x0177b0 thru 0x0235b7 (0xbe08 bytes) 000989 1747848042.446917 trunc from 0x007c3b to 0x008cf9 (0x10bf bytes) 000990 1747848042.449341 trunc from 0x01a489 to 0x01c4f2 (0x206a bytes) 000991 1747848042.451910 punch from 0x00a974 to 0x014394 (0x9a21 bytes) 000993 1747848042.454652 fallocate 0x0145e3 thru 0x021ea5 (0xd8c3 bytes) 000994 1747848042.456914 read 0x0011d8 thru 0x003c04 (0x2a2d bytes) 000995 1747848042.461613 MIRROR_EXTEND to 5 mirrors 000997 1747848042.675224 punch from 0x013d79 to 0x01d64e (0x98d6 bytes) 000998 1747848042.691145 mapwrite 0x00b073 thru 0x013c5b (0x8be9 bytes) 001000 1747848042.713659 mapwrite 0x006f71 thru 0x00b702 (0x4792 bytes) All operations completed A-OK! PASS 70a (30s) == sanity-flr test 71: check mirror extend parent fid ==== 13:20:44 (1747848044) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x464:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 13:20:47 (1747848047) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1384320 .. Extending file size to 7639840 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 1976608 .. Extending file size to 7115264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8676864 .. Extending file size to 8574080 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8855040 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8807968 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8985632 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9209088 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9123392 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9331136 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 31507 31508 31509 31511 31512 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) 192.168.204.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:21:51 (1747848111) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3632160 .. Extending file size to 2522336 .. Extending file size to 4796704 .. Extending file size to 7389952 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8932384 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7522528 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7695232 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9245984 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9275616 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9016384 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 2662 2663 2664 2667 2669 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) 192.168.204.151@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (64s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:22:55 (1747848175) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2812 1284876 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17768 3589252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19924 3587096 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37692 7176348 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2812 1284876 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2408 1285280 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17768 3589252 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19924 3587096 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37692 7176348 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.04621 s, 45.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0679851 s, 30.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:23:01 (1747848181) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0367516 s, 57.1 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 13:23:03 (1747848183) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x99:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:23:05 (1747848185) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0270217 s, 77.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:23:07 (1747848187) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0297133 s, 70.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:23:09 (1747848189) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0363447 s, 57.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4d9:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:23:33 (1747848213) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0233528 s, 44.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d8:0x0] } PASS 204d (33s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:24:06 (1747848246) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0367809 s, 57.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (32s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:24:38 (1747848278) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg451-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg451-server: Pool lustre.test_204f created Adding targets to pool oleg451-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg451-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.035049 s, 59.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x4dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x4d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg451-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg451-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg451-server: Pool lustre.test_204f destroyed PASS 204f (53s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:25:31 (1747848331) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:25:33 (1747848333) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:25:35 (1747848335) oleg451-server: Pool lustre.test_206 created Adding targets to pool oleg451-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg451-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg451-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg451-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg451-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:25:49 (1747848349) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018818 s, 55.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165799 s, 63.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3ec39abd Mirror 3: 0x3ec39abd PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:25:51 (1747848351) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:25:53 (1747848353) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:25:54 (1747848354) Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0126823 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2080000 Read this (4 bytes): BBBB 192.168.204.151@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg451-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:26:09 (1747848369) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00352671 s, 2.3 MB/s the very first read read with warm cache PASS 209b (1s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:26:10 (1747848370) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138448 s, 75.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb6 lmm_fid: [0x200000405:0xb6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb6 lmm_fid: [0x200000405:0xb6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611318 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-21 13:26:11.000000000 -0400 Modify: 2025-05-21 13:26:11.000000000 -0400 Change: 2025-05-21 13:26:11.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:26:13 (1747848373) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100633 s, 104 MB/s oleg451-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg451-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg451-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg451-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg451-server: O/280000401/d24/d8: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:26:27 (1747848387) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0311491 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00527387 s, 777 kB/s size after second write -rw-r--r-- 1 root root 4096 May 21 13:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 21 13:26 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test complete, duration 800 sec ============ 13:26:29 (1747848389) === sanity-flr: start cleanup 13:26:30 (1747848390) === === sanity-flr: finish cleanup 13:26:32 (1747848392) ===