-----============= acceptance-small: sanity-flr ============----- Fri Feb 7 17:15:40 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg355-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:15:45 (1738966545) === oleg355-client.virtnet: executing check_config_client /mnt/lustre oleg355-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg355-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b65f0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b65f0000.idle_timeout=debug disable quota as required oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:15:54 (1738966554) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7617] == sanity-flr test 0a: lfs mirror create with -N option == 17:15:54 (1738966554) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:15:58 (1738966558) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:15:59 (1738966559) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (0s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:15:59 (1738966559) PASS 0d (5s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:16:04 (1738966564) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:16:05 (1738966565) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (0s) == sanity-flr test 0g: lfs mirror create flags support === 17:16:05 (1738966565) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8800b65f0000.stats=clear osc.lustre-OST0001-osc-ffff8800b65f0000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:16:07 (1738966567) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (2s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:16:09 (1738966569) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:16:11 (1738966571) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:16:12 (1738966572) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 17:16:14 (1738966574) PASS 2 (1s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:16:15 (1738966575) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:16:17 (1738966577) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td lfs extend: cannot get the last init comp error: lfs extend: /mnt/lustre/d4.sanity-flr: syserror: Is a directory PASS 4 (1s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:16:18 (1738966578) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:16:21 (1738966581) PASS 7 (1s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:16:22 (1738966582) 18+0 records in 18+0 records out 18874368 bytes (19 MB) copied, 0.299923 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179406 s, 58.4 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:16:24 (1738966584) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0293796 s, 178 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:16:26 (1738966586) Stopping /mnt/lustre-ost1 (opts:) on oleg355-server oleg355-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b65f0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b65f0000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg355-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg355-server oleg355-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b65f0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b65f0000.ost_server_uuid in DISCONN state after 3 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg355-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (23s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:16:49 (1738966609) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.172109 s, 60.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg355-server oleg355-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b65f0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b65f0000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg355-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (9s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:16:58 (1738966618) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg355-server oleg355-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b65f0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b65f0000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg355-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg355-server oleg355-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b65f0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b65f0000.ost_server_uuid in DISCONN state after 4 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg355-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:17:19 (1738966639) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 0.802188 s, 65.4 MB/s -rw-r--r-- 1 root root 52428800 Feb 7 17:17 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.229s user 0m0.000s sys 0m0.106s Stopping /mnt/lustre-ost1 (opts:) on oleg355-server oleg355-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b65f0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b65f0000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.233s user 0m0.000s sys 0m0.110s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg355-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg355-server oleg355-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b65f0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b65f0000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost1)... real 0m0.252s user 0m0.000s sys 0m0.097s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg355-server: oleg355-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg355-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b65f0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b65f0000.ost_server_uuid in FULL state after 0 sec PASS 33b (21s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:17:40 (1738966660) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:17:41 (1738966661) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (0s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:17:41 (1738966661) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 17:17:42 (1738966662) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 17:17:44 (1738966664) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.109253 s, 115 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (2s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:17:46 (1738966666) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00834242 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108958 s, 96.2 MB/s PASS 36b (1s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:17:48 (1738966668) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:17:50 (1738966670) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00050538 s, 2.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000506384 s, 2.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000697455 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000587582 s, 1.7 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 17:17:52 (1738966672) create test files with size 15751647 .. e60d1af6fa63fe5272ee3bcdcc77a5b2 - fc4176d973dc297da4566b4c27867704 - e1db4975c3915ad203b2e26c60d17fea - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg355-client.virtnet /mnt/lustre (opts:) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre PASS 37 (4s) == sanity-flr test 38: resync ============================ 17:17:56 (1738966676) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (2s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:17:58 (1738966678) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:18:00 (1738966680) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164843 s, 63.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00739072 s, 142 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 17:18:02 (1738966682) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0205077 s, 102 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0851257 s, 49.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0336489 s, 62.3 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 17:18:04 (1738966684) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 17:18:04 (1738966684) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:18:05 (1738966685) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00975906 s, 107 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 17:18:07 (1738966687) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0726814 s, 43.3 MB/s PASS 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:18:09 (1738966689) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x46:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x46:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (1s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:18:11 (1738966691) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0642373 s, 163 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg355-client.virtnet /mnt/lustre (opts:) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:18:34 (1738966714) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a950f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0638323 s, 164 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg355-client.virtnet /mnt/lustre (opts:) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff88012be1e000.xattr_cache=1 PASS 44e (3s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:18:37 (1738966717) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (6s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:18:43 (1738966723) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (6s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:18:49 (1738966729) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 17:18:50 (1738966730) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.0592466 s, 53.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.041287 s, 76.2 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: bb07a87940ef981022bd9e0f95e226ab - mirror 1 checksum: b5f6bfedad0beb1943739347c876b6ec - mirror 2 checksum: bb07a87940ef981022bd9e0f95e226ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Feb 7 17:18:51 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Feb 7 17:18:51 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: bb07a87940ef981022bd9e0f95e226ab - mirror 1 checksum: b5f6bfedad0beb1943739347c876b6ec - mirror 2 checksum: b5f6bfedad0beb1943739347c876b6ec - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Feb 7 17:18:51 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x52:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Feb 7 17:18:51 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } PASS 48 (2s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:18:52 (1738966732) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012082 s, 86.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (1s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:18:53 (1738966733) Failing mds1 on oleg355-server Stopping /mnt/lustre-mds1 (opts:) on oleg355-server 17:19:00 (1738966740) shut down facet: mds1 facet_host: oleg355-server facet_failover_host: oleg355-server Failover mds1 to oleg355-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 17:19:13 (1738966753) targets are mounted 17:19:13 (1738966753) facet_failover done oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00812154 s, 129 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:19:20 (1738966760) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00689246 s, 3.0 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00333061 s, 6.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:19:22 (1738966762) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00743105 s, 2.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00311926 s, 6.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:19:24 (1738966764) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00193727 s, 8.5 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:19:26 (1738966766) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00765671 s, 2.7 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:19:27 (1738966767) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00144671 s, 2.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00659634 s, 621 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:19:29 (1738966769) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (3s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:19:32 (1738966772) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:19:50 (1738966790) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:20:07 (1738966807) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (16s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:20:23 (1738966823) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 17:20:25 (1738966825) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 26275 mirror split pid 26276 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 26275 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4866: 26276 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (61s) == sanity-flr test 70a: flr mode fsx test ================ 17:21:26 (1738966886) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 6887 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported 000001 1738966887.443663 write 0x02c689 thru 0x02d018 (0x990 bytes) 000003 1738966887.446818 read 0x0292f9 thru 0x02c914 (0x361c bytes) 000004 1738966887.448582 punch from 0x005fe7 to 0x012cf3 (0xcd0d bytes) 000005 1738966887.449713 write 0x0002d7 thru 0x0026d8 (0x2402 bytes) fallocating to largest ever: 0x3a04d 000006 1738966887.452082 fallocate 0x02a483 thru 0x03a04c (0xfbca bytes) 000007 1738966887.453658 read 0x0127a4 thru 0x019db2 (0x760f bytes) 000009 1738966887.456206 mapread 0x014e77 thru 0x01721b (0x23a5 bytes) 000010 1738966887.456613 fallocate 0x01f8f4 thru 0x02e6f7 (0xee04 bytes) 000011 1738966887.457697 punch from 0x0066d7 to 0x00a7f4 (0x411e bytes) 000012 1738966887.458896 write 0x022379 thru 0x029e1b (0x7aa3 bytes) 000013 1738966887.459456 read 0x01716b thru 0x018931 (0x17c7 bytes) 000014 1738966887.459839 mapread 0x027a3e thru 0x02d018 (0x55db bytes) 000015 1738966887.460248 write 0x00bbd8 thru 0x0182cf (0xc6f8 bytes) 000016 1738966887.461034 read 0x0096fb thru 0x015b0c (0xc412 bytes) 000017 1738966887.462841 mapread 0x029c7d thru 0x02d018 (0x339c bytes) 000018 1738966887.463276 mapwrite 0x02f460 thru 0x03e8ca (0xf46b bytes) 000020 1738966887.477920 write 0x007527 thru 0x00e9ad (0x7487 bytes) fallocating to largest ever: 0x3e47b 000022 1738966887.478773 fallocate 0x032f3d thru 0x03e47a (0xb53e bytes) 000025 1738966887.480843 mapread 0x03c51f thru 0x03e8ca (0x23ac bytes) 000026 1738966887.481312 write 0x0251e6 thru 0x02a449 (0x5264 bytes) 000027 1738966887.481984 write 0x01660b thru 0x02295b (0xc351 bytes) 000028 1738966887.482657 read 0x029abd thru 0x039a54 (0xff98 bytes) 000029 1738966887.483060 write 0x032590 thru 0x038ef9 (0x696a bytes) 000033 1738966887.484294 mapwrite 0x03195b thru 0x036749 (0x4def bytes) 000034 1738966887.489340 mapwrite 0x0259a5 thru 0x02dd45 (0x83a1 bytes) 000036 1738966887.493942 punch from 0x0063b6 to 0x00663f (0x28a bytes) 000037 1738966887.495088 punch from 0x03693b to 0x03e8ca (0x7f90 bytes) 000038 1738966887.496547 mapread 0x03c434 thru 0x03e8ca (0x2497 bytes) 000040 1738966887.499701 punch from 0x03588a to 0x03635d (0xad4 bytes) 000041 1738966887.500838 trunc from 0x03e8cb to 0x000143 (0x3e787 bytes) 000042 1738966887.504940 fallocate 0x023d7a thru 0x02e6d3 (0xa95a bytes) 000044 1738966887.506697 write 0x019df7 thru 0x021c54 (0x7e5e bytes) 000045 1738966887.507312 mapwrite 0x02eb52 thru 0x03042f (0x18de bytes) 000046 1738966887.518287 mapwrite 0x02d70f thru 0x03d2d7 (0xfbc9 bytes) 000048 1738966887.542875 fallocate 0x019ea8 thru 0x022771 (0x88ca bytes) 000050 1738966887.544878 trunc from 0x03d2d8 to 0x03ad28 (0x25af bytes) 000051 1738966887.548233 mapwrite 0x016870 thru 0x026183 (0xf914 bytes) 000052 1738966887.562631 trunc from 0x03ad29 to 0x0199a2 (0x21386 bytes) 000054 1738966887.567022 mapwrite 0x02575e thru 0x034628 (0xeecb bytes) fallocating to largest ever: 0x3fda6 000055 1738966887.591986 fallocate 0x03440d thru 0x03fda5 (0xb999 bytes) 000056 1738966887.593363 fallocate 0x006c9c thru 0x00bc99 (0x4ffe bytes) 000058 1738966887.595083 punch from 0x02953a to 0x02d262 (0x3d29 bytes) 000060 1738966887.596646 trunc from 0x03fda6 to 0x03a66d (0x5738 bytes) 000062 1738966887.600516 trunc from 0x03a66e to 0x0158b5 (0x24db8 bytes) 000065 1738966887.604849 fallocate 0x01f9d9 thru 0x028b04 (0x912c bytes) 000067 1738966887.606457 mapwrite 0x005871 thru 0x011d22 (0xc4b2 bytes) 000068 1738966887.626240 read 0x020380 thru 0x028b04 (0x8785 bytes) 000069 1738966887.628644 write 0x03f54c thru 0x03ffff (0xab4 bytes) 000070 1738966887.629075 mapwrite 0x025216 thru 0x027e92 (0x2c7d bytes) 000071 1738966887.633444 read 0x036fda thru 0x03ffff (0x9026 bytes) 000072 1738966887.636087 punch from 0x037cb2 to 0x03ffff (0x834e bytes) 000073 1738966887.637593 mapwrite 0x025877 thru 0x02eaad (0x9237 bytes) 000074 1738966887.642912 read 0x02ad2f thru 0x02dc6f (0x2f41 bytes) 000075 1738966887.643306 mapwrite 0x0185ae thru 0x0204c6 (0x7f19 bytes) 000076 1738966887.648429 read 0x02e9f0 thru 0x03a9a9 (0xbfba bytes) 000077 1738966887.650651 read 0x01692c thru 0x021c2f (0xb304 bytes) 000078 1738966887.651119 write 0x03c96f thru 0x03ffff (0x3691 bytes) 000079 1738966887.653133 MIRROR_EXTEND to 2 mirrors 000080 1738966887.764091 punch from 0x010f4d to 0x011d41 (0xdf5 bytes) 000081 1738966887.772452 read 0x03d4e5 thru 0x03fdbc (0x28d8 bytes) 000082 1738966887.774441 trunc from 0x040000 to 0x01408f (0x2bf70 bytes) 000084 1738966887.783422 punch from 0x00ced0 to 0x01408f (0x71c0 bytes) 000086 1738966887.785024 trunc from 0x014090 to 0x001e9e (0x121f1 bytes) 000087 1738966887.788416 punch from 0x0010b9 to 0x001e9e (0xde6 bytes) 000088 1738966887.789665 punch from 0x0007c3 to 0x001e9e (0x16dc bytes) 000091 1738966887.791225 mapread 0x001127 thru 0x001e9e (0xd78 bytes) 000092 1738966887.792942 punch from 0x0000d4 to 0x001e9e (0x1dcb bytes) 000094 1738966887.795960 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000095 1738966887.824280 write 0x00a243 thru 0x00dae3 (0x38a1 bytes) 000097 1738966887.825940 mapread 0x00d913 thru 0x00dae3 (0x1d1 bytes) 000098 1738966887.826307 fallocate 0x017975 thru 0x023af1 (0xc17d bytes) 000099 1738966887.827769 punch from 0x01ab13 to 0x01bd9f (0x128d bytes) 000100 1738966887.828986 trunc from 0x023af2 to 0x03f125 (0x1b634 bytes) 000102 1738966887.832354 write 0x037b3f thru 0x038f0b (0x13cd bytes) 000103 1738966887.834996 mapread 0x00cb92 thru 0x01acd1 (0xe140 bytes) 000104 1738966887.839466 read 0x00e90e thru 0x01dba8 (0xf29b bytes) 000106 1738966887.840326 trunc from 0x03f126 to 0x02342f (0x1bcf6 bytes) 000107 1738966887.843981 trunc from 0x023430 to 0x01b409 (0x8026 bytes) 000108 1738966887.847491 write 0x0186a4 thru 0x0231bc (0xab19 bytes) 000112 1738966887.849694 write 0x0063fa thru 0x00b2cd (0x4ed4 bytes) 000113 1738966887.852239 fallocate 0x01dadf thru 0x0272ae (0x97d0 bytes) 000114 1738966887.853708 write 0x030890 thru 0x037d33 (0x74a4 bytes) 000115 1738966887.854596 write 0x028589 thru 0x02b021 (0x2a99 bytes) 000116 1738966887.856696 trunc from 0x037d34 to 0x02f59e (0x8795 bytes) fallocating to largest ever: 0x40000 000118 1738966887.860692 fallocate 0x03a6d0 thru 0x03ffff (0x5930 bytes) 000119 1738966887.862359 write 0x029017 thru 0x038f73 (0xff5d bytes) 000122 1738966887.864090 fallocate 0x004e45 thru 0x006cc0 (0x1e7c bytes) 000124 1738966887.865594 mapwrite 0x029f52 thru 0x02f9bf (0x5a6e bytes) 000125 1738966887.871433 mapread 0x0152d7 thru 0x022ef6 (0xdc20 bytes) 000126 1738966887.871922 read 0x003897 thru 0x007463 (0x3bcd bytes) 000127 1738966887.872233 punch from 0x027829 to 0x02cf29 (0x5701 bytes) 000128 1738966887.873498 punch from 0x0108f4 to 0x01b4cc (0xabd9 bytes) 000129 1738966887.874901 fallocate 0x01472c thru 0x01e2b6 (0x9b8b bytes) 000130 1738966887.876188 mapwrite 0x001348 thru 0x006ed3 (0x5b8c bytes) 000131 1738966887.880634 mapread 0x01c616 thru 0x0279ea (0xb3d5 bytes) 000132 1738966887.882742 write 0x0075ee thru 0x00834b (0xd5e bytes) 000134 1738966887.883363 read 0x001901 thru 0x00d759 (0xbe59 bytes) 000135 1738966887.883640 fallocate 0x03183f thru 0x038b47 (0x7309 bytes) 000136 1738966887.884887 mapread 0x029be2 thru 0x037599 (0xd9b8 bytes) 000137 1738966887.885274 write 0x020ab8 thru 0x025fce (0x5517 bytes) 000140 1738966887.886117 punch from 0x01d612 to 0x02a900 (0xd2ef bytes) 000141 1738966887.889286 trunc from 0x038f74 to 0x0124e1 (0x26a92 bytes) 000142 1738966887.892902 fallocate 0x010c2b thru 0x01b67c (0xaa52 bytes) 000143 1738966887.894346 write 0x000370 thru 0x010064 (0xfcf5 bytes) 000145 1738966887.895546 trunc from 0x01b67d to 0x02a377 (0xecfb bytes) 000146 1738966887.899094 read 0x024b2e thru 0x02a377 (0x584a bytes) 000147 1738966887.902168 mapwrite 0x006984 thru 0x00f328 (0x89a5 bytes) 000149 1738966887.909085 read 0x025a0d thru 0x02a377 (0x496b bytes) 000150 1738966887.909372 mapwrite 0x03a519 thru 0x03ffff (0x5ae7 bytes) 000151 1738966887.919687 trunc from 0x040000 to 0x009a51 (0x365ae bytes) 000153 1738966887.925732 mapwrite 0x0374a8 thru 0x03c441 (0x4f9a bytes) 000154 1738966887.940982 punch from 0x01d607 to 0x026b1b (0x9515 bytes) 000155 1738966887.942147 fallocate 0x0063db thru 0x00a2cd (0x3ef3 bytes) 000156 1738966887.943498 mapread 0x015a19 thru 0x01facd (0xa0b5 bytes) 000157 1738966887.956487 MIRROR_EXTEND to 2 mirrors 000158 1738966888.052854 trunc from 0x03c442 to 0x005a6f (0x369d2 bytes) 000159 1738966888.063622 punch from 0x0035ee to 0x005a6f (0x2482 bytes) 000160 1738966888.064885 punch from 0x0048a3 to 0x005a6f (0x11cd bytes) 000161 1738966888.066115 fallocate 0x01d5e8 thru 0x02c4ca (0xeee3 bytes) 000163 1738966888.067962 write 0x01247a thru 0x01e4dd (0xc064 bytes) 000164 1738966888.070611 fallocate 0x00f12f thru 0x013f8a (0x4e5c bytes) 000165 1738966888.072141 fallocate 0x01fd13 thru 0x02b452 (0xb740 bytes) 000166 1738966888.074964 MIRROR_EXTEND to 3 mirrors 000167 1738966888.180385 write 0x035076 thru 0x037c1f (0x2baa bytes) 000168 1738966888.188543 mapwrite 0x013d96 thru 0x01fd54 (0xbfbf bytes) 000169 1738966888.201771 mapwrite 0x013aeb thru 0x01d907 (0x9e1d bytes) 000172 1738966888.208608 mapwrite 0x004c84 thru 0x008092 (0x340f bytes) 000173 1738966888.216141 read 0x033712 thru 0x037c1f (0x450e bytes) 000174 1738966888.217983 read 0x026a4d thru 0x02dbbc (0x7170 bytes) 000175 1738966888.221639 MIRROR_EXTEND to 4 mirrors 000177 1738966888.332553 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000178 1738966888.383444 write 0x0228c4 thru 0x026ec4 (0x4601 bytes) 000180 1738966888.394342 mapwrite 0x025514 thru 0x0285a8 (0x3095 bytes) 000181 1738966888.401379 punch from 0x027b09 to 0x0338cb (0xbdc3 bytes) 000182 1738966888.403013 trunc from 0x037c20 to 0x0340cd (0x3b52 bytes) 000183 1738966888.410733 fallocate 0x03c6a4 thru 0x03ffff (0x395c bytes) 000185 1738966888.412652 trunc from 0x0340ce to 0x0297a7 (0xa926 bytes) 000186 1738966888.416002 mapread 0x00131c thru 0x00991a (0x85ff bytes) 000187 1738966888.419428 read 0x010f76 thru 0x014215 (0x32a0 bytes) 000188 1738966888.421910 MIRROR_EXTEND to 4 mirrors 000189 1738966888.522329 trunc from 0x0297a8 to 0x009d0a (0x1fa9d bytes) 000191 1738966888.537534 write 0x03bca9 thru 0x03ffff (0x4357 bytes) 000192 1738966888.538430 read 0x01bcb0 thru 0x0245bf (0x8910 bytes) 000193 1738966888.541056 mapread 0x00944e thru 0x00ced7 (0x3a8a bytes) 000194 1738966888.545197 mapread 0x001711 thru 0x00fedf (0xe7cf bytes) 000195 1738966888.547932 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000196 1738966888.593559 read 0x02b255 thru 0x036dba (0xbb66 bytes) 000197 1738966888.595656 read 0x00943c thru 0x017b72 (0xe737 bytes) 000198 1738966888.598690 punch from 0x0247cd to 0x02bb4f (0x7383 bytes) 000200 1738966888.600216 punch from 0x032a1e to 0x03ffff (0xd5e2 bytes) 000201 1738966888.601522 write 0x031f71 thru 0x03ffff (0xe08f bytes) 000202 1738966888.603304 trunc from 0x040000 to 0x00055f (0x3faa0 bytes) 000203 1738966888.608600 MIRROR_RESYNC 3 mirrors 000204 1738966888.683017 trunc from 0x000560 to 0x02898a (0x2842b bytes) 000206 1738966888.696704 punch from 0x01b70e to 0x01c016 (0x909 bytes) 000207 1738966888.697933 mapread 0x010779 thru 0x01521e (0x4aa6 bytes) 000208 1738966888.703772 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000209 1738966888.747121 mapwrite 0x0072ee thru 0x00c28e (0x4fa1 bytes) 000211 1738966888.757405 MIRROR_RESYNC 2 mirrors 000212 1738966888.842944 write 0x0353bc thru 0x037abd (0x2702 bytes) 000214 1738966888.850247 read 0x016faa thru 0x023d3f (0xcd96 bytes) 000215 1738966888.852683 mapread 0x014523 thru 0x01cda6 (0x8884 bytes) 000216 1738966888.855393 write 0x037016 thru 0x03e22e (0x7219 bytes) 000217 1738966888.856260 punch from 0x010665 to 0x01093c (0x2d8 bytes) 000218 1738966888.858584 MIRROR_EXTEND to 3 mirrors 000219 1738966888.954459 punch from 0x02aa44 to 0x02ad2e (0x2eb bytes) 000221 1738966888.963016 punch from 0x018058 to 0x023c3d (0xbbe6 bytes) 000222 1738966888.964457 fallocate 0x02f34f thru 0x038e83 (0x9b35 bytes) 000224 1738966888.966307 fallocate 0x03c436 thru 0x03ffff (0x3bca bytes) 000225 1738966888.968180 fallocate 0x000827 thru 0x00f9ef (0xf1c9 bytes) 000226 1738966888.969686 write 0x026766 thru 0x034792 (0xe02d bytes) 000227 1738966888.973888 mapread 0x026b15 thru 0x031974 (0xae60 bytes) 000228 1738966888.974442 trunc from 0x03e22f to 0x007d7d (0x364b1 bytes) 000229 1738966888.983054 read 0x005a67 thru 0x0073b0 (0x194a bytes) 000231 1738966888.985157 mapread 0x002907 thru 0x0062a6 (0x39a0 bytes) 000232 1738966888.987016 mapwrite 0x009e76 thru 0x0117ad (0x7938 bytes) 000234 1738966889.000394 mapread 0x011392 thru 0x0117ad (0x41c bytes) 000235 1738966889.000828 fallocate 0x02e03b thru 0x02fa9d (0x1a63 bytes) 000236 1738966889.002248 fallocate 0x019a38 thru 0x026943 (0xcf0c bytes) 000239 1738966889.003911 trunc from 0x026944 to 0x0090b4 (0x1d88f bytes) 000241 1738966889.007702 write 0x02692f thru 0x02edb7 (0x8489 bytes) 000243 1738966889.008571 fallocate 0x026890 thru 0x02af70 (0x46e1 bytes) 000244 1738966889.010360 mapwrite 0x02db2c thru 0x03a146 (0xc61b bytes) 000245 1738966889.022612 trunc from 0x03a147 to 0x01856c (0x21bda bytes) 000246 1738966889.026602 mapwrite 0x02eab3 thru 0x038462 (0x99b0 bytes) 000247 1738966889.039241 mapwrite 0x00f463 thru 0x019cd2 (0xa870 bytes) 000248 1738966889.049533 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000249 1738966889.097021 write 0x02ef58 thru 0x02fbc6 (0xc6f bytes) 000251 1738966889.101214 punch from 0x02690c to 0x033b25 (0xd21a bytes) 000253 1738966889.104757 read 0x0026c1 thru 0x003fb3 (0x18f3 bytes) 000254 1738966889.106224 mapread 0x01d6a4 thru 0x02a5f4 (0xcf51 bytes) 000255 1738966889.111056 fallocate 0x01202a thru 0x014795 (0x276c bytes) 000257 1738966889.114483 MIRROR_EXTEND to 3 mirrors 000258 1738966889.222357 MIRROR_EXTEND to 4 mirrors 000259 1738966889.328762 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000262 1738966889.386026 read 0x031faf thru 0x038462 (0x64b4 bytes) 000263 1738966889.389450 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000264 1738966889.450076 punch from 0x02dd8f to 0x038462 (0xa6d4 bytes) 000265 1738966889.457978 trunc from 0x038463 to 0x016832 (0x21c30 bytes) 000267 1738966889.466742 mapread 0x0124e7 thru 0x016832 (0x434c bytes) 000268 1738966889.470845 fallocate 0x006a10 thru 0x00e139 (0x772a bytes) 000269 1738966889.472388 trunc from 0x016833 to 0x03ada6 (0x24574 bytes) 000271 1738966889.476488 read 0x032d63 thru 0x035a9d (0x2d3b bytes) 000273 1738966889.478256 mapwrite 0x00e304 thru 0x011ea2 (0x3b9f bytes) 000274 1738966889.482998 trunc from 0x03ada7 to 0x033d57 (0x704f bytes) 000275 1738966889.486605 mapread 0x0090eb thru 0x013739 (0xa64f bytes) 000278 1738966889.489641 write 0x01505b thru 0x01b727 (0x66cd bytes) 000279 1738966889.491651 read 0x021af0 thru 0x02519b (0x36ac bytes) 000280 1738966889.494350 punch from 0x01839e to 0x02554c (0xd1af bytes) 000281 1738966889.497970 punch from 0x026d11 to 0x02e04d (0x733d bytes) 000284 1738966889.500090 mapread 0x003249 thru 0x0110a7 (0xde5f bytes) 000286 1738966889.501149 read 0x0269a0 thru 0x027b89 (0x11ea bytes) 000287 1738966889.505265 MIRROR_RESYNC 2 mirrors 000289 1738966889.580283 mapwrite 0x0012fb thru 0x00b438 (0xa13e bytes) 000290 1738966889.608744 read 0x015666 thru 0x016113 (0xaae bytes) 000291 1738966889.610305 trunc from 0x033d58 to 0x0258d5 (0xe482 bytes) 000292 1738966889.618496 fallocate 0x02ac47 thru 0x02cb7a (0x1f34 bytes) 000294 1738966889.621841 MIRROR_RESYNC 2 mirrors 000296 1738966889.691730 write 0x016ede thru 0x02398f (0xcab2 bytes) 000298 1738966889.701011 mapwrite 0x0376e6 thru 0x03ffff (0x891a bytes) 000299 1738966889.718099 read 0x0156c8 thru 0x021ad5 (0xc40e bytes) 000300 1738966889.719718 write 0x0149d9 thru 0x01503b (0x663 bytes) 000301 1738966889.723702 MIRROR_RESYNC 2 mirrors 000302 1738966889.803134 mapread 0x02cca1 thru 0x034002 (0x7362 bytes) 000303 1738966889.807009 mapwrite 0x02715f thru 0x035774 (0xe616 bytes) 000304 1738966889.839726 read 0x016053 thru 0x0170d6 (0x1084 bytes) 000307 1738966889.842953 MIRROR_RESYNC 2 mirrors 000308 1738966889.925602 fallocate 0x00cb19 thru 0x0157a8 (0x8c90 bytes) 000309 1738966889.933031 mapwrite 0x039030 thru 0x03ffff (0x6fd0 bytes) 000310 1738966889.942806 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000311 1738966889.970403 mapread 0x02d57d thru 0x037a87 (0xa50b bytes) 000314 1738966889.975812 trunc from 0x040000 to 0x002d47 (0x3d2b8 bytes) 000316 1738966889.984855 MIRROR_EXTEND to 2 mirrors 000318 1738966890.064313 fallocate 0x0055cf thru 0x00aef4 (0x5926 bytes) 000319 1738966890.072515 punch from 0x0013f3 to 0x002d47 (0x1955 bytes) 000321 1738966890.074038 fallocate 0x032492 thru 0x035dea (0x3959 bytes) 000322 1738966890.075615 punch from 0x000be6 to 0x002d47 (0x2162 bytes) 000323 1738966890.077048 read 0x00033b thru 0x002d47 (0x2a0d bytes) 000325 1738966890.078874 mapread 0x00146a thru 0x002d47 (0x18de bytes) 000326 1738966890.079332 read 0x0020c9 thru 0x002d47 (0xc7f bytes) 000328 1738966890.081045 MIRROR_EXTEND to 3 mirrors 000329 1738966890.162835 punch from 0x00200d to 0x002d47 (0xd3b bytes) 000330 1738966890.170193 fallocate 0x0115b5 thru 0x018ac2 (0x750e bytes) 000331 1738966890.171525 punch from 0x00d436 to 0x01757b (0xa146 bytes) 000332 1738966890.172577 read 0x0074fe thru 0x015e0b (0xe90e bytes) 000333 1738966890.174789 write 0x006972 thru 0x014fb0 (0xe63f bytes) 000335 1738966890.178084 mapwrite 0x004023 thru 0x00e6fd (0xa6db bytes) 000336 1738966890.186669 trunc from 0x018ac3 to 0x01a275 (0x17b3 bytes) 000337 1738966890.190756 trunc from 0x01a276 to 0x037f26 (0x1dcb1 bytes) 000339 1738966890.194387 write 0x010072 thru 0x018a3b (0x89ca bytes) 000340 1738966890.196109 read 0x01abd7 thru 0x01ecb7 (0x40e1 bytes) 000342 1738966890.198998 mapread 0x01026e thru 0x0172dc (0x706f bytes) 000343 1738966890.199601 fallocate 0x0006fa thru 0x0091b3 (0x8aba bytes) 000344 1738966890.201185 fallocate 0x01a3f0 thru 0x026d5f (0xc970 bytes) 000345 1738966890.202648 write 0x01aad6 thru 0x028121 (0xd64c bytes) 000346 1738966890.203439 read 0x01bc5a thru 0x02a66d (0xea14 bytes) 000348 1738966890.204071 mapwrite 0x009905 thru 0x00cb82 (0x327e bytes) 000349 1738966890.210575 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000350 1738966890.251929 mapwrite 0x01f3dc thru 0x02883f (0x9464 bytes) 000351 1738966890.260304 write 0x0007b0 thru 0x00ae8d (0xa6de bytes) 000352 1738966890.263568 read 0x00297e thru 0x005cba (0x333d bytes) 000353 1738966890.263873 mapwrite 0x03982a thru 0x03ffff (0x67d6 bytes) 000354 1738966890.275510 mapread 0x025d39 thru 0x026c41 (0xf09 bytes) 000355 1738966890.275953 trunc from 0x040000 to 0x0286df (0x17920 bytes) 000356 1738966890.279319 read 0x005321 thru 0x00a2eb (0x4fcb bytes) 000357 1738966890.279737 fallocate 0x03d1dd thru 0x03ffff (0x2e23 bytes) 000358 1738966890.280923 mapread 0x0254b1 thru 0x0286df (0x322f bytes) 000359 1738966890.281639 trunc from 0x0286e0 to 0x012014 (0x166cb bytes) 000360 1738966890.285323 fallocate 0x01c7e9 thru 0x026a34 (0xa24c bytes) 000361 1738966890.286709 fallocate 0x022aff thru 0x02f262 (0xc764 bytes) 000362 1738966890.288088 mapread 0x0087f9 thru 0x01031f (0x7b27 bytes) 000363 1738966890.290715 punch from 0x003a53 to 0x006e86 (0x3434 bytes) 000364 1738966890.292081 punch from 0x01f19c to 0x02d67f (0xe4e4 bytes) 000365 1738966890.293376 mapwrite 0x033327 thru 0x039d52 (0x6a2c bytes) 000366 1738966890.304530 trunc from 0x039d53 to 0x032bb7 (0x719b bytes) 000367 1738966890.308337 write 0x007ffc thru 0x010fbb (0x8fc0 bytes) 000368 1738966890.309073 write 0x00d9e5 thru 0x0127a3 (0x4dbf bytes) 000369 1738966890.309905 mapwrite 0x011907 thru 0x01f1a1 (0xd89b bytes) 000372 1738966890.318844 fallocate 0x00b179 thru 0x01a5dd (0xf465 bytes) 000373 1738966890.320407 mapwrite 0x022825 thru 0x0285d6 (0x5db2 bytes) 000374 1738966890.331819 write 0x02ad1d thru 0x0385b5 (0xd899 bytes) 000375 1738966890.333647 mapwrite 0x022344 thru 0x02a5cc (0x8289 bytes) 000376 1738966890.339411 mapwrite 0x0108a1 thru 0x01757a (0x6cda bytes) 000377 1738966890.343841 mapread 0x02dc23 thru 0x02f54a (0x1928 bytes) 000378 1738966890.344368 mapread 0x003fc0 thru 0x005f86 (0x1fc7 bytes) 000379 1738966890.347616 trunc from 0x0385b6 to 0x035968 (0x2c4d bytes) 000380 1738966890.351141 read 0x025658 thru 0x026ade (0x1487 bytes) 000381 1738966890.351413 punch from 0x00e59f to 0x019d91 (0xb7f3 bytes) 000382 1738966890.353077 fallocate 0x01b263 thru 0x01ec36 (0x39d4 bytes) 000383 1738966890.355625 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000384 1738966890.382128 punch from 0x0009fb to 0x008997 (0x7f9d bytes) 000386 1738966890.383916 read 0x00f3fa thru 0x011654 (0x225b bytes) 000387 1738966890.385493 mapwrite 0x02e63c thru 0x038e12 (0xa7d7 bytes) 000390 1738966890.399529 mapwrite 0x030fd9 thru 0x03b20d (0xa235 bytes) 000392 1738966890.409116 trunc from 0x03b20e to 0x02e620 (0xcbed bytes) 000393 1738966890.412142 mapwrite 0x01301a thru 0x01c861 (0x9848 bytes) 000394 1738966890.420993 punch from 0x017c5e to 0x01e054 (0x63f7 bytes) 000395 1738966890.422255 trunc from 0x02e621 to 0x00628e (0x28392 bytes) 000396 1738966890.425544 punch from 0x00237a to 0x00628e (0x3f15 bytes) 000398 1738966890.427164 read 0x004a1c thru 0x00628e (0x1873 bytes) 000399 1738966890.429304 trunc from 0x00628f to 0x020214 (0x19f86 bytes) 000400 1738966890.432689 write 0x01b2de thru 0x01da34 (0x2757 bytes) 000402 1738966890.436418 MIRROR_EXTEND to 2 mirrors 000403 1738966890.527797 mapread 0x003610 thru 0x009638 (0x6029 bytes) 000404 1738966890.535010 mapwrite 0x00292e thru 0x00947c (0x6b4f bytes) 000405 1738966890.560920 trunc from 0x020215 to 0x03bab1 (0x1b89d bytes) 000406 1738966890.569368 fallocate 0x0039d4 thru 0x013262 (0xf88f bytes) 000407 1738966890.571964 MIRROR_RESYNC 2 mirrors 000408 1738966890.662665 write 0x0162ca thru 0x0238c3 (0xd5fa bytes) 000409 1738966890.671598 mapwrite 0x01e2d5 thru 0x02d2a7 (0xefd3 bytes) 000410 1738966890.681980 mapread 0x0354a3 thru 0x03bab1 (0x660f bytes) 000411 1738966890.684532 read 0x038009 thru 0x03bab1 (0x3aa9 bytes) 000412 1738966890.685010 write 0x03cd67 thru 0x03ffff (0x3299 bytes) 000413 1738966890.685820 read 0x032469 thru 0x036efa (0x4a92 bytes) 000414 1738966890.687559 MIRROR_RESYNC 2 mirrors 000415 1738966890.783561 punch from 0x003637 to 0x00b424 (0x7dee bytes) 000416 1738966890.790850 mapwrite 0x0168ef thru 0x021b76 (0xb288 bytes) 000417 1738966890.802871 MIRROR_RESYNC 2 mirrors 000418 1738966890.908569 mapread 0x00fdc2 thru 0x019ec6 (0xa105 bytes) 000419 1738966890.914277 mapread 0x037432 thru 0x03ffff (0x8bce bytes) 000420 1738966890.917821 mapread 0x02081a thru 0x02ed84 (0xe56b bytes) 000422 1738966890.921431 mapread 0x023638 thru 0x024065 (0xa2e bytes) 000423 1738966890.921691 fallocate 0x037550 thru 0x03ca76 (0x5527 bytes) 000424 1738966890.928976 fallocate 0x023b83 thru 0x02caa7 (0x8f25 bytes) 000427 1738966890.931725 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000428 1738966890.960371 fallocate 0x027978 thru 0x02c620 (0x4ca9 bytes) 000429 1738966890.961957 fallocate 0x02ec55 thru 0x0310cd (0x2479 bytes) 000430 1738966890.963183 mapwrite 0x00a118 thru 0x00f3d3 (0x52bc bytes) 000431 1738966890.972597 read 0x00d4f1 thru 0x01882b (0xb33b bytes) 000433 1738966890.974399 punch from 0x033402 to 0x0341e1 (0xde0 bytes) 000434 1738966890.975532 punch from 0x001321 to 0x00dc69 (0xc949 bytes) 000435 1738966890.977739 MIRROR_EXTEND to 2 mirrors 000436 1738966891.089877 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000439 1738966891.117259 mapwrite 0x033a69 thru 0x03ffff (0xc597 bytes) 000440 1738966891.131375 trunc from 0x040000 to 0x0371b6 (0x8e49 bytes) 000443 1738966891.136656 read 0x0276dd thru 0x033440 (0xbd64 bytes) 000444 1738966891.139098 fallocate 0x022b70 thru 0x028bc2 (0x6053 bytes) 000446 1738966891.141020 mapread 0x01f6e9 thru 0x020bfe (0x1516 bytes) 000447 1738966891.143424 fallocate 0x02714f thru 0x035d4b (0xebfd bytes) 000449 1738966891.145702 trunc from 0x0371b7 to 0x01f5e4 (0x17bd2 bytes) 000450 1738966891.149247 read 0x0050f6 thru 0x00afb3 (0x5ebe bytes) 000451 1738966891.152553 MIRROR_EXTEND to 2 mirrors 000453 1738966891.245111 write 0x01eaf5 thru 0x024724 (0x5c30 bytes) 000454 1738966891.255429 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000455 1738966891.286546 read 0x01e0ba thru 0x01f228 (0x116f bytes) 000457 1738966891.288649 mapwrite 0x0004d3 thru 0x00cf0d (0xca3b bytes) 000458 1738966891.300238 mapwrite 0x034179 thru 0x03ffff (0xbe87 bytes) 000459 1738966891.317750 fallocate 0x031e79 thru 0x033c37 (0x1dbf bytes) 000460 1738966891.319117 trunc from 0x040000 to 0x007b27 (0x384d8 bytes) 000462 1738966891.322919 mapread 0x0064b7 thru 0x007b27 (0x1671 bytes) 000463 1738966891.323280 mapwrite 0x030c97 thru 0x03205e (0x13c8 bytes) 000464 1738966891.332746 mapread 0x02e9ab thru 0x03205e (0x36b4 bytes) 000465 1738966891.333194 trunc from 0x03205f to 0x024e81 (0xd1dd bytes) 000466 1738966891.336763 read 0x018a1d thru 0x01dc40 (0x5224 bytes) 000467 1738966891.339322 read 0x009881 thru 0x017dce (0xe54e bytes) 000468 1738966891.339728 write 0x00911c thru 0x009876 (0x75b bytes) 000469 1738966891.340124 mapread 0x01c787 thru 0x023c49 (0x74c3 bytes) 000470 1738966891.340528 trunc from 0x024e82 to 0x004096 (0x20deb bytes) 000473 1738966891.345379 trunc from 0x004097 to 0x00147b (0x2c1b bytes) 000474 1738966891.348752 fallocate 0x0358a5 thru 0x03e45d (0x8bb9 bytes) 000476 1738966891.350266 fallocate 0x027e9d thru 0x02bc1c (0x3d80 bytes) 000477 1738966891.351444 mapread 0x0319fb thru 0x0370d7 (0x56dd bytes) 000478 1738966891.354970 trunc from 0x03e45e to 0x01c16e (0x222ef bytes) 000480 1738966891.358728 mapread 0x0043f4 thru 0x00cded (0x89fa bytes) 000481 1738966891.359151 punch from 0x0051b2 to 0x00e5d7 (0x9426 bytes) 000483 1738966891.360714 punch from 0x0098c3 to 0x011836 (0x7f74 bytes) 000484 1738966891.361843 read 0x011292 thru 0x01c16e (0xaedd bytes) 000485 1738966891.363947 punch from 0x0052f4 to 0x00cbd5 (0x78e2 bytes) 000486 1738966891.365208 read 0x017bae thru 0x01c16e (0x45c1 bytes) 000487 1738966891.365580 punch from 0x00f607 to 0x012954 (0x334e bytes) 000491 1738966891.368071 write 0x03c3a1 thru 0x03d2e2 (0xf42 bytes) 000493 1738966891.368875 fallocate 0x0050c0 thru 0x01504f (0xff90 bytes) 000495 1738966891.370808 read 0x020095 thru 0x02096b (0x8d7 bytes) 000496 1738966891.373158 fallocate 0x02fcd4 thru 0x03410b (0x4438 bytes) 000497 1738966891.374366 mapwrite 0x01d764 thru 0x02cf9e (0xf83b bytes) 000498 1738966891.380684 mapread 0x009da3 thru 0x01171f (0x797d bytes) 000501 1738966891.381658 mapread 0x03a533 thru 0x03d2e2 (0x2db0 bytes) 000502 1738966891.382052 trunc from 0x03d2e3 to 0x000ca6 (0x3c63c bytes) 000503 1738966891.385888 fallocate 0x01d2ad thru 0x02184f (0x45a3 bytes) 000504 1738966891.387216 read 0x012818 thru 0x016394 (0x3b7d bytes) 000505 1738966891.389801 read 0x00b668 thru 0x00d83f (0x21d8 bytes) 000507 1738966891.390614 mapwrite 0x0295b3 thru 0x038c2b (0xf679 bytes) 000508 1738966891.401931 write 0x02d91b thru 0x032961 (0x5047 bytes) 000509 1738966891.402417 read 0x014867 thru 0x016ed5 (0x266f bytes) 000510 1738966891.402847 write 0x002521 thru 0x005623 (0x3103 bytes) 000512 1738966891.403727 mapread 0x0183a6 thru 0x02674e (0xe3a9 bytes) 000513 1738966891.404179 fallocate 0x02fbb2 thru 0x0370d7 (0x7526 bytes) 000514 1738966891.405721 trunc from 0x038c2c to 0x00fec3 (0x28d68 bytes) 000515 1738966891.409303 punch from 0x007c7e to 0x00d6b2 (0x5a35 bytes) 000517 1738966891.410820 mapwrite 0x02ea51 thru 0x03decc (0xf47c bytes) 000518 1738966891.423962 write 0x0208c0 thru 0x02a8bb (0x9ffc bytes) 000520 1738966891.424966 fallocate 0x00ba5b thru 0x00f807 (0x3dad bytes) 000521 1738966891.426319 mapwrite 0x02c026 thru 0x0333d0 (0x73ab bytes) 000525 1738966891.431763 punch from 0x001014 to 0x004115 (0x3102 bytes) 000528 1738966891.433831 mapwrite 0x033396 thru 0x03cf5b (0x9bc6 bytes) 000531 1738966891.438968 mapread 0x01eb4b thru 0x027246 (0x86fc bytes) 000533 1738966891.440701 MIRROR_EXTEND to 2 mirrors 000534 1738966891.535928 read 0x00886b thru 0x014cbd (0xc453 bytes) 000536 1738966891.540771 trunc from 0x03decd to 0x0095f4 (0x348d8 bytes) 000537 1738966891.552222 fallocate 0x030bf9 thru 0x03d7d1 (0xcbd9 bytes) 000538 1738966891.553650 fallocate 0x028f1a thru 0x0304d0 (0x75b7 bytes) 000539 1738966891.554887 trunc from 0x03d7d2 to 0x03d6c3 (0x10e bytes) 000540 1738966891.559275 punch from 0x019231 to 0x022845 (0x9615 bytes) 000542 1738966891.560666 mapread 0x01353e thru 0x0173bc (0x3e7f bytes) 000543 1738966891.565277 trunc from 0x03d6c4 to 0x00d3f4 (0x302cf bytes) 000544 1738966891.568431 trunc from 0x00d3f5 to 0x039e46 (0x2ca52 bytes) 000545 1738966891.571285 read 0x027ea9 thru 0x036de9 (0xef41 bytes) 000546 1738966891.574050 mapwrite 0x00ec50 thru 0x013624 (0x49d5 bytes) 000548 1738966891.579039 mapwrite 0x00e261 thru 0x0144af (0x624f bytes) 000549 1738966891.583055 read 0x0208db thru 0x021320 (0xa46 bytes) 000550 1738966891.583323 mapread 0x009074 thru 0x012513 (0x94a0 bytes) 000552 1738966891.584007 read 0x02dc29 thru 0x034f78 (0x7350 bytes) 000553 1738966891.584263 mapread 0x034f15 thru 0x0364e7 (0x15d3 bytes) 000554 1738966891.584621 read 0x003e9f thru 0x012d80 (0xeee2 bytes) 000555 1738966891.585993 MIRROR_RESYNC 2 mirrors 000556 1738966891.674688 punch from 0x010a29 to 0x019169 (0x8741 bytes) 000557 1738966891.682648 mapread 0x01d937 thru 0x021c0a (0x42d4 bytes) 000558 1738966891.688208 MIRROR_EXTEND to 3 mirrors 000559 1738966891.778618 punch from 0x01b469 to 0x02814d (0xcce5 bytes) 000560 1738966891.785958 mapread 0x029686 thru 0x033626 (0x9fa1 bytes) 000563 1738966891.792077 mapwrite 0x007f00 thru 0x0094d4 (0x15d5 bytes) 000564 1738966891.798258 punch from 0x0055a9 to 0x00cbc4 (0x761c bytes) 000565 1738966891.799699 trunc from 0x039e47 to 0x03e593 (0x474d bytes) 000567 1738966891.808001 trunc from 0x03e594 to 0x00adc6 (0x337cd bytes) 000569 1738966891.812008 read 0x006406 thru 0x00adc6 (0x49c1 bytes) 000570 1738966891.814092 punch from 0x007e89 to 0x00adc6 (0x2f3e bytes) 000571 1738966891.815290 punch from 0x003c9c to 0x00adc6 (0x712b bytes) 000572 1738966891.816436 punch from 0x001cf5 to 0x007f84 (0x6290 bytes) 000573 1738966891.817639 punch from 0x0081e2 to 0x0092d2 (0x10f1 bytes) 000574 1738966891.818728 mapread 0x001ae8 thru 0x00adc6 (0x92df bytes) 000575 1738966891.822325 mapwrite 0x01aba5 thru 0x01c1c6 (0x1622 bytes) 000576 1738966891.831651 mapwrite 0x005d74 thru 0x008d5c (0x2fe9 bytes) 000578 1738966891.836133 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000579 1738966891.876243 mapread 0x018278 thru 0x01c1c6 (0x3f4f bytes) 000580 1738966891.881623 MIRROR_EXTEND to 3 mirrors 000581 1738966891.963488 mapread 0x00bf90 thru 0x00e5c7 (0x2638 bytes) 000582 1738966891.967271 read 0x0125d7 thru 0x0151ca (0x2bf4 bytes) 000583 1738966891.968563 mapread 0x01a1e8 thru 0x01c1c6 (0x1fdf bytes) 000585 1738966891.970509 fallocate 0x01dfbf thru 0x02c9f0 (0xea32 bytes) 000586 1738966891.977565 read 0x02aa0d thru 0x02b680 (0xc74 bytes) 000587 1738966891.980118 read 0x012073 thru 0x020d26 (0xecb4 bytes) 000588 1738966891.980442 punch from 0x01513e to 0x019eda (0x4d9d bytes) 000589 1738966891.981687 mapread 0x02c05c thru 0x02c9f0 (0x995 bytes) 000590 1738966891.982109 fallocate 0x0286de thru 0x035fab (0xd8ce bytes) 000591 1738966891.983341 punch from 0x0024f6 to 0x002e26 (0x931 bytes) 000592 1738966891.984497 punch from 0x004378 to 0x00ad2c (0x69b5 bytes) 000593 1738966891.985796 write 0x03af93 thru 0x03ffff (0x506d bytes) 000594 1738966891.987363 fallocate 0x020d82 thru 0x025885 (0x4b04 bytes) 000595 1738966891.988648 trunc from 0x040000 to 0x03ebca (0x1435 bytes) 000597 1738966891.996854 fallocate 0x002d86 thru 0x009076 (0x62f1 bytes) 000600 1738966891.998696 read 0x02cd7d thru 0x035f76 (0x91fa bytes) 000601 1738966892.001032 mapread 0x02c708 thru 0x036f50 (0xa849 bytes) 000602 1738966892.001613 read 0x007e9f thru 0x014e1f (0xcf81 bytes) 000603 1738966892.001915 punch from 0x0330e6 to 0x03a46f (0x738a bytes) 000604 1738966892.004911 read 0x001416 thru 0x00c60f (0xb1fa bytes) 000605 1738966892.005190 trunc from 0x03ebcb to 0x03ed96 (0x1cc bytes) 000606 1738966892.008467 mapread 0x02a2d4 thru 0x032fdb (0x8d08 bytes) 000607 1738966892.009913 MIRROR_RESYNC 3 mirrors 000608 1738966892.118092 mapread 0x01faea thru 0x025d12 (0x6229 bytes) 000609 1738966892.122227 mapwrite 0x03cdb1 thru 0x03ffff (0x324f bytes) 000610 1738966892.141334 read 0x03158e thru 0x039c2c (0x869f bytes) 000611 1738966892.143203 fallocate 0x01b8cd thru 0x02542d (0x9b61 bytes) 000612 1738966892.144619 read 0x00151f thru 0x00a351 (0x8e33 bytes) 000613 1738966892.147840 read 0x00a4fa thru 0x00cad7 (0x25de bytes) 000614 1738966892.148215 mapwrite 0x024adf thru 0x029755 (0x4c77 bytes) 000616 1738966892.153187 write 0x03c962 thru 0x03ffff (0x369e bytes) 000618 1738966892.154292 fallocate 0x025f63 thru 0x02d1a2 (0x7240 bytes) 000621 1738966892.157665 MIRROR_RESYNC 3 mirrors 000622 1738966892.266359 fallocate 0x01d469 thru 0x0274d4 (0xa06c bytes) 000623 1738966892.274677 MIRROR_RESYNC 3 mirrors 000624 1738966892.382018 MIRROR_RESYNC 3 mirrors 000625 1738966892.417652 mapread 0x02ffce thru 0x036b14 (0x6b47 bytes) 000626 1738966892.421618 mapwrite 0x02f2b4 thru 0x03d888 (0xe5d5 bytes) 000627 1738966892.448905 fallocate 0x021187 thru 0x027acd (0x6947 bytes) 000628 1738966892.450500 write 0x0365b5 thru 0x03728f (0xcdb bytes) 000629 1738966892.451033 write 0x013dc5 thru 0x018b2c (0x4d68 bytes) 000630 1738966892.453281 read 0x03def8 thru 0x03ffff (0x2108 bytes) 000632 1738966892.453898 write 0x000ad2 thru 0x005abd (0x4fec bytes) 000633 1738966892.457217 MIRROR_RESYNC 3 mirrors 000634 1738966892.572165 fallocate 0x02ed98 thru 0x035bee (0x6e57 bytes) 000635 1738966892.579568 write 0x02aa2a thru 0x02fa52 (0x5029 bytes) 000636 1738966892.583151 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000638 1738966892.626221 mapwrite 0x012d4a thru 0x01dbcb (0xae82 bytes) 000639 1738966892.640631 MIRROR_RESYNC 2 mirrors 000641 1738966892.715637 read 0x03ee7a thru 0x03ffff (0x1186 bytes) 000642 1738966892.717920 MIRROR_EXTEND to 3 mirrors 000643 1738966892.807600 read 0x0355f2 thru 0x039d50 (0x475f bytes) 000644 1738966892.809467 write 0x0267c7 thru 0x030591 (0x9dcb bytes) 000646 1738966892.819208 mapwrite 0x01eea1 thru 0x029910 (0xaa70 bytes) 000647 1738966892.827022 fallocate 0x02e642 thru 0x039d0c (0xb6cb bytes) 000649 1738966892.828717 fallocate 0x0174b6 thru 0x026d60 (0xf8ab bytes) 000650 1738966892.831170 MIRROR_EXTEND to 4 mirrors 000651 1738966892.926045 mapread 0x001d92 thru 0x0077a3 (0x5a12 bytes) 000652 1738966892.933023 write 0x0344ec thru 0x03501d (0xb32 bytes) 000653 1738966892.942713 read 0x02e915 thru 0x03de03 (0xf4ef bytes) 000654 1738966892.945483 MIRROR_RESYNC 4 mirrors 000655 1738966893.072340 punch from 0x031185 to 0x03ffff (0xee7b bytes) 000657 1738966893.080177 write 0x0293e7 thru 0x03762a (0xe244 bytes) 000658 1738966893.083862 MIRROR_EXTEND to 5 mirrors 000659 1738966893.187032 mapwrite 0x00b4cb thru 0x010a0e (0x5544 bytes) 000660 1738966893.219120 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000662 1738966893.262613 trunc from 0x040000 to 0x036cd5 (0x932a bytes) 000664 1738966893.266967 write 0x00bfa2 thru 0x00f40e (0x346d bytes) 000669 1738966893.270120 write 0x03ed14 thru 0x03ffff (0x12ec bytes) 000670 1738966893.270600 trunc from 0x040000 to 0x02b9cf (0x14630 bytes) 000672 1738966893.274126 punch from 0x00b36b to 0x00dfa4 (0x2c3a bytes) 000673 1738966893.278220 MIRROR_RESYNC 4 mirrors 000674 1738966893.451754 trunc from 0x02b9d0 to 0x02ccc0 (0x12f1 bytes) 000675 1738966893.460017 fallocate 0x005ba9 thru 0x00a8f5 (0x4d4d bytes) 000676 1738966893.461477 mapwrite 0x0325b1 thru 0x03f5aa (0xcffa bytes) 000679 1738966893.477690 fallocate 0x015631 thru 0x01f2f9 (0x9cc9 bytes) 000680 1738966893.479202 mapread 0x0279bb thru 0x02af1d (0x3563 bytes) 000681 1738966893.481201 mapread 0x01e58a thru 0x0226b0 (0x4127 bytes) 000682 1738966893.483063 read 0x01596e thru 0x01ae66 (0x54f9 bytes) 000683 1738966893.484994 trunc from 0x03f5ab to 0x007b9b (0x37a0f bytes) 000684 1738966893.488400 trunc from 0x007b9c to 0x01f7a4 (0x17c09 bytes) 000685 1738966893.491829 mapwrite 0x011d9c thru 0x016443 (0x46a8 bytes) 000686 1738966893.498952 mapwrite 0x01744a thru 0x023c7a (0xc831 bytes) 000687 1738966893.511751 read 0x005542 thru 0x00e223 (0x8ce2 bytes) 000689 1738966893.514591 write 0x02c078 thru 0x02fad2 (0x3a5b bytes) 000690 1738966893.515120 write 0x028e66 thru 0x037589 (0xe724 bytes) 000691 1738966893.517774 mapread 0x004df8 thru 0x014dde (0xffe7 bytes) 000692 1738966893.518318 fallocate 0x03ca38 thru 0x03ffff (0x35c8 bytes) 000694 1738966893.520237 write 0x0189ba thru 0x0253cb (0xca12 bytes) 000696 1738966893.522851 MIRROR_EXTEND to 5 mirrors 000697 1738966893.632574 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000698 1738966893.703063 write 0x03fb06 thru 0x03ffff (0x4fa bytes) 000700 1738966893.712020 trunc from 0x040000 to 0x005bec (0x3a413 bytes) 000701 1738966893.719892 mapwrite 0x031d98 thru 0x035dc6 (0x402f bytes) 000702 1738966893.731026 trunc from 0x035dc7 to 0x03104f (0x4d77 bytes) 000703 1738966893.734737 fallocate 0x00ea7b thru 0x01a6c9 (0xbc4f bytes) 000704 1738966893.736234 read 0x020293 thru 0x02a8dd (0xa64b bytes) 000706 1738966893.739699 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000707 1738966893.779184 trunc from 0x031050 to 0x033dfa (0x2dab bytes) 000708 1738966893.783169 mapread 0x024377 thru 0x02c1b3 (0x7e3d bytes) 000709 1738966893.788121 MIRROR_RESYNC 3 mirrors 000710 1738966893.889505 trunc from 0x033dfb to 0x03788f (0x3a95 bytes) 000712 1738966893.901087 mapread 0x00de89 thru 0x019856 (0xb9ce bytes) 000713 1738966893.907332 write 0x01932e thru 0x01b278 (0x1f4b bytes) 000714 1738966893.907878 mapwrite 0x03771f thru 0x03eca1 (0x7583 bytes) 000715 1738966893.919604 write 0x008bd7 thru 0x014d27 (0xc151 bytes) 000716 1738966893.920296 mapread 0x019cc9 thru 0x01c53b (0x2873 bytes) 000717 1738966893.920677 punch from 0x001c10 to 0x0047b6 (0x2ba7 bytes) 000718 1738966893.922108 mapwrite 0x00b0aa thru 0x00f704 (0x465b bytes) 000719 1738966893.926333 read 0x03897d thru 0x03eca1 (0x6325 bytes) 000720 1738966893.926851 read 0x02467b thru 0x026f3b (0x28c1 bytes) 000723 1738966893.929730 trunc from 0x03eca2 to 0x00a8d3 (0x343ce bytes) 000724 1738966893.933305 write 0x013f51 thru 0x020497 (0xc547 bytes) 000725 1738966893.934067 fallocate 0x01d9b7 thru 0x0242ea (0x6934 bytes) 000726 1738966893.936707 MIRROR_RESYNC 3 mirrors 000727 1738966894.061295 trunc from 0x0242eb to 0x01e04b (0x629f bytes) 000728 1738966894.069542 write 0x032c34 thru 0x03ffff (0xd3cc bytes) 000729 1738966894.070371 read 0x03c5af thru 0x03ffff (0x3a51 bytes) 000730 1738966894.070660 mapread 0x00a0cc thru 0x019b37 (0xfa6c bytes) 000731 1738966894.076478 trunc from 0x040000 to 0x015e63 (0x2a19c bytes) 000733 1738966894.080608 read 0x013c75 thru 0x015e63 (0x21ef bytes) 000734 1738966894.080891 mapread 0x0133fb thru 0x015e63 (0x2a69 bytes) 000735 1738966894.081286 fallocate 0x01e29b thru 0x02d139 (0xee9f bytes) 000736 1738966894.082485 fallocate 0x031340 thru 0x03f698 (0xe359 bytes) 000737 1738966894.083962 trunc from 0x03f699 to 0x0021f8 (0x3d4a0 bytes) 000738 1738966894.087351 write 0x012226 thru 0x01db5a (0xb935 bytes) 000739 1738966894.088078 punch from 0x0162f5 to 0x01db5a (0x7866 bytes) 000740 1738966894.092293 MIRROR_RESYNC 3 mirrors 000741 1738966894.197102 mapread 0x013c3b thru 0x01db5a (0x9f20 bytes) 000742 1738966894.201478 fallocate 0x01c884 thru 0x027e2e (0xb5ab bytes) 000743 1738966894.210154 MIRROR_EXTEND to 4 mirrors 000745 1738966894.312004 fallocate 0x012385 thru 0x016a83 (0x46ff bytes) 000746 1738966894.322164 MIRROR_RESYNC 4 mirrors 000747 1738966894.469040 read 0x010a88 thru 0x015ca4 (0x521d bytes) 000750 1738966894.471347 read 0x01c4fa thru 0x01fd0a (0x3811 bytes) 000752 1738966894.473840 mapread 0x0125e4 thru 0x01851c (0x5f39 bytes) 000753 1738966894.474148 write 0x03d87e thru 0x03ffff (0x2782 bytes) 000757 1738966894.481731 punch from 0x00af83 to 0x016826 (0xb8a4 bytes) 000758 1738966894.483154 punch from 0x024c66 to 0x0252ff (0x69a bytes) 000759 1738966894.484348 read 0x027a2f thru 0x036401 (0xe9d3 bytes) 000761 1738966894.488381 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000762 1738966894.531171 punch from 0x02d64a to 0x03b040 (0xd9f7 bytes) 000763 1738966894.532513 mapread 0x01d0dc thru 0x02bc56 (0xeb7b bytes) 000764 1738966894.539092 mapwrite 0x00a551 thru 0x018504 (0xdfb4 bytes) 000765 1738966894.548329 fallocate 0x01dc43 thru 0x0272cc (0x968a bytes) 000766 1738966894.549881 write 0x02c44d thru 0x02cf29 (0xadd bytes) 000767 1738966894.551725 trunc from 0x040000 to 0x0184e0 (0x27b1f bytes) 000768 1738966894.555428 trunc from 0x0184e1 to 0x037912 (0x1f432 bytes) 000769 1738966894.560030 MIRROR_RESYNC 3 mirrors 000770 1738966894.665536 trunc from 0x037913 to 0x030661 (0x72b1 bytes) 000771 1738966894.678743 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000773 1738966894.719555 mapread 0x00d03f thru 0x015e34 (0x8df6 bytes) 000774 1738966894.725000 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000775 1738966894.749479 trunc from 0x030662 to 0x011849 (0x1ee18 bytes) 000777 1738966894.754040 mapwrite 0x01f235 thru 0x022f4e (0x3d1a bytes) 000779 1738966894.765148 punch from 0x012299 to 0x01f2e3 (0xd04b bytes) 000780 1738966894.766584 mapwrite 0x001e0b thru 0x00e56b (0xc761 bytes) 000781 1738966894.774566 MIRROR_EXTEND to 2 mirrors 000783 1738966894.855545 mapread 0x0050f6 thru 0x0139a1 (0xe8ac bytes) 000784 1738966894.864952 MIRROR_RESYNC 2 mirrors 000785 1738966894.898604 punch from 0x01a5ff to 0x022f4e (0x8950 bytes) 000786 1738966894.906074 mapread 0x0146fa thru 0x01eafa (0xa401 bytes) 000788 1738966894.911963 fallocate 0x02949e thru 0x037983 (0xe4e6 bytes) 000789 1738966894.914533 MIRROR_RESYNC 2 mirrors 000790 1738966894.996497 write 0x00449a thru 0x00ef9e (0xab05 bytes) 000791 1738966895.004421 read 0x011c35 thru 0x01b8fa (0x9cc6 bytes) 000793 1738966895.007713 MIRROR_EXTEND to 3 mirrors 000794 1738966895.093861 punch from 0x00c503 to 0x01851b (0xc019 bytes) 000795 1738966895.100490 fallocate 0x002209 thru 0x00d92e (0xb726 bytes) 000796 1738966895.101796 fallocate 0x01f1e8 thru 0x02371f (0x4538 bytes) 000797 1738966895.103260 trunc from 0x037984 to 0x024e10 (0x12b73 bytes) 000798 1738966895.110523 punch from 0x00ef64 to 0x018fed (0xa08a bytes) 000799 1738966895.111685 punch from 0x00984b to 0x009e6b (0x621 bytes) 000800 1738966895.112854 mapwrite 0x0220b8 thru 0x029d01 (0x7c4a bytes) 000801 1738966895.124008 write 0x00f069 thru 0x00fa77 (0xa0f bytes) 000802 1738966895.126036 read 0x015545 thru 0x01926f (0x3d2b bytes) 000803 1738966895.127849 write 0x00623f thru 0x012113 (0xbed5 bytes) 000804 1738966895.129827 trunc from 0x029d02 to 0x002152 (0x27baf bytes) 000806 1738966895.134525 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000807 1738966895.173608 mapwrite 0x02bca2 thru 0x02cfb1 (0x1310 bytes) 000809 1738966895.184903 MIRROR_RESYNC 2 mirrors 000810 1738966895.250383 punch from 0x00e6dd to 0x01a62c (0xbf50 bytes) 000811 1738966895.257592 mapread 0x020f37 thru 0x022a53 (0x1b1d bytes) 000812 1738966895.263581 MIRROR_RESYNC 2 mirrors 000813 1738966895.330694 trunc from 0x02cfb2 to 0x02c541 (0xa70 bytes) 000814 1738966895.341526 trunc from 0x02c542 to 0x00fb6d (0x1c9d4 bytes) 000815 1738966895.345544 fallocate 0x010f55 thru 0x01750e (0x65ba bytes) 000817 1738966895.347006 trunc from 0x00fb6e to 0x012cda (0x316d bytes) 000818 1738966895.350116 mapread 0x0019a0 thru 0x006e0e (0x546f bytes) 000819 1738966895.354255 write 0x00f55c thru 0x01e48e (0xef33 bytes) 000820 1738966895.355043 mapread 0x005a59 thru 0x008e1b (0x33c3 bytes) 000821 1738966895.355401 mapwrite 0x00604d thru 0x00a621 (0x45d5 bytes) 000824 1738966895.361318 trunc from 0x01e48f to 0x0220f7 (0x3c69 bytes) 000826 1738966895.365108 write 0x0095e7 thru 0x00a926 (0x1340 bytes) 000829 1738966895.365999 fallocate 0x00f196 thru 0x019dba (0xac25 bytes) 000830 1738966895.367237 mapwrite 0x036709 thru 0x037e22 (0x171a bytes) 000833 1738966895.377459 trunc from 0x037e23 to 0x00d048 (0x2adda bytes) 000834 1738966895.381060 write 0x02f5ea thru 0x039aee (0xa505 bytes) 000835 1738966895.381823 punch from 0x02e67d to 0x0366c3 (0x8047 bytes) 000837 1738966895.385216 trunc from 0x039aef to 0x02f68f (0xa45f bytes) 000838 1738966895.388302 fallocate 0x00b57e thru 0x0115ca (0x604d bytes) 000839 1738966895.389597 mapread 0x02b68f thru 0x02f68f (0x4001 bytes) 000840 1738966895.391655 write 0x01cd61 thru 0x01f2dc (0x257c bytes) 000841 1738966895.393845 trunc from 0x02f690 to 0x020811 (0xee7e bytes) 000843 1738966895.397078 punch from 0x01a596 to 0x01de8d (0x38f8 bytes) 000844 1738966895.399524 punch from 0x011e92 to 0x0158c6 (0x3a35 bytes) 000846 1738966895.401134 write 0x0038a6 thru 0x012ef1 (0xf64c bytes) 000847 1738966895.403080 trunc from 0x020812 to 0x02c4af (0xbc9e bytes) 000848 1738966895.407385 MIRROR_RESYNC 2 mirrors 000849 1738966895.487810 punch from 0x0255db to 0x02accf (0x56f5 bytes) 000850 1738966895.494357 fallocate 0x0101be thru 0x0184e5 (0x8328 bytes) 000853 1738966895.496052 write 0x00604c thru 0x00b517 (0x54cc bytes) 000854 1738966895.498357 mapread 0x01608e thru 0x024350 (0xe2c3 bytes) 000855 1738966895.502171 mapread 0x023ddb thru 0x02c4af (0x86d5 bytes) 000862 1738966895.503762 read 0x026d50 thru 0x02c4af (0x5760 bytes) 000864 1738966895.504224 mapwrite 0x000671 thru 0x004336 (0x3cc6 bytes) 000865 1738966895.512869 write 0x03fc2b thru 0x03ffff (0x3d5 bytes) 000866 1738966895.513385 write 0x00b3bb thru 0x012e99 (0x7adf bytes) 000868 1738966895.515047 write 0x029c58 thru 0x030103 (0x64ac bytes) 000869 1738966895.517006 punch from 0x00c41b to 0x01ac3e (0xe824 bytes) 000871 1738966895.521242 trunc from 0x040000 to 0x00f0cb (0x30f34 bytes) 000872 1738966895.529339 fallocate 0x008a7d thru 0x01484d (0xbdd1 bytes) 000873 1738966895.532164 MIRROR_EXTEND to 3 mirrors 000874 1738966895.617558 write 0x00e8ab thru 0x01e098 (0xf7ee bytes) 000876 1738966895.626729 read 0x00339b thru 0x00a9e7 (0x764d bytes) 000877 1738966895.628966 fallocate 0x03d3ac thru 0x03ffff (0x2c54 bytes) 000878 1738966895.630619 mapread 0x01829a thru 0x01e098 (0x5dff bytes) 000879 1738966895.631329 write 0x021d12 thru 0x02204c (0x33b bytes) 000880 1738966895.631871 read 0x013d90 thru 0x01f685 (0xb8f6 bytes) 000881 1738966895.634100 fallocate 0x028d19 thru 0x030364 (0x764c bytes) 000882 1738966895.635749 mapwrite 0x018c22 thru 0x021f26 (0x9305 bytes) 000884 1738966895.642454 punch from 0x0005e8 to 0x002cbd (0x26d6 bytes) 000885 1738966895.643920 mapread 0x010df1 thru 0x01a5e0 (0x97f0 bytes) 000886 1738966895.644589 mapread 0x0193ad thru 0x02204c (0x8ca0 bytes) 000887 1738966895.646560 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000889 1738966895.688504 read 0x017d14 thru 0x02204c (0xa339 bytes) 000890 1738966895.691893 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000891 1738966895.718137 fallocate 0x003d11 thru 0x00990b (0x5bfb bytes) 000894 1738966895.720617 trunc from 0x02204d to 0x02adf9 (0x8dad bytes) 000895 1738966895.727733 trunc from 0x02adfa to 0x03be7e (0x11085 bytes) 000896 1738966895.730953 fallocate 0x010e07 thru 0x01c484 (0xb67e bytes) 000897 1738966895.732395 write 0x0238d7 thru 0x02f2ea (0xba14 bytes) 000898 1738966895.735508 fallocate 0x03ad75 thru 0x03ffff (0x528b bytes) 000899 1738966895.737010 write 0x03ac92 thru 0x03ffff (0x536e bytes) 000900 1738966895.739499 mapread 0x0133c1 thru 0x0165c9 (0x3209 bytes) 000901 1738966895.741846 fallocate 0x02a182 thru 0x031e81 (0x7d00 bytes) 000904 1738966895.744086 trunc from 0x040000 to 0x02ea5e (0x115a1 bytes) 000906 1738966895.747737 read 0x01f0e3 thru 0x020997 (0x18b5 bytes) 000908 1738966895.750103 read 0x019846 thru 0x01a264 (0xa1f bytes) 000909 1738966895.750855 trunc from 0x02ea5f to 0x02e394 (0x6ca bytes) 000911 1738966895.754194 trunc from 0x02e395 to 0x0087cb (0x25bc9 bytes) 000914 1738966895.757648 mapwrite 0x01e5c2 thru 0x028258 (0x9c97 bytes) 000916 1738966895.769301 fallocate 0x03deef thru 0x03ffff (0x2111 bytes) 000918 1738966895.771257 write 0x00885f thru 0x013d56 (0xb4f8 bytes) 000919 1738966895.772252 write 0x035ed0 thru 0x03ffff (0xa130 bytes) 000921 1738966895.775466 read 0x01a0e5 thru 0x020f0d (0x6e29 bytes) 000922 1738966895.777037 MIRROR_EXTEND to 2 mirrors 000924 1738966895.881495 mapwrite 0x008804 thru 0x00b2c0 (0x2abd bytes) 000925 1738966895.910389 mapwrite 0x004a07 thru 0x008f3f (0x4539 bytes) 000926 1738966895.915621 MIRROR_RESYNC 2 mirrors 000927 1738966896.012998 mapread 0x02b822 thru 0x02ff0e (0x46ed bytes) 000928 1738966896.016978 trunc from 0x040000 to 0x020116 (0x1fee9 bytes) 000929 1738966896.029076 punch from 0x006e3e to 0x00da48 (0x6c0b bytes) 000930 1738966896.030319 mapwrite 0x023a91 thru 0x02b7ca (0x7d3a bytes) 000932 1738966896.041487 punch from 0x0187bb to 0x01e58e (0x5dd4 bytes) 000933 1738966896.042891 fallocate 0x007960 thru 0x017709 (0xfdaa bytes) 000934 1738966896.044234 write 0x002138 thru 0x00b1f7 (0x90c0 bytes) 000936 1738966896.046588 write 0x023456 thru 0x030c7d (0xd828 bytes) 000937 1738966896.047250 write 0x027add thru 0x028d4f (0x1273 bytes) 000938 1738966896.047642 read 0x006e17 thru 0x0154bf (0xe6a9 bytes) 000939 1738966896.049584 write 0x0179e7 thru 0x0226fd (0xad17 bytes) 000940 1738966896.050488 read 0x0045ac thru 0x010290 (0xbce5 bytes) 000942 1738966896.051287 write 0x01ccb3 thru 0x0246f5 (0x7a43 bytes) 000943 1738966896.053414 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000944 1738966896.082678 trunc from 0x030c7e to 0x0048ab (0x2c3d2 bytes) 000945 1738966896.087011 read 0x002107 thru 0x0048ab (0x27a5 bytes) 000946 1738966896.088467 mapwrite 0x01c0d3 thru 0x022e47 (0x6d75 bytes) 000947 1738966896.100131 punch from 0x00175c to 0x006554 (0x4df9 bytes) 000948 1738966896.101455 punch from 0x01547b to 0x01e81b (0x93a1 bytes) 000949 1738966896.102773 mapread 0x02107b thru 0x022409 (0x138f bytes) 000950 1738966896.103142 read 0x008865 thru 0x014a0e (0xc1aa bytes) 000951 1738966896.105651 read 0x004ff8 thru 0x00d51c (0x8525 bytes) 000954 1738966896.106647 trunc from 0x022e48 to 0x023754 (0x90d bytes) 000955 1738966896.110151 read 0x0099b1 thru 0x018af0 (0xf140 bytes) 000956 1738966896.110484 trunc from 0x023755 to 0x00341b (0x20339 bytes) 000957 1738966896.113897 write 0x01d48e thru 0x020ae6 (0x3659 bytes) 000960 1738966896.116253 MIRROR_EXTEND to 2 mirrors 000961 1738966896.199999 trunc from 0x020ae7 to 0x018f4b (0x7b9b bytes) 000962 1738966896.212354 mapread 0x018709 thru 0x018f4b (0x843 bytes) 000963 1738966896.214165 fallocate 0x01d7b5 thru 0x0275cd (0x9e19 bytes) 000965 1738966896.216118 punch from 0x01f567 to 0x0275cd (0x8067 bytes) 000966 1738966896.217354 punch from 0x008dba to 0x0182bd (0xf504 bytes) 000967 1738966896.218693 punch from 0x011cb4 to 0x01c68d (0xa9da bytes) 000968 1738966896.219891 write 0x009ba0 thru 0x016877 (0xccd8 bytes) 000969 1738966896.222470 punch from 0x003ada to 0x0091f7 (0x571e bytes) 000970 1738966896.226798 MIRROR_EXTEND to 3 mirrors 000971 1738966896.317333 MIRROR_EXTEND to 4 mirrors 000972 1738966896.410614 MIRROR_RESYNC 4 mirrors 000973 1738966896.514888 trunc from 0x0275ce to 0x0249dd (0x2bf0 bytes) 000975 1738966896.527577 trunc from 0x0249de to 0x000e91 (0x23b4c bytes) 000976 1738966896.531936 mapread 0x000c58 thru 0x000e91 (0x23a bytes) 000979 1738966896.533948 write 0x02ae08 thru 0x03280f (0x7a08 bytes) 000980 1738966896.534630 punch from 0x000345 to 0x0029a6 (0x2662 bytes) 000981 1738966896.536045 fallocate 0x0048a5 thru 0x00d0cc (0x8828 bytes) 000982 1738966896.539045 MIRROR_RESYNC 4 mirrors 000984 1738966896.652068 fallocate 0x0152fd thru 0x0230d3 (0xddd7 bytes) 000985 1738966896.659590 read 0x02cec0 thru 0x03280f (0x5950 bytes) 000986 1738966896.661670 write 0x016fad thru 0x024fe8 (0xe03c bytes) 000988 1738966896.664358 fallocate 0x02fe3c thru 0x03ae2a (0xafef bytes) 000989 1738966896.665677 read 0x02989d thru 0x03494e (0xb0b2 bytes) 000991 1738966896.668333 trunc from 0x03ae2b to 0x004326 (0x36b04 bytes) 000992 1738966896.677209 MIRROR_RESYNC 4 mirrors 000993 1738966896.774580 punch from 0x00220e to 0x004326 (0x2119 bytes) 000994 1738966896.781504 fallocate 0x0208a0 thru 0x02ce6c (0xc5cd bytes) 000995 1738966896.782957 mapread 0x0216bd thru 0x0255e4 (0x3f28 bytes) 000996 1738966896.786841 mapwrite 0x03a453 thru 0x03ad64 (0x912 bytes) 000997 1738966896.799389 punch from 0x00dae0 to 0x0159e3 (0x7f04 bytes) 000998 1738966896.800590 write 0x02f94c thru 0x030742 (0xdf7 bytes) 000999 1738966896.802440 trunc from 0x03ad65 to 0x021dd7 (0x18f8d bytes) All operations completed A-OK! PASS 70a (11s) == sanity-flr test 71: check mirror extend parent fid ==== 17:21:37 (1738966897) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x7e:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (2s) == sanity-flr test 200a: stress test ===================== 17:21:40 (1738966900) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre2 Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1935200 .. Extending file size to 2284000 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6742944 .. Extending file size to 2513504 .. Extending file size to 5183168 .. Extending file size to 7284896 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8593760 .. Extending file size to 8253760 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8956096 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8830016 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9052000 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9288512 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9000992 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9065088 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 5205 5206 5207 5209 5212 done 192.168.203.155@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg355-client.virtnet /mnt/lustre2 (opts:) 192.168.203.155@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg355-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:22:43 (1738966963) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre2 Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3545184 .. Extending file size to 5525216 .. failed Extending file size to 4430240 .. Extending file size to 5850784 .. Extending file size to 5130176 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7470848 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7039008 .. Extending file size to 7980736 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8029728 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8269248 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8692352 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8903008 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8718400 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8930656 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9235552 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 9393 9394 9397 9399 9401 192.168.203.155@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg355-client.virtnet /mnt/lustre2 (opts:) 192.168.203.155@tcp:/lustre /mnt/lustre3 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg355-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (63s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:23:46 (1738967026) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4228 1283460 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1285656 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14856 3592164 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22816 3584204 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37672 7176368 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4228 1283460 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2032 1285656 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14856 3592164 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 22816 3584204 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37672 7176368 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0340241 s, 61.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0404221 s, 51.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.155@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg355-client.virtnet /mnt/lustre2 (opts:) PASS 200c (3s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:23:50 (1738967030) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0253861 s, 82.6 MB/s PASS 202 (1s) == sanity-flr test 203: mirror file preserve mirror ID === 17:23:51 (1738967031) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb90:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10cf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10cf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d0:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:23:53 (1738967033) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0251852 s, 83.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (1s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:23:54 (1738967034) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0247357 s, 84.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d4:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:23:56 (1738967036) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.020256 s, 104 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d6:0x0] } PASS 204c (22s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:24:19 (1738967059) Waiting for MDT destroys to complete oleg355-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg355-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg355-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112392 s, 93.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d6:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:24:54 (1738967094) Waiting for MDT destroys to complete oleg355-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg355-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg355-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0251467 s, 83.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:25:29 (1738967129) Waiting for MDT destroys to complete oleg355-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg355-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg355-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg355-server: Pool lustre.test_204f created Adding targets to pool oleg355-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg355-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0236716 s, 88.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg355-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg355-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg355-server: Pool lustre.test_204f destroyed PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:26:16 (1738967176) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:26:18 (1738967178) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:26:20 (1738967180) oleg355-server: Pool lustre.test_206 created Adding targets to pool oleg355-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg355-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg355-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg355-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg355-server: Pool lustre.test_206 destroyed PASS 206 (11s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:26:31 (1738967191) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0185007 s, 56.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00960842 s, 109 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x3995362f Mirror 3: 0x3995362f PASS 207 (2s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:26:33 (1738967193) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:26:34 (1738967194) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:26:35 (1738967195) Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0089505 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1810000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1eb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1eb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1eb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2460000 Read this (4 bytes): BBBB 192.168.203.155@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg355-client.virtnet /mnt/lustre2 (opts:) PASS 209a (13s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:26:48 (1738967208) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.00263516 s, 3.1 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:26:50 (1738967210) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00902976 s, 116 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbb5 lmm_fid: [0x200000405:0xbb5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbb5 lmm_fid: [0x200000405:0xbb5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e5:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339614133 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-02-07 17:26:50.000000000 -0500 Modify: 2025-02-07 17:26:50.000000000 -0500 Change: 2025-02-07 17:26:50.000000000 -0500 Birth: - PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:26:52 (1738967212) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00742848 s, 141 MB/s oleg355-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg355-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg355-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg355-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 oleg355-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg355-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg355-server: O/280000401/d3/10e3: File not found by ext2_lookup PASS 210b (14s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:27:06 (1738967226) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0331785 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00791746 s, 517 kB/s size after second write -rw-r--r-- 1 root root 4096 Feb 7 17:27 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbba:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 7 17:27 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 689 sec ============ 17:27:09 (1738967229) === sanity-flr: start cleanup 17:27:09 (1738967229) === === sanity-flr: finish cleanup 17:27:12 (1738967232) ===