-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 07:09:41 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg432-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:10:06 (1788520206) === oleg432-client.virtnet: executing check_config_client /mnt/lustre oleg432-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg432-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff961250acd800.idle_timeout=debug osc.lustre-OST0001-osc-ffff961250acd800.idle_timeout=debug disable quota as required oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:10:49 (1788520249) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7626] == sanity-flr test 0a: lfs mirror create with -N option == 07:10:52 (1788520252) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:11:13 (1788520273) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:11:17 (1788520277) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:11:21 (1788520281) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:11:51 (1788520311) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:11:56 (1788520316) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 07:11:59 (1788520319) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff961250acd800.stats=clear osc.lustre-OST0001-osc-ffff961250acd800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:12:09 (1788520329) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:12:23 (1788520343) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:12:33 (1788520353) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:12:37 (1788520357) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:12:47 (1788520367) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 07:12:57 (1788520377) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:13:06 (1788520386) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:13:11 (1788520391) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:13:20 (1788520400) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:13:32 (1788520412) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:13:39 (1788520419) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 2.30176 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213287 s, 4.9 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:13:52 (1788520432) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.303848 s, 34.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:14:02 (1788520442) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:14:15 (1788520455) Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff961250acd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff961250acd800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff961250acd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff961250acd800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (88s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:15:43 (1788520543) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.363386 s, 17.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff961250acd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff961250acd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:16:33 (1788520593) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff961250acd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff961250acd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff961250acd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff961250acd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:17:55 (1788520675) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.83736 s, 9.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 07:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.089s user 0m0.000s sys 0m0.209s Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff961250acd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff961250acd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.286s user 0m0.001s sys 0m0.209s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff961250acd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff961250acd800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.643s user 0m0.003s sys 0m0.211s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff961250acd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff961250acd800.ost_server_uuid in FULL state after 0 sec PASS 33b (116s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:19:51 (1788520791) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:19:55 (1788520795) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:19:59 (1788520799) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 07:20:02 (1788520802) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 07:20:11 (1788520811) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.680096 s, 18.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:20:25 (1788520825) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404344 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745032 s, 14.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:20:33 (1788520833) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:20:43 (1788520843) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00174906 s, 585 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00380962 s, 269 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011975 s, 855 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.002036 s, 503 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 07:20:58 (1788520858) create test files with size 15753476 .. 36f221a0c8d6550e77448dfbf881c623 - 50e5d4df949d5e4e264d3fbd41ce287c - e961b8a907aea3d0694da95c1bc52bd1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 07:21:27 (1788520887) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:21:43 (1788520903) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:21:51 (1788520911) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263349 s, 4.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312844 s, 33.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 07:22:03 (1788520923) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118122 s, 17.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.308937 s, 13.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.423548 s, 5.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 07:22:24 (1788520944) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 07:22:28 (1788520948) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:22:32 (1788520952) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550577 s, 19.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 07:22:42 (1788520962) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:22:46 (1788520966) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xcf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:22:59 (1788520979) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.428062 s, 24.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:23:15 (1788520995) llite.lustre-ffff9612501ad800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.421175 s, 24.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff96124a509000.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:23:35 (1788521015) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:24:20 (1788521060) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:24:59 (1788521099) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 07:25:04 (1788521104) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.285591 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.185931 s, 16.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7afde43 (nosync,stale) Mirror 2: 0x7afde43 (nosync,stale) ** verify mirror 2 doesn't change original checksum: c192df05496cd49f865b2b464a0728ab - mirror 1 checksum: 6c208a920534927d7b811b86e809d4ad - mirror 2 checksum: c192df05496cd49f865b2b464a0728ab - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T07:25:08-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T07:25:08-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c192df05496cd49f865b2b464a0728ab - mirror 1 checksum: 6c208a920534927d7b811b86e809d4ad - mirror 2 checksum: 6c208a920534927d7b811b86e809d4ad - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T07:25:08-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T07:25:08-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:25:20 (1788521120) Failing mds1 on oleg432-server Stopping /mnt/lustre-mds1 (opts:) on oleg432-server 07:25:28 (1788521128) shut down facet: mds1 facet_host: oleg432-server facet_failover_host: oleg432-server Failover mds1 to oleg432-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 07:26:04 (1788521164) targets are mounted 07:26:04 (1788521164) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599645 s, 17.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (65s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:26:26 (1788521186) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:26:31 (1788521191) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (7s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:26:37 (1788521197) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0182064 s, 900 kB/s PASS 50c (13s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:26:51 (1788521211) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:26:56 (1788521216) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00869966 s, 471 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.084464 s, 48.5 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:27:06 (1788521226) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:27:35 (1788521255) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:28:06 (1788521286) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:28:32 (1788521312) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:28:59 (1788521339) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 07:29:07 (1788521347) mirror create pid 54348 mirror split pid 54349 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54348 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54349 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 07:30:16 (1788521416) Seed set to 1418 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1149b 000003 1788521418.862501 trunc from 00000000 to 0x01149a (0x1149b bytes) 000005 1788521418.913780 mapread 0x00b339 thru 0x011474 (0x613c bytes) 000006 1788521418.991871 read 0x00f5f3 thru 0x01149a (0x1ea8 bytes) 000007 1788521418.998260 mapwrite 0x005b26 thru 0x00e9e7 (0x8ec2 bytes) 000008 1788521419.132266 punch from 0x002687 to 0x00532e (0x2ca8 bytes) 000009 1788521419.137642 read 0x0043aa thru 0x008181 (0x3dd8 bytes) 000010 1788521419.148760 mapread 0x006824 thru 0x00f8e7 (0x90c4 bytes) 000011 1788521419.150709 punch from 0x0077c6 to 0x00b90c (0x4147 bytes) 000013 1788521419.156875 mapread 0x00f603 thru 0x01149a (0x1e98 bytes) 000014 1788521419.157918 punch from 0x004d6f to 0x009581 (0x4813 bytes) 000015 1788521419.163898 mapread 0x0104e6 thru 0x01149a (0xfb5 bytes) 000017 1788521419.166873 write 0x0069c5 thru 0x00c8c7 (0x5f03 bytes) 000018 1788521419.172990 trunc from 0x01149b to 0x001b24 (0xf976 bytes) 000019 1788521419.196622 mapwrite 0x033060 thru 0x03eb14 (0xbab5 bytes) 000020 1788521419.380933 trunc from 0x03eb15 to 0x0001f7 (0x3e91d bytes) 000021 1788521419.402187 read 0x0001e9 thru 0x0001f7 (0xf bytes) 000022 1788521419.403148 mapread 0x000131 thru 0x0001f7 (0xc7 bytes) 000023 1788521419.405712 mapread 0x0001d6 thru 0x0001f7 (0x22 bytes) 000024 1788521419.407722 mapread 0x000168 thru 0x0001f7 (0x90 bytes) 000025 1788521419.409556 read 0x0001e0 thru 0x0001f7 (0x18 bytes) 000026 1788521419.410622 punch from 0x000038 to 0x0001f7 (0x1c0 bytes) 000027 1788521419.415881 punch from 0x00017c to 0x0001f7 (0x7c bytes) 000029 1788521419.424450 read 0x000068 thru 0x0001f7 (0x190 bytes) 000031 1788521419.449064 read 0x00014b thru 0x0001f7 (0xad bytes) 000032 1788521419.450748 write 0x0334ef thru 0x03ffff (0xcb11 bytes) 000033 1788521419.454113 mapwrite 0x02fa5d thru 0x036885 (0x6e29 bytes) 000039 1788521419.572762 mapwrite 0x02559e thru 0x03097e (0xb3e1 bytes) truncating to largest ever: 0x3ad9a 000040 1788521419.666582 trunc from 0x040000 to 0x03ad99 (0x5266 bytes) 000042 1788521419.691026 read 0x011f93 thru 0x019064 (0x70d2 bytes) 000047 1788521419.695391 mapread 0x023e74 thru 0x03359f (0xf72c bytes) 000048 1788521419.697249 write 0x034943 thru 0x038f75 (0x4633 bytes) 000049 1788521419.698536 mapread 0x015eea thru 0x018b64 (0x2c7b bytes) truncating to largest ever: 0x3c544 000050 1788521419.699966 trunc from 0x03ad9a to 0x03c543 (0x17aa bytes) 000052 1788521419.724530 mapwrite 0x0195bc thru 0x01b9c0 (0x2405 bytes) 000053 1788521419.790549 write 0x035c0e thru 0x0364e9 (0x8dc bytes) 000054 1788521419.794497 mapwrite 0x03e9a7 thru 0x03fd9c (0x13f6 bytes) 000055 1788521419.918008 trunc from 0x03fd9d to 0x0139e1 (0x2c3bb bytes) 000056 1788521419.949717 mapread 0x0067a6 thru 0x0120ab (0xb906 bytes) 000059 1788521419.953637 read 0x00a76e thru 0x00ec20 (0x44b3 bytes) 000062 1788521419.955970 mapread 0x008ccb thru 0x0139e1 (0xad17 bytes) 000063 1788521419.957400 trunc from 0x0139e2 to 0x0029c8 (0x11019 bytes) 000066 1788521419.984564 punch from 0x001ee2 to 0x0029c8 (0xae7 bytes) 000070 1788521419.990919 punch from 0x00074b to 0x0029c8 (0x227e bytes) 000072 1788521419.996131 write 0x0230e4 thru 0x029955 (0x6872 bytes) 000073 1788521420.002756 MIRROR_EXTEND to 2 mirrors 000074 1788521421.325373 trunc from 0x029956 to 0x03512e (0xb7d9 bytes) 000076 1788521421.446928 read 0x008729 thru 0x016ba2 (0xe47a bytes) 000077 1788521421.472781 mapread 0x0058d0 thru 0x0133b3 (0xdae4 bytes) 000078 1788521421.531873 mapread 0x015d7d thru 0x02342b (0xd6af bytes) 000079 1788521421.587756 MIRROR_RESYNC 2 mirrors 000080 1788521422.865504 read 0x01a4b9 thru 0x020c0a (0x6752 bytes) 000081 1788521422.881652 mapwrite 0x00d1fe thru 0x015af3 (0x88f6 bytes) 000082 1788521423.228068 write 0x01a607 thru 0x0288cf (0xe2c9 bytes) truncating to largest ever: 0x3f637 000083 1788521423.242962 trunc from 0x03512f to 0x03f636 (0xa508 bytes) 000084 1788521423.282095 punch from 0x01ca76 to 0x025d87 (0x9312 bytes) 000086 1788521423.315152 mapwrite 0x004db0 thru 0x011c82 (0xced3 bytes) 000087 1788521423.456630 trunc from 0x03f637 to 0x033ee2 (0xb754 bytes) 000088 1788521423.544066 write 0x0000b4 thru 0x00a3b3 (0xa300 bytes) 000089 1788521423.569444 write 0x03f469 thru 0x03ffff (0xb97 bytes) 000091 1788521423.571898 punch from 0x0352e3 to 0x039b84 (0x48a2 bytes) 000092 1788521423.622826 trunc from 0x040000 to 0x021978 (0x1e687 bytes) 000093 1788521423.692952 write 0x01dc95 thru 0x0226a9 (0x4a15 bytes) 000095 1788521423.710596 read 0x001bdb thru 0x00cd3f (0xb165 bytes) 000097 1788521423.720553 read 0x01a0bc thru 0x0226a9 (0x85ee bytes) 000098 1788521423.772809 read 0x00d3bc thru 0x01875f (0xb3a4 bytes) 000099 1788521423.775905 trunc from 0x0226aa to 0x01287f (0xfe2a bytes) 000100 1788521423.889344 mapread 0x007cc0 thru 0x010381 (0x86c2 bytes) 000101 1788521423.892164 punch from 0x00a7f3 to 0x00eae0 (0x42ee bytes) 000102 1788521423.925645 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000103 1788521424.207589 mapread 0x00752f thru 0x01054e (0x9020 bytes) 000104 1788521424.235333 read 0x003539 thru 0x011470 (0xdf38 bytes) 000106 1788521424.236967 read 0x00dc32 thru 0x01287f (0x4c4e bytes) 000107 1788521424.247399 trunc from 0x012880 to 0x020eb8 (0xe639 bytes) 000108 1788521424.272308 mapread 0x017daa thru 0x01c58e (0x47e5 bytes) 000111 1788521424.286819 punch from 0x001086 to 0x010b8b (0xfb06 bytes) 000113 1788521424.294010 write 0x030f53 thru 0x036282 (0x5330 bytes) 000114 1788521424.295682 mapread 0x02db74 thru 0x036282 (0x870f bytes) 000117 1788521424.307258 mapwrite 0x00b21d thru 0x017015 (0xbdf9 bytes) 000118 1788521424.433052 punch from 0x02fca1 to 0x034758 (0x4ab8 bytes) 000123 1788521424.456216 mapwrite 0x00e79b thru 0x0154aa (0x6d10 bytes) 000124 1788521424.636515 read 0x004ad0 thru 0x0140e1 (0xf612 bytes) 000125 1788521424.658667 punch from 0x02ff21 to 0x034e4c (0x4f2c bytes) 000126 1788521424.668759 trunc from 0x036283 to 0x014621 (0x21c61 bytes) 000127 1788521424.713007 read 0x002b6c thru 0x003a97 (0xf2c bytes) 000128 1788521424.715596 punch from 0x00409d to 0x00ba6a (0x79ce bytes) 000130 1788521424.729252 mapwrite 0x01384e thru 0x019ece (0x6681 bytes) 000131 1788521425.047142 mapread 0x008238 thru 0x01708c (0xee55 bytes) 000132 1788521425.048315 trunc from 0x019ecf to 0x037ad7 (0x1dc09 bytes) 000133 1788521425.102452 punch from 0x009c9d to 0x00ad50 (0x10b4 bytes) 000134 1788521425.110764 mapread 0x01b57e thru 0x01e769 (0x31ec bytes) 000138 1788521425.148952 mapread 0x0104f8 thru 0x01beb8 (0xb9c1 bytes) 000141 1788521425.151777 punch from 0x01fd38 to 0x02a0cf (0xa398 bytes) 000144 1788521425.172426 punch from 0x01a4f4 to 0x01ce33 (0x2940 bytes) 000145 1788521425.193916 write 0x035cc7 thru 0x03ffff (0xa339 bytes) 000146 1788521425.196640 read 0x03df8d thru 0x03ffff (0x2073 bytes) 000147 1788521425.197775 write 0x031557 thru 0x0339cc (0x2476 bytes) 000148 1788521425.199116 read 0x007a22 thru 0x013baa (0xc189 bytes) 000149 1788521425.217377 mapwrite 0x0259c1 thru 0x02996d (0x3fad bytes) 000151 1788521425.354066 trunc from 0x040000 to 0x0030e9 (0x3cf16 bytes) 000154 1788521425.407447 punch from 0x002daa to 0x0030e9 (0x340 bytes) 000155 1788521425.414593 punch from 0x0003f7 to 0x0030e9 (0x2cf3 bytes) 000158 1788521425.448696 mapread 0x00099a thru 0x0030e9 (0x2750 bytes) 000159 1788521425.497560 mapread 0x000e19 thru 0x0030e9 (0x22d1 bytes) 000160 1788521425.499929 write 0x01aa13 thru 0x022cdb (0x82c9 bytes) 000161 1788521425.506128 write 0x02398f thru 0x02ec94 (0xb306 bytes) 000164 1788521425.514328 mapread 0x02cc91 thru 0x02ec94 (0x2004 bytes) 000165 1788521425.515689 read 0x0078ee thru 0x016916 (0xf029 bytes) 000166 1788521425.542264 write 0x019152 thru 0x020c31 (0x7ae0 bytes) 000167 1788521425.545339 read 0x0129e0 thru 0x017023 (0x4644 bytes) 000169 1788521425.563927 write 0x0217ba thru 0x02f3c9 (0xdc10 bytes) 000170 1788521425.607081 write 0x01be81 thru 0x01c32e (0x4ae bytes) 000171 1788521425.609321 mapwrite 0x03b7fe thru 0x03ffff (0x4802 bytes) 000172 1788521425.758327 write 0x02de39 thru 0x03b1ce (0xd396 bytes) 000173 1788521425.769044 mapwrite 0x0258d2 thru 0x027729 (0x1e58 bytes) 000174 1788521425.893086 read 0x01af58 thru 0x01fcfc (0x4da5 bytes) 000175 1788521425.906418 mapwrite 0x01bac0 thru 0x01fa08 (0x3f49 bytes) 000176 1788521426.005559 mapread 0x00e11d thru 0x012906 (0x47ea bytes) 000177 1788521426.007765 read 0x009cad thru 0x012ee7 (0x923b bytes) 000178 1788521426.012879 punch from 0x00ca6e to 0x019cac (0xd23f bytes) 000179 1788521426.032543 write 0x00c880 thru 0x01c4b7 (0xfc38 bytes) 000180 1788521426.054442 mapread 0x00af9b thru 0x0164c4 (0xb52a bytes) 000181 1788521426.064453 trunc from 0x040000 to 0x00e5c3 (0x31a3c bytes) 000182 1788521426.088863 write 0x02f9a0 thru 0x038fb7 (0x9618 bytes) 000185 1788521426.097538 mapread 0x015977 thru 0x022086 (0xc710 bytes) 000186 1788521426.242863 mapread 0x0363ae thru 0x038fb7 (0x2c0a bytes) 000189 1788521426.261447 punch from 0x02811e to 0x033110 (0xaff3 bytes) 000190 1788521426.293625 mapread 0x028de9 thru 0x0355c1 (0xc7d9 bytes) 000192 1788521426.464305 mapread 0x01c821 thru 0x025f78 (0x9758 bytes) 000193 1788521426.497610 write 0x01b005 thru 0x023318 (0x8314 bytes) 000195 1788521426.499376 read 0x001635 thru 0x00a64b (0x9017 bytes) 000197 1788521426.501770 read 0x0301a2 thru 0x038fb7 (0x8e16 bytes) 000200 1788521426.504563 mapwrite 0x012b17 thru 0x0229f4 (0xfede bytes) 000201 1788521426.648371 read 0x005d6c thru 0x00f219 (0x94ae bytes) 000202 1788521426.649304 trunc from 0x038fb8 to 0x025701 (0x138b6 bytes) 000203 1788521426.673772 trunc from 0x025702 to 0x0117f8 (0x13f09 bytes) 000205 1788521426.753347 mapread 0x001c2f thru 0x009ba1 (0x7f73 bytes) 000206 1788521426.763871 mapwrite 0x034538 thru 0x03e031 (0x9afa bytes) 000208 1788521426.974009 mapread 0x0045f2 thru 0x00671c (0x212b bytes) 000212 1788521426.979132 read 0x00ea66 thru 0x0153f5 (0x6990 bytes) 000213 1788521426.979978 write 0x01c945 thru 0x02a232 (0xd8ee bytes) 000221 1788521426.985756 mapread 0x0281eb thru 0x02a20c (0x2022 bytes) 000224 1788521426.987971 trunc from 0x03e032 to 0x018a11 (0x25620 bytes) 000225 1788521427.006913 mapread 0x00ec53 thru 0x016bf9 (0x7fa7 bytes) 000228 1788521427.011726 mapwrite 0x01da49 thru 0x0219d2 (0x3f8a bytes) 000230 1788521427.109727 write 0x011322 thru 0x01983b (0x851a bytes) 000233 1788521427.112678 read 0x01ce62 thru 0x0219d2 (0x4b71 bytes) 000235 1788521427.113956 mapwrite 0x00306c thru 0x004a4d (0x19e2 bytes) 000236 1788521427.189248 write 0x0338c3 thru 0x03ffff (0xc73d bytes) 000237 1788521427.192373 write 0x001fb2 thru 0x00c672 (0xa6c1 bytes) 000239 1788521427.198418 write 0x02ae46 thru 0x035250 (0xa40b bytes) 000241 1788521427.227662 mapwrite 0x00ebcf thru 0x016f8b (0x83bd bytes) 000242 1788521427.327049 trunc from 0x040000 to 0x010e52 (0x2f1ad bytes) 000243 1788521427.353468 mapread 0x004350 thru 0x009544 (0x51f5 bytes) 000244 1788521427.372594 MIRROR_EXTEND to 2 mirrors 000246 1788521428.676826 write 0x02b3ee thru 0x030885 (0x5498 bytes) 000247 1788521428.827570 trunc from 0x030886 to 0x01088d (0x1fff8 bytes) 000248 1788521428.915742 mapread 0x00337b thru 0x0065ae (0x3234 bytes) 000250 1788521428.977745 read 0x0055a4 thru 0x01088d (0xb2ea bytes) 000252 1788521428.980684 read 0x002188 thru 0x006ce6 (0x4b5f bytes) 000253 1788521428.981589 trunc from 0x01088e to 0x00d23a (0x3653 bytes) 000255 1788521429.026952 write 0x0388f1 thru 0x03ac72 (0x2382 bytes) 000256 1788521429.034179 write 0x00bc0a thru 0x01a9e7 (0xedde bytes) 000259 1788521429.060090 punch from 0x012c3a to 0x01f43c (0xc803 bytes) 000262 1788521429.100888 write 0x02cfd2 thru 0x037523 (0xa552 bytes) 000263 1788521429.141695 punch from 0x029dfb to 0x032eea (0x90f0 bytes) 000264 1788521429.177367 mapread 0x009923 thru 0x0130f5 (0x97d3 bytes) 000266 1788521429.182550 mapread 0x016cc9 thru 0x0216a1 (0xa9d9 bytes) 000268 1788521429.194222 punch from 0x02e08b to 0x03ac72 (0xcbe8 bytes) 000269 1788521429.205445 trunc from 0x03ac73 to 0x034d9b (0x5ed7 bytes) 000270 1788521429.222265 punch from 0x01fe7e to 0x023ab1 (0x3c34 bytes) 000271 1788521429.227867 punch from 0x0200e3 to 0x02d5f2 (0xd510 bytes) 000273 1788521429.237991 trunc from 0x034d9c to 0x00d075 (0x27d26 bytes) 000274 1788521429.263285 mapwrite 0x009f75 thru 0x016e7f (0xcf0b bytes) 000275 1788521429.547353 read 0x00b0c6 thru 0x016150 (0xb08b bytes) 000276 1788521429.548412 write 0x009f72 thru 0x00c8a8 (0x2937 bytes) 000279 1788521429.552130 mapwrite 0x00cd2f thru 0x00dcc2 (0xf94 bytes) 000281 1788521429.679487 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16e80): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16e80): Mirror 2: 0x517991dc (stale) CRC-32 checksum value for chunk [0, 0x16e80): Mirror 1: 0xc4dedd8d Mirror 3: 0xc4dedd8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16e80: skipped 000282 1788521430.626780 punch from 0x00b916 to 0x00fd6a (0x4455 bytes) 000283 1788521430.708688 write 0x02ed6c thru 0x036971 (0x7c06 bytes) 000284 1788521430.711418 mapread 0x00bc51 thru 0x0134dd (0x788d bytes) 000285 1788521430.729590 trunc from 0x036972 to 0x0390ca (0x2759 bytes) 000286 1788521430.751195 punch from 0x0153ce to 0x01982d (0x4460 bytes) 000290 1788521430.766804 write 0x026f77 thru 0x02b55b (0x45e5 bytes) 000292 1788521430.778978 write 0x026f63 thru 0x02d048 (0x60e6 bytes) 000294 1788521430.786388 trunc from 0x0390cb to 0x01453f (0x24b8b bytes) 000295 1788521430.806514 write 0x01764f thru 0x0199aa (0x235c bytes) 000297 1788521430.810825 punch from 0x006970 to 0x015fb1 (0xf642 bytes) 000298 1788521430.822950 mapwrite 0x02200f thru 0x02c41d (0xa40f bytes) 000301 1788521430.918509 mapread 0x01d772 thru 0x0271c9 (0x9a58 bytes) 000303 1788521430.921580 write 0x00b550 thru 0x010131 (0x4be2 bytes) 000304 1788521430.936825 mapwrite 0x01495d thru 0x020697 (0xbd3b bytes) 000305 1788521431.026173 MIRROR_RESYNC 3 mirrors 000306 1788521432.996829 mapwrite 0x026318 thru 0x03443d (0xe126 bytes) 000307 1788521433.594349 write 0x02bc1d thru 0x037fa2 (0xc386 bytes) 000310 1788521433.600497 mapread 0x009ded thru 0x017946 (0xdb5a bytes) 000311 1788521433.668047 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fa3): Mirror 3: 0xf19d0135 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fa3): Mirror 3: 0xf19d0135 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fa3: skipped 000313 1788521434.216938 trunc from 0x037fa3 to 0x026ce7 (0x112bb bytes) 000314 1788521434.252167 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000315 1788521434.522499 write 0x033ef4 thru 0x03802d (0x413a bytes) 000317 1788521434.526068 mapwrite 0x029499 thru 0x02ed9d (0x5905 bytes) 000319 1788521434.672038 punch from 0x01979e to 0x0238cf (0xa132 bytes) 000321 1788521434.684653 trunc from 0x03802e to 0x00bb23 (0x2c50a bytes) 000322 1788521434.709665 write 0x011332 thru 0x0206ba (0xf389 bytes) 000323 1788521434.714662 mapread 0x01bb6b thru 0x01fe73 (0x4309 bytes) 000325 1788521434.716940 read 0x01b562 thru 0x0206ba (0x5159 bytes) 000326 1788521434.718034 punch from 0x00155e to 0x006c1f (0x56c2 bytes) 000330 1788521434.778073 mapread 0x002ac8 thru 0x00beff (0x9438 bytes) 000331 1788521434.792325 punch from 0x01feb3 to 0x020158 (0x2a6 bytes) 000332 1788521434.798326 punch from 0x000b08 to 0x007471 (0x696a bytes) 000333 1788521434.807231 mapwrite 0x02dabc thru 0x02f573 (0x1ab8 bytes) 000334 1788521434.929625 punch from 0x0296e0 to 0x02e4b2 (0x4dd3 bytes) 000337 1788521434.937497 trunc from 0x02f574 to 0x03ceb0 (0xd93d bytes) 000338 1788521434.958762 write 0x021478 thru 0x024d8e (0x3917 bytes) 000339 1788521434.971392 punch from 0x01e403 to 0x02aad7 (0xc6d5 bytes) 000340 1788521434.989948 punch from 0x02cd56 to 0x031817 (0x4ac2 bytes) 000341 1788521435.013825 MIRROR_EXTEND to 2 mirrors 000344 1788521436.718742 write 0x02e406 thru 0x034b0b (0x6706 bytes) 000345 1788521436.835353 mapread 0x008b6f thru 0x00cc53 (0x40e5 bytes) 000346 1788521436.866673 MIRROR_RESYNC 2 mirrors 000347 1788521438.830536 write 0x031c70 thru 0x03edb8 (0xd149 bytes) 000348 1788521438.960262 mapwrite 0x02f06c thru 0x036fe9 (0x7f7e bytes) 000349 1788521439.109105 punch from 0x02aee8 to 0x02fc2d (0x4d46 bytes) 000350 1788521439.122576 write 0x005163 thru 0x014085 (0xef23 bytes) 000351 1788521439.141133 trunc from 0x03edb9 to 0x011910 (0x2d4a8 bytes) 000354 1788521439.186704 write 0x0248a1 thru 0x02ae86 (0x65e6 bytes) 000359 1788521439.209733 read 0x01687b thru 0x017a7e (0x1204 bytes) 000360 1788521439.245227 punch from 0x00e63d to 0x01e362 (0xfd26 bytes) 000361 1788521439.324513 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ae87): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ae87): Mirror 2: 0xcab283d1 (stale) CRC-32 checksum value for chunk [0, 0x2ae87): Mirror 1: 0xbace5a1b Mirror 3: 0xbace5a1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ae87: skipped 000362 1788521442.436510 punch from 0x00d92b to 0x0120af (0x4785 bytes) 000365 1788521442.543385 mapread 0x00a197 thru 0x011fb9 (0x7e23 bytes) 000366 1788521442.564437 trunc from 0x02ae87 to 0x033d5d (0x8ed7 bytes) 000368 1788521442.634412 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33d5e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d5e): Mirror 2: 0xf8da712c (stale) Mirror 3: 0x7a83e9be (stale) CRC-32 checksum value for chunk [0, 0x33d5e): Mirror 1: 0x95dbf0ae Mirror 4: 0x95dbf0ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d5e: skipped 000369 1788521444.509897 read 0x0297a8 thru 0x033d5d (0xa5b6 bytes) 000370 1788521444.546619 mapread 0x023902 thru 0x02f992 (0xc091 bytes) 000371 1788521444.627237 mapwrite 0x0331de thru 0x03ffff (0xce22 bytes) 000372 1788521444.890073 punch from 0x0307e5 to 0x0333e6 (0x2c02 bytes) 000373 1788521444.900445 read 0x02321a thru 0x028981 (0x5768 bytes) 000374 1788521444.925741 write 0x03e42e thru 0x03e4f5 (0xc8 bytes) 000375 1788521444.940466 read 0x01acdb thru 0x025f28 (0xb24e bytes) 000376 1788521444.949915 MIRROR_RESYNC 4 mirrors 000377 1788521447.744186 mapwrite 0x023784 thru 0x033464 (0xfce1 bytes) 000378 1788521447.992261 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xee5c1534 (stale) Mirror 3: 0xee5c1534 (stale) Mirror 4: 0xee5c1534 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8df6ed8a Mirror 5: 0x8df6ed8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000380 1788521449.582014 read 0x01b4a3 thru 0x024c6c (0x97ca bytes) 000381 1788521449.602613 punch from 0x027f55 to 0x036eb4 (0xef60 bytes) 000388 1788521449.744379 write 0x01e6ee thru 0x02568e (0x6fa1 bytes) 000389 1788521449.764368 read 0x01f68c thru 0x024312 (0x4c87 bytes) 000390 1788521449.765400 punch from 0x00a651 to 0x018c29 (0xe5d9 bytes) 000393 1788521449.814288 mapread 0x02f680 thru 0x03c562 (0xcee3 bytes) 000395 1788521449.841768 punch from 0x022e4a to 0x02a007 (0x71be bytes) 000396 1788521449.873895 mapread 0x004b7b thru 0x006115 (0x159b bytes) 000397 1788521449.874766 punch from 0x01f29c to 0x024360 (0x50c5 bytes) 000398 1788521449.883296 mapread 0x0104c7 thru 0x01f62a (0xf164 bytes) 000399 1788521449.912464 MIRROR_RESYNC 5 mirrors 000400 1788521454.402371 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000402 1788521455.072167 MIRROR_RESYNC 4 mirrors 000403 1788521455.916794 read 0x03ea79 thru 0x03ffff (0x1587 bytes) 000405 1788521455.928747 punch from 0x0265f8 to 0x032d7b (0xc784 bytes) 000408 1788521456.070800 punch from 0x0225f8 to 0x029e46 (0x784f bytes) 000409 1788521456.080731 mapwrite 0x009bb0 thru 0x015a21 (0xbe72 bytes) 000410 1788521456.217266 write 0x02c4fd thru 0x035a08 (0x950c bytes) 000411 1788521456.242198 read 0x02102b thru 0x030b8f (0xfb65 bytes) 000414 1788521456.277434 punch from 0x0038bd to 0x004e58 (0x159c bytes) 000415 1788521456.314178 trunc from 0x040000 to 0x003d58 (0x3c2a7 bytes) 000416 1788521456.366740 mapwrite 0x005e43 thru 0x00a913 (0x4ad1 bytes) 000419 1788521456.589029 punch from 0x000f9d to 0x007529 (0x658d bytes) 000420 1788521456.595901 write 0x033797 thru 0x03ffd1 (0xc83b bytes) 000422 1788521456.598701 write 0x02719a thru 0x028b4c (0x19b3 bytes) 000423 1788521456.609258 mapwrite 0x03916b thru 0x03a6bf (0x1555 bytes) 000424 1788521456.709775 punch from 0x017071 to 0x0252eb (0xe27b bytes) 000425 1788521456.725246 write 0x01ed36 thru 0x02e1f3 (0xf4be bytes) 000427 1788521456.766131 mapwrite 0x021b38 thru 0x022c5d (0x1126 bytes) 000431 1788521456.926105 trunc from 0x03ffd2 to 0x02cd8f (0x13242 bytes) 000433 1788521456.981543 trunc from 0x02cd90 to 0x000bbc (0x2c1d3 bytes) 000434 1788521457.043169 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbbd): Mirror 4: 0x97685a3a (stale) Mirror 5: 0x97685a3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbbd): Mirror 4: 0x97685a3a (stale) Mirror 5: 0x97685a3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbbd: skipped 000435 1788521457.670077 write 0x03d402 thru 0x03ffff (0x2bfe bytes) 000436 1788521457.671558 read 0x0200b0 thru 0x02d427 (0xd378 bytes) 000440 1788521457.693589 mapwrite 0x01725d thru 0x02429e (0xd042 bytes) 000441 1788521457.806627 mapread 0x021590 thru 0x02fc73 (0xe6e4 bytes) 000445 1788521457.823288 mapread 0x000839 thru 0x00b8ad (0xb075 bytes) 000447 1788521457.841995 punch from 0x02f9dd to 0x03c32c (0xc950 bytes) 000448 1788521457.849559 write 0x000c82 thru 0x001208 (0x587 bytes) 000451 1788521457.852555 mapwrite 0x019360 thru 0x01ca93 (0x3734 bytes) 000452 1788521457.923392 punch from 0x0209b3 to 0x027746 (0x6d94 bytes) 000453 1788521457.940664 MIRROR_RESYNC 3 mirrors 000454 1788521459.524018 trunc from 0x040000 to 0x024929 (0x1b6d6 bytes) 000456 1788521459.611861 mapwrite 0x005a00 thru 0x01468e (0xec8f bytes) 000461 1788521459.707400 read 0x00d8dc thru 0x0112b6 (0x39db bytes) 000462 1788521459.710441 trunc from 0x02492a to 0x037fb7 (0x1368e bytes) 000463 1788521459.751373 punch from 0x032ab8 to 0x0365d7 (0x3b20 bytes) 000464 1788521459.755387 mapwrite 0x026c54 thru 0x028f98 (0x2345 bytes) 000466 1788521459.822448 read 0x032dcf thru 0x037fb7 (0x51e9 bytes) 000468 1788521459.839904 write 0x00b7f0 thru 0x019ff6 (0xe807 bytes) 000470 1788521459.845071 read 0x02d025 thru 0x034562 (0x753e bytes) 000472 1788521459.849889 mapread 0x016de1 thru 0x0250c7 (0xe2e7 bytes) 000473 1788521459.856751 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fb8): Mirror 5: 0x651f4dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fb8): Mirror 5: 0x651f4dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fb8: skipped 000474 1788521460.292424 read 0x01113f thru 0x015f37 (0x4df9 bytes) 000475 1788521460.307034 write 0x000a5f thru 0x0066ee (0x5c90 bytes) 000476 1788521460.325569 mapread 0x008e80 thru 0x00d3e3 (0x4564 bytes) 000480 1788521460.360311 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37fb8): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fb8): Mirror 5: 0x651f4dc (stale) CRC-32 checksum value for chunk [0, 0x37fb8): Mirror 1: 0x2097149a Mirror 6: 0x2097149a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fb8: skipped 000482 1788521462.114674 mapread 0x00d2a9 thru 0x017b88 (0xa8e0 bytes) 000484 1788521462.168742 punch from 0x023aac to 0x02fc50 (0xc1a5 bytes) 000485 1788521462.262915 punch from 0x020e90 to 0x02e8d6 (0xda47 bytes) 000486 1788521462.268434 punch from 0x0355bb to 0x037fb7 (0x29fd bytes) 000488 1788521462.274564 trunc from 0x037fb8 to 0x00f713 (0x288a4 bytes) 000490 1788521462.298937 read 0x0071cd thru 0x007574 (0x3a8 bytes) 000491 1788521462.305773 trunc from 0x00f714 to 0x024204 (0x14af1 bytes) 000492 1788521462.328117 punch from 0x0097a5 to 0x0132c4 (0x9b20 bytes) 000493 1788521462.333785 mapwrite 0x034a06 thru 0x03ffff (0xb5fa bytes) 000495 1788521462.465573 mapread 0x031317 thru 0x032ffe (0x1ce8 bytes) 000497 1788521462.477129 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xace1be70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xace1be70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3fa65 000498 1788521462.910689 trunc from 0x040000 to 0x03fa64 (0x59b bytes) 000499 1788521462.934987 punch from 0x0261ae to 0x0315bf (0xb412 bytes) 000502 1788521462.942278 mapread 0x03b862 thru 0x03fa64 (0x4203 bytes) 000504 1788521463.002082 MIRROR_RESYNC 2 mirrors 000505 1788521464.574141 write 0x02378b thru 0x02df4c (0xa7c2 bytes) 000506 1788521464.749335 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000507 1788521464.915220 punch from 0x0185ed to 0x022c25 (0xa639 bytes) 000510 1788521464.923530 mapread 0x000ac9 thru 0x00f70c (0xec44 bytes) 000512 1788521464.966700 mapwrite 0x020b82 thru 0x02f2c0 (0xe73f bytes) 000514 1788521465.155639 mapread 0x03c371 thru 0x03fa64 (0x36f4 bytes) 000515 1788521465.176544 trunc from 0x03fa65 to 0x03ab49 (0x4f1b bytes) 000516 1788521465.225788 read 0x033a33 thru 0x039706 (0x5cd4 bytes) 000517 1788521465.243322 punch from 0x018784 to 0x0252bf (0xcb3c bytes) 000518 1788521465.254174 write 0x001df8 thru 0x006cf4 (0x4efd bytes) 000519 1788521465.255399 punch from 0x0377f2 to 0x03ab49 (0x3358 bytes) 000520 1788521465.297198 read 0x03283d thru 0x03ab49 (0x830d bytes) 000521 1788521465.326817 read 0x01f6bb thru 0x02c316 (0xcc5c bytes) 000524 1788521465.333534 trunc from 0x03ab4a to 0x021106 (0x19a43 bytes) 000526 1788521465.379623 write 0x03c518 thru 0x03ffff (0x3ae8 bytes) 000527 1788521465.383797 trunc from 0x040000 to 0x0087cf (0x37830 bytes) 000530 1788521465.410067 read 0x003f88 thru 0x0087cf (0x4848 bytes) 000533 1788521465.412668 trunc from 0x0087d0 to 0x00948e (0xcbf bytes) 000534 1788521465.438936 punch from 0x0093ce to 0x00948e (0xc1 bytes) 000535 1788521465.444352 trunc from 0x00948f to 0x00c98e (0x3500 bytes) 000536 1788521465.476979 punch from 0x004b29 to 0x00966e (0x4b46 bytes) 000537 1788521465.488308 write 0x02ec06 thru 0x037975 (0x8d70 bytes) 000540 1788521465.491406 mapread 0x01d229 thru 0x0252bb (0x8093 bytes) 000541 1788521465.516429 write 0x01f855 thru 0x020654 (0xe00 bytes) 000546 1788521465.523218 write 0x02f76f thru 0x0352fa (0x5b8c bytes) 000548 1788521465.526261 read 0x00df18 thru 0x016025 (0x810e bytes) 000549 1788521465.528796 mapwrite 0x000cf6 thru 0x0035b7 (0x28c2 bytes) 000550 1788521465.597180 mapread 0x0373a4 thru 0x037975 (0x5d2 bytes) 000551 1788521465.598253 mapwrite 0x0278a9 thru 0x02d220 (0x5978 bytes) 000552 1788521465.737382 mapread 0x0308bb thru 0x037975 (0x70bb bytes) 000553 1788521465.738495 read 0x006984 thru 0x00db03 (0x7180 bytes) 000554 1788521465.740218 write 0x00d116 thru 0x00fdc8 (0x2cb3 bytes) 000556 1788521465.757151 trunc from 0x037976 to 0x029061 (0xe914 bytes) 000557 1788521465.804861 punch from 0x015a52 to 0x018a6d (0x301c bytes) 000558 1788521465.823050 write 0x021f5a thru 0x02201a (0xc1 bytes) 000559 1788521465.824599 write 0x0051c6 thru 0x007f21 (0x2d5c bytes) 000561 1788521465.826673 read 0x002069 thru 0x00d610 (0xb5a8 bytes) 000562 1788521465.827358 trunc from 0x029062 to 0x00b34a (0x1dd17 bytes) 000563 1788521465.876568 trunc from 0x00b34b to 0x023b2b (0x187e1 bytes) 000565 1788521465.926663 read 0x01fcdb thru 0x023b2b (0x3e51 bytes) 000567 1788521465.985018 MIRROR_EXTEND to 2 mirrors 000568 1788521467.393578 read 0x013ad9 thru 0x01c6aa (0x8bd2 bytes) 000570 1788521467.502924 trunc from 0x023b2c to 0x025ae1 (0x1fb6 bytes) 000571 1788521467.662796 mapwrite 0x0202dc thru 0x021a23 (0x1748 bytes) 000575 1788521467.742674 write 0x02d222 thru 0x038056 (0xae35 bytes) 000577 1788521467.746324 read 0x01ab0b thru 0x01b311 (0x807 bytes) 000578 1788521467.765358 trunc from 0x038057 to 0x030892 (0x77c4 bytes) 000579 1788521467.802512 mapread 0x00029f thru 0x0054ba (0x521c bytes) 000580 1788521467.804730 trunc from 0x030893 to 0x01f61b (0x11277 bytes) 000585 1788521467.836743 mapwrite 0x001b12 thru 0x00e2a1 (0xc790 bytes) 000586 1788521467.913666 read 0x00f6ce thru 0x017094 (0x79c7 bytes) 000588 1788521467.914959 mapwrite 0x011a76 thru 0x01bcd8 (0xa263 bytes) 000589 1788521467.997671 punch from 0x00c352 to 0x015f21 (0x9bd0 bytes) 000591 1788521468.007622 mapread 0x00ae2a thru 0x00d786 (0x295d bytes) 000592 1788521468.026712 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f61c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f61c): Mirror 2: 0xa35eaf37 (stale) CRC-32 checksum value for chunk [0, 0x1f61c): Mirror 1: 0xd62a9510 Mirror 3: 0xd62a9510 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f61c: skipped 000593 1788521469.630693 write 0x025aa1 thru 0x02bf6c (0x64cc bytes) 000594 1788521469.751452 write 0x035782 thru 0x035f7e (0x7fd bytes) 000595 1788521469.759350 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35f7f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f7f): Mirror 2: 0xae97947a (stale) Mirror 3: 0xd72032a2 (stale) CRC-32 checksum value for chunk [0, 0x35f7f): Mirror 1: 0xf4c4e68a Mirror 4: 0xf4c4e68a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f7f: skipped 000596 1788521471.399229 read 0x000794 thru 0x00d36d (0xcbda bytes) 000598 1788521471.463534 write 0x031f87 thru 0x032782 (0x7fc bytes) 000599 1788521471.570521 read 0x0070b1 thru 0x010160 (0x90b0 bytes) 000600 1788521471.587376 punch from 0x01eced to 0x0218e2 (0x2bf6 bytes) 000601 1788521471.604509 mapwrite 0x02d8c1 thru 0x0378b9 (0x9ff9 bytes) 000602 1788521471.694242 read 0x01ba77 thru 0x01c0dd (0x667 bytes) 000603 1788521471.700751 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x378ba): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378ba): Mirror 2: 0x668f9db6 (stale) Mirror 3: 0x88847853 (stale) Mirror 4: 0xbef1cacd (stale) CRC-32 checksum value for chunk [0, 0x378ba): Mirror 1: 0xc647c6a8 Mirror 5: 0xc647c6a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378ba: skipped 000604 1788521473.408601 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x378ba): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378ba): Mirror 2: 0x668f9db6 (stale) Mirror 3: 0x88847853 (stale) Mirror 4: 0xbef1cacd (stale) CRC-32 checksum value for chunk [0, 0x378ba): Mirror 1: 0xc647c6a8 Mirror 5: 0xc647c6a8 Mirror 6: 0xc647c6a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378ba: skipped 000605 1788521474.924713 punch from 0x025f9c to 0x0359c4 (0xfa29 bytes) 000606 1788521475.163587 write 0x00b38e thru 0x0188c9 (0xd53c bytes) 000608 1788521475.195511 mapwrite 0x00a3a5 thru 0x0191db (0xee37 bytes) 000610 1788521475.386133 read 0x016bfa thru 0x023b4b (0xcf52 bytes) 000611 1788521475.400889 trunc from 0x0378ba to 0x035b6c (0x1d4d bytes) 000612 1788521475.433479 write 0x020cb3 thru 0x0308c2 (0xfc10 bytes) 000613 1788521475.452795 write 0x01edac thru 0x02c4d2 (0xd727 bytes) 000614 1788521475.458943 mapwrite 0x01bfb7 thru 0x02a051 (0xe09b bytes) 000615 1788521475.577488 trunc from 0x035b6d to 0x00a9b9 (0x2b1b3 bytes) 000616 1788521475.626336 trunc from 0x00a9ba to 0x039d91 (0x2f3d8 bytes) 000617 1788521475.669635 trunc from 0x039d92 to 0x02cf42 (0xce4f bytes) 000618 1788521475.700647 read 0x02b839 thru 0x02cf42 (0x170a bytes) 000619 1788521475.716062 write 0x016f9d thru 0x02697b (0xf9df bytes) 000620 1788521475.724364 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cf43): Mirror 3: 0x1a5d1a3b (stale) Mirror 4: 0x6a292d2a (stale) Mirror 5: 0x6a292d2a (stale) Mirror 6: 0x6a292d2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cf43): Mirror 3: 0x1a5d1a3b (stale) Mirror 4: 0x6a292d2a (stale) Mirror 5: 0x6a292d2a (stale) Mirror 6: 0x6a292d2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cf43: skipped 000621 1788521476.373335 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cf43): Mirror 3: 0x1a5d1a3b (stale) Mirror 5: 0x6a292d2a (stale) Mirror 6: 0x6a292d2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cf43): Mirror 3: 0x1a5d1a3b (stale) Mirror 5: 0x6a292d2a (stale) Mirror 6: 0x6a292d2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cf43: skipped 000622 1788521476.811510 trunc from 0x02cf43 to 0x0245cb (0x8977 bytes) 000624 1788521476.837615 trunc from 0x0245cc to 0x03b8de (0x17313 bytes) 000627 1788521476.858558 write 0x004a3b thru 0x011e1d (0xd3e3 bytes) 000628 1788521476.874649 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8df): Mirror 5: 0xfb0f1b5f (stale) Mirror 6: 0xfb0f1b5f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8df): Mirror 5: 0xfb0f1b5f (stale) Mirror 6: 0xfb0f1b5f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8df: skipped 000629 1788521477.371689 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8df): Mirror 6: 0xfb0f1b5f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8df): Mirror 6: 0xfb0f1b5f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8df: skipped 000631 1788521477.824026 punch from 0x036642 to 0x03b8de (0x529d bytes) 000632 1788521477.833757 read 0x0265c6 thru 0x035136 (0xeb71 bytes) 000633 1788521477.854483 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000634 1788521478.101713 mapwrite 0x03ce46 thru 0x03f5c8 (0x2783 bytes) 000637 1788521478.321531 write 0x014ec0 thru 0x01a008 (0x5149 bytes) 000638 1788521478.348744 punch from 0x00ca59 to 0x0178d4 (0xae7c bytes) 000639 1788521478.373109 mapwrite 0x025f74 thru 0x02c632 (0x66bf bytes) 000641 1788521478.537429 trunc from 0x03f5c9 to 0x03206c (0xd55c bytes) 000642 1788521478.571780 mapread 0x025a20 thru 0x02a472 (0x4a53 bytes) 000643 1788521478.572735 mapread 0x027f59 thru 0x02db3e (0x5be6 bytes) 000646 1788521478.579549 mapwrite 0x0148da thru 0x020537 (0xbc5e bytes) 000649 1788521478.791284 mapread 0x003307 thru 0x011031 (0xdd2b bytes) 000650 1788521478.810396 trunc from 0x03206d to 0x0053ab (0x2ccc1 bytes) 000651 1788521478.841198 mapwrite 0x001731 thru 0x00357d (0x1e4d bytes) 000652 1788521478.996539 mapread 0x00064c thru 0x0021f5 (0x1baa bytes) 000653 1788521479.026989 punch from 0x004867 to 0x0053ab (0xb45 bytes) 000654 1788521479.036982 mapwrite 0x0325d9 thru 0x03932e (0x6d56 bytes) 000655 1788521479.220313 read 0x0124c1 thru 0x017d3d (0x587d bytes) 000656 1788521479.242021 mapread 0x003ae3 thru 0x00a771 (0x6c8f bytes) 000657 1788521479.261301 write 0x03e87b thru 0x03ffff (0x1785 bytes) 000659 1788521479.264685 read 0x0391a3 thru 0x03ffff (0x6e5d bytes) 000660 1788521479.275182 read 0x005547 thru 0x009531 (0x3feb bytes) 000661 1788521479.275724 mapread 0x008bfa thru 0x01077d (0x7b84 bytes) 000663 1788521479.277123 write 0x01a64e thru 0x026180 (0xbb33 bytes) 000665 1788521479.280044 punch from 0x024819 to 0x02fe78 (0xb660 bytes) 000667 1788521479.312541 read 0x01869d thru 0x01ec6d (0x65d1 bytes) 000668 1788521479.313593 punch from 0x02af6e to 0x03283a (0x78cd bytes) 000669 1788521479.322458 mapread 0x019d92 thru 0x028e5f (0xf0ce bytes) 000670 1788521479.334481 punch from 0x01f144 to 0x02b72a (0xc5e7 bytes) 000671 1788521479.341283 trunc from 0x040000 to 0x015700 (0x2a8ff bytes) 000674 1788521479.357827 mapread 0x01037b thru 0x0128e0 (0x2566 bytes) 000675 1788521479.358697 read 0x006ad2 thru 0x00fd76 (0x92a5 bytes) 000679 1788521479.361567 write 0x02034b thru 0x027b60 (0x7816 bytes) 000681 1788521479.365315 mapwrite 0x0250a1 thru 0x02dc12 (0x8b72 bytes) 000682 1788521479.468992 write 0x03362f thru 0x03e5c2 (0xaf94 bytes) 000684 1788521479.471765 punch from 0x029288 to 0x02d667 (0x43e0 bytes) 000685 1788521479.488614 write 0x01540b thru 0x0199b7 (0x45ad bytes) 000687 1788521479.490148 mapwrite 0x001e33 thru 0x009c0d (0x7ddb bytes) 000691 1788521479.568375 punch from 0x00eada to 0x019ad1 (0xaff8 bytes) 000693 1788521479.574599 read 0x004ced thru 0x00ad3a (0x604e bytes) 000695 1788521479.577015 mapwrite 0x03ca45 thru 0x03ffff (0x35bb bytes) 000697 1788521479.674894 trunc from 0x040000 to 0x03b582 (0x4a7d bytes) 000699 1788521479.701630 mapwrite 0x010f48 thru 0x011643 (0x6fc bytes) 000700 1788521479.761009 trunc from 0x03b583 to 0x03e5d1 (0x304f bytes) 000701 1788521479.798287 write 0x0164e5 thru 0x022f52 (0xca6e bytes) 000702 1788521479.800044 trunc from 0x03e5d2 to 0x010eab (0x2d726 bytes) 000703 1788521479.832207 read 0x00daa7 thru 0x010eab (0x3405 bytes) 000706 1788521479.839988 mapwrite 0x019d10 thru 0x0226a5 (0x8996 bytes) 000707 1788521480.013815 write 0x02c896 thru 0x035490 (0x8bfb bytes) 000708 1788521480.029390 MIRROR_EXTEND to 2 mirrors 000710 1788521481.591184 read 0x019ece thru 0x01e4f9 (0x462c bytes) 000711 1788521481.623344 mapwrite 0x031f97 thru 0x03df33 (0xbf9d bytes) 000713 1788521481.888497 trunc from 0x03df34 to 0x038053 (0x5ee0 bytes) 000714 1788521481.926819 trunc from 0x038054 to 0x038d56 (0xd03 bytes) 000715 1788521481.954385 write 0x012640 thru 0x01c9cf (0xa390 bytes) 000716 1788521481.973739 mapread 0x0164ac thru 0x01cf45 (0x6a9a bytes) 000717 1788521481.974968 mapread 0x01ef11 thru 0x021931 (0x2a21 bytes) 000718 1788521481.993035 trunc from 0x038d57 to 0x010505 (0x28851 bytes) 000720 1788521482.020181 read 0x0040d9 thru 0x00eb69 (0xaa91 bytes) 000721 1788521482.037585 read 0x008c52 thru 0x010505 (0x78b4 bytes) 000723 1788521482.038701 mapwrite 0x0292b0 thru 0x03447d (0xb1ce bytes) 000724 1788521482.158423 read 0x03174c thru 0x03447d (0x2d32 bytes) 000729 1788521482.162991 mapread 0x0289d2 thru 0x029085 (0x6b4 bytes) 000731 1788521482.165428 trunc from 0x03447e to 0x00637d (0x2e100 bytes) 000732 1788521482.183431 write 0x0285fe thru 0x0350ad (0xcab0 bytes) 000735 1788521482.188213 punch from 0x001f9d to 0x0108dc (0xe940 bytes) 000736 1788521482.204478 punch from 0x0016c8 to 0x00df19 (0xc852 bytes) 000737 1788521482.209940 mapread 0x008098 thru 0x00c89d (0x4806 bytes) 000740 1788521482.228579 punch from 0x003bca to 0x00fd55 (0xc18c bytes) 000742 1788521482.235966 mapread 0x028091 thru 0x02db2a (0x5a9a bytes) 000744 1788521482.238558 trunc from 0x0350ae to 0x00fd9c (0x25311 bytes) 000745 1788521482.264648 read 0x000c5c thru 0x00e235 (0xd5da bytes) 000748 1788521482.274973 mapread 0x0002e3 thru 0x00fd9c (0xfaba bytes) 000750 1788521482.276919 write 0x01f3e4 thru 0x022323 (0x2f40 bytes) 000755 1788521482.283918 write 0x023e86 thru 0x029e8b (0x6006 bytes) 000756 1788521482.286036 write 0x01d37a thru 0x02bc89 (0xe910 bytes) 000757 1788521482.306214 write 0x004c06 thru 0x0102ef (0xb6ea bytes) 000758 1788521482.313703 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000759 1788521482.478401 punch from 0x0228eb to 0x026e7a (0x4590 bytes) 000760 1788521482.484916 write 0x0181d5 thru 0x019bf0 (0x1a1c bytes) 000761 1788521482.499558 write 0x03d6e5 thru 0x03ffff (0x291b bytes) 000762 1788521482.501605 mapwrite 0x01bd45 thru 0x01f979 (0x3c35 bytes) 000763 1788521482.597984 write 0x013748 thru 0x017487 (0x3d40 bytes) 000765 1788521482.605785 mapwrite 0x018070 thru 0x021e61 (0x9df2 bytes) 000766 1788521482.659561 mapread 0x014e66 thru 0x0162d7 (0x1472 bytes) 000767 1788521482.660730 mapwrite 0x008310 thru 0x0132a4 (0xaf95 bytes) 000768 1788521482.751444 write 0x03ce7e thru 0x03d6f8 (0x87b bytes) 000769 1788521482.757631 punch from 0x016872 to 0x017822 (0xfb1 bytes) 000771 1788521482.774318 trunc from 0x040000 to 0x001cbf (0x3e340 bytes) 000772 1788521482.805331 punch from 0x0010ae to 0x001cbf (0xc12 bytes) 000774 1788521482.811048 punch from 0x0018c7 to 0x001cbf (0x3f9 bytes) 000775 1788521482.816458 mapwrite 0x037aeb thru 0x03ffff (0x8515 bytes) 000776 1788521482.916394 mapread 0x015b31 thru 0x01605b (0x52b bytes) 000778 1788521482.927595 punch from 0x015328 to 0x021628 (0xc301 bytes) 000779 1788521482.933226 punch from 0x033e8d to 0x0374d2 (0x3646 bytes) 000781 1788521482.939432 trunc from 0x040000 to 0x00a8e7 (0x35718 bytes) 000783 1788521482.970091 mapwrite 0x001a7b thru 0x007a16 (0x5f9c bytes) 000785 1788521483.050118 mapread 0x0006e7 thru 0x00a8e7 (0xa201 bytes) 000786 1788521483.069431 mapread 0x002661 thru 0x008aed (0x648d bytes) 000787 1788521483.071783 read 0x00886c thru 0x00a8e7 (0x207c bytes) 000788 1788521483.073934 write 0x01ac9d thru 0x0204d5 (0x5839 bytes) 000789 1788521483.076812 read 0x006b56 thru 0x0157bb (0xec66 bytes) 000790 1788521483.115425 trunc from 0x0204d6 to 0x0077d2 (0x18d03 bytes) 000791 1788521483.180365 read 0x00331b thru 0x0077d2 (0x44b8 bytes) 000792 1788521483.181773 punch from 0x005d61 to 0x0077d2 (0x1a72 bytes) 000795 1788521483.198639 write 0x027ee4 thru 0x035ddd (0xdefa bytes) 000796 1788521483.204712 punch from 0x028a84 to 0x02cb9a (0x4117 bytes) 000800 1788521483.233719 write 0x0256c5 thru 0x034474 (0xedb0 bytes) 000802 1788521483.252726 mapread 0x035046 thru 0x035ddd (0xd98 bytes) 000805 1788521483.260570 mapwrite 0x03e25b thru 0x03ffff (0x1da5 bytes) 000808 1788521483.452090 mapwrite 0x0276de thru 0x034b0d (0xd430 bytes) 000810 1788521483.581707 mapwrite 0x038c8e thru 0x03ffff (0x7372 bytes) 000811 1788521483.684209 trunc from 0x040000 to 0x035649 (0xa9b6 bytes) 000812 1788521483.730759 punch from 0x014e62 to 0x01a418 (0x55b7 bytes) 000814 1788521483.740629 punch from 0x00290b to 0x00729b (0x4991 bytes) 000816 1788521483.745735 read 0x0004d9 thru 0x008ed9 (0x8a01 bytes) 000817 1788521483.755479 write 0x034380 thru 0x03e9dc (0xa65d bytes) 000818 1788521483.759464 write 0x02e0d5 thru 0x03ac51 (0xcb7d bytes) 000819 1788521483.760802 read 0x036897 thru 0x03e9dc (0x8146 bytes) 000820 1788521483.762137 read 0x021840 thru 0x026ddf (0x55a0 bytes) 000821 1788521483.762874 read 0x0230ad thru 0x027107 (0x405b bytes) 000823 1788521483.764142 mapread 0x009d17 thru 0x0104e3 (0x67cd bytes) 000826 1788521483.766798 mapwrite 0x028a60 thru 0x0291d1 (0x772 bytes) 000829 1788521483.823713 punch from 0x03d9cf to 0x03e9dc (0x100e bytes) 000831 1788521483.837156 read 0x03a033 thru 0x03ac26 (0xbf4 bytes) 000832 1788521483.837723 punch from 0x01236b to 0x018f38 (0x6bce bytes) 000835 1788521483.854440 punch from 0x002f16 to 0x006e70 (0x3f5b bytes) 000839 1788521483.864924 mapwrite 0x00ed4b thru 0x016585 (0x783b bytes) 000840 1788521484.012980 mapwrite 0x0175bb thru 0x0179ab (0x3f1 bytes) 000841 1788521484.131045 write 0x0168b3 thru 0x01e4d4 (0x7c22 bytes) 000843 1788521484.141617 mapread 0x02f158 thru 0x0351d5 (0x607e bytes) 000844 1788521484.146931 punch from 0x01712f to 0x020e23 (0x9cf5 bytes) 000845 1788521484.174245 read 0x023c4d thru 0x02f371 (0xb725 bytes) 000846 1788521484.175014 mapwrite 0x00ee77 thru 0x010492 (0x161c bytes) 000847 1788521484.250027 mapread 0x0279ea thru 0x02eb53 (0x716a bytes) 000849 1788521484.251467 mapwrite 0x022b1f thru 0x02ab74 (0x8056 bytes) 000850 1788521484.313110 mapwrite 0x0367da thru 0x03845a (0x1c81 bytes) 000851 1788521484.418575 MIRROR_EXTEND to 2 mirrors 000852 1788521485.722482 read 0x03ca6c thru 0x03e9dc (0x1f71 bytes) 000855 1788521485.791416 write 0x00c828 thru 0x019de3 (0xd5bc bytes) 000857 1788521485.912521 read 0x01ca58 thru 0x02ba95 (0xf03e bytes) 000858 1788521485.936519 mapwrite 0x02d537 thru 0x034883 (0x734d bytes) 000859 1788521486.029693 trunc from 0x03e9dd to 0x0109cb (0x2e011 bytes) 000860 1788521486.077963 mapwrite 0x030672 thru 0x037b92 (0x7521 bytes) 000861 1788521486.293552 mapwrite 0x027739 thru 0x02f049 (0x7911 bytes) 000862 1788521486.491822 mapwrite 0x027789 thru 0x02be49 (0x46c1 bytes) 000863 1788521486.594368 MIRROR_RESYNC 2 mirrors 000864 1788521488.339430 trunc from 0x037b93 to 0x033739 (0x4459 bytes) 000867 1788521488.431816 MIRROR_RESYNC 2 mirrors 000868 1788521489.924785 read 0x00f3aa thru 0x0162d1 (0x6f28 bytes) 000869 1788521489.945166 read 0x024b97 thru 0x033739 (0xeba3 bytes) 000870 1788521490.021501 trunc from 0x03373a to 0x00ceb0 (0x26889 bytes) 000871 1788521490.120182 trunc from 0x00ceb1 to 0x0353b9 (0x28509 bytes) 000872 1788521490.149568 mapread 0x01b2fa thru 0x027662 (0xc369 bytes) 000873 1788521490.245382 punch from 0x01f6fd to 0x0279bb (0x82bf bytes) 000874 1788521490.250715 mapwrite 0x02350a thru 0x025580 (0x2077 bytes) 000877 1788521490.332986 write 0x0129c4 thru 0x01a4f3 (0x7b30 bytes) 000878 1788521490.344924 write 0x025259 thru 0x034302 (0xf0aa bytes) 000879 1788521490.347466 mapwrite 0x025f90 thru 0x02bb60 (0x5bd1 bytes) 000880 1788521490.406282 write 0x03810f thru 0x03ffff (0x7ef1 bytes) 000881 1788521490.407734 trunc from 0x040000 to 0x0223b9 (0x1dc46 bytes) 000883 1788521490.464121 mapwrite 0x0225ab thru 0x0275f4 (0x504a bytes) 000884 1788521490.570078 read 0x01d2eb thru 0x02229d (0x4fb3 bytes) 000885 1788521490.581575 mapwrite 0x034676 thru 0x03a539 (0x5ec4 bytes) 000887 1788521490.684102 mapread 0x01cea3 thru 0x024fb2 (0x8110 bytes) 000888 1788521490.685498 mapread 0x0020a8 thru 0x00bb90 (0x9ae9 bytes) 000890 1788521490.688577 mapread 0x02f718 thru 0x03a539 (0xae22 bytes) 000891 1788521490.699050 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000892 1788521490.871757 MIRROR_EXTEND to 2 mirrors 000895 1788521492.275720 read 0x015d38 thru 0x02418d (0xe456 bytes) 000896 1788521492.293500 punch from 0x02fa34 to 0x037dd4 (0x83a1 bytes) 000897 1788521492.414499 punch from 0x023efa to 0x024911 (0xa18 bytes) 000899 1788521492.448957 read 0x014112 thru 0x01fa3b (0xb92a bytes) 000902 1788521492.495620 trunc from 0x03a53a to 0x0112ca (0x2926f bytes) 000904 1788521492.536171 read 0x00205d thru 0x00c8aa (0xa84e bytes) 000905 1788521492.537500 mapwrite 0x03d72b thru 0x03ffff (0x28d5 bytes) 000906 1788521492.661933 read 0x00b0c6 thru 0x01446f (0x93aa bytes) 000907 1788521492.663211 read 0x005ccd thru 0x00e130 (0x8464 bytes) 000909 1788521492.665411 write 0x011b43 thru 0x0142b3 (0x2771 bytes) 000910 1788521492.667403 mapread 0x00ff90 thru 0x01bfff (0xc070 bytes) 000912 1788521492.671114 mapread 0x027bb3 thru 0x0345fb (0xca49 bytes) 000913 1788521492.673007 mapwrite 0x02b143 thru 0x036bcb (0xba89 bytes) 000914 1788521492.751669 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x302e7b6f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3616f668 Mirror 3: 0x3616f668 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000915 1788521494.253641 trunc from 0x040000 to 0x03d862 (0x279d bytes) 000916 1788521494.435344 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d863): Mirror 3: 0xd9f6090f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d863): Mirror 3: 0xd9f6090f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d863: skipped 000917 1788521494.838890 trunc from 0x03d863 to 0x03d10c (0x756 bytes) 000918 1788521494.868339 write 0x00a1fb thru 0x00c752 (0x2558 bytes) 000919 1788521494.888583 punch from 0x01d210 to 0x024ff0 (0x7de1 bytes) 000920 1788521494.902181 read 0x01ab68 thru 0x026724 (0xbbbd bytes) 000921 1788521494.915455 mapwrite 0x007911 thru 0x01152d (0x9c1d bytes) 000922 1788521495.051642 mapread 0x030809 thru 0x0340c2 (0x38ba bytes) 000923 1788521495.067132 trunc from 0x03d10d to 0x0204cb (0x1cc41 bytes) 000925 1788521495.089767 read 0x0139e6 thru 0x013ce1 (0x2fc bytes) 000926 1788521495.104347 write 0x024de4 thru 0x0307be (0xb9db bytes) 000928 1788521495.107711 trunc from 0x0307bf to 0x01b30b (0x154b3 bytes) 000929 1788521495.130178 punch from 0x00f95c to 0x0103f1 (0xa96 bytes) 000931 1788521495.135587 punch from 0x012d26 to 0x01b30b (0x85e6 bytes) 000932 1788521495.142590 write 0x03b2b3 thru 0x03ffff (0x4d4d bytes) 000934 1788521495.145734 trunc from 0x040000 to 0x007919 (0x386e6 bytes) 000935 1788521495.175323 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000936 1788521495.395754 trunc from 0x00791a to 0x02a9bb (0x230a2 bytes) 000941 1788521495.432383 mapread 0x0155d3 thru 0x020d4b (0xb779 bytes) 000942 1788521495.473434 trunc from 0x02a9bc to 0x03258f (0x7bd4 bytes) 000943 1788521495.496918 punch from 0x01f230 to 0x01fa3e (0x80f bytes) 000945 1788521495.504608 trunc from 0x032590 to 0x01fe15 (0x1277a bytes) 000947 1788521495.524408 trunc from 0x01fe16 to 0x00981c (0x165f9 bytes) 000950 1788521495.544573 write 0x02e902 thru 0x036a3c (0x813b bytes) 000951 1788521495.547407 trunc from 0x036a3d to 0x00a7ac (0x2c290 bytes) 000954 1788521495.573969 write 0x03dbe3 thru 0x03ffff (0x241d bytes) 000957 1788521495.576955 mapwrite 0x00a59a thru 0x0137be (0x9225 bytes) 000958 1788521495.652759 mapwrite 0x012de8 thru 0x0213b5 (0xe5ce bytes) 000959 1788521495.814021 mapwrite 0x01924c thru 0x01e7d7 (0x558c bytes) 000960 1788521495.883467 read 0x01edc0 thru 0x020b7c (0x1dbd bytes) 000962 1788521495.886676 mapwrite 0x025a49 thru 0x03025e (0xa816 bytes) 000964 1788521496.099342 mapread 0x03b48c thru 0x03fd59 (0x48ce bytes) 000965 1788521496.127144 punch from 0x021aa4 to 0x02e61e (0xcb7b bytes) 000968 1788521496.142021 punch from 0x026c1c to 0x02c90e (0x5cf3 bytes) 000970 1788521496.150922 mapread 0x007259 thru 0x011be2 (0xa98a bytes) 000971 1788521496.153316 read 0x03aefa thru 0x03ffff (0x5106 bytes) 000972 1788521496.159057 read 0x0162c1 thru 0x017816 (0x1556 bytes) 000973 1788521496.160055 mapread 0x0341f2 thru 0x03ffff (0xbe0e bytes) 000977 1788521496.175192 trunc from 0x040000 to 0x022e7c (0x1d183 bytes) 000978 1788521496.200735 write 0x0182b8 thru 0x01a05f (0x1da8 bytes) 000979 1788521496.202452 mapwrite 0x033cc9 thru 0x03eb6d (0xaea5 bytes) 000980 1788521496.362329 punch from 0x029955 to 0x035bfa (0xc2a6 bytes) 000981 1788521496.369169 mapread 0x037075 thru 0x03eb6d (0x7af9 bytes) 000982 1788521496.371072 mapread 0x0291b6 thru 0x036c53 (0xda9e bytes) 000983 1788521496.477408 write 0x01246d thru 0x01f891 (0xd425 bytes) 000984 1788521496.479435 mapwrite 0x02be9d thru 0x02d6ed (0x1851 bytes) 000985 1788521496.530477 read 0x0133b6 thru 0x016c3c (0x3887 bytes) 000987 1788521496.531536 mapwrite 0x023a60 thru 0x0242e8 (0x889 bytes) 000988 1788521496.585823 mapread 0x00edee thru 0x0134a6 (0x46b9 bytes) 000989 1788521496.587798 trunc from 0x03eb6e to 0x030797 (0xe3d6 bytes) 000990 1788521496.617372 read 0x00fe56 thru 0x011bc3 (0x1d6e bytes) 000991 1788521496.619173 trunc from 0x030798 to 0x01a5e2 (0x161b5 bytes) 000992 1788521496.635881 punch from 0x014d61 to 0x0182b0 (0x3550 bytes) 000994 1788521496.643592 trunc from 0x01a5e3 to 0x00875e (0x11e84 bytes) 000996 1788521496.660642 punch from 0x0002a4 to 0x00875e (0x84bb bytes) 000997 1788521496.668505 mapread 0x003866 thru 0x00875e (0x4ef9 bytes) 000998 1788521496.679149 read 0x006890 thru 0x00875e (0x1ecf bytes) 000999 1788521496.679937 trunc from 0x00875f to 0x025cc0 (0x1d562 bytes) 001000 1788521496.695453 trunc from 0x025cc1 to 0x03d327 (0x17667 bytes) All operations completed A-OK! PASS 70a (87s) == sanity-flr test 71: check mirror extend parent fid ==== 07:31:43 (1788521503) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 07:31:47 (1788521507) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.387666 s, 5.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361247 s, 2.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x279:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27b:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 07:31:59 (1788521519) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5687136 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3484960 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8515584 .. Extending file size to 5312224 .. Extending file size to 7753280 .. Extending file size to 8169696 .. Extending file size to 8547552 .. Extending file size to 9055296 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57205 57206 57209 57210 57211 failed 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) 192.168.204.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:33:23 (1788521603) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3132544 .. Extending file size to 3990176 .. failed Extending file size to 7213600 .. Extending file size to 4921024 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7024960 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7877952 .. Extending file size to 7752320 .. Extending file size to 8533376 .. Extending file size to 8700128 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8883328 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58601 58602 58603 58605 58606 done 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) 192.168.204.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:34:45 (1788521685) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228852 s, 9.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.445853 s, 4.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xbc6ae06c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:35:10 (1788521710) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0913081 s, 23.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 07:35:18 (1788521718) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:35:26 (1788521726) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247401 s, 8.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x292:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:35:36 (1788521736) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.410356 s, 5.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x298:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x293:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x298:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x293:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x294:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:35:46 (1788521746) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.342049 s, 6.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x299:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x295:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x299:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x295:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x296:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:36:21 (1788521781) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177363 s, 5.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x298:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x298:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29b:0x0] } PASS 204d (67s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:37:28 (1788521848) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x299:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.312655 s, 6.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x299:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x299:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:38:29 (1788521909) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. Creating new pool test_204f oleg432-server: Pool lustre.test_204f created Adding targets to pool oleg432-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg432-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229134 s, 9.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg432-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg432-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg432-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:39:54 (1788521994) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:40:02 (1788522002) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:40:10 (1788522010) oleg432-server: Pool lustre.test_206 created Adding targets to pool oleg432-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg432-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg432-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg432-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg432-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:40:38 (1788522038) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932762 s, 11.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055755 s, 18.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe5ee67f4 Mirror 3: 0xe5ee67f4 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:40:50 (1788522050) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:40:53 (1788522053) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:40:57 (1788522057) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.168611 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2410000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1230000 Read this (4 bytes): BBBB 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:41:36 (1788522096) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.020645 s, 397 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:41:44 (1788522104) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068018 s, 15.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c3 lmm_fid: [0x200000404:0x1c3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c3 lmm_fid: [0x200000404:0x1c3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834371 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 07:41:46.000000000 -0400 Modify: 2026-09-04 07:41:46.000000000 -0400 Change: 2026-09-04 07:41:48.000000000 -0400 Birth: 2026-09-04 07:41:47.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:41:54 (1788522114) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:41:58 (1788522118) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.710123 s, 57.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0724409 s, 56.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 07:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 07:42 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:42:09 (1788522129) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:42:18 (1788522138) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1966 sec =========== 07:42:29 (1788522149) === sanity-flr: start cleanup 07:42:31 (1788522151) === === sanity-flr: finish cleanup 07:42:39 (1788522159) ===