-----============= acceptance-small: sanity-flr ============----- Tue Apr 28 20:42:26 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg143-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:42:33 (1777423353) === oleg143-client.virtnet: executing check_config_client /mnt/lustre oleg143-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg143-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99568a8d6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99568a8d6800.idle_timeout=debug disable quota as required oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:42:46 (1777423366) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7562] == sanity-flr test 0a: lfs mirror create with -N option == 20:42:47 (1777423367) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:42:53 (1777423373) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:42:54 (1777423374) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:42:56 (1777423376) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:43:04 (1777423384) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:43:05 (1777423385) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 20:43:06 (1777423386) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99568a8d6800.stats=clear osc.lustre-OST0001-osc-ffff99568a8d6800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:43:09 (1777423389) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:43:13 (1777423393) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:43:16 (1777423396) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:43:17 (1777423397) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 20:43:20 (1777423400) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:43:23 (1777423403) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:43:24 (1777423404) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:43:27 (1777423407) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:43:31 (1777423411) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:43:33 (1777423413) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.751731 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648092 s, 16.2 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:43:37 (1777423417) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.17515 s, 114 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:43:41 (1777423421) Stopping /mnt/lustre-ost1 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99568a8d6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99568a8d6800.ost_server_uuid in DISCONN state after 3 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg143-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99568a8d6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99568a8d6800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg143-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (31s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:44:11 (1777423451) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0580038 s, 54.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99568a8d6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99568a8d6800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg143-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:44:26 (1777423466) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99568a8d6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99568a8d6800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg143-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99568a8d6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99568a8d6800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg143-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:44:52 (1777423492) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2334 s, 42.5 MB/s -rw-r--r-- 1 root root 52428800 Apr 28 20:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.315s user 0m0.001s sys 0m0.144s Stopping /mnt/lustre-ost1 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99568a8d6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99568a8d6800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.275s user 0m0.001s sys 0m0.143s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg143-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg143-server oleg143-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99568a8d6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99568a8d6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.392s user 0m0.001s sys 0m0.151s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg143-server: oleg143-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg143-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg143-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99568a8d6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99568a8d6800.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:45:20 (1777423520) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:45:21 (1777423521) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:45:22 (1777423522) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 20:45:23 (1777423523) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 20:45:26 (1777423526) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.162878 s, 77.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:45:30 (1777423530) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00986721 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122725 s, 85.4 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:45:32 (1777423532) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:45:36 (1777423536) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000669284 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000612917 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000713486 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00070498 s, 1.5 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 20:45:39 (1777423539) create test files with size 15745467 .. 1b43b09a98d20a9df0f40c8eade86a52 - 80a6be51abda8d9f2492e5a500f661d7 - b76be333a941d0ad97b3038b177ede61 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg143-client.virtnet /mnt/lustre (opts:) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 20:45:45 (1777423545) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:45:49 (1777423549) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:45:51 (1777423551) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334997 s, 31.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113525 s, 92.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 20:45:54 (1777423554) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0439606 s, 47.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.113458 s, 37.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0603022 s, 34.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 20:45:59 (1777423559) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 20:46:00 (1777423560) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:46:01 (1777423561) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012708 s, 82.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 20:46:04 (1777423564) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:46:05 (1777423565) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:46:08 (1777423568) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0895329 s, 117 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg143-client.virtnet /mnt/lustre (opts:) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:46:28 (1777423588) llite.lustre-ffff9956aa7a8000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0899616 s, 117 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg143-client.virtnet /mnt/lustre (opts:) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff995684efe800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:46:32 (1777423592) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:46:43 (1777423603) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:46:54 (1777423614) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 20:46:55 (1777423615) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0642383 s, 49.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0516917 s, 60.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4d8afeb7 (stale) Mirror 2: 0x4d8afeb7 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 67010b49711904722ced53cf092910f1 - mirror 1 checksum: 8f43ea79aa5e2a2be317927ea5acb40a - mirror 2 checksum: 67010b49711904722ced53cf092910f1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 28 20:46:55 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Apr 28 20:46:55 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 67010b49711904722ced53cf092910f1 - mirror 1 checksum: 8f43ea79aa5e2a2be317927ea5acb40a - mirror 2 checksum: 8f43ea79aa5e2a2be317927ea5acb40a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 28 20:46:55 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Apr 28 20:46:55 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (2s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:46:58 (1777423618) Failing mds1 on oleg143-server Stopping /mnt/lustre-mds1 (opts:) on oleg143-server 20:47:00 (1777423620) shut down facet: mds1 facet_host: oleg143-server facet_failover_host: oleg143-server Failover mds1 to oleg143-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 20:47:14 (1777423634) targets are mounted 20:47:14 (1777423634) facet_failover done oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113651 s, 92.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:47:20 (1777423640) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:47:21 (1777423641) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:47:22 (1777423642) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00230031 s, 7.1 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:47:24 (1777423644) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:47:25 (1777423645) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:47:26 (1777423646) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:47:33 (1777423653) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:47:52 (1777423672) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:48:09 (1777423689) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:48:27 (1777423707) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 20:48:29 (1777423709) mirror create pid 52280 mirror split pid 52281 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52280 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52281 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 20:49:32 (1777423772) Seed set to 3772 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005 1777423772.925986 write 0x02a2eb thru 0x03490b (0xa621 bytes) 000007 1777423772.928757 read 0x011c62 thru 0x021ab7 (0xfe56 bytes) 000008 1777423772.934304 write 0x01b683 thru 0x02943c (0xddba bytes) 000010 1777423772.938306 write 0x013f4d thru 0x01cebe (0x8f72 bytes) 000011 1777423772.939561 read 0x023afd thru 0x02fcc7 (0xc1cb bytes) 000016 1777423772.942091 mapread 0x016c16 thru 0x01c923 (0x5d0e bytes) 000019 1777423772.944537 write 0x01c2a4 thru 0x024206 (0x7f63 bytes) 000022 1777423772.946697 mapwrite 0x028f4a thru 0x02e1d8 (0x528f bytes) 000023 1777423772.977866 read 0x003fa4 thru 0x008501 (0x455e bytes) truncating to largest ever: 0x3b4d9 000025 1777423772.983664 trunc from 0x03490c to 0x03b4d8 (0x6bcd bytes) 000028 1777423772.994481 mapwrite 0x03f7e8 thru 0x03ffff (0x818 bytes) 000030 1777423773.017321 write 0x008fff thru 0x00f0f1 (0x60f3 bytes) 000032 1777423773.018864 mapread 0x006eca thru 0x00b20c (0x4343 bytes) 000034 1777423773.020297 write 0x03b8b1 thru 0x03ffff (0x474f bytes) 000036 1777423773.021717 trunc from 0x040000 to 0x02353e (0x1cac1 bytes) 000039 1777423773.029280 mapread 0x009d1f thru 0x017695 (0xd977 bytes) 000040 1777423773.030147 trunc from 0x02353f to 0x00a25e (0x192e0 bytes) 000043 1777423773.037026 mapwrite 0x01474a thru 0x01afbc (0x6873 bytes) 000044 1777423773.099989 mapwrite 0x009c34 thru 0x015cf5 (0xc0c2 bytes) 000045 1777423773.156684 mapread 0x013e47 thru 0x017ca3 (0x3e5d bytes) 000046 1777423773.157367 read 0x01635b thru 0x01afbc (0x4c62 bytes) 000047 1777423773.157772 trunc from 0x01afbd to 0x0122cd (0x8cef bytes) 000048 1777423773.165456 write 0x013311 thru 0x01e0f3 (0xade3 bytes) 000049 1777423773.166996 write 0x03992c thru 0x03ffff (0x66d4 bytes) 000051 1777423773.168662 mapwrite 0x000c3e thru 0x00ae01 (0xa1c4 bytes) 000055 1777423773.190480 read 0x02f64d thru 0x035341 (0x5cf5 bytes) 000058 1777423773.196603 mapwrite 0x036737 thru 0x03fdb3 (0x967d bytes) 000062 1777423773.230207 mapwrite 0x03c5d0 thru 0x03ffff (0x3a30 bytes) 000063 1777423773.247896 mapwrite 0x02b2da thru 0x0388b2 (0xd5d9 bytes) 000067 1777423773.267819 mapwrite 0x03794b thru 0x038068 (0x71e bytes) 000070 1777423773.291083 read 0x02ee2d thru 0x036a4b (0x7c1f bytes) 000072 1777423773.292392 write 0x019621 thru 0x01beaf (0x288f bytes) 000074 1777423773.294202 read 0x0078fb thru 0x00ee1a (0x7520 bytes) 000075 1777423773.294698 write 0x01bfe0 thru 0x01fbcc (0x3bed bytes) 000079 1777423773.297122 write 0x001cab thru 0x00470c (0x2a62 bytes) 000080 1777423773.298111 read 0x01c079 thru 0x01f64b (0x35d3 bytes) 000083 1777423773.299373 read 0x010ab7 thru 0x01601c (0x5566 bytes) 000084 1777423773.299923 mapread 0x02d862 thru 0x035ded (0x858c bytes) 000087 1777423773.301922 read 0x03696d thru 0x03ffff (0x9693 bytes) 000088 1777423773.302558 read 0x000f85 thru 0x00e40e (0xd48a bytes) 000091 1777423773.303840 write 0x017322 thru 0x01edfb (0x7ada bytes) truncating to largest ever: 0x3b929 000093 1777423773.305465 trunc from 0x040000 to 0x03b928 (0x46d7 bytes) 000094 1777423773.312656 write 0x03de5d thru 0x03ffff (0x21a3 bytes) 000099 1777423773.316166 read 0x008cf7 thru 0x012bb0 (0x9eba bytes) 000103 1777423773.318743 read 0x03fd56 thru 0x03ffff (0x2aa bytes) 000105 1777423773.319904 trunc from 0x040000 to 0x0050fc (0x3af03 bytes) 000106 1777423773.329655 MIRROR_EXTEND to 2 mirrors 000107 1777423773.574703 MIRROR_EXTEND to 3 mirrors 000108 1777423773.831523 mapread 0x004b43 thru 0x0050fc (0x5ba bytes) 000109 1777423773.848605 write 0x01e330 thru 0x02c9fa (0xe6cb bytes) 000111 1777423773.889654 read 0x01964b thru 0x01d487 (0x3e3d bytes) 000112 1777423773.893932 read 0x00c9ea thru 0x01c3b5 (0xf9cc bytes) 000115 1777423773.900241 trunc from 0x02c9fb to 0x007376 (0x25684 bytes) 000116 1777423773.919487 trunc from 0x007377 to 0x018020 (0x10caa bytes) 000117 1777423773.925669 write 0x00bb9d thru 0x00e362 (0x27c6 bytes) 000120 1777423773.930876 write 0x005977 thru 0x00919a (0x3824 bytes) 000122 1777423773.940964 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18021): Mirror 3: 0xf1dc1b5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18021): Mirror 3: 0xf1dc1b5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18021: skipped 000127 1777423774.089182 write 0x026255 thru 0x033111 (0xcebd bytes) 000128 1777423774.090724 read 0x0197b7 thru 0x0231c9 (0x9a13 bytes) 000132 1777423774.098824 mapwrite 0x021ea2 thru 0x02ffcd (0xe12c bytes) 000135 1777423774.134022 write 0x03528a thru 0x03ff67 (0xacde bytes) 000136 1777423774.135282 mapwrite 0x00a6a0 thru 0x012b91 (0x84f2 bytes) 000139 1777423774.159536 write 0x01bedc thru 0x01f33b (0x3460 bytes) 000142 1777423774.161802 write 0x00f5ef thru 0x01eeb3 (0xf8c5 bytes) 000143 1777423774.163215 mapwrite 0x010a8b thru 0x01c755 (0xbccb bytes) 000145 1777423774.193631 write 0x032191 thru 0x038e25 (0x6c95 bytes) 000146 1777423774.194774 write 0x02d783 thru 0x02e7e5 (0x1063 bytes) 000147 1777423774.196175 read 0x03772d thru 0x03ff67 (0x883b bytes) 000148 1777423774.196943 write 0x03ee46 thru 0x03ffff (0x11ba bytes) 000149 1777423774.197846 trunc from 0x040000 to 0x00acc8 (0x35337 bytes) 000152 1777423774.215408 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xacc9): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xacc9): Mirror 3: 0x41be7453 (stale) CRC-32 checksum value for chunk [0, 0xacc9): Mirror 1: 0x4b6d8282 Mirror 4: 0x4b6d8282 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xacc9: skipped 000162 1777423774.530262 write 0x00aea0 thru 0x014889 (0x99ea bytes) 000163 1777423774.560161 trunc from 0x01488a to 0x01e149 (0x98c0 bytes) 000166 1777423774.581732 MIRROR_RESYNC 3 mirrors 000167 1777423774.876444 MIRROR_EXTEND to 4 mirrors 000168 1777423775.096977 write 0x016b71 thru 0x01b23f (0x46cf bytes) 000169 1777423775.129276 mapwrite 0x02e79b thru 0x02f2ea (0xb50 bytes) 000170 1777423775.178616 write 0x0009e3 thru 0x00f66c (0xec8a bytes) 000172 1777423775.183319 trunc from 0x02f2eb to 0x00b6dd (0x23c0d bytes) 000174 1777423775.192174 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb6de): Mirror 3: 0xf645f84a (stale) Mirror 5: 0xf645f84a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb6de): Mirror 3: 0xf645f84a (stale) Mirror 5: 0xf645f84a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb6de: skipped 000175 1777423775.348181 MIRROR_RESYNC 3 mirrors 000176 1777423775.596210 trunc from 0x00b6de to 0x018ceb (0xd60e bytes) 000178 1777423775.615712 read 0x0083e4 thru 0x00dd0c (0x5929 bytes) 000181 1777423775.621441 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18cec): Mirror 3: 0xae20cc44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18cec): Mirror 3: 0xae20cc44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18cec: skipped 000182 1777423775.714871 read 0x00f682 thru 0x0176fe (0x807d bytes) 000183 1777423775.719467 trunc from 0x018cec to 0x03668a (0x1d99f bytes) 000184 1777423775.727962 mapwrite 0x01b96d thru 0x020618 (0x4cac bytes) 000185 1777423775.755756 trunc from 0x03668b to 0x006cd4 (0x2f9b6 bytes) 000186 1777423775.763739 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000187 1777423775.810775 mapwrite 0x003536 thru 0x004582 (0x104d bytes) 000190 1777423775.833916 mapread 0x002a96 thru 0x006cd4 (0x423f bytes) 000193 1777423775.842595 mapread 0x00501f thru 0x006cd4 (0x1cb6 bytes) 000194 1777423775.843383 trunc from 0x006cd5 to 0x02dbc2 (0x26eee bytes) 000195 1777423775.850875 trunc from 0x02dbc3 to 0x022757 (0xb46b bytes) 000197 1777423775.859814 mapwrite 0x03940c thru 0x03ffff (0x6bf4 bytes) 000199 1777423775.883084 write 0x020b7d thru 0x02ad97 (0xa21b bytes) 000200 1777423775.891214 MIRROR_EXTEND to 2 mirrors 000201 1777423776.119549 read 0x03aa1f thru 0x03ffff (0x55e1 bytes) 000203 1777423776.127838 write 0x01d893 thru 0x02c464 (0xebd2 bytes) 000204 1777423776.155241 read 0x03336a thru 0x03b258 (0x7eef bytes) 000206 1777423776.160809 trunc from 0x040000 to 0x012fdf (0x2d020 bytes) 000208 1777423776.169675 MIRROR_RESYNC 2 mirrors 000209 1777423776.355254 trunc from 0x012fe0 to 0x0378b1 (0x248d2 bytes) 000210 1777423776.382280 mapread 0x01949c thru 0x01e79b (0x5300 bytes) 000211 1777423776.389045 trunc from 0x0378b2 to 0x031e6c (0x5a45 bytes) 000212 1777423776.396523 mapread 0x01b876 thru 0x01e084 (0x280f bytes) 000213 1777423776.397513 mapwrite 0x016ba2 thru 0x01a55a (0x39b9 bytes) 000214 1777423776.412932 write 0x0010ca thru 0x00569a (0x45d1 bytes) 000215 1777423776.416664 trunc from 0x031e6d to 0x028415 (0x9a57 bytes) 000219 1777423776.427459 read 0x021564 thru 0x028415 (0x6eb2 bytes) 000220 1777423776.431007 write 0x0106d3 thru 0x0140ec (0x3a1a bytes) 000221 1777423776.433886 mapwrite 0x03d446 thru 0x03ffff (0x2bba bytes) 000225 1777423776.459857 write 0x019ac1 thru 0x01eafc (0x503c bytes) 000226 1777423776.463747 MIRROR_RESYNC 2 mirrors 000230 1777423776.704578 read 0x0310ce thru 0x036ec0 (0x5df3 bytes) 000232 1777423776.709015 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000234 1777423776.765966 write 0x02af9b thru 0x036cca (0xbd30 bytes) 000235 1777423776.774933 mapwrite 0x009fcd thru 0x018b99 (0xebcd bytes) 000236 1777423776.807532 MIRROR_EXTEND to 2 mirrors 000237 1777423777.029885 trunc from 0x040000 to 0x0397d0 (0x682f bytes) 000238 1777423777.060239 trunc from 0x0397d1 to 0x0366b1 (0x311f bytes) 000239 1777423777.068726 MIRROR_RESYNC 2 mirrors 000240 1777423777.288503 write 0x00a06c thru 0x00cb96 (0x2b2b bytes) 000241 1777423777.315966 read 0x02216a thru 0x02ea83 (0xc91a bytes) 000243 1777423777.323922 mapwrite 0x020920 thru 0x025a82 (0x5163 bytes) 000246 1777423777.356832 mapread 0x00f0ea thru 0x017a79 (0x8990 bytes) 000247 1777423777.362630 mapread 0x014e96 thru 0x02475e (0xf8c9 bytes) 000248 1777423777.366731 write 0x03d30a thru 0x03ffff (0x2cf6 bytes) 000249 1777423777.367798 mapwrite 0x02e920 thru 0x032fa3 (0x4684 bytes) 000250 1777423777.395252 write 0x00d18d thru 0x019081 (0xbef5 bytes) 000251 1777423777.396221 mapread 0x00d0b3 thru 0x00d609 (0x557 bytes) 000254 1777423777.397806 mapwrite 0x033720 thru 0x03b7eb (0x80cc bytes) 000256 1777423777.419021 mapread 0x031099 thru 0x033e14 (0x2d7c bytes) 000257 1777423777.419997 read 0x004f12 thru 0x0052c6 (0x3b5 bytes) 000262 1777423777.425829 mapwrite 0x03ba02 thru 0x03ffff (0x45fe bytes) 000263 1777423777.441702 trunc from 0x040000 to 0x00b9d4 (0x3462b bytes) 000264 1777423777.452703 mapwrite 0x038288 thru 0x03fb5a (0x78d3 bytes) 000265 1777423777.493281 read 0x03a75c thru 0x03fb5a (0x53ff bytes) 000267 1777423777.494211 trunc from 0x03fb5b to 0x00bb1c (0x3403e bytes) 000268 1777423777.499820 mapwrite 0x003b0c thru 0x011f52 (0xe447 bytes) 000269 1777423777.522655 mapwrite 0x008ee5 thru 0x00d8ee (0x4a0a bytes) 000271 1777423777.539073 mapwrite 0x01038f thru 0x01b283 (0xaef5 bytes) 000273 1777423777.569453 write 0x005632 thru 0x014a25 (0xf3f4 bytes) 000274 1777423777.570995 mapread 0x0119ef thru 0x01b283 (0x9895 bytes) 000275 1777423777.572275 write 0x0247f1 thru 0x02b42c (0x6c3c bytes) 000276 1777423777.573587 read 0x018319 thru 0x01befb (0x3be3 bytes) 000277 1777423777.574219 mapwrite 0x03dd1d thru 0x03ffff (0x22e3 bytes) 000279 1777423777.613387 mapread 0x02d567 thru 0x035d7f (0x8819 bytes) 000280 1777423777.614228 mapwrite 0x009364 thru 0x0147bf (0xb45c bytes) 000281 1777423777.632313 write 0x020773 thru 0x026eae (0x673c bytes) 000282 1777423777.633489 mapwrite 0x036c87 thru 0x03ffff (0x9379 bytes) 000283 1777423777.650772 read 0x02a181 thru 0x034d70 (0xabf0 bytes) 000286 1777423777.652513 mapwrite 0x02ecd0 thru 0x035a77 (0x6da8 bytes) 000288 1777423777.674930 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x92d0a838 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xafc458c3 Mirror 3: 0xafc458c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000290 1777423777.948033 write 0x016080 thru 0x01939f (0x3320 bytes) 000293 1777423777.977988 mapwrite 0x00537d thru 0x010bd5 (0xb859 bytes) 000296 1777423778.006158 MIRROR_RESYNC 3 mirrors 000297 1777423778.255681 mapwrite 0x017c94 thru 0x026324 (0xe691 bytes) 000299 1777423778.330121 trunc from 0x040000 to 0x0286ff (0x17900 bytes) 000300 1777423778.341130 write 0x01c3be thru 0x02849d (0xc0e0 bytes) 000302 1777423778.342668 trunc from 0x028700 to 0x010483 (0x1827c bytes) 000304 1777423778.351445 mapread 0x00e042 thru 0x00e3d7 (0x396 bytes) 000308 1777423778.358427 read 0x005876 thru 0x010483 (0xac0e bytes) 000311 1777423778.362231 trunc from 0x010484 to 0x00beda (0x45a9 bytes) 000317 1777423778.371771 mapwrite 0x001b09 thru 0x008225 (0x671d bytes) 000320 1777423778.395565 MIRROR_RESYNC 3 mirrors 000321 1777423778.598735 mapread 0x009bef thru 0x00beda (0x22ec bytes) 000322 1777423778.612665 MIRROR_RESYNC 3 mirrors 000327 1777423778.760325 read 0x007130 thru 0x00beda (0x4dab bytes) 000328 1777423778.762303 MIRROR_RESYNC 3 mirrors 000333 1777423778.931409 mapread 0x002012 thru 0x00beda (0x9ec9 bytes) 000335 1777423778.939017 mapwrite 0x031f74 thru 0x036867 (0x48f4 bytes) 000337 1777423779.003066 read 0x029d6b thru 0x036867 (0xcafd bytes) 000338 1777423779.006413 write 0x021b95 thru 0x02974c (0x7bb8 bytes) 000342 1777423779.017996 read 0x013d79 thru 0x01bbe2 (0x7e6a bytes) 000343 1777423779.026238 write 0x017e5b thru 0x01fa78 (0x7c1e bytes) 000344 1777423779.028039 write 0x03dde8 thru 0x03ffff (0x2218 bytes) 000345 1777423779.034782 MIRROR_RESYNC 3 mirrors 000346 1777423779.325271 mapwrite 0x025658 thru 0x02b4e8 (0x5e91 bytes) 000347 1777423779.398063 read 0x01e2d5 thru 0x02819b (0x9ec7 bytes) 000349 1777423779.400810 mapwrite 0x01ed2f thru 0x0242e3 (0x55b5 bytes) 000350 1777423779.420634 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x46284cd4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x46284cd4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000353 1777423779.544555 trunc from 0x040000 to 0x012b0e (0x2d4f1 bytes) 000355 1777423779.555731 write 0x029900 thru 0x034b73 (0xb274 bytes) 000356 1777423779.556968 mapwrite 0x03bf0e thru 0x03ffff (0x40f2 bytes) 000357 1777423779.588244 write 0x031e90 thru 0x03ffff (0xe170 bytes) 000358 1777423779.589732 trunc from 0x040000 to 0x01111b (0x2eee4 bytes) 000363 1777423779.597993 mapread 0x00e145 thru 0x01111b (0x2fd7 bytes) 000366 1777423779.605603 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000368 1777423779.673859 write 0x001e69 thru 0x00c1f6 (0xa38e bytes) 000371 1777423779.681267 read 0x00553b thru 0x005c30 (0x6f6 bytes) 000374 1777423779.683230 read 0x001f58 thru 0x003721 (0x17ca bytes) 000375 1777423779.683893 write 0x003105 thru 0x005eb9 (0x2db5 bytes) 000377 1777423779.685710 read 0x002660 thru 0x004686 (0x2027 bytes) 000381 1777423779.687238 trunc from 0x01111c to 0x014244 (0x3129 bytes) 000382 1777423779.696177 write 0x00ddb0 thru 0x01323e (0x548f bytes) 000383 1777423779.705383 trunc from 0x014245 to 0x02da26 (0x197e2 bytes) 000384 1777423779.714117 MIRROR_EXTEND to 2 mirrors 000386 1777423779.920621 MIRROR_EXTEND to 3 mirrors 000387 1777423780.156533 MIRROR_EXTEND to 4 mirrors 000388 1777423780.391978 MIRROR_EXTEND to 5 mirrors 000389 1777423780.599085 write 0x012fd1 thru 0x01b4b6 (0x84e6 bytes) 000390 1777423780.625748 write 0x024798 thru 0x02c6fb (0x7f64 bytes) 000391 1777423780.629109 write 0x034783 thru 0x03ffff (0xb87d bytes) 000392 1777423780.630559 read 0x0195ed thru 0x025f57 (0xc96b bytes) 000393 1777423780.633145 trunc from 0x040000 to 0x00f0c3 (0x30f3c bytes) 000396 1777423780.657029 trunc from 0x00f0c4 to 0x03266e (0x235ab bytes) 000398 1777423780.663363 read 0x02dd25 thru 0x03266e (0x494a bytes) 000399 1777423780.669436 MIRROR_RESYNC 5 mirrors 000400 1777423780.944618 read 0x025c1f thru 0x03266e (0xca50 bytes) 000403 1777423780.947923 mapwrite 0x0207bf thru 0x020c43 (0x485 bytes) 000404 1777423781.002019 MIRROR_RESYNC 5 mirrors 000405 1777423781.262718 write 0x010da9 thru 0x01c2a5 (0xb4fd bytes) 000406 1777423781.285650 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3266f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3266f): Mirror 2: 0x69c72fee (stale) Mirror 3: 0x69c72fee (stale) Mirror 4: 0x69c72fee (stale) Mirror 5: 0x69c72fee (stale) CRC-32 checksum value for chunk [0, 0x3266f): Mirror 1: 0x51aa5e5c Mirror 6: 0x51aa5e5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3266f: skipped 000407 1777423781.545640 mapread 0x00c2b7 thru 0x01a46f (0xe1b9 bytes) 000409 1777423781.561658 mapread 0x0051b9 thru 0x00cc37 (0x7a7f bytes) 000411 1777423781.573404 trunc from 0x03266f to 0x02ca2b (0x5c43 bytes) 000415 1777423781.600630 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ca2c): Mirror 3: 0x6acccb27 (stale) Mirror 4: 0x6acccb27 (stale) Mirror 5: 0x6acccb27 (stale) Mirror 6: 0x2583979c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca2c): Mirror 3: 0x6acccb27 (stale) Mirror 4: 0x6acccb27 (stale) Mirror 5: 0x6acccb27 (stale) Mirror 6: 0x2583979c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca2c: skipped 000417 1777423781.707904 trunc from 0x02ca2c to 0x029f91 (0x2a9a bytes) 000418 1777423781.713985 mapread 0x00605f thru 0x008d02 (0x2ca4 bytes) 000420 1777423781.721735 read 0x005327 thru 0x013396 (0xe070 bytes) 000430 1777423781.730502 mapread 0x025bf2 thru 0x026d0d (0x111c bytes) 000431 1777423781.734552 read 0x00aba6 thru 0x01ab39 (0xff94 bytes) 000434 1777423781.738120 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29f92): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29f92): Mirror 3: 0x2bc3e9e0 (stale) Mirror 4: 0x2bc3e9e0 (stale) Mirror 5: 0x2bc3e9e0 (stale) Mirror 6: 0xec619138 (stale) CRC-32 checksum value for chunk [0, 0x29f92): Mirror 1: 0xec619138 Mirror 7: 0xec619138 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29f92: skipped 000438 1777423781.985686 write 0x007249 thru 0x015301 (0xe0b9 bytes) 000439 1777423782.009174 write 0x009c32 thru 0x0172c0 (0xd68f bytes) 000440 1777423782.012924 read 0x014b19 thru 0x018534 (0x3a1c bytes) 000441 1777423782.013455 mapread 0x02582d thru 0x0279b2 (0x2186 bytes) 000444 1777423782.018780 mapwrite 0x00ed40 thru 0x0166ed (0x79ae bytes) 000445 1777423782.039571 write 0x03f164 thru 0x03ffff (0xe9c bytes) 000446 1777423782.040515 trunc from 0x040000 to 0x01d243 (0x22dbc bytes) 000447 1777423782.053161 read 0x014866 thru 0x01d243 (0x89de bytes) 000450 1777423782.054677 trunc from 0x01d244 to 0x00f637 (0xdc0c bytes) 000451 1777423782.061678 write 0x031b03 thru 0x03ffff (0xe4fd bytes) 000453 1777423782.063265 mapwrite 0x03667e thru 0x039ce0 (0x3663 bytes) 000454 1777423782.079379 write 0x02bc10 thru 0x02eb6b (0x2f5c bytes) 000455 1777423782.085487 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd6d5c434 (stale) Mirror 4: 0xd6d5c434 (stale) Mirror 5: 0xd6d5c434 (stale) Mirror 7: 0x852325c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd6d5c434 (stale) Mirror 4: 0xd6d5c434 (stale) Mirror 5: 0xd6d5c434 (stale) Mirror 7: 0x852325c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000456 1777423782.209199 mapread 0x036eeb thru 0x03cdf2 (0x5f08 bytes) 000457 1777423782.216094 mapwrite 0x035761 thru 0x0360bb (0x95b bytes) 000458 1777423782.228813 mapread 0x00edb5 thru 0x019a2f (0xac7b bytes) 000459 1777423782.236221 read 0x01b484 thru 0x0233cc (0x7f49 bytes) 000462 1777423782.242061 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd6d5c434 (stale) Mirror 4: 0xd6d5c434 (stale) Mirror 5: 0xd6d5c434 (stale) Mirror 7: 0x852325c4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd1c0567 Mirror 8: 0xdd1c0567 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000465 1777423782.509877 write 0x033022 thru 0x03cefc (0x9edb bytes) 000466 1777423782.539869 write 0x037daf thru 0x03ffff (0x8251 bytes) 000467 1777423782.540958 write 0x00c7ac thru 0x010618 (0x3e6d bytes) truncating to largest ever: 0x3cdf8 000468 1777423782.544635 trunc from 0x040000 to 0x03cdf7 (0x3208 bytes) 000469 1777423782.562544 mapread 0x002892 thru 0x004fc3 (0x2732 bytes) 000472 1777423782.566667 read 0x028205 thru 0x030aef (0x88eb bytes) 000473 1777423782.570826 write 0x0078c9 thru 0x009df9 (0x2531 bytes) 000474 1777423782.571986 mapwrite 0x03d82e thru 0x03ffff (0x27d2 bytes) 000475 1777423782.605913 read 0x0190bb thru 0x0257ee (0xc734 bytes) 000480 1777423782.611714 mapwrite 0x021ca8 thru 0x023f97 (0x22f0 bytes) 000481 1777423782.626667 write 0x033cd0 thru 0x03dff4 (0xa325 bytes) 000482 1777423782.627663 trunc from 0x040000 to 0x0014e1 (0x3eb1e bytes) 000485 1777423782.635125 write 0x00e397 thru 0x01d848 (0xf4b2 bytes) 000486 1777423782.636893 trunc from 0x01d849 to 0x022db8 (0x5570 bytes) 000487 1777423782.645068 trunc from 0x022db9 to 0x03a01c (0x17264 bytes) 000490 1777423782.656272 trunc from 0x03a01d to 0x006945 (0x336d7 bytes) 000491 1777423782.662035 read 0x005da5 thru 0x006945 (0xba1 bytes) 000492 1777423782.665347 read 0x001194 thru 0x006945 (0x57b2 bytes) 000494 1777423782.666225 read 0x0045e5 thru 0x006945 (0x2361 bytes) 000496 1777423782.666922 mapread 0x0054c0 thru 0x00601f (0xb60 bytes) 000497 1777423782.670660 MIRROR_RESYNC 6 mirrors 000500 1777423783.026033 mapread 0x0051ef thru 0x006945 (0x1757 bytes) 000501 1777423783.031449 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000503 1777423783.152433 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000508 1777423783.259378 read 0x00557c thru 0x006945 (0x13ca bytes) 000509 1777423783.267813 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000511 1777423783.373502 mapwrite 0x03f133 thru 0x03ffff (0xecd bytes) 000512 1777423783.424056 trunc from 0x040000 to 0x00526a (0x3ad95 bytes) 000513 1777423783.431294 mapread 0x0032ce thru 0x00526a (0x1f9d bytes) 000514 1777423783.437519 write 0x0097cd thru 0x00c514 (0x2d48 bytes) 000516 1777423783.438867 trunc from 0x00c515 to 0x0383d8 (0x2bec4 bytes) 000517 1777423783.444721 write 0x000106 thru 0x009f43 (0x9e3e bytes) 000518 1777423783.448831 mapwrite 0x02427f thru 0x02f3ec (0xb16e bytes) 000521 1777423783.477920 read 0x019d12 thru 0x02257a (0x8869 bytes) 000522 1777423783.481293 trunc from 0x0383d9 to 0x0303e2 (0x7ff6 bytes) 000523 1777423783.492162 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x303e3): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x303e3): Mirror 3: 0x2a82f5d7 (stale) Mirror 5: 0x2a82f5d7 (stale) CRC-32 checksum value for chunk [0, 0x303e3): Mirror 1: 0x21114c12 Mirror 6: 0x21114c12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x303e3: skipped 000524 1777423783.758649 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x303e3): Mirror 5: 0x2a82f5d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x303e3): Mirror 5: 0x2a82f5d7 (stale) CRC-32 checksum value for chunk [0, 0x303e3): Mirror 1: 0x21114c12 Mirror 6: 0x21114c12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x303e3: skipped 000525 1777423783.943243 mapread 0x02bf44 thru 0x02fee0 (0x3f9d bytes) 000526 1777423783.958690 MIRROR_RESYNC 3 mirrors 000527 1777423784.183042 mapwrite 0x011e80 thru 0x01529b (0x341c bytes) 000529 1777423784.238445 mapread 0x017167 thru 0x01a935 (0x37cf bytes) 000530 1777423784.239129 mapwrite 0x0353f7 thru 0x037bdb (0x27e5 bytes) 000532 1777423784.264749 trunc from 0x037bdc to 0x03933c (0x1761 bytes) 000534 1777423784.271894 MIRROR_RESYNC 3 mirrors 000535 1777423784.479436 mapread 0x011f08 thru 0x016b30 (0x4c29 bytes) 000538 1777423784.486752 read 0x00b676 thru 0x019515 (0xdea0 bytes) 000539 1777423784.488872 read 0x032bee thru 0x03933c (0x674f bytes) 000540 1777423784.493909 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000541 1777423784.590350 write 0x021619 thru 0x0246b4 (0x309c bytes) 000542 1777423784.611146 mapwrite 0x03dcda thru 0x03ffff (0x2326 bytes) 000544 1777423784.634619 mapread 0x0032ce thru 0x00af24 (0x7c57 bytes) 000547 1777423784.640599 mapwrite 0x001f2e thru 0x002629 (0x6fc bytes) 000548 1777423784.659067 write 0x007548 thru 0x008890 (0x1349 bytes) 000549 1777423784.661923 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000550 1777423784.711870 mapwrite 0x01bbfc thru 0x022c66 (0x706b bytes) 000551 1777423784.731063 mapread 0x019df0 thru 0x01e15e (0x436f bytes) 000557 1777423784.733745 mapread 0x01277f thru 0x01e5f4 (0xbe76 bytes) 000561 1777423784.739266 write 0x00767c thru 0x014209 (0xcb8e bytes) 000562 1777423784.743571 mapread 0x01b616 thru 0x027cbf (0xc6aa bytes) 000563 1777423784.748672 MIRROR_EXTEND to 2 mirrors 000568 1777423784.946764 read 0x03b59f thru 0x03c2b7 (0xd19 bytes) 000569 1777423784.952313 write 0x033a1b thru 0x03403f (0x625 bytes) 000572 1777423784.977703 mapread 0x025869 thru 0x032587 (0xcd1f bytes) 000574 1777423784.987263 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000576 1777423785.030481 trunc from 0x040000 to 0x02b6d9 (0x14926 bytes) 000578 1777423785.036287 read 0x00fa1e thru 0x01c329 (0xc90c bytes) 000580 1777423785.041235 write 0x007d75 thru 0x009ff5 (0x2281 bytes) 000581 1777423785.045610 trunc from 0x02b6da to 0x0227db (0x8efe bytes) 000583 1777423785.053142 read 0x00c762 thru 0x0137da (0x7079 bytes) 000586 1777423785.059780 trunc from 0x0227dc to 0x008279 (0x1a562 bytes) 000588 1777423785.067147 mapread 0x00406b thru 0x005e72 (0x1e08 bytes) 000589 1777423785.067854 trunc from 0x00827a to 0x0097a4 (0x152b bytes) 000590 1777423785.072665 mapread 0x005e88 thru 0x00964a (0x37c3 bytes) 000594 1777423785.077755 read 0x007b21 thru 0x0097a4 (0x1c84 bytes) 000595 1777423785.078415 write 0x00abba thru 0x0110d2 (0x6519 bytes) 000599 1777423785.080443 mapread 0x00fc7a thru 0x0110d2 (0x1459 bytes) truncating to largest ever: 0x3e716 000604 1777423785.083314 trunc from 0x0110d3 to 0x03e715 (0x2d643 bytes) 000607 1777423785.092448 MIRROR_EXTEND to 2 mirrors 000608 1777423785.272759 trunc from 0x03e716 to 0x01ea3e (0x1fcd7 bytes) 000609 1777423785.303825 mapwrite 0x0020b7 thru 0x008eda (0x6e24 bytes) 000611 1777423785.326226 mapread 0x006233 thru 0x0097b0 (0x357e bytes) 000612 1777423785.327266 mapwrite 0x010a2e thru 0x01a72d (0x9d00 bytes) 000613 1777423785.354201 mapwrite 0x0338a7 thru 0x03f787 (0xbee1 bytes) 000614 1777423785.394267 read 0x02df44 thru 0x031c35 (0x3cf2 bytes) 000615 1777423785.395005 trunc from 0x03f788 to 0x03af57 (0x4830 bytes) 000616 1777423785.400224 read 0x01b316 thru 0x021b50 (0x683b bytes) 000617 1777423785.403308 trunc from 0x03af58 to 0x02dd49 (0xd20e bytes) 000618 1777423785.408922 mapread 0x02a342 thru 0x02dd49 (0x3a08 bytes) 000619 1777423785.409906 trunc from 0x02dd4a to 0x03247a (0x4731 bytes) 000620 1777423785.414659 mapread 0x00221c thru 0x00ba24 (0x9809 bytes) 000622 1777423785.415855 trunc from 0x03247b to 0x02feb5 (0x25c5 bytes) 000625 1777423785.421637 mapread 0x00222e thru 0x002a03 (0x7d6 bytes) 000626 1777423785.422606 write 0x00565a thru 0x00cc33 (0x75da bytes) 000627 1777423785.423728 read 0x02558f thru 0x02feb5 (0xa927 bytes) 000631 1777423785.427910 trunc from 0x02feb6 to 0x00adf8 (0x250bd bytes) 000633 1777423785.436148 mapwrite 0x024d6e thru 0x026355 (0x15e8 bytes) 000635 1777423785.469643 read 0x006ed0 thru 0x00dc95 (0x6dc6 bytes) 000636 1777423785.474486 mapread 0x019d43 thru 0x020cc2 (0x6f80 bytes) 000638 1777423785.476034 read 0x002b9a thru 0x00627c (0x36e3 bytes) 000639 1777423785.476620 write 0x03193e thru 0x035b57 (0x421a bytes) 000640 1777423785.477643 write 0x037ad2 thru 0x03ffff (0x852e bytes) 000642 1777423785.479225 read 0x03d1c4 thru 0x03de48 (0xc85 bytes) 000644 1777423785.480043 write 0x014d58 thru 0x018da7 (0x4050 bytes) 000645 1777423785.480795 mapread 0x028515 thru 0x02a6ad (0x2199 bytes) 000646 1777423785.488048 mapread 0x01d774 thru 0x02a2b0 (0xcb3d bytes) 000647 1777423785.488957 trunc from 0x040000 to 0x00fa91 (0x3056e bytes) 000648 1777423785.498643 read 0x007046 thru 0x00fa91 (0x8a4c bytes) 000649 1777423785.499381 write 0x017650 thru 0x023ae0 (0xc491 bytes) 000650 1777423785.503535 MIRROR_RESYNC 2 mirrors 000651 1777423785.696170 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) 000652 1777423785.746636 write 0x0021c7 thru 0x01126f (0xf0a9 bytes) 000653 1777423785.754788 write 0x03b5cb thru 0x03ffff (0x4a35 bytes) 000654 1777423785.755893 mapwrite 0x02a701 thru 0x02f67b (0x4f7b bytes) 000655 1777423785.778323 mapwrite 0x013afd thru 0x019fb1 (0x64b5 bytes) 000656 1777423785.800249 write 0x02c9ae thru 0x02f297 (0x28ea bytes) 000658 1777423785.801783 mapread 0x027cbd thru 0x02cc48 (0x4f8c bytes) 000659 1777423785.802609 write 0x035c24 thru 0x03ffff (0xa3dc bytes) 000660 1777423785.807289 write 0x0261d9 thru 0x035566 (0xf38e bytes) 000663 1777423785.809773 write 0x011d9d thru 0x01d50d (0xb771 bytes) 000665 1777423785.813490 trunc from 0x040000 to 0x01a3bc (0x25c43 bytes) 000667 1777423785.827475 trunc from 0x01a3bd to 0x019bcb (0x7f1 bytes) 000668 1777423785.837388 write 0x00a9bb thru 0x00d51d (0x2b63 bytes) 000669 1777423785.838420 read 0x00ce15 thru 0x013aff (0x6ceb bytes) 000670 1777423785.838986 write 0x0359d5 thru 0x03f2fa (0x9926 bytes) 000671 1777423785.840052 read 0x00d150 thru 0x0144fe (0x73af bytes) 000674 1777423785.841485 read 0x028d56 thru 0x038b8b (0xfe36 bytes) 000675 1777423785.847414 write 0x02d99a thru 0x02fa76 (0x20dd bytes) 000676 1777423785.848459 write 0x036bc3 thru 0x03dc98 (0x70d6 bytes) 000681 1777423785.851687 mapwrite 0x030874 thru 0x03f4b4 (0xec41 bytes) 000682 1777423785.885854 trunc from 0x03f4b5 to 0x02c193 (0x13321 bytes) 000684 1777423785.892053 mapwrite 0x00834f thru 0x01483e (0xc4f0 bytes) 000685 1777423785.913561 read 0x008d63 thru 0x01614c (0xd3ea bytes) 000689 1777423785.915795 trunc from 0x02c194 to 0x01afdb (0x111b8 bytes) 000692 1777423785.924968 mapwrite 0x00dc18 thru 0x015db0 (0x8199 bytes) 000693 1777423785.939749 write 0x03d451 thru 0x03ffff (0x2baf bytes) 000694 1777423785.940842 write 0x0326b3 thru 0x03ffff (0xd94d bytes) 000695 1777423785.945757 mapwrite 0x02a1c7 thru 0x0395bc (0xf3f6 bytes) 000696 1777423785.964107 read 0x004bca thru 0x013525 (0xe95c bytes) 000699 1777423785.965837 mapwrite 0x022453 thru 0x0273e9 (0x4f97 bytes) 000702 1777423785.990892 write 0x000499 thru 0x0033e6 (0x2f4e bytes) 000704 1777423785.992274 trunc from 0x040000 to 0x01ee9f (0x21160 bytes) 000705 1777423786.000041 mapwrite 0x037feb thru 0x03fedb (0x7ef1 bytes) 000706 1777423786.035398 write 0x003192 thru 0x00b1df (0x804e bytes) 000709 1777423786.037307 write 0x000573 thru 0x00c3a7 (0xbe35 bytes) 000712 1777423786.040410 write 0x034ba8 thru 0x03ffff (0xb458 bytes) 000713 1777423786.041667 mapwrite 0x00e203 thru 0x019182 (0xaf80 bytes) 000715 1777423786.059440 mapwrite 0x023d69 thru 0x02d341 (0x95d9 bytes) 000720 1777423786.076509 trunc from 0x040000 to 0x011b68 (0x2e497 bytes) 000721 1777423786.082156 mapwrite 0x00b2ce thru 0x00bd75 (0xaa8 bytes) 000722 1777423786.096314 write 0x00467c thru 0x00af81 (0x6906 bytes) 000723 1777423786.097375 mapread 0x00c287 thru 0x011b68 (0x58e2 bytes) 000725 1777423786.098644 trunc from 0x011b69 to 0x00514e (0xca1a bytes) 000726 1777423786.105707 mapread 0x0045e9 thru 0x00514e (0xb66 bytes) 000728 1777423786.106866 read 0x000f24 thru 0x00514e (0x422b bytes) 000729 1777423786.107275 read 0x002268 thru 0x00514e (0x2ee7 bytes) 000730 1777423786.107778 write 0x014df1 thru 0x023cd4 (0xeee4 bytes) 000731 1777423786.109235 write 0x01ff35 thru 0x023aff (0x3bcb bytes) 000732 1777423786.110180 mapread 0x0070f1 thru 0x015d7d (0xec8d bytes) 000733 1777423786.114169 mapwrite 0x00178d thru 0x00f42d (0xdca1 bytes) 000735 1777423786.133664 read 0x016321 thru 0x01e609 (0x82e9 bytes) 000737 1777423786.135129 mapread 0x023b95 thru 0x023cd4 (0x140 bytes) 000738 1777423786.136045 mapwrite 0x015a74 thru 0x020d53 (0xb2e0 bytes) 000739 1777423786.153723 mapwrite 0x001bc2 thru 0x011249 (0xf688 bytes) 000740 1777423786.173744 trunc from 0x023cd5 to 0x00f62a (0x146aa bytes) 000741 1777423786.179201 trunc from 0x00f62b to 0x0259f2 (0x163c8 bytes) 000742 1777423786.184856 mapwrite 0x015ca4 thru 0x015da2 (0xff bytes) 000743 1777423786.202721 trunc from 0x0259f3 to 0x010355 (0x1569d bytes) 000744 1777423786.212065 read 0x006fab thru 0x00d628 (0x667e bytes) 000745 1777423786.212883 write 0x013f74 thru 0x017213 (0x32a0 bytes) 000747 1777423786.214827 write 0x0366c9 thru 0x03fa4c (0x9384 bytes) 000748 1777423786.216193 mapwrite 0x026160 thru 0x030bcc (0xaa6d bytes) 000749 1777423786.245759 mapwrite 0x03be47 thru 0x03ffff (0x41b9 bytes) 000750 1777423786.276307 mapwrite 0x00250f thru 0x010b3b (0xe62d bytes) 000751 1777423786.297371 read 0x035f6d thru 0x03dc6a (0x7cfe bytes) 000753 1777423786.298664 trunc from 0x040000 to 0x03001c (0xffe3 bytes) 000754 1777423786.304272 read 0x0128cb thru 0x0183e0 (0x5b16 bytes) 000755 1777423786.304799 read 0x01446e thru 0x015705 (0x1298 bytes) 000756 1777423786.305302 trunc from 0x03001d to 0x01f4da (0x10b42 bytes) 000763 1777423786.317651 mapwrite 0x034bf9 thru 0x03ffff (0xb407 bytes) 000764 1777423786.352470 trunc from 0x040000 to 0x025f99 (0x1a066 bytes) 000769 1777423786.361703 read 0x009ec7 thru 0x0132b1 (0x93eb bytes) 000771 1777423786.362742 trunc from 0x025f9a to 0x026cbd (0xd24 bytes) 000773 1777423786.371208 mapwrite 0x00b6e1 thru 0x00bb8f (0x4af bytes) 000774 1777423786.384979 trunc from 0x026cbe to 0x03b4af (0x147f2 bytes) 000777 1777423786.393881 mapwrite 0x0060d7 thru 0x00b233 (0x515d bytes) 000778 1777423786.406675 read 0x014209 thru 0x017086 (0x2e7e bytes) 000783 1777423786.409021 read 0x033269 thru 0x0346a7 (0x143f bytes) 000784 1777423786.412199 mapwrite 0x024388 thru 0x024ba3 (0x81c bytes) 000786 1777423786.424133 mapread 0x01db2c thru 0x025082 (0x7557 bytes) 000788 1777423786.425373 trunc from 0x03b4b0 to 0x008425 (0x3308a bytes) 000789 1777423786.431054 trunc from 0x008426 to 0x007991 (0xa94 bytes) 000790 1777423786.436776 mapwrite 0x0245ce thru 0x02d2ed (0x8d20 bytes) 000791 1777423786.488794 mapwrite 0x006234 thru 0x009761 (0x352e bytes) 000793 1777423786.512971 mapwrite 0x0077fe thru 0x0087d7 (0xfda bytes) 000796 1777423786.525264 mapwrite 0x012a12 thru 0x01edf9 (0xc3e8 bytes) 000797 1777423786.556443 mapread 0x029d40 thru 0x02d2ed (0x35ae bytes) 000798 1777423786.557375 trunc from 0x02d2ee to 0x034490 (0x71a3 bytes) 000802 1777423786.565146 mapread 0x01aafa thru 0x025e22 (0xb329 bytes) 000803 1777423786.575103 trunc from 0x034491 to 0x037543 (0x30b3 bytes) 000804 1777423786.581375 read 0x033935 thru 0x037543 (0x3c0f bytes) 000806 1777423786.587064 mapread 0x017ef0 thru 0x01de26 (0x5f37 bytes) 000810 1777423786.589629 trunc from 0x037544 to 0x01b0f0 (0x1c453 bytes) 000812 1777423786.598474 mapwrite 0x022d62 thru 0x026388 (0x3627 bytes) 000813 1777423786.638158 mapwrite 0x015be1 thru 0x019c3a (0x405a bytes) 000814 1777423786.653995 write 0x01333f thru 0x020586 (0xd248 bytes) 000816 1777423786.658364 trunc from 0x026389 to 0x01ef75 (0x7413 bytes) 000817 1777423786.668297 read 0x0046c4 thru 0x00921e (0x4b5b bytes) 000819 1777423786.669663 mapread 0x01a5dc thru 0x01ef75 (0x499a bytes) 000821 1777423786.675775 read 0x019a9b thru 0x01a89e (0xe04 bytes) 000823 1777423786.676967 mapwrite 0x03c418 thru 0x03ce7d (0xa66 bytes) 000831 1777423786.716947 trunc from 0x03ce7e to 0x02ec93 (0xe1ea bytes) 000832 1777423786.723944 read 0x015759 thru 0x025488 (0xfd30 bytes) 000836 1777423786.726319 mapwrite 0x009a73 thru 0x00d4ff (0x3a8d bytes) 000837 1777423786.750319 mapwrite 0x001873 thru 0x00ed6b (0xd4f9 bytes) 000839 1777423786.769735 trunc from 0x02ec94 to 0x028c70 (0x6023 bytes) 000843 1777423786.779591 mapwrite 0x014a62 thru 0x0168f3 (0x1e92 bytes) 000844 1777423786.792487 mapread 0x003613 thru 0x012465 (0xee53 bytes) 000845 1777423786.793635 trunc from 0x028c71 to 0x03b4a6 (0x12836 bytes) 000847 1777423786.800110 write 0x00e522 thru 0x014612 (0x60f1 bytes) 000848 1777423786.803021 MIRROR_EXTEND to 2 mirrors 000850 1777423786.985363 mapwrite 0x0186a9 thru 0x02033a (0x7c92 bytes) 000853 1777423787.045791 read 0x00aafb thru 0x01036a (0x5870 bytes) 000854 1777423787.051008 read 0x0260fe thru 0x02d7dd (0x76e0 bytes) 000856 1777423787.055373 mapread 0x01100c thru 0x016726 (0x571b bytes) 000857 1777423787.058754 MIRROR_RESYNC 2 mirrors 000859 1777423787.264412 mapwrite 0x02bcb8 thru 0x02f1e3 (0x352c bytes) 000862 1777423787.328855 read 0x03374f thru 0x03b4a6 (0x7d58 bytes) 000864 1777423787.333171 mapwrite 0x0047ad thru 0x008d23 (0x4577 bytes) 000866 1777423787.359365 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b4a7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4a7): Mirror 2: 0x63cb82ff (stale) CRC-32 checksum value for chunk [0, 0x3b4a7): Mirror 1: 0x8fbd3baf Mirror 3: 0x8fbd3baf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4a7: skipped 000867 1777423787.597354 mapread 0x019ab5 thru 0x022252 (0x879e bytes) 000868 1777423787.615824 mapwrite 0x00cf5b thru 0x00fd45 (0x2deb bytes) 000869 1777423787.680421 mapwrite 0x005ecc thru 0x0134bc (0xd5f1 bytes) 000871 1777423787.698275 write 0x02a30d thru 0x037a78 (0xd76c bytes) 000873 1777423787.708366 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b4a7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4a7): Mirror 2: 0x63cb82ff (stale) Mirror 3: 0x8fbd3baf (stale) CRC-32 checksum value for chunk [0, 0x3b4a7): Mirror 1: 0x89e8264d Mirror 4: 0x89e8264d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4a7: skipped 000874 1777423787.959245 mapread 0x003ff2 thru 0x00a25e (0x626d bytes) 000877 1777423787.975410 trunc from 0x03b4a7 to 0x024141 (0x17365 bytes) 000878 1777423788.018105 write 0x037f9e thru 0x03ffff (0x8062 bytes) 000879 1777423788.019308 mapread 0x03e776 thru 0x03ffff (0x188a bytes) 000881 1777423788.020918 mapwrite 0x027bf3 thru 0x02aa5f (0x2e6d bytes) 000883 1777423788.043703 MIRROR_RESYNC 4 mirrors 000889 1777423788.279795 read 0x03b1fa thru 0x03ffff (0x4e06 bytes) 000890 1777423788.287357 write 0x038aed thru 0x03ffff (0x7513 bytes) 000892 1777423788.316436 mapread 0x03c77d thru 0x03e6e3 (0x1f67 bytes) 000894 1777423788.317841 trunc from 0x040000 to 0x00f8ff (0x30700 bytes) 000895 1777423788.334967 mapread 0x00209b thru 0x00f8ff (0xd865 bytes) 000896 1777423788.340711 mapwrite 0x028a1d thru 0x035529 (0xcb0d bytes) 000897 1777423788.369020 mapread 0x00694f thru 0x012b78 (0xc22a bytes) 000899 1777423788.372788 write 0x03328b thru 0x033abd (0x833 bytes) 000900 1777423788.373564 write 0x00a800 thru 0x010528 (0x5d29 bytes) 000901 1777423788.374229 write 0x018813 thru 0x028131 (0xf91f bytes) 000902 1777423788.377271 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3552a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3552a): Mirror 2: 0x6ccafc48 (stale) Mirror 3: 0x6ccafc48 (stale) Mirror 4: 0x6ccafc48 (stale) CRC-32 checksum value for chunk [0, 0x3552a): Mirror 1: 0x5d67f582 Mirror 5: 0x5d67f582 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3552a: skipped 000903 1777423788.608142 read 0x0249b0 thru 0x0282b0 (0x3901 bytes) 000904 1777423788.610973 mapwrite 0x019001 thru 0x024cd7 (0xbcd7 bytes) 000905 1777423788.684455 mapread 0x00aa4f thru 0x00d58b (0x2b3d bytes) 000907 1777423788.691915 mapwrite 0x03e779 thru 0x03fb90 (0x1418 bytes) 000908 1777423788.728193 mapread 0x029cb8 thru 0x02ebdd (0x4f26 bytes) 000909 1777423788.737062 MIRROR_RESYNC 5 mirrors 000910 1777423789.044123 trunc from 0x03fb91 to 0x031ca4 (0xdeec bytes) 000912 1777423789.083057 mapread 0x029c44 thru 0x02cf6d (0x332a bytes) 000913 1777423789.089854 trunc from 0x031ca5 to 0x03b447 (0x97a3 bytes) 000914 1777423789.097641 read 0x032c98 thru 0x03b447 (0x87b0 bytes) 000916 1777423789.101315 write 0x035493 thru 0x03ffff (0xab6d bytes) 000917 1777423789.102609 mapread 0x032f7d thru 0x03b62c (0x86b0 bytes) 000920 1777423789.104137 write 0x01a437 thru 0x01cd6e (0x2938 bytes) 000922 1777423789.108954 trunc from 0x040000 to 0x00ce71 (0x3318e bytes) 000924 1777423789.115388 mapwrite 0x00a869 thru 0x015c64 (0xb3fc bytes) 000926 1777423789.141583 mapread 0x005187 thru 0x0098fd (0x4777 bytes) 000928 1777423789.143234 mapread 0x0154eb thru 0x015c64 (0x77a bytes) 000931 1777423789.145129 write 0x00d060 thru 0x01cd64 (0xfd05 bytes) 000933 1777423789.147086 read 0x000b60 thru 0x0052e6 (0x4787 bytes) 000934 1777423789.149921 mapwrite 0x032955 thru 0x03f79e (0xce4a bytes) 000937 1777423789.188710 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f79f): Mirror 3: 0x61534be8 (stale) Mirror 4: 0x61534be8 (stale) Mirror 5: 0x61534be8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f79f): Mirror 3: 0x61534be8 (stale) Mirror 4: 0x61534be8 (stale) Mirror 5: 0x61534be8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f79f: skipped 000938 1777423789.313685 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f79f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f79f): Mirror 3: 0x61534be8 (stale) Mirror 4: 0x61534be8 (stale) Mirror 5: 0x61534be8 (stale) CRC-32 checksum value for chunk [0, 0x3f79f): Mirror 1: 0xee42d260 Mirror 6: 0xee42d260 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f79f: skipped 000939 1777423789.600204 mapwrite 0x01e8a8 thru 0x0288eb (0xa044 bytes) 000941 1777423789.658873 read 0x0011d8 thru 0x00303a (0x1e63 bytes) 000942 1777423789.661324 read 0x01e4b2 thru 0x02d6a8 (0xf1f7 bytes) 000943 1777423789.661859 mapread 0x026e9b thru 0x035a52 (0xebb8 bytes) 000945 1777423789.666364 mapread 0x01f44b thru 0x02a1a2 (0xad58 bytes) 000946 1777423789.667176 trunc from 0x03f79f to 0x03786a (0x7f34 bytes) 000947 1777423789.679573 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3786b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3786b): Mirror 3: 0x8272b82f (stale) Mirror 4: 0x8272b82f (stale) Mirror 5: 0x8272b82f (stale) Mirror 6: 0xb4d7153f (stale) CRC-32 checksum value for chunk [0, 0x3786b): Mirror 1: 0x793abcbf Mirror 7: 0x793abcbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3786b: skipped 000948 1777423789.924012 mapwrite 0x015695 thru 0x015bcb (0x537 bytes) 000949 1777423789.982490 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3786b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3786b): Mirror 3: 0x8272b82f (stale) Mirror 4: 0x8272b82f (stale) Mirror 5: 0x8272b82f (stale) Mirror 6: 0xb4d7153f (stale) Mirror 7: 0x793abcbf (stale) CRC-32 checksum value for chunk [0, 0x3786b): Mirror 1: 0x2ff7c254 Mirror 8: 0x2ff7c254 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3786b: skipped 000950 1777423790.271593 read 0x010648 thru 0x015516 (0x4ecf bytes) 000952 1777423790.274902 mapwrite 0x02cd27 thru 0x031561 (0x483b bytes) 000953 1777423790.334287 write 0x00ef70 thru 0x011d8c (0x2e1d bytes) 000954 1777423790.339890 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3786b): Mirror 4: 0x8272b82f (stale) Mirror 5: 0x8272b82f (stale) Mirror 6: 0xb4d7153f (stale) Mirror 7: 0x793abcbf (stale) Mirror 8: 0x2ff7c254 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3786b): Mirror 4: 0x8272b82f (stale) Mirror 5: 0x8272b82f (stale) Mirror 6: 0xb4d7153f (stale) Mirror 7: 0x793abcbf (stale) Mirror 8: 0x2ff7c254 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3786b: skipped 000956 1777423790.476528 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3786b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3786b): Mirror 4: 0x8272b82f (stale) Mirror 5: 0x8272b82f (stale) Mirror 6: 0xb4d7153f (stale) Mirror 7: 0x793abcbf (stale) Mirror 8: 0x2ff7c254 (stale) CRC-32 checksum value for chunk [0, 0x3786b): Mirror 1: 0x40175ccb Mirror 9: 0x40175ccb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3786b: skipped 000959 1777423790.749311 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3786b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3786b): Mirror 4: 0x8272b82f (stale) Mirror 5: 0x8272b82f (stale) Mirror 6: 0xb4d7153f (stale) Mirror 7: 0x793abcbf (stale) Mirror 8: 0x2ff7c254 (stale) CRC-32 checksum value for chunk [0, 0x3786b): Mirror 1: 0x40175ccb Mirror 9: 0x40175ccb Mirror 10: 0x40175ccb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3786b: skipped 000960 1777423791.060115 mapread 0x001ef8 thru 0x010fc6 (0xf0cf bytes) 000963 1777423791.081020 trunc from 0x03786b to 0x031a97 (0x5dd3 bytes) 000965 1777423791.104096 mapwrite 0x03830d thru 0x03bc76 (0x396a bytes) 000968 1777423791.139575 write 0x0207a9 thru 0x027179 (0x69d1 bytes) 000969 1777423791.145988 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3bc77): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc77): Mirror 4: 0x543c564e (stale) Mirror 5: 0x543c564e (stale) Mirror 6: 0xfd1f36b (stale) Mirror 7: 0x9852a3d1 (stale) Mirror 8: 0xd273285f (stale) Mirror 9: 0xaa957a74 (stale) Mirror 10: 0xaa957a74 (stale) CRC-32 checksum value for chunk [0, 0x3bc77): Mirror 1: 0x1c986e4a Mirror 11: 0x1c986e4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc77: skipped 000970 1777423791.418463 mapread 0x013e41 thru 0x0228e5 (0xeaa5 bytes) 000971 1777423791.429490 write 0x032b08 thru 0x03916f (0x6668 bytes) 000972 1777423791.461507 read 0x02d35f thru 0x03bc76 (0xe918 bytes) 000974 1777423791.464576 trunc from 0x03bc77 to 0x01549c (0x267da bytes) 000976 1777423791.485817 trunc from 0x01549d to 0x014698 (0xe04 bytes) 000977 1777423791.491580 mapread 0x00f805 thru 0x014698 (0x4e94 bytes) 000978 1777423791.498053 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x14699): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14699): Mirror 4: 0x5e3ade44 (stale) Mirror 5: 0x5e3ade44 (stale) Mirror 6: 0xc8d6eae1 (stale) Mirror 7: 0xc8d6eae1 (stale) Mirror 8: 0xc8d6eae1 (stale) Mirror 9: 0xb8f14ab3 (stale) Mirror 10: 0xb8f14ab3 (stale) Mirror 11: 0xb8f14ab3 (stale) CRC-32 checksum value for chunk [0, 0x14699): Mirror 1: 0xb8f14ab3 Mirror 12: 0xb8f14ab3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14699: skipped 000980 1777423791.788573 read 0x010c7a thru 0x014698 (0x3a1f bytes) 000981 1777423791.792881 write 0x029432 thru 0x0353e4 (0xbfb3 bytes) 000983 1777423791.815108 mapread 0x0328eb thru 0x0353e4 (0x2afa bytes) 000988 1777423791.818034 write 0x002178 thru 0x0103e3 (0xe26c bytes) 000989 1777423791.824593 mapwrite 0x02709c thru 0x031688 (0xa5ed bytes) 000990 1777423791.842844 mapwrite 0x03f288 thru 0x03ffff (0xd78 bytes) 000991 1777423791.875381 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x530:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x680250da (stale) Mirror 5: 0x680250da (stale) Mirror 6: 0x46f338e3 (stale) Mirror 7: 0x438769ce (stale) Mirror 8: 0x79dbfd2e (stale) Mirror 9: 0x1f18838f (stale) Mirror 10: 0x1f18838f (stale) Mirror 12: 0x64d48267 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x680250da (stale) Mirror 5: 0x680250da (stale) Mirror 6: 0x46f338e3 (stale) Mirror 7: 0x438769ce (stale) Mirror 8: 0x79dbfd2e (stale) Mirror 9: 0x1f18838f (stale) Mirror 10: 0x1f18838f (stale) Mirror 12: 0x64d48267 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1777423792.084284 read 0x01f668 thru 0x02c152 (0xcaeb bytes) 000995 1777423792.089691 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x680250da (stale) Mirror 5: 0x680250da (stale) Mirror 6: 0x46f338e3 (stale) Mirror 7: 0x438769ce (stale) Mirror 8: 0x79dbfd2e (stale) Mirror 9: 0x1f18838f (stale) Mirror 10: 0x1f18838f (stale) Mirror 12: 0x64d48267 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9d04e8c Mirror 13: 0xd9d04e8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000996 1777423792.412935 read 0x02ab34 thru 0x032052 (0x751f bytes) 000997 1777423792.417512 mapread 0x02cd1e thru 0x03bff3 (0xf2d6 bytes) All operations completed A-OK! PASS 70a (22s) == sanity-flr test 71: check mirror extend parent fid ==== 20:49:54 (1777423794) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 20:49:55 (1777423795) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x752:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x74e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100223 s, 20.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x752:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x753:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x74e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700466 s, 15.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x752:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x753:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x74f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x74e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x752:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x753:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x74f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x74e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x754:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 20:49:58 (1777423798) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre2 Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5699296 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1918432 .. done Extending file size to 5666656 .. Extending file size to 7524064 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7608288 .. Extending file size to 7895744 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7701792 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8691040 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8824096 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8826432 .. Extending file size to 8933120 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9119424 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57688 57689 57691 57693 57695 done 192.168.201.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:) 192.168.201.143@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:51:04 (1777423864) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre2 Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1605920 .. Extending file size to 1408768 .. done Extending file size to 5665536 .. Extending file size to 8609728 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8348256 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8737280 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9261248 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9150464 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9268928 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60682 60683 60685 60686 60689 192.168.201.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:) 192.168.201.143@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:52:09 (1777423929) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3456 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3456 2204800 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0396176 s, 52.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0813482 s, 25.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xeacaadbe (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:52:15 (1777423935) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0277943 s, 75.5 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 20:52:17 (1777423937) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x572:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x765:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x765:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x766:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x769:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:52:20 (1777423940) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0456457 s, 45.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x767:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x76a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x76b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:52:22 (1777423942) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0437087 s, 48.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x768:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x769:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x76c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x768:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x769:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x76c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x76d:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:52:24 (1777423944) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0551089 s, 38.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x76e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x76e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x76f:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:52:48 (1777423968) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg143-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195793 s, 53.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x771:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x770:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x771:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x770:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76c:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:53:32 (1777424012) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg143-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0390643 s, 53.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x772:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x772:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:54:18 (1777424058) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg143-server mds-ost sync done. Creating new pool test_204f oleg143-server: Pool lustre.test_204f created Adding targets to pool oleg143-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg143-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0368077 s, 57.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x773:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x76f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x773:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg143-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg143-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg143-server: Pool lustre.test_204f destroyed PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:55:18 (1777424118) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:55:20 (1777424120) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:55:23 (1777424123) oleg143-server: Pool lustre.test_206 created Adding targets to pool oleg143-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg143-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg143-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg143-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg143-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:55:37 (1777424137) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329245 s, 31.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136846 s, 76.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xcb20213e Mirror 3: 0xcb20213e PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:55:40 (1777424140) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:55:41 (1777424141) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:55:42 (1777424142) Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0337767 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1730000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1dc0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1dc0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1dc0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x6c0000 Read this (4 bytes): BBBB 192.168.201.143@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg143-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:55:58 (1777424158) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00353701 s, 2.3 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:56:00 (1777424160) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141145 s, 74.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x597 lmm_fid: [0x200000404:0x597:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x597 lmm_fid: [0x200000404:0x597:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x77e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835351 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-28 20:56:01.000000000 -0400 Modify: 2026-04-28 20:56:01.000000000 -0400 Change: 2026-04-28 20:56:01.000000000 -0400 Birth: 2026-04-28 20:56:00.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:56:03 (1777424163) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:56:04 (1777424164) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.133019 s, 308 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128268 s, 319 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 28 20:56 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x599:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 28 20:56 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:56:06 (1777424166) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:56:09 (1777424169) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 825 sec ============ 20:56:11 (1777424171) === sanity-flr: start cleanup 20:56:12 (1777424172) === === sanity-flr: finish cleanup 20:56:14 (1777424174) ===