-----============= acceptance-small: sanity-flr ============----- Thu Aug 6 11:46:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg119-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:46:39 (1786031199) === oleg119-client.virtnet: executing check_config_client /mnt/lustre oleg119-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg119-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ed88313d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ed88313d800.idle_timeout=debug disable quota as required oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:47:25 (1786031245) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7641] == sanity-flr test 0a: lfs mirror create with -N option == 11:47:28 (1786031248) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:47:50 (1786031270) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:47:54 (1786031274) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:47:57 (1786031277) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:48:26 (1786031306) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:48:30 (1786031310) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 11:48:34 (1786031314) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ed88313d800.stats=clear osc.lustre-OST0001-osc-ffff9ed88313d800.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:48:46 (1786031326) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (17s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:49:02 (1786031342) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:49:13 (1786031353) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:49:16 (1786031356) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:49:26 (1786031366) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 11:49:37 (1786031377) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:49:45 (1786031385) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:49:49 (1786031389) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:49:57 (1786031397) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:50:09 (1786031409) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:50:17 (1786031417) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 1.16975 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210492 s, 5.0 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:50:30 (1786031430) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.365035 s, 17.2 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:50:40 (1786031440) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:50:54 (1786031454) Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ed88313d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ed88313d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ed88313d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ed88313d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (90s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:52:25 (1786031545) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.701917 s, 16.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ed88313d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ed88313d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:53:12 (1786031592) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ed88313d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ed88313d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ed88313d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ed88313d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:54:38 (1786031678) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.24204 s, 12.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 6 11:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.196s user 0m0.000s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ed88313d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ed88313d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.444s user 0m0.001s sys 0m0.206s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg119-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg119-server oleg119-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ed88313d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ed88313d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.721s user 0m0.000s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg119-server: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg119-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg119-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ed88313d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ed88313d800.ost_server_uuid in FULL state after 0 sec PASS 33b (114s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:56:32 (1786031792) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:56:37 (1786031797) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:56:42 (1786031802) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 11:56:48 (1786031808) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 11:56:58 (1786031818) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.748 s, 16.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:57:13 (1786031833) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385561 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275071 s, 38.1 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:57:22 (1786031842) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:57:31 (1786031851) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00187387 s, 546 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00277104 s, 370 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103217 s, 992 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115489 s, 887 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 11:57:45 (1786031865) create test files with size 15729238 .. 9d7d5407260e09897ca678f91c91c58a - 2420de9f83ba93d9e0f8687406b21cfe - b43af63f9567e7f8fa858edc9f7079e0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 11:58:10 (1786031890) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:58:25 (1786031905) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:58:33 (1786031913) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243074 s, 4.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368362 s, 28.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 11:58:43 (1786031923) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160671 s, 13.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.324417 s, 12.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2175 s, 9.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 11:59:01 (1786031941) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 11:59:05 (1786031945) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:59:08 (1786031948) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302035 s, 34.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 11:59:16 (1786031956) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:59:20 (1786031960) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:59:32 (1786031972) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.332445 s, 31.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:59:47 (1786031987) llite.lustre-ffff9ed885845000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.290078 s, 36.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg119-client.virtnet /mnt/lustre (opts:) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9ed8860cb800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:00:07 (1786032007) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:00:53 (1786032053) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:01:36 (1786032096) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 12:01:40 (1786032100) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.239531 s, 13.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.174521 s, 18.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x63cced38 (nosync,stale) Mirror 2: 0x63cced38 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5701bafce2039246d69cd633016cbc02 - mirror 1 checksum: 5591d0e3c5a3d695632743fb8ba73e93 - mirror 2 checksum: 5701bafce2039246d69cd633016cbc02 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-06T12:01:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-06T12:01:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5701bafce2039246d69cd633016cbc02 - mirror 1 checksum: 5591d0e3c5a3d695632743fb8ba73e93 - mirror 2 checksum: 5591d0e3c5a3d695632743fb8ba73e93 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-06T12:01:43-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-06T12:01:43-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:01:55 (1786032115) Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 12:02:02 (1786032122) shut down facet: mds1 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds1 to oleg119-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 12:02:34 (1786032154) targets are mounted 12:02:34 (1786032154) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679585 s, 15.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:02:55 (1786032175) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:03:00 (1786032180) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:03:04 (1786032184) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00747052 s, 2.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:03:14 (1786032194) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:03:18 (1786032198) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00955448 s, 429 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.132623 s, 30.9 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:03:29 (1786032209) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:03:55 (1786032235) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:04:28 (1786032268) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:04:52 (1786032292) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (31s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:05:23 (1786032323) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 12:05:33 (1786032333) mirror create pid 54365 mirror split pid 54368 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54365 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54368 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 12:06:42 (1786032402) Seed set to 2404 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1786032405.092826 mapwrite 0x02617d thru 0x027dc8 (0x1c4c bytes) 000004 1786032405.305041 mapwrite 0x01871f thru 0x01e970 (0x6252 bytes) truncating to largest ever: 0x257ad 000007 1786032405.467532 trunc from 0x027dc9 to 0x0257ac (0x261c bytes) 000010 1786032405.497819 mapread 0x017caa thru 0x0257ac (0xdb03 bytes) 000011 1786032405.513210 mapread 0x00098b thru 0x00da7c (0xd0f2 bytes) 000012 1786032405.525877 write 0x01a990 thru 0x01b746 (0xdb7 bytes) truncating to largest ever: 0x2d5fd 000013 1786032405.527629 trunc from 0x0257ad to 0x02d5fc (0x7e50 bytes) 000016 1786032405.551437 MIRROR_EXTEND to 2 mirrors 000017 1786032406.725003 read 0x011778 thru 0x01ad47 (0x95d0 bytes) 000018 1786032406.743580 read 0x01a585 thru 0x01c609 (0x2085 bytes) 000019 1786032406.777340 trunc from 0x02d5fd to 0x00a993 (0x22c69 bytes) 000020 1786032407.002159 MIRROR_RESYNC 2 mirrors 000021 1786032408.258018 mapwrite 0x0305b0 thru 0x03ef46 (0xe997 bytes) 000022 1786032408.518740 mapwrite 0x00b004 thru 0x015953 (0xa950 bytes) 000023 1786032408.716362 MIRROR_RESYNC 2 mirrors 000024 1786032410.566616 MIRROR_EXTEND to 3 mirrors 000025 1786032411.659660 read 0x00c136 thru 0x0132c2 (0x718d bytes) 000027 1786032411.673084 mapread 0x00174f thru 0x011368 (0xfc1a bytes) 000029 1786032411.721759 MIRROR_RESYNC 3 mirrors 000033 1786032412.558219 mapread 0x03b874 thru 0x03ef46 (0x36d3 bytes) 000036 1786032412.609874 mapwrite 0x013e9e thru 0x020af1 (0xcc54 bytes) 000037 1786032412.952971 mapwrite 0x020c4e thru 0x025c7d (0x5030 bytes) 000038 1786032413.070003 write 0x039ac3 thru 0x03ffff (0x653d bytes) 000039 1786032413.091662 mapwrite 0x01314d thru 0x01fb5e (0xca12 bytes) 000040 1786032413.224060 mapwrite 0x035329 thru 0x03561e (0x2f6 bytes) 000041 1786032413.335062 write 0x0310eb thru 0x03e53a (0xd450 bytes) 000046 1786032413.348088 trunc from 0x040000 to 0x0182c0 (0x27d3f bytes) 000047 1786032413.405585 trunc from 0x0182c1 to 0x02b1b2 (0x12ef2 bytes) 000048 1786032413.453880 read 0x00319a thru 0x01315f (0xffc6 bytes) 000052 1786032413.517434 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b1b3): Mirror 3: 0xc563d774 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1b3): Mirror 3: 0xc563d774 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1b3: skipped 000054 1786032413.943002 trunc from 0x02b1b3 to 0x02969b (0x1b17 bytes) 000056 1786032413.987365 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000058 1786032414.245562 read 0x022f5b thru 0x02969b (0x6741 bytes) 000060 1786032414.257778 write 0x01df5d thru 0x02b823 (0xd8c7 bytes) 000063 1786032414.275509 read 0x02a4d8 thru 0x02ad3d (0x866 bytes) 000064 1786032414.276731 read 0x01473f thru 0x019bd1 (0x5493 bytes) 000065 1786032414.300202 write 0x00933a thru 0x00c8dc (0x35a3 bytes) 000068 1786032414.311130 mapwrite 0x00351c thru 0x00b7af (0x8294 bytes) 000069 1786032414.427224 read 0x0026f4 thru 0x00a617 (0x7f24 bytes) 000070 1786032414.434918 MIRROR_EXTEND to 2 mirrors 000072 1786032415.823811 read 0x005be6 thru 0x00d433 (0x784e bytes) 000075 1786032415.848883 MIRROR_EXTEND to 3 mirrors 000076 1786032416.955084 mapread 0x022b6f thru 0x02567c (0x2b0e bytes) 000077 1786032416.972215 trunc from 0x02b824 to 0x00712f (0x246f4 bytes) 000078 1786032417.073219 trunc from 0x007130 to 0x0068fa (0x835 bytes) 000079 1786032417.097186 mapwrite 0x002ae7 thru 0x00b9e5 (0x8eff bytes) 000080 1786032417.239636 write 0x027902 thru 0x02bba1 (0x42a0 bytes) 000081 1786032417.246763 MIRROR_RESYNC 3 mirrors 000082 1786032418.972940 mapwrite 0x02d065 thru 0x03820a (0xb1a6 bytes) 000083 1786032419.261845 mapread 0x0268bf thru 0x02bf61 (0x56a3 bytes) 000084 1786032419.284589 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3820b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3820b): Mirror 2: 0x19e34ca7 (stale) Mirror 3: 0x19e34ca7 (stale) CRC-32 checksum value for chunk [0, 0x3820b): Mirror 1: 0x2553ea1 Mirror 4: 0x2553ea1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3820b: skipped 000085 1786032420.551779 mapread 0x020667 thru 0x02a543 (0x9edd bytes) 000088 1786032420.624580 trunc from 0x03820b to 0x005cea (0x32520 bytes) 000089 1786032420.804572 mapread 0x000147 thru 0x005cea (0x5ba4 bytes) 000091 1786032420.820644 mapwrite 0x01bbcf thru 0x020489 (0x48bb bytes) 000092 1786032420.973092 mapwrite 0x01bd36 thru 0x025d8c (0xa057 bytes) 000093 1786032421.104210 mapwrite 0x00176d thru 0x00334b (0x1bdf bytes) 000094 1786032421.187863 write 0x0022fc thru 0x00ce36 (0xab3b bytes) 000095 1786032421.201329 write 0x03bb98 thru 0x03fb66 (0x3fcf bytes) 000097 1786032421.207169 trunc from 0x03fb67 to 0x01612c (0x29a3a bytes) 000098 1786032421.255498 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1612d): Mirror 2: 0x2b66c399 (stale) Mirror 4: 0x2b66c399 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1612d): Mirror 2: 0x2b66c399 (stale) Mirror 4: 0x2b66c399 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1612d: skipped 000099 1786032421.823899 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1612d): Mirror 2: 0x2b66c399 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1612d): Mirror 2: 0x2b66c399 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1612d: skipped 000100 1786032422.304704 mapwrite 0x0007bf thru 0x0103a3 (0xfbe5 bytes) 000103 1786032422.412483 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000105 1786032422.622041 trunc from 0x01612d to 0x0015fd (0x14b2f bytes) 000106 1786032422.659704 write 0x00b4c0 thru 0x012c86 (0x77c7 bytes) 000107 1786032422.662998 trunc from 0x012c87 to 0x01663a (0x39b4 bytes) 000108 1786032422.688182 mapwrite 0x005137 thru 0x005785 (0x64f bytes) 000109 1786032422.813432 mapwrite 0x033568 thru 0x03abd7 (0x7670 bytes) 000110 1786032422.968484 MIRROR_EXTEND to 2 mirrors 000111 1786032424.694317 read 0x02ce04 thru 0x03abd7 (0xddd4 bytes) 000113 1786032424.714709 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000115 1786032424.919905 read 0x004a7d thru 0x009b5b (0x50df bytes) truncating to largest ever: 0x3b571 000116 1786032424.932747 trunc from 0x03abd8 to 0x03b570 (0x999 bytes) 000118 1786032424.957667 write 0x01bd6a thru 0x0228c9 (0x6b60 bytes) 000120 1786032424.966430 write 0x03a550 thru 0x03a965 (0x416 bytes) 000122 1786032424.972079 write 0x028c5c thru 0x034396 (0xb73b bytes) 000123 1786032424.989843 trunc from 0x03b571 to 0x00cfc9 (0x2e5a7 bytes) 000124 1786032425.010018 read 0x0023fe thru 0x006b5e (0x4761 bytes) 000125 1786032425.022669 mapread 0x0059a4 thru 0x00cfc9 (0x7626 bytes) 000127 1786032425.026139 write 0x01f4bf thru 0x0203d2 (0xf14 bytes) 000130 1786032425.030419 mapread 0x01b8d1 thru 0x0203d2 (0x4b02 bytes) 000131 1786032425.042428 read 0x0154c3 thru 0x01dc53 (0x8791 bytes) 000134 1786032425.045495 mapwrite 0x023e49 thru 0x0291eb (0x53a3 bytes) 000138 1786032425.244029 write 0x037499 thru 0x03ffff (0x8b67 bytes) 000139 1786032425.254150 mapwrite 0x026348 thru 0x02b925 (0x55de bytes) 000144 1786032425.362883 write 0x01425f thru 0x022785 (0xe527 bytes) 000145 1786032425.366890 trunc from 0x040000 to 0x00fec4 (0x3013b bytes) 000146 1786032425.391649 read 0x00331d thru 0x0072a6 (0x3f8a bytes) 000147 1786032425.392694 mapwrite 0x02795d thru 0x0370a1 (0xf745 bytes) 000148 1786032425.523928 trunc from 0x0370a2 to 0x031f19 (0x5188 bytes) 000152 1786032425.561862 write 0x019607 thru 0x01a7a9 (0x11a3 bytes) 000153 1786032425.563297 write 0x01c0cc thru 0x01e748 (0x267d bytes) 000154 1786032425.570938 mapread 0x02fac1 thru 0x031f19 (0x2459 bytes) 000155 1786032425.574918 trunc from 0x031f1a to 0x00f96f (0x225aa bytes) 000156 1786032425.606104 mapread 0x00b730 thru 0x00c27e (0xb4f bytes) 000157 1786032425.607415 mapwrite 0x03e50c thru 0x03ffff (0x1af4 bytes) 000162 1786032425.733427 read 0x02babf thru 0x03296d (0x6eaf bytes) 000164 1786032425.767801 read 0x0028b5 thru 0x00e064 (0xb7b0 bytes) 000165 1786032425.768889 read 0x012651 thru 0x01b82a (0x91da bytes) 000166 1786032425.771696 trunc from 0x040000 to 0x0084fd (0x37b02 bytes) 000168 1786032425.799762 mapread 0x004d10 thru 0x007cc5 (0x2fb6 bytes) 000169 1786032425.804701 write 0x0286e8 thru 0x0329f9 (0xa312 bytes) 000173 1786032425.815072 write 0x0386e8 thru 0x03c8ef (0x4208 bytes) 000174 1786032425.817052 trunc from 0x03c8f0 to 0x02ed0e (0xdbe1 bytes) 000175 1786032425.845028 mapwrite 0x02ae1b thru 0x03735e (0xc544 bytes) 000176 1786032425.986062 trunc from 0x03735f to 0x02a41f (0xcf3f bytes) 000177 1786032426.018366 mapwrite 0x008b87 thru 0x018975 (0xfdef bytes) 000181 1786032426.133851 mapwrite 0x033d5c thru 0x03ee98 (0xb13d bytes) 000183 1786032426.358772 write 0x03cdcc thru 0x03ffff (0x3234 bytes) 000184 1786032426.408308 trunc from 0x040000 to 0x00a464 (0x35b9b bytes) 000186 1786032426.460760 trunc from 0x00a465 to 0x004811 (0x5c53 bytes) 000187 1786032426.502633 mapread 0x002041 thru 0x004811 (0x27d1 bytes) 000188 1786032426.506664 write 0x035a9e thru 0x03deef (0x8452 bytes) 000189 1786032426.519064 mapread 0x00749a thru 0x012acb (0xb632 bytes) 000190 1786032426.662482 read 0x026d99 thru 0x0343f0 (0xd658 bytes) 000192 1786032426.676394 read 0x01a314 thru 0x01b2bd (0xfaa bytes) 000195 1786032426.680793 read 0x010293 thru 0x0159c5 (0x5733 bytes) 000196 1786032426.682050 write 0x029879 thru 0x02a041 (0x7c9 bytes) 000197 1786032426.692616 MIRROR_EXTEND to 2 mirrors 000198 1786032428.485416 read 0x00bf1b thru 0x019246 (0xd32c bytes) 000199 1786032428.508896 trunc from 0x03def0 to 0x01e900 (0x1f5ef bytes) 000200 1786032428.689980 trunc from 0x01e901 to 0x02f1f1 (0x108f1 bytes) 000201 1786032428.728705 trunc from 0x02f1f2 to 0x01c3b0 (0x12e41 bytes) 000203 1786032428.766143 trunc from 0x01c3b1 to 0x029311 (0xcf61 bytes) 000204 1786032428.799621 MIRROR_RESYNC 2 mirrors 000205 1786032430.191574 write 0x011ce1 thru 0x013a73 (0x1d93 bytes) 000207 1786032430.292041 read 0x01aea7 thru 0x01dce1 (0x2e3b bytes) 000209 1786032430.301008 mapwrite 0x0173e1 thru 0x01c468 (0x5088 bytes) 000211 1786032430.397074 mapread 0x00917c thru 0x00ca79 (0x38fe bytes) 000214 1786032430.415330 read 0x01e99f thru 0x029311 (0xa973 bytes) 000215 1786032430.441579 mapread 0x0169d5 thru 0x02338a (0xc9b6 bytes) 000217 1786032430.451768 trunc from 0x029312 to 0x015269 (0x140a8 bytes) 000218 1786032430.501294 trunc from 0x01526a to 0x01c7f1 (0x7588 bytes) 000219 1786032430.541597 mapread 0x005180 thru 0x007ea3 (0x2d24 bytes) 000222 1786032430.545531 read 0x000600 thru 0x00f5f6 (0xeff7 bytes) 000225 1786032430.551433 read 0x01432d thru 0x01c7f1 (0x84c5 bytes) 000229 1786032430.561767 mapwrite 0x034658 thru 0x03ed68 (0xa711 bytes) 000231 1786032430.639798 write 0x02b437 thru 0x033042 (0x7c0c bytes) 000232 1786032430.642739 mapwrite 0x016b77 thru 0x01c1fd (0x5687 bytes) 000233 1786032430.705638 mapread 0x004466 thru 0x006c7c (0x2817 bytes) 000236 1786032430.708865 mapread 0x033df3 thru 0x039f82 (0x6190 bytes) 000237 1786032430.710847 trunc from 0x03ed69 to 0x039ad8 (0x5290 bytes) 000239 1786032430.735563 MIRROR_RESYNC 2 mirrors 000242 1786032432.000340 write 0x015dc3 thru 0x01a051 (0x428f bytes) 000246 1786032432.148972 read 0x00740d thru 0x010834 (0x9428 bytes) 000249 1786032432.172151 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39ad9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ad9): Mirror 2: 0xce5a0a57 (stale) CRC-32 checksum value for chunk [0, 0x39ad9): Mirror 1: 0xcf5292a9 Mirror 3: 0xcf5292a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ad9: skipped 000251 1786032433.499757 mapwrite 0x02fdde thru 0x034550 (0x4773 bytes) 000253 1786032433.767680 mapread 0x012149 thru 0x016701 (0x45b9 bytes) 000254 1786032433.798931 MIRROR_RESYNC 3 mirrors 000255 1786032435.568178 read 0x0075f3 thru 0x01336b (0xbd79 bytes) 000256 1786032435.578631 mapwrite 0x01050f thru 0x0106be (0x1b0 bytes) 000257 1786032435.734387 mapwrite 0x0325d5 thru 0x037a54 (0x5480 bytes) 000258 1786032435.826872 read 0x038590 thru 0x039ad8 (0x1549 bytes) 000260 1786032435.829153 trunc from 0x039ad9 to 0x03157c (0x855c bytes) 000261 1786032435.870886 write 0x01e24e thru 0x02d4a3 (0xf256 bytes) 000262 1786032435.896901 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3157d): Mirror 3: 0xc6fce106 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3157d): Mirror 3: 0xc6fce106 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3157d: skipped 000264 1786032436.381907 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3157d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3157d): Mirror 3: 0xc6fce106 (stale) CRC-32 checksum value for chunk [0, 0x3157d): Mirror 1: 0x6e7072c0 Mirror 4: 0x6e7072c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3157d: skipped 000265 1786032438.574744 write 0x02b122 thru 0x02fc05 (0x4ae4 bytes) 000271 1786032438.779125 mapwrite 0x01f78e thru 0x024ccc (0x553f bytes) 000274 1786032438.870623 trunc from 0x03157d to 0x015eda (0x1b6a2 bytes) 000275 1786032438.910708 write 0x01eeea thru 0x02a06c (0xb183 bytes) 000276 1786032438.912420 trunc from 0x02a06d to 0x01f8ce (0xa79e bytes) 000279 1786032438.941066 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f8cf): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f8cf): Mirror 3: 0xb47f1565 (stale) Mirror 4: 0x63c43608 (stale) CRC-32 checksum value for chunk [0, 0x1f8cf): Mirror 1: 0x26d6c178 Mirror 5: 0x26d6c178 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f8cf: skipped 000283 1786032440.594864 mapread 0x0167b2 thru 0x018e0d (0x265c bytes) 000284 1786032440.639969 mapwrite 0x0149ee thru 0x01c7d8 (0x7deb bytes) 000287 1786032440.928632 write 0x00584e thru 0x006fa1 (0x1754 bytes) 000289 1786032440.943361 mapwrite 0x028089 thru 0x037c73 (0xfbeb bytes) 000290 1786032441.160177 write 0x01f99d thru 0x02f1e1 (0xf845 bytes) 000291 1786032441.189524 MIRROR_RESYNC 4 mirrors 000293 1786032443.373798 mapread 0x003013 thru 0x00aaf6 (0x7ae4 bytes) 000294 1786032443.409304 read 0x00c217 thru 0x0126cb (0x64b5 bytes) 000295 1786032443.432512 write 0x01138c thru 0x01b075 (0x9cea bytes) 000296 1786032443.534202 trunc from 0x037c74 to 0x039e6d (0x21fa bytes) 000297 1786032443.583063 trunc from 0x039e6e to 0x029465 (0x10a08 bytes) 000298 1786032443.606491 MIRROR_RESYNC 4 mirrors 000299 1786032445.483168 mapwrite 0x00f5a3 thru 0x018105 (0x8b63 bytes) 000300 1786032445.720789 mapread 0x015ef7 thru 0x017181 (0x128b bytes) 000301 1786032445.721903 trunc from 0x029466 to 0x026587 (0x2ede bytes) 000303 1786032445.797736 read 0x00d746 thru 0x01397c (0x6237 bytes) 000304 1786032445.799510 mapwrite 0x0344d9 thru 0x03c1d8 (0x7d00 bytes) 000305 1786032445.952481 read 0x018d04 thru 0x027ab6 (0xedb3 bytes) 000307 1786032445.974488 write 0x027fb8 thru 0x032838 (0xa881 bytes) 000309 1786032445.991459 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c1d9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1d9): Mirror 3: 0xa5b94ae5 (stale) Mirror 4: 0xa5b94ae5 (stale) Mirror 5: 0xa5b94ae5 (stale) CRC-32 checksum value for chunk [0, 0x3c1d9): Mirror 1: 0xd2e39510 Mirror 6: 0xd2e39510 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1d9: skipped 000310 1786032447.665321 read 0x02a7ad thru 0x03a4fc (0xfd50 bytes) truncating to largest ever: 0x3cf4e 000312 1786032447.693409 trunc from 0x03c1d9 to 0x03cf4d (0xd75 bytes) 000316 1786032447.825871 trunc from 0x03cf4e to 0x000e1d (0x3c130 bytes) 000317 1786032447.850091 mapwrite 0x0360c0 thru 0x0367be (0x6ff bytes) 000318 1786032447.913199 mapread 0x01036d thru 0x01fad4 (0xf768 bytes) 000319 1786032447.937333 read 0x008a4d thru 0x016295 (0xd849 bytes) 000321 1786032447.948086 mapwrite 0x03fd75 thru 0x03ffff (0x28b bytes) 000322 1786032448.025755 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8428c48e (stale) Mirror 5: 0x8428c48e (stale) Mirror 6: 0xee3c6ec4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8428c48e (stale) Mirror 5: 0x8428c48e (stale) Mirror 6: 0xee3c6ec4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000325 1786032448.491433 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8428c48e (stale) Mirror 5: 0x8428c48e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8428c48e (stale) Mirror 5: 0x8428c48e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000326 1786032448.803522 read 0x034fd2 thru 0x03ffff (0xb02e bytes) 000327 1786032448.810941 mapwrite 0x001fa1 thru 0x0035d5 (0x1635 bytes) 000328 1786032448.875053 write 0x02d3d3 thru 0x0317c4 (0x43f2 bytes) 000329 1786032448.900881 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8428c48e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8428c48e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000334 1786032449.317390 read 0x039e4c thru 0x03ffff (0x61b4 bytes) 000335 1786032449.360813 trunc from 0x040000 to 0x03964d (0x69b2 bytes) 000338 1786032449.410026 trunc from 0x03964e to 0x005fe5 (0x33668 bytes) 000339 1786032449.470724 trunc from 0x005fe6 to 0x03466c (0x2e687 bytes) 000341 1786032449.507848 write 0x027824 thru 0x02ecdb (0x74b8 bytes) 000342 1786032449.528958 write 0x00badb thru 0x010d09 (0x522f bytes) 000343 1786032449.555806 trunc from 0x03466d to 0x033908 (0xd64 bytes) 000344 1786032449.601443 read 0x00a09e thru 0x013492 (0x93f5 bytes) 000347 1786032449.617320 read 0x006d92 thru 0x00ce18 (0x6087 bytes) 000348 1786032449.618795 trunc from 0x033909 to 0x02439f (0xf569 bytes) 000349 1786032449.657908 mapwrite 0x037f37 thru 0x03f5e5 (0x76af bytes) 000351 1786032449.790630 mapwrite 0x021f0d thru 0x024a35 (0x2b29 bytes) 000354 1786032449.883503 write 0x01aa9f thru 0x01e4f2 (0x3a54 bytes) 000355 1786032449.889167 mapwrite 0x031a62 thru 0x032b70 (0x110f bytes) 000359 1786032450.030945 write 0x02c770 thru 0x036934 (0xa1c5 bytes) 000361 1786032450.034268 read 0x0348b2 thru 0x03f5e5 (0xad34 bytes) 000363 1786032450.035834 write 0x00d179 thru 0x016112 (0x8f9a bytes) 000364 1786032450.037790 mapwrite 0x022525 thru 0x029cc4 (0x77a0 bytes) 000366 1786032450.098660 mapread 0x03d8a4 thru 0x03f5e5 (0x1d42 bytes) 000367 1786032450.100315 mapwrite 0x029448 thru 0x03493a (0xb4f3 bytes) 000371 1786032450.154407 write 0x03ed6f thru 0x03ffff (0x1291 bytes) 000373 1786032450.157467 write 0x00ec3a thru 0x0141ee (0x55b5 bytes) 000374 1786032450.159334 write 0x00be6e thru 0x018bde (0xcd71 bytes) 000375 1786032450.161477 read 0x0285d7 thru 0x028646 (0x70 bytes) 000378 1786032450.163317 mapread 0x01ca9e thru 0x01ce1b (0x37e bytes) 000381 1786032450.173328 MIRROR_RESYNC 2 mirrors 000382 1786032451.395443 write 0x020aa5 thru 0x02838d (0x78e9 bytes) 000383 1786032451.552303 mapwrite 0x00c6af thru 0x017076 (0xa9c8 bytes) 000384 1786032451.670610 read 0x03db65 thru 0x03ffff (0x249b bytes) 000385 1786032451.682007 write 0x007178 thru 0x010c6b (0x9af4 bytes) 000386 1786032451.683419 read 0x008a20 thru 0x016a61 (0xe042 bytes) 000387 1786032451.684126 trunc from 0x040000 to 0x032a6a (0xd595 bytes) 000393 1786032451.740405 write 0x02dead thru 0x0398e0 (0xba34 bytes) 000395 1786032451.755762 read 0x01d111 thru 0x02333f (0x622f bytes) 000396 1786032451.757196 mapwrite 0x016f28 thru 0x0188bd (0x1996 bytes) 000397 1786032451.904138 mapwrite 0x0039b8 thru 0x00e33a (0xa983 bytes) 000399 1786032452.019539 read 0x012941 thru 0x021f21 (0xf5e1 bytes) 000400 1786032452.035442 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000401 1786032452.286534 trunc from 0x0398e1 to 0x01f0c3 (0x1a81d bytes) 000402 1786032452.364742 mapwrite 0x01a7f0 thru 0x01f33e (0x4b4f bytes) 000403 1786032452.586859 trunc from 0x01f33f to 0x0229ea (0x36ac bytes) 000404 1786032452.622519 read 0x0224f6 thru 0x0229ea (0x4f5 bytes) 000406 1786032452.638089 write 0x01864b thru 0x01cfb6 (0x496c bytes) 000408 1786032452.646312 mapread 0x00a60b thru 0x0138c3 (0x92b9 bytes) 000410 1786032452.699298 read 0x0205d7 thru 0x0229ea (0x2414 bytes) 000412 1786032452.729509 trunc from 0x0229eb to 0x011ea2 (0x10b48 bytes) 000413 1786032452.759208 trunc from 0x011ea3 to 0x01ccde (0xae3c bytes) 000416 1786032452.802106 write 0x01adc9 thru 0x01faf6 (0x4d2e bytes) truncating to largest ever: 0x3d330 000417 1786032452.809572 trunc from 0x01faf7 to 0x03d32f (0x1d839 bytes) 000418 1786032452.841991 mapwrite 0x004696 thru 0x007f4e (0x38b9 bytes) 000420 1786032452.933292 mapwrite 0x015cc5 thru 0x01d1fc (0x7538 bytes) 000422 1786032453.027517 trunc from 0x03d330 to 0x025e8b (0x174a4 bytes) 000423 1786032453.046599 mapread 0x011582 thru 0x01c430 (0xaeaf bytes) 000424 1786032453.066712 write 0x03f605 thru 0x03ffff (0x9fb bytes) 000425 1786032453.067780 mapread 0x027a66 thru 0x033ddd (0xc378 bytes) 000426 1786032453.131606 write 0x0239d3 thru 0x02b989 (0x7fb7 bytes) 000429 1786032453.135817 read 0x00cb40 thru 0x0108ac (0x3d6d bytes) 000430 1786032453.136677 write 0x027498 thru 0x02a178 (0x2ce1 bytes) 000432 1786032453.139380 mapwrite 0x03b220 thru 0x03e6f9 (0x34da bytes) 000434 1786032453.222991 mapwrite 0x01a28f thru 0x0256d7 (0xb449 bytes) 000435 1786032453.315950 write 0x0222e9 thru 0x0276ba (0x53d2 bytes) 000436 1786032453.317102 write 0x03cf0d thru 0x03ffff (0x30f3 bytes) 000438 1786032453.321008 write 0x02955d thru 0x036691 (0xd135 bytes) 000440 1786032453.331012 mapread 0x01999d thru 0x025a29 (0xc08d bytes) 000442 1786032453.339756 trunc from 0x040000 to 0x0283f6 (0x17c09 bytes) 000443 1786032453.357939 trunc from 0x0283f7 to 0x0376ec (0xf2f6 bytes) 000445 1786032453.378400 trunc from 0x0376ed to 0x017c5f (0x1fa8d bytes) 000447 1786032453.399343 mapread 0x00915d thru 0x00fb11 (0x69b5 bytes) 000449 1786032453.402171 mapwrite 0x01d651 thru 0x02a51f (0xcecf bytes) 000450 1786032453.663746 read 0x01a1fe thru 0x01af58 (0xd5b bytes) 000451 1786032453.677984 mapread 0x004c0d thru 0x01001a (0xb40e bytes) 000453 1786032453.688900 MIRROR_EXTEND to 2 mirrors 000454 1786032454.937870 write 0x012638 thru 0x0130c5 (0xa8e bytes) 000458 1786032455.099888 mapwrite 0x03634b thru 0x03ffff (0x9cb5 bytes) 000459 1786032455.265286 mapwrite 0x026abb thru 0x0310c8 (0xa60e bytes) 000460 1786032455.452358 mapread 0x00a641 thru 0x01822a (0xdbea bytes) 000463 1786032455.503263 mapwrite 0x008c57 thru 0x008ead (0x257 bytes) 000464 1786032455.632199 read 0x0140a9 thru 0x01f0a1 (0xaff9 bytes) 000467 1786032455.643073 read 0x03aa42 thru 0x03f8a9 (0x4e68 bytes) 000468 1786032455.646765 write 0x029c25 thru 0x02c980 (0x2d5c bytes) 000469 1786032455.649157 read 0x03cb67 thru 0x03ffff (0x3499 bytes) 000471 1786032455.652129 trunc from 0x040000 to 0x014615 (0x2b9ea bytes) 000472 1786032455.672060 read 0x000fb3 thru 0x007fee (0x703c bytes) 000473 1786032455.678719 mapread 0x014399 thru 0x014615 (0x27d bytes) 000474 1786032455.680594 write 0x02f85d thru 0x034583 (0x4d27 bytes) 000475 1786032455.682080 write 0x03d190 thru 0x03ffff (0x2e70 bytes) 000476 1786032455.684104 trunc from 0x040000 to 0x01d699 (0x22966 bytes) 000477 1786032455.721598 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d69a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d69a): Mirror 2: 0xcfcf5afe (stale) CRC-32 checksum value for chunk [0, 0x1d69a): Mirror 1: 0x6bb8be25 Mirror 3: 0x6bb8be25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d69a: skipped 000478 1786032456.904526 read 0x00840d thru 0x009bca (0x17be bytes) 000480 1786032456.910907 mapread 0x007a9b thru 0x00ac09 (0x316f bytes) 000481 1786032456.921967 mapwrite 0x030a20 thru 0x032084 (0x1665 bytes) 000482 1786032457.053455 trunc from 0x032085 to 0x00d9f9 (0x2468b bytes) 000483 1786032457.079775 write 0x03c31a thru 0x03ffff (0x3ce6 bytes) 000484 1786032457.081019 read 0x01887d thru 0x0217a2 (0x8f26 bytes) 000485 1786032457.091459 read 0x0373b2 thru 0x039119 (0x1d68 bytes) 000486 1786032457.092213 mapread 0x02b8c7 thru 0x02ceda (0x1614 bytes) 000487 1786032457.093869 mapwrite 0x03a5d9 thru 0x03e994 (0x43bc bytes) 000488 1786032457.148212 MIRROR_RESYNC 3 mirrors 000489 1786032458.562043 trunc from 0x040000 to 0x026ea5 (0x1915a bytes) 000490 1786032458.647740 trunc from 0x026ea6 to 0x004d0a (0x2219b bytes) 000493 1786032458.680467 read 0x003043 thru 0x004d0a (0x1cc8 bytes) 000495 1786032458.696318 MIRROR_RESYNC 3 mirrors 000497 1786032459.398971 write 0x03d08a thru 0x03ffff (0x2f76 bytes) 000498 1786032459.469835 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4651976d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4651976d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000499 1786032459.791892 write 0x002471 thru 0x0110d1 (0xec61 bytes) 000500 1786032459.807330 mapread 0x00bdcb thru 0x017757 (0xb98d bytes) 000502 1786032459.831046 mapwrite 0x024c59 thru 0x02ee5f (0xa207 bytes) 000503 1786032459.945356 trunc from 0x040000 to 0x038a4e (0x75b1 bytes) 000504 1786032459.990049 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) 000505 1786032460.159280 mapwrite 0x01c2bf thru 0x01d197 (0xed9 bytes) 000506 1786032460.206431 write 0x029b61 thru 0x039023 (0xf4c3 bytes) 000509 1786032460.214165 write 0x03e55e thru 0x03ffff (0x1aa2 bytes) 000510 1786032460.216844 mapwrite 0x00928b thru 0x0103ea (0x7160 bytes) 000513 1786032460.310216 write 0x01b000 thru 0x021459 (0x645a bytes) 000519 1786032460.339775 write 0x02704d thru 0x02b279 (0x422d bytes) 000520 1786032460.345816 mapwrite 0x015390 thru 0x0220cf (0xcd40 bytes) 000521 1786032460.457948 mapread 0x00b2bd thru 0x00e42d (0x3171 bytes) 000522 1786032460.460175 trunc from 0x040000 to 0x006944 (0x396bb bytes) 000523 1786032460.486688 read 0x001830 thru 0x006944 (0x5115 bytes) 000524 1786032460.491981 read 0x006706 thru 0x006944 (0x23f bytes) 000525 1786032460.493790 trunc from 0x006945 to 0x034a53 (0x2e10f bytes) 000528 1786032460.513816 trunc from 0x034a54 to 0x016210 (0x1e843 bytes) 000529 1786032460.537519 trunc from 0x016211 to 0x026a00 (0x107f0 bytes) 000530 1786032460.560321 mapread 0x014380 thru 0x01fefd (0xbb7e bytes) 000531 1786032460.570932 write 0x027b84 thru 0x02aa95 (0x2f12 bytes) 000533 1786032460.572644 mapwrite 0x01bdc7 thru 0x025ea5 (0xa0df bytes) 000534 1786032460.631861 mapwrite 0x02bfe6 thru 0x02caf0 (0xb0b bytes) 000536 1786032460.704122 trunc from 0x02caf1 to 0x01bc3c (0x10eb4 bytes) 000537 1786032460.724905 mapread 0x019895 thru 0x01ae32 (0x159e bytes) 000538 1786032460.725942 mapread 0x0010fc thru 0x008384 (0x7289 bytes) 000539 1786032460.728891 write 0x0383e5 thru 0x03ffff (0x7c1b bytes) 000541 1786032460.732450 write 0x021c67 thru 0x027315 (0x56af bytes) 000542 1786032460.744678 mapwrite 0x02b02d thru 0x02eb95 (0x3b69 bytes) 000543 1786032460.842848 read 0x02dc68 thru 0x03a648 (0xc9e1 bytes) truncating to largest ever: 0x3f2b4 000546 1786032460.850913 trunc from 0x040000 to 0x03f2b3 (0xd4c bytes) 000548 1786032460.874169 mapwrite 0x0005da thru 0x00f63b (0xf062 bytes) 000549 1786032460.965814 write 0x009967 thru 0x0125f0 (0x8c8a bytes) 000550 1786032460.970392 mapread 0x029087 thru 0x02b52b (0x24a5 bytes) 000551 1786032460.981899 read 0x010756 thru 0x01ac94 (0xa53f bytes) 000553 1786032460.983376 mapread 0x00e921 thru 0x01a859 (0xbf39 bytes) 000554 1786032460.985832 trunc from 0x03f2b4 to 0x012d38 (0x2c57b bytes) 000555 1786032461.007490 MIRROR_EXTEND to 2 mirrors 000556 1786032462.095661 mapread 0x001144 thru 0x0026f7 (0x15b4 bytes) 000559 1786032462.111054 read 0x00965b thru 0x012d38 (0x96de bytes) 000561 1786032462.121632 mapwrite 0x002c5e thru 0x012b0a (0xfead bytes) truncating to largest ever: 0x3fcc5 000562 1786032462.432627 trunc from 0x012d39 to 0x03fcc4 (0x2cf8c bytes) 000563 1786032462.503740 read 0x02fbf1 thru 0x03cbb3 (0xcfc3 bytes) 000564 1786032462.521917 trunc from 0x03fcc5 to 0x000db1 (0x3ef13 bytes) 000569 1786032462.551655 trunc from 0x000db2 to 0x01711f (0x1636e bytes) 000570 1786032462.578596 MIRROR_RESYNC 2 mirrors 000571 1786032463.369525 mapread 0x014729 thru 0x01711f (0x29f7 bytes) 000572 1786032463.392305 mapread 0x016d01 thru 0x01711f (0x41f bytes) 000573 1786032463.394135 mapread 0x00e67f thru 0x01711f (0x8aa1 bytes) 000574 1786032463.405573 MIRROR_EXTEND to 3 mirrors 000575 1786032464.424042 write 0x01a0a1 thru 0x02882c (0xe78c bytes) 000576 1786032464.593567 mapread 0x010333 thru 0x011fbd (0x1c8b bytes) 000578 1786032464.617993 mapwrite 0x030efb thru 0x03e981 (0xda87 bytes) 000579 1786032464.856172 read 0x01ab26 thru 0x0253b0 (0xa88b bytes) 000582 1786032464.858154 trunc from 0x03e982 to 0x02f5fd (0xf384 bytes) 000584 1786032464.894547 read 0x00e8cd thru 0x01d43f (0xeb73 bytes) 000585 1786032464.906677 write 0x03a4ee thru 0x03ffff (0x5b12 bytes) 000586 1786032464.908047 write 0x02b2da thru 0x03a50f (0xf236 bytes) 000587 1786032464.910594 mapread 0x0338fa thru 0x03beaf (0x85b6 bytes) 000592 1786032464.915891 trunc from 0x040000 to 0x038904 (0x76fb bytes) 000594 1786032464.971719 mapwrite 0x00b0c9 thru 0x0131e3 (0x811b bytes) 000596 1786032465.060169 mapwrite 0x02c0e3 thru 0x037a69 (0xb987 bytes) 000597 1786032465.186004 trunc from 0x038905 to 0x039a26 (0x1122 bytes) 000599 1786032465.250846 mapread 0x03800b thru 0x038ee0 (0xed6 bytes) 000604 1786032465.296405 mapwrite 0x012afb thru 0x01fc5c (0xd162 bytes) 000605 1786032465.454751 trunc from 0x039a27 to 0x00fc83 (0x29da3 bytes) 000606 1786032465.495291 write 0x0038b3 thru 0x00469a (0xde8 bytes) 000608 1786032465.498748 write 0x035963 thru 0x03a8cd (0x4f6b bytes) 000610 1786032465.504055 read 0x00140f thru 0x00e469 (0xd05b bytes) 000611 1786032465.505760 trunc from 0x03a8ce to 0x0290d6 (0x117f7 bytes) 000615 1786032465.563094 MIRROR_RESYNC 3 mirrors 000616 1786032466.869768 read 0x021e60 thru 0x0290d6 (0x7277 bytes) 000618 1786032466.891217 mapwrite 0x0290a1 thru 0x0296c3 (0x623 bytes) 000620 1786032467.092956 trunc from 0x0296c4 to 0x0119c5 (0x17cfe bytes) 000621 1786032467.127398 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x119c6): Mirror 3: 0x571dbe3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x119c6): Mirror 3: 0x571dbe3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x119c6: skipped 000622 1786032467.599831 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x119c6): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x119c6): Mirror 3: 0x571dbe3 (stale) CRC-32 checksum value for chunk [0, 0x119c6): Mirror 1: 0x571dbe3 Mirror 4: 0x571dbe3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x119c6: skipped 000624 1786032469.396292 trunc from 0x0119c6 to 0x002d5d (0xec68 bytes) 000626 1786032469.494273 mapread 0x002a97 thru 0x002d5d (0x2c7 bytes) 000627 1786032469.503987 read 0x0005de thru 0x002d5d (0x2780 bytes) 000628 1786032469.512594 read 0x002d0e thru 0x002d5d (0x50 bytes) 000631 1786032469.516111 trunc from 0x002d5e to 0x008764 (0x5a07 bytes) 000632 1786032469.546519 trunc from 0x008765 to 0x03607b (0x2d917 bytes) 000633 1786032469.569992 trunc from 0x03607c to 0x039256 (0x31db bytes) 000634 1786032469.591619 read 0x01632b thru 0x0226ca (0xc3a0 bytes) 000635 1786032469.614407 mapwrite 0x0386a1 thru 0x03c333 (0x3c93 bytes) 000636 1786032469.703536 mapread 0x02751f thru 0x02de13 (0x68f5 bytes) 000637 1786032469.713213 read 0x02845c thru 0x02d000 (0x4ba5 bytes) 000638 1786032469.723229 mapwrite 0x012bfb thru 0x021020 (0xe426 bytes) 000639 1786032469.967465 read 0x009258 thru 0x00cd00 (0x3aa9 bytes) 000643 1786032469.972163 trunc from 0x03c334 to 0x038827 (0x3b0c bytes) 000648 1786032470.039132 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38828): Mirror 3: 0xcc99189a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38828): Mirror 3: 0xcc99189a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38828: skipped 000649 1786032470.741495 MIRROR_RESYNC 2 mirrors 000650 1786032472.044634 write 0x03a467 thru 0x03ffff (0x5b99 bytes) 000651 1786032472.151642 trunc from 0x040000 to 0x0327d6 (0xd829 bytes) 000652 1786032472.188941 read 0x02a46b thru 0x0327d6 (0x836c bytes) 000653 1786032472.199178 mapwrite 0x01277d thru 0x013cff (0x1583 bytes) 000654 1786032472.287317 read 0x029649 thru 0x0327d6 (0x918e bytes) 000655 1786032472.305192 trunc from 0x0327d7 to 0x01f8a3 (0x12f33 bytes) 000656 1786032472.332653 mapread 0x00a532 thru 0x00bdec (0x18bb bytes) 000657 1786032472.336140 read 0x00fba0 thru 0x00fd1b (0x17c bytes) 000658 1786032472.337290 trunc from 0x01f8a4 to 0x007048 (0x1885b bytes) 000659 1786032472.364433 trunc from 0x007049 to 0x02c176 (0x2512e bytes) 000660 1786032472.397054 mapread 0x01d964 thru 0x0222e9 (0x4986 bytes) 000661 1786032472.426782 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c177): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c177): Mirror 3: 0xaaecc6e5 (stale) CRC-32 checksum value for chunk [0, 0x2c177): Mirror 1: 0xd886a075 Mirror 4: 0xd886a075 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c177: skipped 000663 1786032474.025206 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c177): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c177): Mirror 3: 0xaaecc6e5 (stale) CRC-32 checksum value for chunk [0, 0x2c177): Mirror 1: 0xd886a075 Mirror 4: 0xd886a075 Mirror 5: 0xd886a075 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c177: skipped 000665 1786032475.272623 read 0x016b8a thru 0x025095 (0xe50c bytes) 000667 1786032475.302570 mapread 0x00fccb thru 0x010da3 (0x10d9 bytes) 000670 1786032475.359988 mapread 0x00c832 thru 0x0192bb (0xca8a bytes) 000671 1786032475.395839 write 0x0289f5 thru 0x02defd (0x5509 bytes) 000674 1786032475.531006 mapwrite 0x002b45 thru 0x00aa8c (0x7f48 bytes) 000675 1786032475.613757 read 0x0060f6 thru 0x0141b2 (0xe0bd bytes) 000676 1786032475.624992 trunc from 0x02defe to 0x02a8fb (0x3602 bytes) 000677 1786032475.674884 trunc from 0x02a8fc to 0x00f386 (0x1b575 bytes) 000678 1786032475.699114 read 0x004e49 thru 0x009318 (0x44d0 bytes) 000679 1786032475.702701 read 0x001334 thru 0x00f386 (0xe053 bytes) 000681 1786032475.704900 write 0x00f155 thru 0x013089 (0x3f35 bytes) 000682 1786032475.715953 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1308a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1308a): Mirror 3: 0x6b5056f7 (stale) Mirror 4: 0x90ee7f75 (stale) Mirror 5: 0x90ee7f75 (stale) CRC-32 checksum value for chunk [0, 0x1308a): Mirror 1: 0x5e151af1 Mirror 6: 0x5e151af1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1308a: skipped 000684 1786032477.596168 read 0x00f42b thru 0x013089 (0x3c5f bytes) 000685 1786032477.624786 mapread 0x00a093 thru 0x00a56e (0x4dc bytes) 000686 1786032477.678557 write 0x024b43 thru 0x02e0fa (0x95b8 bytes) 000687 1786032477.809052 mapread 0x002c09 thru 0x00ccef (0xa0e7 bytes) 000689 1786032477.852100 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e0fb): Mirror 4: 0x1e609370 (stale) Mirror 5: 0x1e609370 (stale) Mirror 6: 0x339b2ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0fb): Mirror 4: 0x1e609370 (stale) Mirror 5: 0x1e609370 (stale) Mirror 6: 0x339b2ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0fb: skipped 000690 1786032478.423567 mapread 0x008a54 thru 0x0105f4 (0x7ba1 bytes) 000691 1786032478.458087 read 0x0217ca thru 0x02a1da (0x8a11 bytes) 000693 1786032478.468377 write 0x0246c0 thru 0x02a5b6 (0x5ef7 bytes) 000694 1786032478.469582 trunc from 0x02e0fb to 0x03bd4d (0xdc53 bytes) 000695 1786032478.510876 mapread 0x01e84c thru 0x023ecd (0x5682 bytes) 000697 1786032478.525371 trunc from 0x03bd4e to 0x01c5b3 (0x1f79a bytes) 000701 1786032478.565530 mapwrite 0x00ca2d thru 0x01c08a (0xf65e bytes) 000702 1786032478.670675 mapwrite 0x0336f4 thru 0x03f4e3 (0xbdf0 bytes) 000704 1786032478.874913 write 0x02df0d thru 0x03a1a3 (0xc297 bytes) 000705 1786032478.880463 mapwrite 0x03488f thru 0x039855 (0x4fc7 bytes) 000709 1786032478.942904 trunc from 0x03f4e4 to 0x03231a (0xd1c9 bytes) 000710 1786032478.996992 mapwrite 0x02e449 thru 0x03661c (0x81d4 bytes) 000711 1786032479.155132 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3661d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3661d): Mirror 4: 0x2367a4dc (stale) Mirror 5: 0x2367a4dc (stale) Mirror 6: 0xa2f9075c (stale) CRC-32 checksum value for chunk [0, 0x3661d): Mirror 1: 0x4a87dff5 Mirror 7: 0x4a87dff5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3661d: skipped 000712 1786032481.131050 trunc from 0x03661d to 0x0299b9 (0xcc63 bytes) 000713 1786032481.280908 write 0x010aae thru 0x01d606 (0xcb59 bytes) 000714 1786032481.295410 trunc from 0x0299ba to 0x02cbde (0x3225 bytes) 000719 1786032481.326771 mapread 0x025b64 thru 0x028b0e (0x2fab bytes) 000720 1786032481.360704 mapwrite 0x00f09a thru 0x017efe (0x8e65 bytes) 000721 1786032481.457555 mapwrite 0x03c38d thru 0x03cf24 (0xb98 bytes) 000723 1786032481.589932 mapwrite 0x023cc1 thru 0x029e09 (0x6149 bytes) 000726 1786032481.702056 read 0x027edf thru 0x02a424 (0x2546 bytes) 000727 1786032481.703588 trunc from 0x03cf25 to 0x021ff7 (0x1af2d bytes) 000728 1786032481.762456 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21ff8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ff8): Mirror 4: 0xea4e76f7 (stale) Mirror 5: 0xea4e76f7 (stale) Mirror 6: 0x71189217 (stale) Mirror 7: 0xdc5fb791 (stale) CRC-32 checksum value for chunk [0, 0x21ff8): Mirror 1: 0xed602d62 Mirror 8: 0xed602d62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ff8: skipped 000731 1786032484.057707 mapwrite 0x01513e thru 0x0183ba (0x327d bytes) 000734 1786032484.352470 mapwrite 0x01bb88 thru 0x01d45f (0x18d8 bytes) 000737 1786032484.415482 mapwrite 0x007684 thru 0x014e61 (0xd7de bytes) 000739 1786032484.528675 MIRROR_RESYNC 6 mirrors 000740 1786032486.663928 trunc from 0x021ff8 to 0x03e197 (0x1c1a0 bytes) 000741 1786032486.727089 mapwrite 0x03d286 thru 0x03ffff (0x2d7a bytes) 000743 1786032486.813086 mapread 0x0204ea thru 0x028c05 (0x871c bytes) 000744 1786032486.827396 mapwrite 0x02389a thru 0x0325ca (0xed31 bytes) 000745 1786032486.933111 trunc from 0x040000 to 0x01e169 (0x21e96 bytes) 000746 1786032486.962727 mapwrite 0x01836a thru 0x018b40 (0x7d7 bytes) 000747 1786032487.022491 write 0x01f801 thru 0x025281 (0x5a81 bytes) 000748 1786032487.024083 read 0x01d979 thru 0x024f46 (0x75ce bytes) 000750 1786032487.027565 read 0x01f19e thru 0x025281 (0x60e4 bytes) 000751 1786032487.028768 write 0x023700 thru 0x02b3a6 (0x7ca7 bytes) 000753 1786032487.033380 write 0x029035 thru 0x02f73c (0x6708 bytes) 000754 1786032487.036171 write 0x03c503 thru 0x03ffff (0x3afd bytes) 000755 1786032487.038763 read 0x015c31 thru 0x020b56 (0xaf26 bytes) 000757 1786032487.041764 mapread 0x01c77b thru 0x0282d9 (0xbb5f bytes) 000758 1786032487.044801 trunc from 0x040000 to 0x009995 (0x3666a bytes) 000760 1786032487.062000 read 0x006bf9 thru 0x009995 (0x2d9d bytes) 000761 1786032487.072756 mapread 0x000acf thru 0x003e00 (0x3332 bytes) 000762 1786032487.074207 read 0x006c56 thru 0x009995 (0x2d40 bytes) 000763 1786032487.076146 read 0x00018e thru 0x009995 (0x9808 bytes) 000765 1786032487.079045 write 0x02d07c thru 0x0356b1 (0x8636 bytes) 000769 1786032487.097942 read 0x0251e6 thru 0x02d814 (0x862f bytes) 000770 1786032487.114047 read 0x0041d0 thru 0x00a3a5 (0x61d6 bytes) 000771 1786032487.122017 mapwrite 0x02f7ef thru 0x037a70 (0x8282 bytes) 000777 1786032487.308803 mapwrite 0x02baab thru 0x0399e4 (0xdf3a bytes) 000778 1786032487.493919 mapread 0x032c4e thru 0x0343de (0x1791 bytes) 000780 1786032487.515895 MIRROR_RESYNC 6 mirrors 000781 1786032490.110859 mapread 0x01015f thru 0x01034c (0x1ee bytes) 000782 1786032490.120175 write 0x00d641 thru 0x011d3e (0x46fe bytes) 000785 1786032490.343013 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x399e5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x399e5): Mirror 4: 0xcef5edc0 (stale) Mirror 5: 0xcef5edc0 (stale) Mirror 6: 0xcef5edc0 (stale) Mirror 7: 0xcef5edc0 (stale) Mirror 8: 0xcef5edc0 (stale) CRC-32 checksum value for chunk [0, 0x399e5): Mirror 1: 0x2feff02f Mirror 9: 0x2feff02f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x399e5: skipped 000786 1786032492.348995 write 0x03d7be thru 0x03ffff (0x2842 bytes) 000788 1786032492.410469 MIRROR_RESYNC 7 mirrors 000789 1786032496.362398 mapread 0x03dc13 thru 0x03ffff (0x23ed bytes) 000793 1786032496.428756 trunc from 0x040000 to 0x0368c0 (0x973f bytes) 000795 1786032496.575877 write 0x02961b thru 0x02f38d (0x5d73 bytes) 000796 1786032496.592596 mapread 0x00c564 thru 0x00e4d4 (0x1f71 bytes) 000797 1786032496.609116 trunc from 0x0368c1 to 0x038f95 (0x26d5 bytes) 000798 1786032496.656176 read 0x017738 thru 0x01a4a5 (0x2d6e bytes) 000800 1786032496.674561 mapread 0x00259b thru 0x0081fb (0x5c61 bytes) 000801 1786032496.705475 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38f96): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f96): Mirror 4: 0xf3a7dc91 (stale) Mirror 5: 0xf3a7dc91 (stale) Mirror 6: 0xf3a7dc91 (stale) Mirror 7: 0xf3a7dc91 (stale) Mirror 8: 0xf3a7dc91 (stale) Mirror 9: 0xf3a7dc91 (stale) CRC-32 checksum value for chunk [0, 0x38f96): Mirror 1: 0x7d2757c Mirror 10: 0x7d2757c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f96: skipped 000803 1786032499.380036 trunc from 0x038f96 to 0x0267de (0x127b7 bytes) 000804 1786032499.486046 read 0x00c4d5 thru 0x00dc55 (0x1781 bytes) 000806 1786032499.494025 mapwrite 0x012df6 thru 0x0224a6 (0xf6b1 bytes) 000807 1786032499.618609 mapread 0x0212ba thru 0x02457f (0x32c6 bytes) 000808 1786032499.621921 read 0x01e73f thru 0x01f4a9 (0xd6b bytes) 000809 1786032499.623173 write 0x0287a3 thru 0x034041 (0xb89f bytes) 000812 1786032499.629716 mapread 0x015484 thru 0x01be3b (0x69b8 bytes) 000815 1786032499.636581 mapread 0x025b77 thru 0x0309b7 (0xae41 bytes) 000816 1786032499.649742 mapwrite 0x00ca7b thru 0x01ae5a (0xe3e0 bytes) 000819 1786032499.757947 write 0x021209 thru 0x029e84 (0x8c7c bytes) 000820 1786032499.760152 mapread 0x020656 thru 0x03050a (0xfeb5 bytes) 000821 1786032499.762115 trunc from 0x034042 to 0x028093 (0xbfae bytes) 000823 1786032499.797063 mapread 0x009f9a thru 0x0111f4 (0x725b bytes) 000825 1786032499.808378 MIRROR_RESYNC 8 mirrors 000826 1786032502.472348 mapwrite 0x01cceb thru 0x02bffd (0xf313 bytes) 000828 1786032502.857732 mapread 0x00df34 thru 0x01695d (0x8a2a bytes) 000830 1786032502.879539 mapread 0x021ae6 thru 0x02801e (0x6539 bytes) 000831 1786032502.883464 write 0x0002ac thru 0x00222c (0x1f81 bytes) 000834 1786032502.904343 MIRROR_RESYNC 8 mirrors 000837 1786032506.070478 mapread 0x0100c0 thru 0x015587 (0x54c8 bytes) 000839 1786032506.137004 trunc from 0x02bffe to 0x01c9db (0xf622 bytes) 000843 1786032506.355470 write 0x02e5f6 thru 0x035f63 (0x796e bytes) 000848 1786032506.363948 read 0x02fde2 thru 0x035f63 (0x6182 bytes) 000854 1786032506.388703 read 0x004623 thru 0x008d2a (0x4708 bytes) 000855 1786032506.407782 trunc from 0x035f64 to 0x027aa4 (0xe4bf bytes) 000857 1786032506.485897 read 0x004a63 thru 0x010e22 (0xc3c0 bytes) 000861 1786032506.515644 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27aa5): Mirror 4: 0x401cf9fd (stale) Mirror 5: 0x401cf9fd (stale) Mirror 6: 0x401cf9fd (stale) Mirror 7: 0x401cf9fd (stale) Mirror 9: 0x401cf9fd (stale) Mirror 10: 0x401cf9fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27aa5): Mirror 4: 0x401cf9fd (stale) Mirror 5: 0x401cf9fd (stale) Mirror 6: 0x401cf9fd (stale) Mirror 7: 0x401cf9fd (stale) Mirror 9: 0x401cf9fd (stale) Mirror 10: 0x401cf9fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27aa5: skipped 000864 1786032507.208022 read 0x024dbe thru 0x027aa4 (0x2ce7 bytes) 000866 1786032507.235778 mapread 0x00edec thru 0x01370e (0x4923 bytes) 000867 1786032507.261090 mapwrite 0x011c59 thru 0x01bad3 (0x9e7b bytes) 000868 1786032507.360708 mapread 0x01ff05 thru 0x027aa4 (0x7ba0 bytes) 000870 1786032507.371889 mapwrite 0x01da14 thru 0x01e711 (0xcfe bytes) 000871 1786032507.507375 read 0x0041ef thru 0x00b5fb (0x740d bytes) 000873 1786032507.523458 mapwrite 0x02fdf3 thru 0x03612e (0x633c bytes) 000875 1786032507.695672 trunc from 0x03612f to 0x01b35f (0x1adcf bytes) 000878 1786032507.759033 write 0x005256 thru 0x00e707 (0x94b2 bytes) 000881 1786032507.779739 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b360): Mirror 4: 0xbd02bd6c (stale) Mirror 6: 0xbd02bd6c (stale) Mirror 7: 0xbd02bd6c (stale) Mirror 9: 0xbd02bd6c (stale) Mirror 10: 0xbd02bd6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b360): Mirror 4: 0xbd02bd6c (stale) Mirror 6: 0xbd02bd6c (stale) Mirror 7: 0xbd02bd6c (stale) Mirror 9: 0xbd02bd6c (stale) Mirror 10: 0xbd02bd6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b360: skipped 000882 1786032508.506201 read 0x017b00 thru 0x01b35f (0x3860 bytes) 000885 1786032508.562663 write 0x03c811 thru 0x03ffff (0x37ef bytes) 000886 1786032508.586058 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc1693b15 (stale) Mirror 6: 0xc1693b15 (stale) Mirror 7: 0xc1693b15 (stale) Mirror 9: 0xc1693b15 (stale) Mirror 10: 0xc1693b15 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x568c6272 Mirror 11: 0x568c6272 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000887 1786032510.261214 write 0x032be8 thru 0x032c9d (0xb6 bytes) 000890 1786032510.425633 mapwrite 0x038722 thru 0x03adfd (0x26dc bytes) 000891 1786032510.567015 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc1693b15 (stale) Mirror 6: 0xc1693b15 (stale) Mirror 7: 0xc1693b15 (stale) Mirror 9: 0xc1693b15 (stale) Mirror 10: 0xc1693b15 (stale) Mirror 11: 0x568c6272 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb5aba015 Mirror 12: 0xb5aba015 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000892 1786032511.829040 mapwrite 0x02c63e thru 0x032492 (0x5e55 bytes) 000893 1786032512.013376 mapwrite 0x01a33c thru 0x01c886 (0x254b bytes) 000896 1786032512.066316 mapwrite 0x00f483 thru 0x013669 (0x41e7 bytes) 000897 1786032512.128351 write 0x033c8a thru 0x03ffff (0xc376 bytes) 000898 1786032512.130886 trunc from 0x040000 to 0x018828 (0x277d7 bytes) 000899 1786032512.154748 read 0x0039c1 thru 0x00ef21 (0xb561 bytes) 000901 1786032512.163856 write 0x0062f1 thru 0x01140b (0xb11b bytes) 000902 1786032512.165585 read 0x00cdae thru 0x01428d (0x74e0 bytes) 000903 1786032512.166739 write 0x036129 thru 0x03ffff (0x9ed7 bytes) 000904 1786032512.168595 mapread 0x01baff thru 0x025cb1 (0xa1b3 bytes) 000905 1786032512.179655 trunc from 0x040000 to 0x039fb6 (0x6049 bytes) 000906 1786032512.212411 MIRROR_RESYNC 8 mirrors 000907 1786032515.590416 write 0x033c72 thru 0x03882d (0x4bbc bytes) 000908 1786032515.801796 mapwrite 0x02b594 thru 0x0321fc (0x6c69 bytes) 000909 1786032516.088744 mapwrite 0x028794 thru 0x03131b (0x8b88 bytes) 000910 1786032516.199500 read 0x00d072 thru 0x018927 (0xb8b6 bytes) 000912 1786032516.213483 mapread 0x00d382 thru 0x00d7fa (0x479 bytes) 000913 1786032516.214461 trunc from 0x039fb7 to 0x00d985 (0x2c631 bytes) 000917 1786032516.268902 mapread 0x0014b6 thru 0x00d985 (0xc4d0 bytes) 000918 1786032516.300134 mapread 0x0006b8 thru 0x0026be (0x2007 bytes) 000919 1786032516.310897 mapwrite 0x03acff thru 0x03f953 (0x4c55 bytes) 000921 1786032516.542634 write 0x02e9d8 thru 0x0371e8 (0x8811 bytes) 000923 1786032516.551349 mapread 0x0162f8 thru 0x016c5a (0x963 bytes) 000925 1786032516.596951 trunc from 0x03f954 to 0x03c80d (0x3146 bytes) 000926 1786032516.692361 mapread 0x00e3d4 thru 0x0178ce (0x94fb bytes) 000929 1786032516.710200 mapread 0x03121d thru 0x036153 (0x4f37 bytes) 000930 1786032516.712473 mapread 0x01a922 thru 0x024c68 (0xa347 bytes) 000931 1786032516.754888 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c80e): Mirror 6: 0xba128115 (stale) Mirror 7: 0xba128115 (stale) Mirror 9: 0xba128115 (stale) Mirror 10: 0xba128115 (stale) Mirror 11: 0xba128115 (stale) Mirror 12: 0xba128115 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c80e): Mirror 6: 0xba128115 (stale) Mirror 7: 0xba128115 (stale) Mirror 9: 0xba128115 (stale) Mirror 10: 0xba128115 (stale) Mirror 11: 0xba128115 (stale) Mirror 12: 0xba128115 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c80e: skipped 000932 1786032517.746997 mapread 0x034d57 thru 0x039b95 (0x4e3f bytes) 000934 1786032517.782668 trunc from 0x03c80e to 0x01d0f5 (0x1f718 bytes) 000935 1786032517.830685 write 0x02a334 thru 0x02f88a (0x5557 bytes) 000936 1786032517.833445 trunc from 0x02f88b to 0x011bb6 (0x1dcd4 bytes) 000937 1786032517.867857 trunc from 0x011bb7 to 0x003c1f (0xdf97 bytes) 000938 1786032517.889094 trunc from 0x003c20 to 0x031619 (0x2d9fa bytes) 000939 1786032517.931154 read 0x029bf4 thru 0x02ba72 (0x1e7f bytes) 000940 1786032517.938287 trunc from 0x03161a to 0x03ac94 (0x967b bytes) 000941 1786032517.955445 trunc from 0x03ac95 to 0x018f8f (0x21d05 bytes) 000942 1786032517.972764 write 0x03f555 thru 0x03ffff (0xaab bytes) 000943 1786032517.975620 mapread 0x01a661 thru 0x0235e8 (0x8f88 bytes) 000948 1786032518.011435 read 0x01eb90 thru 0x02a2a7 (0xb718 bytes) 000949 1786032518.013267 read 0x028aa4 thru 0x02cabe (0x401b bytes) 000950 1786032518.052842 MIRROR_RESYNC 7 mirrors 000952 1786032520.664914 mapwrite 0x02b72f thru 0x02f72a (0x3ffc bytes) 000954 1786032520.984307 write 0x02b9cd thru 0x02dafa (0x212e bytes) 000955 1786032520.988991 mapwrite 0x01d9dc thru 0x025764 (0x7d89 bytes) 000956 1786032521.123138 trunc from 0x040000 to 0x023fe3 (0x1c01c bytes) 000957 1786032521.157713 mapwrite 0x0269f5 thru 0x035ad3 (0xf0df bytes) 000958 1786032521.292390 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35ad4): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ad4): Mirror 6: 0x68210e9a (stale) Mirror 7: 0x68210e9a (stale) Mirror 9: 0x68210e9a (stale) Mirror 10: 0x68210e9a (stale) Mirror 11: 0x68210e9a (stale) Mirror 12: 0x68210e9a (stale) CRC-32 checksum value for chunk [0, 0x35ad4): Mirror 1: 0xa63acd17 Mirror 13: 0xa63acd17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ad4: skipped 000959 1786032522.582991 read 0x012d6f thru 0x02273e (0xf9d0 bytes) 000960 1786032522.615340 read 0x012785 thru 0x01b71c (0x8f98 bytes) 000962 1786032522.644355 write 0x039c1e thru 0x03d752 (0x3b35 bytes) 000963 1786032522.772062 mapread 0x01ebb5 thru 0x024ecb (0x6317 bytes) 000966 1786032522.790021 read 0x00e4e8 thru 0x019b96 (0xb6af bytes) 000967 1786032522.790817 write 0x000970 thru 0x00e953 (0xdfe4 bytes) 000968 1786032522.810198 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d753): Mirror 6: 0x441aee3b (stale) Mirror 7: 0x441aee3b (stale) Mirror 10: 0x441aee3b (stale) Mirror 11: 0x441aee3b (stale) Mirror 12: 0x441aee3b (stale) Mirror 13: 0x92a1f760 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d753): Mirror 6: 0x441aee3b (stale) Mirror 7: 0x441aee3b (stale) Mirror 10: 0x441aee3b (stale) Mirror 11: 0x441aee3b (stale) Mirror 12: 0x441aee3b (stale) Mirror 13: 0x92a1f760 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d753: skipped 000970 1786032523.472991 trunc from 0x03d753 to 0x01aef3 (0x2285f bytes) 000971 1786032523.520992 mapread 0x01923d thru 0x01aef3 (0x1cb7 bytes) 000972 1786032523.539341 mapwrite 0x03bea2 thru 0x03ffff (0x415e bytes) 000975 1786032523.634228 trunc from 0x040000 to 0x009b1c (0x364e3 bytes) 000978 1786032523.657440 read 0x00218c thru 0x00850f (0x6384 bytes) 000979 1786032523.669950 trunc from 0x009b1d to 0x030698 (0x26b7c bytes) 000980 1786032523.687205 read 0x0156b2 thru 0x01c9d2 (0x7321 bytes) 000982 1786032523.700391 read 0x00f686 thru 0x01ce4a (0xd7c5 bytes) 000983 1786032523.702661 trunc from 0x030699 to 0x0255ad (0xb0eb bytes) 000984 1786032523.729508 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x255ae): Mirror 6: 0x622a73d6 (stale) Mirror 7: 0x622a73d6 (stale) Mirror 10: 0x622a73d6 (stale) Mirror 12: 0x622a73d6 (stale) Mirror 13: 0xfb067854 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x255ae): Mirror 6: 0x622a73d6 (stale) Mirror 7: 0x622a73d6 (stale) Mirror 10: 0x622a73d6 (stale) Mirror 12: 0x622a73d6 (stale) Mirror 13: 0xfb067854 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x255ae: skipped 000986 1786032524.316289 read 0x021e0a thru 0x0255ad (0x37a4 bytes) 000987 1786032524.345079 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x255ae): Mirror 6: 0x622a73d6 (stale) Mirror 7: 0x622a73d6 (stale) Mirror 10: 0x622a73d6 (stale) Mirror 13: 0xfb067854 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x255ae): Mirror 6: 0x622a73d6 (stale) Mirror 7: 0x622a73d6 (stale) Mirror 10: 0x622a73d6 (stale) Mirror 13: 0xfb067854 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x255ae: skipped 000988 1786032524.896971 write 0x02ed7d thru 0x03b945 (0xcbc9 bytes) 000989 1786032524.914596 write 0x000511 thru 0x00b3e1 (0xaed1 bytes) 000990 1786032524.984665 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b946): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b946): Mirror 6: 0x670e3455 (stale) Mirror 7: 0x670e3455 (stale) Mirror 10: 0x670e3455 (stale) Mirror 13: 0xb5a155a (stale) CRC-32 checksum value for chunk [0, 0x3b946): Mirror 1: 0xe515347d Mirror 14: 0xe515347d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b946: skipped 000993 1786032526.810162 read 0x025e9a thru 0x02df32 (0x8099 bytes) 000994 1786032526.817545 trunc from 0x03b946 to 0x021eb8 (0x19a8d bytes) 000995 1786032526.920572 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x181:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21eb9): Mirror 7: 0x87e17b57 (stale) Mirror 10: 0x87e17b57 (stale) Mirror 13: 0x7fd4d858 (stale) Mirror 14: 0x31553294 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21eb9): Mirror 7: 0x87e17b57 (stale) Mirror 10: 0x87e17b57 (stale) Mirror 13: 0x7fd4d858 (stale) Mirror 14: 0x31553294 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21eb9: skipped 000996 1786032527.481647 read 0x00563f thru 0x00643e (0xe00 bytes) 000997 1786032527.496655 read 0x012c02 thru 0x01dc7f (0xb07e bytes) 000998 1786032527.509710 write 0x0171d0 thru 0x019a53 (0x2884 bytes) 001000 1786032527.521706 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21eb9): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21eb9): Mirror 7: 0x87e17b57 (stale) Mirror 10: 0x87e17b57 (stale) Mirror 13: 0x7fd4d858 (stale) Mirror 14: 0x31553294 (stale) CRC-32 checksum value for chunk [0, 0x21eb9): Mirror 1: 0x67476039 Mirror 15: 0x67476039 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21eb9: skipped All operations completed A-OK! PASS 70a (133s) == sanity-flr test 71: check mirror extend parent fid ==== 12:08:56 (1786032536) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 12:09:00 (1786032540) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251595 s, 8.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21863 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 12:09:11 (1786032551) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre2 Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3625248 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1150720 .. Extending file size to 6425216 .. Extending file size to 3237472 .. Extending file size to 7199264 .. Extending file size to 7409536 .. Extending file size to 6820000 .. Extending file size to 7474656 .. Extending file size to 7034432 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7878016 .. Extending file size to 8111424 .. Extending file size to 8159584 .. Extending file size to 8595104 .. Extending file size to 8977760 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9039712 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9126944 .. Waiting 57311 57312 57314 57315 57316 done 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) 192.168.201.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:10:35 (1786032635) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre2 Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2100480 .. Extending file size to 2649504 .. Extending file size to 9100640 .. Extending file size to 8068128 .. Extending file size to 8284832 .. Extending file size to 8879136 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9048992 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58779 58780 58781 58782 58784 done 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) 192.168.201.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (89s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:12:04 (1786032724) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143103 s, 14.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.406608 s, 5.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x79238aad (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) PASS 200c (24s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:12:31 (1786032751) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0849701 s, 24.7 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 12:12:41 (1786032761) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:12:51 (1786032771) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28716 s, 7.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:13:02 (1786032782) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.361468 s, 5.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } PASS 204b (12s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:13:14 (1786032794) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.38474 s, 5.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:13:50 (1786032830) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199889 s, 5.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:14:55 (1786032895) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.409194 s, 5.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:15:58 (1786032958) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg119-server mds-ost sync done. Creating new pool test_204f oleg119-server: Pool lustre.test_204f created Adding targets to pool oleg119-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg119-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.445374 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg119-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg119-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg119-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:17:19 (1786033039) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:17:31 (1786033051) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:17:41 (1786033061) oleg119-server: Pool lustre.test_206 created Adding targets to pool oleg119-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg119-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg119-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg119-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg119-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:18:10 (1786033090) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930362 s, 11.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702316 s, 14.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb9a27442 Mirror 3: 0xb9a27442 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:18:23 (1786033103) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:18:28 (1786033108) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:18:32 (1786033112) Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.123996 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2050000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1580000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1580000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1580000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x18b0000 Read this (4 bytes): BBBB 192.168.201.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg119-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:19:15 (1786033155) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0322158 s, 254 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:19:25 (1786033165) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511016 s, 20.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ea lmm_fid: [0x200000404:0x1ea:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ea lmm_fid: [0x200000404:0x1ea:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834410 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-06 12:19:26.000000000 -0400 Modify: 2026-08-06 12:19:26.000000000 -0400 Change: 2026-08-06 12:19:28.000000000 -0400 Birth: 2026-08-06 12:19:26.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:19:34 (1786033174) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:19:38 (1786033178) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.828277 s, 49.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0511481 s, 80.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 6 12:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ec:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 6 12:19 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:19:48 (1786033188) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:19:56 (1786033196) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2030 sec =========== 12:20:07 (1786033207) === sanity-flr: start cleanup 12:20:09 (1786033209) === === sanity-flr: finish cleanup 12:20:16 (1786033216) ===