-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 12:18:53 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg615-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:19:12 (1787847552) === oleg615-client.virtnet: executing check_config_client /mnt/lustre oleg615-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg615-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e16435e8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e16435e8800.idle_timeout=debug disable quota as required oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:19:45 (1787847585) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7647] == sanity-flr test 0a: lfs mirror create with -N option == 12:19:48 (1787847588) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:20:05 (1787847605) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:20:09 (1787847609) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:20:12 (1787847612) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:20:39 (1787847639) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:20:42 (1787847642) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 12:20:46 (1787847646) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e16435e8800.stats=clear osc.lustre-OST0001-osc-ffff8e16435e8800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:20:54 (1787847654) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:21:05 (1787847665) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:21:13 (1787847673) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:21:16 (1787847676) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:21:24 (1787847684) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 12:21:35 (1787847695) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:21:42 (1787847702) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:21:45 (1787847705) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:21:53 (1787847713) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:22:02 (1787847722) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:22:08 (1787847728) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.68375 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149125 s, 7.0 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:22:19 (1787847739) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.494698 s, 36.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:22:27 (1787847747) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:22:37 (1787847757) Stopping /mnt/lustre-ost1 (opts:) on oleg615-server oleg615-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e16435e8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e16435e8800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg615-server: oleg615-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg615-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg615-server oleg615-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e16435e8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e16435e8800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg615-server: oleg615-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg615-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:23:46 (1787847826) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.325414 s, 19.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg615-server oleg615-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e16435e8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e16435e8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg615-server: oleg615-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg615-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:24:26 (1787847866) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg615-server oleg615-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e16435e8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e16435e8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg615-server: oleg615-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg615-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg615-server oleg615-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e16435e8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e16435e8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg615-server: oleg615-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg615-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:25:41 (1787847941) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.17113 s, 10.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 12:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.980s user 0m0.000s sys 0m0.196s Stopping /mnt/lustre-ost1 (opts:) on oleg615-server oleg615-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e16435e8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e16435e8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.223s user 0m0.001s sys 0m0.191s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg615-server: oleg615-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg615-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg615-server oleg615-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e16435e8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e16435e8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.520s user 0m0.000s sys 0m0.160s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg615-server: oleg615-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg615-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg615-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e16435e8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e16435e8800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:27:22 (1787848042) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:27:25 (1787848045) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:27:29 (1787848049) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:27:33 (1787848053) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 12:27:43 (1787848063) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.425442 s, 29.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:27:57 (1787848077) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689974 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317851 s, 33.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:28:06 (1787848086) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:28:15 (1787848095) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00160048 s, 640 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00623836 s, 164 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00126012 s, 813 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000987497 s, 1.0 MB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 12:28:29 (1787848109) create test files with size 15747198 .. 97f33072228ea1537f2a17c9fb3ee27c - ac33357d349f74fb0c42734af00ddcc8 - 1a1cae368944b8420d96860a2ff3b6f5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg615-client.virtnet /mnt/lustre (opts:) Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 12:28:54 (1787848134) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:29:11 (1787848151) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:29:19 (1787848159) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194607 s, 5.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224594 s, 46.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 12:29:30 (1787848170) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.215478 s, 9.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.287069 s, 14.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.238134 s, 8.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 12:29:51 (1787848191) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 12:29:55 (1787848195) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:29:59 (1787848199) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383053 s, 27.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 12:30:08 (1787848208) llite.lustre-ffff8e1643e6b800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 12:30:16 (1787848216) llite.lustre-ffff8e1643e6b800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273297 s, 38.4 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419898 s, 25.0 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 12:30:26 (1787848226) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:30:29 (1787848229) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:30:40 (1787848240) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.348933 s, 30.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg615-client.virtnet /mnt/lustre (opts:) Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:30:52 (1787848252) llite.lustre-ffff8e1644839000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.274693 s, 38.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 4102 flags: 1 Stopping client oleg615-client.virtnet /mnt/lustre (opts:) Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 4102 flags: 1 llite.lustre-ffff8e164de22000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:31:06 (1787848266) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:31:43 (1787848303) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:32:18 (1787848338) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 12:32:22 (1787848342) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.286671 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.165432 s, 19.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3ca80804 (nosync,stale) Mirror 2: 0x3ca80804 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5a8ba06ccd0c5dd0d99bb8ec0224b03d - mirror 1 checksum: 67c78341244fd4b8e87b137f143ba5e3 - mirror 2 checksum: 5a8ba06ccd0c5dd0d99bb8ec0224b03d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T12:32:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T12:32:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5a8ba06ccd0c5dd0d99bb8ec0224b03d - mirror 1 checksum: 67c78341244fd4b8e87b137f143ba5e3 - mirror 2 checksum: 67c78341244fd4b8e87b137f143ba5e3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T12:32:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T12:32:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:32:36 (1787848356) Failing mds1 on oleg615-server Stopping /mnt/lustre-mds1 (opts:) on oleg615-server 12:32:43 (1787848363) shut down facet: mds1 facet_host: oleg615-server facet_failover_host: oleg615-server Failover mds1 to oleg615-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg615-server: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg615-client: oleg615-server: ssh exited with exit code 1 Started lustre-MDT0000 12:33:16 (1787848396) targets are mounted 12:33:16 (1787848396) facet_failover done oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325934 s, 32.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:33:35 (1787848415) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:33:38 (1787848418) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:33:41 (1787848421) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0201371 s, 814 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:33:51 (1787848431) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:33:54 (1787848434) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174419 s, 235 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.09205 s, 44.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:34:02 (1787848442) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:34:26 (1787848466) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:34:56 (1787848496) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:35:20 (1787848520) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:35:44 (1787848544) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 12:35:51 (1787848551) mirror create pid 55585 mirror split pid 55587 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55585 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55587 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 12:36:59 (1787848619) Seed set to 8621 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005 1787848621.872842 MIRROR_EXTEND to 2 mirrors skipping zero size read skipping zero size read 000014 1787848622.379686 write 0x013179 thru 0x014cf5 (0x1b7d bytes) 000015 1787848622.566516 read 0x00cef1 thru 0x00e396 (0x14a6 bytes) 000016 1787848622.605314 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x14cf6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14cf6): Mirror 2: 0x713fc0d8 (stale) CRC-32 checksum value for chunk [0, 0x14cf6): Mirror 1: 0x72c4bac1 Mirror 3: 0x72c4bac1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14cf6: skipped 000019 1787848624.046596 write 0x0061cd thru 0x012007 (0xbe3b bytes) 000023 1787848624.176505 mapread 0x001f92 thru 0x00f8ea (0xd959 bytes) 000025 1787848624.205079 read 0x002d47 thru 0x010fa6 (0xe260 bytes) 000026 1787848624.216885 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x14cf6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14cf6): Mirror 2: 0x713fc0d8 (stale) Mirror 3: 0x72c4bac1 (stale) CRC-32 checksum value for chunk [0, 0x14cf6): Mirror 1: 0x17c72d50 Mirror 4: 0x17c72d50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14cf6: skipped truncating to largest ever: 0x1857f 000029 1787848625.596462 trunc from 0x014cf6 to 0x01857e (0x3889 bytes) 000031 1787848625.722552 trunc from 0x01857f to 0x00b7ed (0xcd91 bytes) 000032 1787848625.748075 mapwrite 0x0105d7 thru 0x015363 (0x4d8d bytes) 000033 1787848625.847022 write 0x020f0a thru 0x021fa7 (0x109e bytes) truncating to largest ever: 0x2732c 000038 1787848625.858840 trunc from 0x021fa8 to 0x02732b (0x5384 bytes) 000039 1787848625.915887 write 0x00f2ce thru 0x0152db (0x600e bytes) 000040 1787848625.917002 mapread 0x01b023 thru 0x024979 (0x9957 bytes) 000041 1787848625.945341 mapwrite 0x02cac6 thru 0x036a98 (0x9fd3 bytes) 000042 1787848626.113795 mapwrite 0x035d3e thru 0x03ffff (0xa2c2 bytes) 000044 1787848626.251307 mapread 0x006c99 thru 0x0108a3 (0x9c0b bytes) 000045 1787848626.274050 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x63cce7ad (stale) Mirror 4: 0x643cd9a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf7b52070 Mirror 5: 0xf7b52070 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000046 1787848627.727927 write 0x00b6bc thru 0x01023f (0x4b84 bytes) 000047 1787848627.830727 MIRROR_RESYNC 5 mirrors 000048 1787848629.967268 mapwrite 0x039dbb thru 0x03ffff (0x6245 bytes) 000049 1787848630.231204 mapwrite 0x014620 thru 0x023eab (0xf88c bytes) 000050 1787848630.362065 write 0x0022ac thru 0x00ded5 (0xbc2a bytes) 000053 1787848630.369817 mapwrite 0x022357 thru 0x02aecd (0x8b77 bytes) 000055 1787848630.464409 mapwrite 0x0124f1 thru 0x0141c1 (0x1cd1 bytes) 000056 1787848630.512692 read 0x00d310 thru 0x01af83 (0xdc74 bytes) 000057 1787848630.513644 mapwrite 0x01507c thru 0x023b5f (0xeae4 bytes) 000058 1787848630.555720 mapwrite 0x0333d5 thru 0x03fd2f (0xc95b bytes) 000061 1787848630.621398 read 0x013c63 thru 0x020de9 (0xd187 bytes) 000065 1787848630.624430 mapwrite 0x029cce thru 0x02c404 (0x2737 bytes) 000067 1787848630.670268 trunc from 0x040000 to 0x01e3a4 (0x21c5b bytes) 000069 1787848630.689622 write 0x01a65c thru 0x01e6ef (0x4094 bytes) 000070 1787848630.691159 mapread 0x00f4a3 thru 0x01a576 (0xb0d4 bytes) 000071 1787848630.692991 write 0x00e0ce thru 0x014327 (0x625a bytes) 000072 1787848630.698334 MIRROR_RESYNC 5 mirrors 000074 1787848632.169474 mapread 0x008a35 thru 0x01261e (0x9bea bytes) 000075 1787848632.205598 mapread 0x004939 thru 0x004a8d (0x155 bytes) truncating to largest ever: 0x2d93a 000078 1787848632.214303 trunc from 0x01e6f0 to 0x02d939 (0xf24a bytes) 000079 1787848632.379411 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d93a): Mirror 3: 0xa4b25ed0 (stale) Mirror 4: 0xa4b25ed0 (stale) Mirror 5: 0xa4b25ed0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d93a): Mirror 3: 0xa4b25ed0 (stale) Mirror 4: 0xa4b25ed0 (stale) Mirror 5: 0xa4b25ed0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d93a: skipped 000080 1787848632.949249 mapwrite 0x00e85a thru 0x0195b5 (0xad5c bytes) 000082 1787848633.079446 trunc from 0x02d93a to 0x02189e (0xc09b bytes) 000084 1787848633.108676 mapread 0x010957 thru 0x01823c (0x78e6 bytes) 000085 1787848633.110548 read 0x00e04a thru 0x011b2f (0x3ae6 bytes) 000086 1787848633.111594 trunc from 0x02189f to 0x00cc06 (0x14c98 bytes) 000089 1787848633.143027 mapread 0x001189 thru 0x002bfc (0x1a74 bytes) 000091 1787848633.158410 trunc from 0x00cc07 to 0x00eb9d (0x1f97 bytes) 000092 1787848633.187641 mapread 0x0049bf thru 0x00eb9d (0xa1df bytes) 000093 1787848633.208626 write 0x0223b5 thru 0x02bf31 (0x9b7d bytes) 000094 1787848633.221243 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2bf32): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf32): Mirror 3: 0x82e562a5 (stale) Mirror 4: 0x82e562a5 (stale) Mirror 5: 0x82e562a5 (stale) CRC-32 checksum value for chunk [0, 0x2bf32): Mirror 1: 0x71235af7 Mirror 6: 0x71235af7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf32: skipped 000095 1787848634.332428 mapread 0x01c295 thru 0x025cc6 (0x9a32 bytes) 000097 1787848634.348757 trunc from 0x02bf32 to 0x011bc1 (0x1a370 bytes) 000098 1787848634.466049 read 0x006b66 thru 0x00c70a (0x5ba5 bytes) 000100 1787848634.474922 mapread 0x00bbe4 thru 0x010eb9 (0x52d6 bytes) 000102 1787848634.485098 read 0x001d7d thru 0x0095c2 (0x7846 bytes) 000103 1787848634.493489 trunc from 0x011bc2 to 0x02ce68 (0x1b2a7 bytes) 000105 1787848634.514942 trunc from 0x02ce69 to 0x0288ea (0x457e bytes) 000106 1787848634.530762 mapwrite 0x01661a thru 0x01b711 (0x50f8 bytes) 000107 1787848634.592795 trunc from 0x0288eb to 0x0136a7 (0x15243 bytes) 000108 1787848634.611690 write 0x02c70d thru 0x0332f3 (0x6be7 bytes) 000110 1787848634.617668 mapwrite 0x00fbe5 thru 0x011980 (0x1d9c bytes) 000111 1787848634.682532 trunc from 0x0332f4 to 0x01d87f (0x15a74 bytes) 000112 1787848634.700265 trunc from 0x01d880 to 0x01ae87 (0x29f8 bytes) 000114 1787848634.723928 read 0x00dae1 thru 0x01adef (0xd30f bytes) 000115 1787848634.758789 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1ae88): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ae88): Mirror 3: 0xd337b802 (stale) Mirror 4: 0xd337b802 (stale) Mirror 5: 0xd337b802 (stale) Mirror 6: 0xe5d7b3a (stale) CRC-32 checksum value for chunk [0, 0x1ae88): Mirror 1: 0xde8c534f Mirror 7: 0xde8c534f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ae88: skipped 000116 1787848635.923304 trunc from 0x01ae88 to 0x0216b9 (0x6832 bytes) 000117 1787848636.016091 mapread 0x01df49 thru 0x0216b9 (0x3771 bytes) 000118 1787848636.041352 mapread 0x0137dd thru 0x015174 (0x1998 bytes) 000119 1787848636.054548 mapwrite 0x0186f4 thru 0x01efbe (0x68cb bytes) 000120 1787848636.148268 mapread 0x00f063 thru 0x01b467 (0xc405 bytes) 000123 1787848636.153627 mapwrite 0x02333a thru 0x024d68 (0x1a2f bytes) 000124 1787848636.279022 write 0x03aea2 thru 0x03b80d (0x96c bytes) 000125 1787848636.280404 mapread 0x01d097 thru 0x01ee97 (0x1e01 bytes) 000127 1787848636.288156 mapread 0x0081b7 thru 0x01251b (0xa365 bytes) truncating to largest ever: 0x3dcf1 000129 1787848636.311854 trunc from 0x03b80e to 0x03dcf0 (0x24e3 bytes) 000130 1787848636.343682 MIRROR_RESYNC 6 mirrors 000132 1787848639.470428 read 0x00aabc thru 0x00d635 (0x2b7a bytes) 000135 1787848639.479815 write 0x0247e6 thru 0x02f2a8 (0xaac3 bytes) 000136 1787848639.568856 mapwrite 0x036d3d thru 0x03ffff (0x92c3 bytes) 000138 1787848639.673835 mapwrite 0x03e041 thru 0x03ffff (0x1fbf bytes) 000139 1787848639.731054 read 0x029dc1 thru 0x035e92 (0xc0d2 bytes) 000140 1787848639.751046 mapread 0x00cc23 thru 0x012e6b (0x6249 bytes) 000142 1787848639.752467 trunc from 0x040000 to 0x01c645 (0x239ba bytes) 000144 1787848639.772192 mapwrite 0x003e7f thru 0x00b915 (0x7a97 bytes) 000149 1787848639.859842 mapwrite 0x0341a4 thru 0x03fa32 (0xb88f bytes) 000150 1787848640.074402 write 0x001a90 thru 0x002142 (0x6b3 bytes) 000151 1787848640.075333 write 0x01c6f9 thru 0x0260a4 (0x99ac bytes) 000152 1787848640.085852 trunc from 0x03fa33 to 0x0210b6 (0x1e97c bytes) 000153 1787848640.116354 read 0x00ceef thru 0x012176 (0x5288 bytes) 000154 1787848640.119241 mapread 0x00ecdc thru 0x01c41d (0xd742 bytes) 000156 1787848640.121695 mapread 0x002b44 thru 0x005d79 (0x3236 bytes) 000157 1787848640.129797 MIRROR_RESYNC 6 mirrors 000159 1787848641.721582 mapread 0x008445 thru 0x00a32e (0x1eea bytes) 000160 1787848641.744671 read 0x00f118 thru 0x01827c (0x9165 bytes) 000161 1787848641.751485 read 0x0044ef thru 0x013991 (0xf4a3 bytes) 000164 1787848641.752467 trunc from 0x0210b7 to 0x022918 (0x1862 bytes) 000167 1787848641.836397 MIRROR_RESYNC 6 mirrors 000168 1787848643.628750 read 0x020916 thru 0x022918 (0x2003 bytes) 000169 1787848643.640423 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000170 1787848644.062463 trunc from 0x022919 to 0x018260 (0xa6b8 bytes) 000171 1787848644.148417 trunc from 0x018261 to 0x030fe2 (0x18d82 bytes) 000173 1787848644.180649 MIRROR_RESYNC 5 mirrors 000175 1787848645.828656 mapwrite 0x01f9a2 thru 0x020754 (0xdb3 bytes) 000176 1787848646.067686 mapread 0x02db18 thru 0x030fe2 (0x34cb bytes) 000178 1787848646.107734 MIRROR_RESYNC 5 mirrors 000179 1787848648.262419 read 0x01a46a thru 0x022849 (0x83e0 bytes) 000180 1787848648.281752 read 0x00af6e thru 0x019060 (0xe0f3 bytes) 000183 1787848648.297980 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000184 1787848648.610169 trunc from 0x030fe3 to 0x002b86 (0x2e45c bytes) 000186 1787848648.703998 write 0x02444c thru 0x02586f (0x1424 bytes) 000187 1787848648.709671 MIRROR_RESYNC 4 mirrors 000193 1787848650.136470 mapread 0x01deb1 thru 0x02586f (0x79bf bytes) 000194 1787848650.163678 mapwrite 0x035cac thru 0x039051 (0x33a6 bytes) 000196 1787848650.399431 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39052): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39052): Mirror 3: 0x46b0d5f9 (stale) Mirror 4: 0x46b0d5f9 (stale) Mirror 7: 0x46b0d5f9 (stale) CRC-32 checksum value for chunk [0, 0x39052): Mirror 1: 0x58abc319 Mirror 8: 0x58abc319 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39052: skipped 000197 1787848651.719276 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39052): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39052): Mirror 3: 0x46b0d5f9 (stale) Mirror 4: 0x46b0d5f9 (stale) Mirror 7: 0x46b0d5f9 (stale) CRC-32 checksum value for chunk [0, 0x39052): Mirror 1: 0x58abc319 Mirror 8: 0x58abc319 Mirror 9: 0x58abc319 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39052: skipped 000198 1787848653.204508 trunc from 0x039052 to 0x02628c (0x12dc5 bytes) 000199 1787848653.308808 mapwrite 0x0046e4 thru 0x008852 (0x416f bytes) 000200 1787848653.381935 trunc from 0x02628d to 0x02a6b8 (0x442c bytes) 000202 1787848653.403376 trunc from 0x02a6b9 to 0x001c29 (0x28a8f bytes) 000203 1787848653.440363 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c2a): Mirror 3: 0x197d848e (stale) Mirror 4: 0x197d848e (stale) Mirror 8: 0x197d848e (stale) Mirror 9: 0x197d848e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c2a): Mirror 3: 0x197d848e (stale) Mirror 4: 0x197d848e (stale) Mirror 8: 0x197d848e (stale) Mirror 9: 0x197d848e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c2a: skipped 000204 1787848654.024402 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c2a): Mirror 4: 0x197d848e (stale) Mirror 8: 0x197d848e (stale) Mirror 9: 0x197d848e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c2a): Mirror 4: 0x197d848e (stale) Mirror 8: 0x197d848e (stale) Mirror 9: 0x197d848e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c2a: skipped 000206 1787848654.366161 mapread 0x0005e0 thru 0x001c29 (0x164a bytes) 000210 1787848654.379091 trunc from 0x001c2a to 0x03460c (0x329e3 bytes) 000213 1787848654.402560 read 0x01437a thru 0x0165fa (0x2281 bytes) 000214 1787848654.409180 write 0x012b9a thru 0x013fb9 (0x1420 bytes) 000218 1787848654.420905 mapwrite 0x015e65 thru 0x01bb5e (0x5cfa bytes) 000219 1787848654.478732 mapwrite 0x00656b thru 0x01429d (0xdd33 bytes) 000220 1787848654.569931 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3460d): Mirror 4: 0x91e31340 (stale) Mirror 8: 0x91e31340 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3460d): Mirror 4: 0x91e31340 (stale) Mirror 8: 0x91e31340 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3460d: skipped 000221 1787848654.896054 mapwrite 0x033d17 thru 0x0381d9 (0x44c3 bytes) 000222 1787848654.973553 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x381da): Mirror 8: 0x3cc016d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381da): Mirror 8: 0x3cc016d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381da: skipped 000223 1787848655.280635 mapread 0x00e00f thru 0x010543 (0x2535 bytes) 000224 1787848655.296360 mapread 0x02a101 thru 0x030127 (0x6027 bytes) 000225 1787848655.305037 read 0x01ae8e thru 0x0275b1 (0xc724 bytes) 000227 1787848655.313345 mapwrite 0x01edda thru 0x0220bd (0x32e4 bytes) 000228 1787848655.357479 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000230 1787848655.482369 MIRROR_EXTEND to 2 mirrors 000231 1787848656.519672 read 0x02af3a thru 0x0345cb (0x9692 bytes) 000234 1787848656.528397 mapwrite 0x020034 thru 0x028821 (0x87ee bytes) 000235 1787848656.814410 read 0x01d653 thru 0x02b399 (0xdd47 bytes) 000236 1787848656.832335 mapread 0x00bd37 thru 0x00d10d (0x13d7 bytes) 000237 1787848656.833499 mapread 0x00c9e6 thru 0x01b271 (0xe88c bytes) 000238 1787848656.837334 read 0x012d93 thru 0x02234a (0xf5b8 bytes) 000239 1787848656.847108 MIRROR_RESYNC 2 mirrors 000240 1787848658.069040 trunc from 0x0381da to 0x01e4f4 (0x19ce5 bytes) 000242 1787848658.159555 trunc from 0x01e4f5 to 0x007756 (0x16d9e bytes) 000244 1787848658.186342 mapwrite 0x025899 thru 0x02798f (0x20f7 bytes) 000245 1787848658.276548 mapread 0x01df5e thru 0x02798f (0x9a32 bytes) 000247 1787848658.287637 mapread 0x012e94 thru 0x01f205 (0xc372 bytes) 000248 1787848658.296110 mapwrite 0x0352e2 thru 0x03cc71 (0x7990 bytes) 000249 1787848658.379556 read 0x002cab thru 0x00fea8 (0xd1fe bytes) 000250 1787848658.388374 mapread 0x026728 thru 0x035615 (0xeeee bytes) 000252 1787848658.397503 write 0x02d636 thru 0x032417 (0x4de2 bytes) 000253 1787848658.398686 read 0x010c80 thru 0x01a9a1 (0x9d22 bytes) 000260 1787848658.409582 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000261 1787848658.578782 write 0x007e20 thru 0x01231c (0xa4fd bytes) 000262 1787848658.591283 mapwrite 0x029a62 thru 0x02d6ab (0x3c4a bytes) 000264 1787848658.655404 trunc from 0x03cc72 to 0x038a37 (0x423a bytes) 000266 1787848658.681157 mapwrite 0x027963 thru 0x02ad4d (0x33eb bytes) 000269 1787848658.723011 mapwrite 0x024a99 thru 0x02a189 (0x56f1 bytes) 000270 1787848658.785760 write 0x005758 thru 0x013bc6 (0xe46f bytes) 000273 1787848658.804492 read 0x007311 thru 0x01585a (0xe54a bytes) 000274 1787848658.813869 mapwrite 0x02d4ae thru 0x034dfe (0x7951 bytes) 000275 1787848658.876160 write 0x03f48c thru 0x03ffff (0xb74 bytes) 000277 1787848658.878217 trunc from 0x040000 to 0x00220a (0x3ddf5 bytes) 000278 1787848658.898576 mapwrite 0x0399ed thru 0x03ffff (0x6613 bytes) 000279 1787848658.989254 mapwrite 0x00eb1b thru 0x015fa3 (0x7489 bytes) 000281 1787848659.062280 trunc from 0x040000 to 0x02d90e (0x126f1 bytes) 000282 1787848659.092283 mapwrite 0x004b8f thru 0x00c53e (0x79b0 bytes) 000283 1787848659.146430 mapread 0x004c98 thru 0x00bb88 (0x6ef1 bytes) 000284 1787848659.147473 mapread 0x012f4b thru 0x0137d0 (0x886 bytes) 000285 1787848659.148327 read 0x011d02 thru 0x01c209 (0xa508 bytes) 000286 1787848659.159593 mapwrite 0x00a67a thru 0x017603 (0xcf8a bytes) 000287 1787848659.214643 write 0x01333f thru 0x02310d (0xfdcf bytes) 000289 1787848659.227577 write 0x0158d2 thru 0x018045 (0x2774 bytes) 000291 1787848659.230262 write 0x016249 thru 0x020bad (0xa965 bytes) 000292 1787848659.233003 write 0x03c5da thru 0x03ffff (0x3a26 bytes) 000294 1787848659.236539 mapread 0x008c0e thru 0x0184e0 (0xf8d3 bytes) 000296 1787848659.239147 trunc from 0x040000 to 0x00df38 (0x320c7 bytes) 000297 1787848659.267348 trunc from 0x00df39 to 0x01c2ff (0xe3c7 bytes) 000298 1787848659.295459 write 0x017485 thru 0x01ecf0 (0x786c bytes) 000299 1787848659.302397 mapread 0x0088f9 thru 0x011fdb (0x96e3 bytes) 000301 1787848659.325443 read 0x0014b9 thru 0x00d559 (0xc0a1 bytes) 000304 1787848659.327465 mapwrite 0x005622 thru 0x00ac5d (0x563c bytes) 000305 1787848659.374527 write 0x01790d thru 0x026be0 (0xf2d4 bytes) 000306 1787848659.377268 read 0x00c9d8 thru 0x0192da (0xc903 bytes) 000310 1787848659.388279 mapread 0x007820 thru 0x007cab (0x48c bytes) 000311 1787848659.389531 trunc from 0x026be1 to 0x038803 (0x11c23 bytes) 000312 1787848659.405612 write 0x02cfed thru 0x02e2ec (0x1300 bytes) 000315 1787848659.417602 write 0x0367b0 thru 0x0393de (0x2c2f bytes) 000316 1787848659.419569 mapwrite 0x01cfcb thru 0x02389c (0x68d2 bytes) 000317 1787848659.473218 write 0x03d885 thru 0x03f026 (0x17a2 bytes) 000320 1787848659.476820 write 0x028dbe thru 0x031bb9 (0x8dfc bytes) 000321 1787848659.479454 trunc from 0x03f027 to 0x0041a8 (0x3ae7e bytes) 000322 1787848659.502694 write 0x033dd9 thru 0x039ebd (0x60e5 bytes) 000324 1787848659.506606 write 0x03241f thru 0x03b7f9 (0x93db bytes) 000326 1787848659.514034 mapwrite 0x01c4d9 thru 0x0238fa (0x7422 bytes) 000328 1787848659.623665 write 0x00bf5e thru 0x00c602 (0x6a5 bytes) 000331 1787848659.635861 mapwrite 0x014d26 thru 0x020c4e (0xbf29 bytes) 000333 1787848659.808234 MIRROR_EXTEND to 2 mirrors 000337 1787848661.013747 mapread 0x0000b9 thru 0x0023c0 (0x2308 bytes) 000338 1787848661.037499 MIRROR_RESYNC 2 mirrors 000339 1787848661.548108 trunc from 0x03b7fa to 0x019f49 (0x218b0 bytes) 000343 1787848661.659293 mapread 0x012d07 thru 0x019f49 (0x7243 bytes) 000345 1787848661.691516 mapread 0x000571 thru 0x001576 (0x1006 bytes) 000346 1787848661.704013 mapread 0x014001 thru 0x019f49 (0x5f49 bytes) 000347 1787848661.708781 mapread 0x002a10 thru 0x010e31 (0xe422 bytes) 000348 1787848661.712701 mapwrite 0x01b2c8 thru 0x024d4b (0x9a84 bytes) 000349 1787848661.863455 write 0x03e94f thru 0x03ffff (0x16b1 bytes) 000350 1787848661.866109 trunc from 0x040000 to 0x010683 (0x2f97c bytes) 000352 1787848661.896998 write 0x025e9d thru 0x027839 (0x199d bytes) 000353 1787848661.906468 MIRROR_RESYNC 2 mirrors 000354 1787848663.006527 trunc from 0x02783a to 0x01e362 (0x94d7 bytes) 000357 1787848663.203919 mapread 0x01a9f7 thru 0x01c624 (0x1c2e bytes) 000358 1787848663.236915 trunc from 0x01e363 to 0x013b00 (0xa862 bytes) 000359 1787848663.268844 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x13b01): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13b01): Mirror 2: 0xb4d823d9 (stale) CRC-32 checksum value for chunk [0, 0x13b01): Mirror 1: 0xb4d823d9 Mirror 3: 0xb4d823d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13b01: skipped 000361 1787848664.402261 mapwrite 0x038255 thru 0x03ffff (0x7dab bytes) 000362 1787848664.604431 trunc from 0x040000 to 0x017744 (0x288bb bytes) 000363 1787848664.627410 mapwrite 0x00a80e thru 0x00d6c5 (0x2eb8 bytes) 000365 1787848664.691413 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x17745): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17745): Mirror 2: 0x19ae2c5b (stale) Mirror 3: 0x19ae2c5b (stale) CRC-32 checksum value for chunk [0, 0x17745): Mirror 1: 0xb5054d2d Mirror 4: 0xb5054d2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17745: skipped 000367 1787848666.028713 trunc from 0x017745 to 0x00eae8 (0x8c5c bytes) 000368 1787848666.147615 trunc from 0x00eae9 to 0x01a3ce (0xb8e6 bytes) 000372 1787848666.197637 read 0x01184d thru 0x01936c (0x7b20 bytes) 000373 1787848666.208658 read 0x009b88 thru 0x018501 (0xe97a bytes) 000374 1787848666.224606 mapread 0x008d3e thru 0x0158d0 (0xcb93 bytes) 000375 1787848666.227109 trunc from 0x01a3cf to 0x0272d4 (0xcf06 bytes) 000377 1787848666.250561 mapread 0x004369 thru 0x00c743 (0x83db bytes) 000378 1787848666.251679 write 0x005c1e thru 0x008f1c (0x32ff bytes) 000379 1787848666.254460 trunc from 0x0272d5 to 0x01a6fd (0xcbd7 bytes) 000380 1787848666.293218 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a6fe): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a6fe): Mirror 2: 0x71f51641 (stale) Mirror 3: 0x71f51641 (stale) Mirror 4: 0x6d8c74d9 (stale) CRC-32 checksum value for chunk [0, 0x1a6fe): Mirror 1: 0x511e8f21 Mirror 5: 0x511e8f21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a6fe: skipped 000381 1787848667.462632 write 0x01f013 thru 0x02ec35 (0xfc23 bytes) 000382 1787848667.594353 trunc from 0x02ec36 to 0x010d4a (0x1deeb bytes) 000384 1787848667.658341 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10d4b): Mirror 3: 0x22f9807b (stale) Mirror 4: 0x9d99d5e8 (stale) Mirror 5: 0xcc3523a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10d4b): Mirror 3: 0x22f9807b (stale) Mirror 4: 0x9d99d5e8 (stale) Mirror 5: 0xcc3523a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10d4b: skipped 000385 1787848668.043805 read 0x005547 thru 0x010d4a (0xb804 bytes) 000386 1787848668.055025 mapwrite 0x012b7f thru 0x01f210 (0xc692 bytes) 000387 1787848668.139780 write 0x019296 thru 0x01bf73 (0x2cde bytes) 000390 1787848668.154999 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f211): Mirror 4: 0x4aa1b8ba (stale) Mirror 5: 0x149fc349 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f211): Mirror 4: 0x4aa1b8ba (stale) Mirror 5: 0x149fc349 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f211: skipped 000391 1787848668.526891 write 0x022891 thru 0x0311d0 (0xe940 bytes) 000392 1787848668.530027 mapread 0x0200ce thru 0x024f7b (0x4eae bytes) 000393 1787848668.558039 read 0x011936 thru 0x01e161 (0xc82c bytes) 000394 1787848668.577452 mapwrite 0x010502 thru 0x012861 (0x2360 bytes) 000395 1787848668.637446 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x311d1): Mirror 5: 0xf892b487 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x311d1): Mirror 5: 0xf892b487 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x311d1: skipped 000398 1787848669.003490 write 0x0209bb thru 0x026181 (0x57c7 bytes) 000399 1787848669.018504 mapwrite 0x03e555 thru 0x03ffff (0x1aab bytes) 000400 1787848669.086986 write 0x01b338 thru 0x01ed43 (0x3a0c bytes) 000401 1787848669.098317 mapread 0x00ce42 thru 0x016855 (0x9a14 bytes) 000403 1787848669.134768 trunc from 0x040000 to 0x024143 (0x1bebc bytes) 000404 1787848669.154651 mapwrite 0x008705 thru 0x00bc91 (0x358d bytes) 000406 1787848669.244540 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24144): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24144): Mirror 5: 0x9c5aed7 (stale) CRC-32 checksum value for chunk [0, 0x24144): Mirror 1: 0x8f9789c5 Mirror 6: 0x8f9789c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24144: skipped 000407 1787848670.653895 read 0x00b935 thru 0x00c925 (0xff1 bytes) 000408 1787848670.667581 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000413 1787848671.157241 mapwrite 0x01cc09 thru 0x02ca6d (0xfe65 bytes) 000414 1787848671.437531 write 0x011bae thru 0x01cd68 (0xb1bb bytes) 000415 1787848671.444859 mapwrite 0x024802 thru 0x0275fc (0x2dfb bytes) 000418 1787848671.492358 write 0x012406 thru 0x01ece8 (0xc8e3 bytes) 000420 1787848671.496367 read 0x015187 thru 0x02054c (0xb3c6 bytes) 000422 1787848671.498300 trunc from 0x02ca6e to 0x02ac7d (0x1df0 bytes) 000423 1787848671.522821 write 0x0311e7 thru 0x034f2d (0x3d47 bytes) 000424 1787848671.524080 trunc from 0x034f2e to 0x03bf72 (0x7045 bytes) 000426 1787848671.548619 read 0x00afe5 thru 0x012dbc (0x7dd8 bytes) 000427 1787848671.565769 MIRROR_RESYNC 2 mirrors 000428 1787848673.078058 trunc from 0x03bf73 to 0x002339 (0x39c39 bytes) 000432 1787848673.222591 mapread 0x000c7e thru 0x002339 (0x16bc bytes) 000433 1787848673.262227 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000436 1787848673.482643 write 0x0049bd thru 0x00d10b (0x874f bytes) 000439 1787848673.489104 mapread 0x007a9d thru 0x00d10b (0x566f bytes) 000441 1787848673.498598 MIRROR_EXTEND to 2 mirrors 000443 1787848674.682309 MIRROR_RESYNC 2 mirrors 000444 1787848675.090621 MIRROR_RESYNC 2 mirrors 000445 1787848675.417322 MIRROR_EXTEND to 3 mirrors 000447 1787848676.294636 mapwrite 0x03aa2c thru 0x03ffff (0x55d4 bytes) 000448 1787848676.439614 write 0x01ad2a thru 0x024b60 (0x9e37 bytes) 000449 1787848676.450012 mapwrite 0x00e737 thru 0x010704 (0x1fce bytes) 000450 1787848676.525952 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8474d648 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8474d648 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000452 1787848676.891984 MIRROR_RESYNC 2 mirrors 000453 1787848678.296594 write 0x02706a thru 0x027a1f (0x9b6 bytes) 000454 1787848678.452342 write 0x00582b thru 0x01141a (0xbbf0 bytes) 000455 1787848678.467248 trunc from 0x040000 to 0x038042 (0x7fbd bytes) 000456 1787848678.508998 mapwrite 0x038a9c thru 0x03a688 (0x1bed bytes) 000457 1787848678.626272 write 0x008dff thru 0x0138b3 (0xaab5 bytes) 000458 1787848678.647989 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000459 1787848678.877455 read 0x002c03 thru 0x006531 (0x392f bytes) 000460 1787848678.896657 write 0x02cf51 thru 0x03738f (0xa43f bytes) 000461 1787848678.925269 read 0x003215 thru 0x010a48 (0xd834 bytes) 000465 1787848678.948821 read 0x02852d thru 0x03005b (0x7b2f bytes) 000467 1787848678.951198 read 0x022ac6 thru 0x02f9bb (0xcef6 bytes) 000469 1787848678.953831 mapread 0x01a454 thru 0x029704 (0xf2b1 bytes) 000470 1787848678.955764 trunc from 0x03a689 to 0x0380b7 (0x25d1 bytes) 000472 1787848678.979963 read 0x038044 thru 0x0380b7 (0x74 bytes) 000473 1787848678.980890 mapread 0x007a91 thru 0x00c4a7 (0x4a17 bytes) 000475 1787848678.983871 read 0x024d10 thru 0x0294a1 (0x4792 bytes) 000476 1787848678.984792 read 0x011e21 thru 0x015680 (0x3860 bytes) 000477 1787848678.986206 trunc from 0x0380b8 to 0x01bc80 (0x1c437 bytes) 000478 1787848679.004099 trunc from 0x01bc81 to 0x00d1f0 (0xea90 bytes) 000479 1787848679.025405 mapread 0x005b94 thru 0x00d1f0 (0x765d bytes) 000480 1787848679.028546 mapread 0x0034b1 thru 0x0041b2 (0xd02 bytes) 000481 1787848679.033315 write 0x000ca7 thru 0x0016d4 (0xa2e bytes) 000482 1787848679.034570 trunc from 0x00d1f1 to 0x02784e (0x1a65e bytes) 000485 1787848679.058025 trunc from 0x02784f to 0x02512b (0x2723 bytes) 000487 1787848679.093199 mapwrite 0x0312e0 thru 0x03a677 (0x9398 bytes) 000489 1787848679.243323 mapwrite 0x00648e thru 0x00aa6d (0x45e0 bytes) 000490 1787848679.341017 mapread 0x03704f thru 0x03a677 (0x3629 bytes) 000491 1787848679.350553 mapwrite 0x01160d thru 0x016644 (0x5038 bytes) 000497 1787848679.451755 read 0x0067e8 thru 0x012948 (0xc161 bytes) 000499 1787848679.456394 trunc from 0x03a678 to 0x00631b (0x3435c bytes) 000500 1787848679.485026 write 0x00bd1c thru 0x016505 (0xa7ea bytes) 000502 1787848679.493411 write 0x01ff9f thru 0x022fc6 (0x3028 bytes) 000506 1787848679.503533 write 0x01d4f1 thru 0x0271a2 (0x9cb2 bytes) 000509 1787848679.533577 mapwrite 0x03ff75 thru 0x03ffff (0x8b bytes) 000510 1787848679.619291 read 0x039011 thru 0x03f227 (0x6217 bytes) 000511 1787848679.620274 trunc from 0x040000 to 0x03dbc6 (0x2439 bytes) 000514 1787848679.660090 write 0x002a8f thru 0x004353 (0x18c5 bytes) 000518 1787848679.663351 mapread 0x01c863 thru 0x023954 (0x70f2 bytes) 000519 1787848679.664328 trunc from 0x03dbc7 to 0x011336 (0x2c890 bytes) 000520 1787848679.695326 MIRROR_EXTEND to 2 mirrors 000521 1787848680.706735 mapwrite 0x0014e6 thru 0x007790 (0x62ab bytes) 000523 1787848680.896425 mapread 0x006fdb thru 0x0098be (0x28e4 bytes) 000524 1787848680.901771 mapwrite 0x02c262 thru 0x039b58 (0xd8f7 bytes) 000525 1787848681.038371 read 0x01a800 thru 0x01bfda (0x17db bytes) 000528 1787848681.046279 mapwrite 0x03f449 thru 0x03ffff (0xbb7 bytes) 000533 1787848681.182633 mapread 0x014ca6 thru 0x0217b6 (0xcb11 bytes) 000535 1787848681.220677 trunc from 0x040000 to 0x010aee (0x2f511 bytes) 000537 1787848681.256479 trunc from 0x010aef to 0x02378b (0x12c9d bytes) 000538 1787848681.307460 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000541 1787848681.498633 mapwrite 0x025bf8 thru 0x02beae (0x62b7 bytes) 000542 1787848681.602196 mapwrite 0x0008e5 thru 0x00ad0c (0xa428 bytes) 000544 1787848681.738474 write 0x02bf0d thru 0x030fdb (0x50cf bytes) 000546 1787848681.741621 read 0x029e51 thru 0x02e09b (0x424b bytes) 000547 1787848681.744752 mapwrite 0x01a847 thru 0x022eea (0x86a4 bytes) 000549 1787848681.854469 mapread 0x0109f8 thru 0x0186d0 (0x7cd9 bytes) 000551 1787848681.881419 mapread 0x016378 thru 0x016e3e (0xac7 bytes) 000552 1787848681.887722 trunc from 0x030fdc to 0x0335ed (0x2612 bytes) 000555 1787848681.931057 mapwrite 0x03945f thru 0x03ffff (0x6ba1 bytes) 000557 1787848682.082690 mapwrite 0x00d085 thru 0x0169ed (0x9969 bytes) 000558 1787848682.156922 write 0x0334d5 thru 0x03f694 (0xc1c0 bytes) 000559 1787848682.170373 read 0x00e61c thru 0x00f97f (0x1364 bytes) 000560 1787848682.171434 trunc from 0x040000 to 0x0316ce (0xe931 bytes) 000564 1787848682.214911 mapread 0x00bc27 thru 0x018e5c (0xd236 bytes) 000568 1787848682.226189 write 0x026b21 thru 0x0358c7 (0xeda7 bytes) 000569 1787848682.228543 trunc from 0x0358c8 to 0x000ec7 (0x34a00 bytes) 000571 1787848682.275962 read 0x000aef thru 0x000ec7 (0x3d9 bytes) 000575 1787848682.283486 trunc from 0x000ec8 to 0x0181fc (0x17335 bytes) 000576 1787848682.321791 mapread 0x001e77 thru 0x00a0da (0x8264 bytes) 000578 1787848682.342471 mapread 0x010241 thru 0x0181fc (0x7fbc bytes) 000579 1787848682.344196 write 0x01ee9a thru 0x0226ea (0x3851 bytes) 000583 1787848682.356654 read 0x00075d thru 0x00e9a8 (0xe24c bytes) 000588 1787848682.364099 trunc from 0x0226eb to 0x026b5c (0x4472 bytes) 000591 1787848682.419479 read 0x0191d3 thru 0x026398 (0xd1c6 bytes) 000593 1787848682.450416 write 0x01d757 thru 0x0243f0 (0x6c9a bytes) 000595 1787848682.454056 write 0x03b760 thru 0x03ffff (0x48a0 bytes) 000597 1787848682.471684 MIRROR_EXTEND to 2 mirrors 000599 1787848683.692024 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000600 1787848683.878579 mapread 0x00072f thru 0x009964 (0x9236 bytes) 000606 1787848683.935661 mapread 0x00cc8a thru 0x018cfd (0xc074 bytes) 000612 1787848683.958851 mapwrite 0x027676 thru 0x030e87 (0x9812 bytes) 000613 1787848684.079753 trunc from 0x040000 to 0x007e10 (0x381ef bytes) 000614 1787848684.096544 write 0x02d4be thru 0x034ed5 (0x7a18 bytes) 000616 1787848684.100235 read 0x02c785 thru 0x034ed5 (0x8751 bytes) 000624 1787848684.114934 trunc from 0x034ed6 to 0x032dac (0x2129 bytes) 000625 1787848684.130616 mapread 0x00342d thru 0x00f79a (0xc36e bytes) 000629 1787848684.140447 write 0x0368c0 thru 0x03d94f (0x7090 bytes) 000631 1787848684.147515 MIRROR_EXTEND to 2 mirrors 000634 1787848685.418199 read 0x03a37d thru 0x03d94f (0x35d3 bytes) 000635 1787848685.435893 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000636 1787848685.684233 read 0x008e14 thru 0x018007 (0xf1f4 bytes) 000637 1787848685.702276 read 0x038821 thru 0x03ab4d (0x232d bytes) 000639 1787848685.729373 read 0x03c81c thru 0x03d94f (0x1134 bytes) 000641 1787848685.733957 mapwrite 0x03a330 thru 0x03ffff (0x5cd0 bytes) 000644 1787848685.881244 mapwrite 0x03afb5 thru 0x03ffff (0x504b bytes) 000646 1787848686.001539 mapread 0x01d219 thru 0x01f4a9 (0x2291 bytes) 000648 1787848686.005710 read 0x03b4b7 thru 0x03f3af (0x3ef9 bytes) 000650 1787848686.009220 write 0x01733c thru 0x01f76e (0x8433 bytes) 000652 1787848686.016548 write 0x03893f thru 0x03e900 (0x5fc2 bytes) 000653 1787848686.018670 read 0x02b60f thru 0x038a77 (0xd469 bytes) 000654 1787848686.028098 MIRROR_EXTEND to 2 mirrors 000655 1787848687.138598 mapwrite 0x02f26f thru 0x039ff7 (0xad89 bytes) 000656 1787848687.404095 mapwrite 0x01883b thru 0x018a86 (0x24c bytes) 000657 1787848687.470081 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000658 1787848687.638777 mapread 0x02df60 thru 0x039fcf (0xc070 bytes) 000659 1787848687.672782 read 0x029afa thru 0x036637 (0xcb3e bytes) 000660 1787848687.674056 read 0x016c5a thru 0x02267e (0xba25 bytes) 000661 1787848687.684867 mapwrite 0x03a710 thru 0x03ffff (0x58f0 bytes) 000663 1787848687.738640 trunc from 0x040000 to 0x00ec95 (0x3136a bytes) 000664 1787848687.795988 MIRROR_EXTEND to 2 mirrors 000665 1787848688.780320 write 0x005a59 thru 0x00da4e (0x7ff6 bytes) 000666 1787848688.881069 trunc from 0x00ec96 to 0x016253 (0x75be bytes) 000667 1787848688.901294 trunc from 0x016254 to 0x01d157 (0x6f04 bytes) 000668 1787848688.936259 MIRROR_RESYNC 2 mirrors 000669 1787848690.006465 mapwrite 0x02f2c4 thru 0x0329c5 (0x3702 bytes) 000671 1787848690.175472 write 0x01f7c4 thru 0x029652 (0x9e8f bytes) 000673 1787848690.186968 read 0x02b94e thru 0x0329c5 (0x7078 bytes) 000675 1787848690.192383 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x329c6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329c6): Mirror 2: 0xb523734b (stale) CRC-32 checksum value for chunk [0, 0x329c6): Mirror 1: 0x8a48f11f Mirror 3: 0x8a48f11f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329c6: skipped 000676 1787848691.497834 mapwrite 0x021c1d thru 0x030c50 (0xf034 bytes) 000677 1787848691.756262 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x329c6): Mirror 3: 0x8a48f11f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329c6): Mirror 3: 0x8a48f11f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329c6: skipped 000679 1787848692.105449 mapwrite 0x00416a thru 0x00cf2c (0x8dc3 bytes) 000681 1787848692.167087 write 0x022627 thru 0x0291ac (0x6b86 bytes) 000682 1787848692.175999 write 0x01db66 thru 0x01f85c (0x1cf7 bytes) 000683 1787848692.207335 write 0x027a81 thru 0x031312 (0x9892 bytes) 000686 1787848692.216892 mapread 0x0037ff thru 0x0115ba (0xddbc bytes) 000687 1787848692.248411 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000688 1787848692.400066 write 0x0179c9 thru 0x01b755 (0x3d8d bytes) 000692 1787848692.419068 MIRROR_EXTEND to 2 mirrors 000693 1787848693.467953 mapread 0x01bdf9 thru 0x0222ae (0x64b6 bytes) 000694 1787848693.498236 mapread 0x018b1b thru 0x021bd3 (0x90b9 bytes) 000695 1787848693.507264 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000699 1787848693.717803 read 0x0133fe thru 0x02064c (0xd24f bytes) 000701 1787848693.746082 write 0x01d339 thru 0x025876 (0x853e bytes) 000702 1787848693.786452 mapwrite 0x015ff6 thru 0x022e75 (0xce80 bytes) 000704 1787848693.873321 write 0x004091 thru 0x009e5f (0x5dcf bytes) 000705 1787848693.883616 mapwrite 0x012dda thru 0x01e3d1 (0xb5f8 bytes) 000706 1787848693.931082 trunc from 0x0329c6 to 0x03d867 (0xaea2 bytes) 000708 1787848693.948087 mapwrite 0x027e8a thru 0x037cd5 (0xfe4c bytes) 000711 1787848694.041251 mapwrite 0x018002 thru 0x020633 (0x8632 bytes) 000713 1787848694.083940 mapwrite 0x0088cf thru 0x008991 (0xc3 bytes) 000714 1787848694.118297 write 0x004c3e thru 0x00836d (0x3730 bytes) 000715 1787848694.126679 MIRROR_EXTEND to 2 mirrors 000716 1787848695.462366 trunc from 0x03d868 to 0x00a3a8 (0x334bf bytes) 000717 1787848695.557288 trunc from 0x00a3a9 to 0x01b5c1 (0x11219 bytes) 000718 1787848695.575200 trunc from 0x01b5c2 to 0x01a021 (0x15a0 bytes) 000720 1787848695.598658 read 0x002279 thru 0x008052 (0x5dda bytes) 000722 1787848695.607028 mapread 0x007fb3 thru 0x00f57a (0x75c8 bytes) 000723 1787848695.631353 write 0x001dec thru 0x00979a (0x79af bytes) 000724 1787848695.640600 mapwrite 0x00045d thru 0x008139 (0x7cdd bytes) 000727 1787848695.707942 read 0x00f9a1 thru 0x010c77 (0x12d7 bytes) 000730 1787848695.709879 mapwrite 0x02ea37 thru 0x038272 (0x983c bytes) 000732 1787848695.794590 MIRROR_RESYNC 2 mirrors 000733 1787848696.749077 mapwrite 0x0047d1 thru 0x006967 (0x2197 bytes) 000734 1787848696.931352 mapread 0x005081 thru 0x00eafc (0x9a7c bytes) 000735 1787848696.934942 mapread 0x025f4f thru 0x03185d (0xb90f bytes) 000736 1787848696.956206 write 0x016dc2 thru 0x017f06 (0x1145 bytes) 000737 1787848696.969503 write 0x034178 thru 0x03ffff (0xbe88 bytes) 000739 1787848696.978993 MIRROR_RESYNC 2 mirrors 000740 1787848697.989779 write 0x00b2fb thru 0x01b0e2 (0xfde8 bytes) 000741 1787848698.108580 mapwrite 0x001aee thru 0x00545a (0x396d bytes) 000743 1787848698.266977 mapread 0x00b7cb thru 0x014431 (0x8c67 bytes) 000744 1787848698.276846 read 0x039d05 thru 0x03db6c (0x3e68 bytes) 000745 1787848698.297971 mapwrite 0x0027b0 thru 0x006913 (0x4164 bytes) 000746 1787848698.377219 MIRROR_RESYNC 2 mirrors 000748 1787848699.061023 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000749 1787848699.172075 trunc from 0x040000 to 0x013397 (0x2cc68 bytes) 000751 1787848699.227653 read 0x0127fb thru 0x013397 (0xb9d bytes) 000752 1787848699.232991 write 0x006c6f thru 0x010f92 (0xa324 bytes) 000753 1787848699.241691 mapwrite 0x024d07 thru 0x0264a7 (0x17a1 bytes) 000755 1787848699.326654 read 0x014af2 thru 0x01a257 (0x5766 bytes) 000759 1787848699.343360 mapwrite 0x020fa1 thru 0x026e82 (0x5ee2 bytes) 000760 1787848699.406086 write 0x03212c thru 0x03ebc7 (0xca9c bytes) 000762 1787848699.409316 mapread 0x0264da thru 0x03109e (0xabc5 bytes) 000763 1787848699.415981 mapwrite 0x02bbdf thru 0x02f1a0 (0x35c2 bytes) 000764 1787848699.461818 write 0x03968d thru 0x03d866 (0x41da bytes) 000765 1787848699.462842 trunc from 0x03ebc8 to 0x02f1be (0xfa09 bytes) 000766 1787848699.477704 mapwrite 0x035ca8 thru 0x03ffff (0xa358 bytes) 000769 1787848699.628487 trunc from 0x040000 to 0x000c60 (0x3f39f bytes) 000770 1787848699.666011 read 0x0002b2 thru 0x000c60 (0x9af bytes) 000771 1787848699.667856 mapread 0x0008bb thru 0x000c60 (0x3a6 bytes) 000773 1787848699.673231 mapread 0x000a36 thru 0x000c60 (0x22b bytes) 000776 1787848699.677712 mapread 0x0002d1 thru 0x000c60 (0x990 bytes) 000777 1787848699.678658 mapread 0x000483 thru 0x000c60 (0x7de bytes) 000778 1787848699.679407 trunc from 0x000c61 to 0x01ddba (0x1d15a bytes) 000779 1787848699.700858 mapread 0x0163b1 thru 0x01ddba (0x7a0a bytes) 000783 1787848699.717480 write 0x0027fe thru 0x0065d6 (0x3dd9 bytes) 000785 1787848699.721892 read 0x008038 thru 0x00d2e7 (0x52b0 bytes) 000788 1787848699.727453 read 0x015a57 thru 0x016af7 (0x10a1 bytes) 000789 1787848699.728013 mapread 0x013f30 thru 0x0192b3 (0x5384 bytes) 000791 1787848699.730524 read 0x001962 thru 0x008749 (0x6de8 bytes) 000795 1787848699.733452 mapwrite 0x004180 thru 0x013d7b (0xfbfc bytes) 000797 1787848699.854629 trunc from 0x01ddbb to 0x039e63 (0x1c0a9 bytes) 000798 1787848699.878678 write 0x030099 thru 0x03d7b3 (0xd71b bytes) 000799 1787848699.892965 read 0x02b46f thru 0x02c429 (0xfbb bytes) 000800 1787848699.894875 read 0x03bd91 thru 0x03d7b3 (0x1a23 bytes) 000801 1787848699.914360 MIRROR_EXTEND to 2 mirrors 000803 1787848700.963423 mapread 0x035754 thru 0x03d7b3 (0x8060 bytes) 000805 1787848700.978640 read 0x030bd2 thru 0x03576c (0x4b9b bytes) 000806 1787848700.979250 mapread 0x012718 thru 0x0191a2 (0x6a8b bytes) 000807 1787848700.988023 write 0x000752 thru 0x000c07 (0x4b6 bytes) 000811 1787848701.069220 trunc from 0x03d7b4 to 0x02b4ac (0x12307 bytes) 000812 1787848701.098598 read 0x02927d thru 0x029802 (0x586 bytes) 000813 1787848701.112020 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) 000814 1787848701.298683 write 0x0315a4 thru 0x036360 (0x4dbd bytes) 000815 1787848701.302184 mapwrite 0x00895b thru 0x00b5bf (0x2c65 bytes) 000816 1787848701.409116 mapread 0x01a8b6 thru 0x024a3a (0xa185 bytes) 000822 1787848701.435298 MIRROR_EXTEND to 2 mirrors 000823 1787848702.438221 mapread 0x027b4c thru 0x029d14 (0x21c9 bytes) 000824 1787848702.479642 MIRROR_EXTEND to 3 mirrors 000826 1787848703.735988 MIRROR_EXTEND to 4 mirrors 000827 1787848705.090700 trunc from 0x036361 to 0x039dcc (0x3a6c bytes) 000828 1787848705.172974 read 0x00aadc thru 0x013ffe (0x9523 bytes) 000829 1787848705.182291 mapwrite 0x021382 thru 0x02c3e8 (0xb067 bytes) 000830 1787848705.272267 write 0x0266f3 thru 0x027c65 (0x1573 bytes) 000833 1787848705.275277 trunc from 0x039dcd to 0x03a3c4 (0x5f8 bytes) 000834 1787848705.302041 trunc from 0x03a3c5 to 0x01cff7 (0x1d3cd bytes) 000835 1787848705.319899 read 0x012af0 thru 0x01cff7 (0xa508 bytes) 000836 1787848705.337176 read 0x005eff thru 0x005fe9 (0xeb bytes) 000838 1787848705.342197 trunc from 0x01cff8 to 0x023d9b (0x6da4 bytes) 000840 1787848705.362349 trunc from 0x023d9c to 0x011763 (0x12638 bytes) 000841 1787848705.391086 mapread 0x00ef4e thru 0x011763 (0x2816 bytes) 000842 1787848705.402690 mapread 0x00b9e6 thru 0x011763 (0x5d7e bytes) 000843 1787848705.403875 write 0x032907 thru 0x03d0e1 (0xa7db bytes) 000845 1787848705.406500 mapread 0x017617 thru 0x026a2a (0xf414 bytes) 000848 1787848705.432587 mapread 0x01b9dc thru 0x01e30f (0x2934 bytes) 000849 1787848705.434555 mapwrite 0x01617c thru 0x01b91c (0x57a1 bytes) 000853 1787848705.532194 read 0x034e9d thru 0x03bc1c (0x6d80 bytes) 000855 1787848705.535019 trunc from 0x03d0e2 to 0x00cadb (0x30606 bytes) 000856 1787848705.563555 mapread 0x005d4d thru 0x00cadb (0x6d8f bytes) 000857 1787848705.565387 write 0x01edaf thru 0x02b05a (0xc2ac bytes) 000858 1787848705.568838 mapread 0x00b06e thru 0x014dd6 (0x9d69 bytes) 000859 1787848705.585171 trunc from 0x02b05b to 0x01f385 (0xbcd5 bytes) 000860 1787848705.626221 mapread 0x0169e9 thru 0x01a431 (0x3a49 bytes) 000862 1787848705.631932 read 0x01cca5 thru 0x01f385 (0x26e1 bytes) 000864 1787848705.634517 read 0x000dd7 thru 0x010797 (0xf9c1 bytes) 000865 1787848705.643824 MIRROR_RESYNC 4 mirrors 000866 1787848707.290040 write 0x01fdac thru 0x0293d1 (0x9626 bytes) 000867 1787848707.427373 trunc from 0x0293d2 to 0x034f04 (0xbb33 bytes) 000869 1787848707.460409 trunc from 0x034f05 to 0x015d09 (0x1f1fb bytes) 000873 1787848707.492070 write 0x031880 thru 0x032f7a (0x16fb bytes) 000875 1787848707.495046 mapread 0x02687c thru 0x02b799 (0x4f1e bytes) 000876 1787848707.515107 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f7b): Mirror 2: 0x3977c134 (stale) Mirror 3: 0x3977c134 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f7b): Mirror 2: 0x3977c134 (stale) Mirror 3: 0x3977c134 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f7b: skipped 000877 1787848707.928411 write 0x02b7e6 thru 0x02c772 (0xf8d bytes) 000878 1787848707.943831 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f7b): Mirror 3: 0x3977c134 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f7b): Mirror 3: 0x3977c134 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f7b: skipped 000879 1787848708.239064 write 0x0302a7 thru 0x0355f2 (0x534c bytes) 000880 1787848708.246924 mapwrite 0x0241fd thru 0x02edb8 (0xabbc bytes) 000881 1787848708.348948 write 0x005cec thru 0x0144cf (0xe7e4 bytes) 000882 1787848708.380532 write 0x00a7b9 thru 0x017741 (0xcf89 bytes) 000884 1787848708.394752 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x355f3): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x355f3): Mirror 3: 0x3909b191 (stale) CRC-32 checksum value for chunk [0, 0x355f3): Mirror 1: 0xd18c0a8c Mirror 4: 0xd18c0a8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x355f3: skipped 000885 1787848709.691734 mapwrite 0x02d884 thru 0x03b13f (0xd8bc bytes) 000886 1787848709.896026 trunc from 0x03b140 to 0x0126a3 (0x28a9c bytes) 000887 1787848709.944926 MIRROR_RESYNC 3 mirrors 000890 1787848710.688966 trunc from 0x0126a4 to 0x014d0e (0x266b bytes) 000892 1787848710.756727 trunc from 0x014d0f to 0x02933b (0x1462d bytes) 000895 1787848710.777503 mapread 0x00ae2d thru 0x00d28e (0x2462 bytes) 000896 1787848710.794402 mapwrite 0x01a068 thru 0x0271a9 (0xd142 bytes) 000897 1787848710.907575 write 0x00f2b9 thru 0x01a96f (0xb6b7 bytes) 000899 1787848710.910629 write 0x00a766 thru 0x016dc7 (0xc662 bytes) 000905 1787848710.919811 mapwrite 0x00a0b3 thru 0x015de0 (0xbd2e bytes) 000906 1787848710.971726 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2933c): Mirror 4: 0x88172856 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2933c): Mirror 4: 0x88172856 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2933c: skipped 000911 1787848711.311596 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2933c): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2933c): Mirror 4: 0x88172856 (stale) CRC-32 checksum value for chunk [0, 0x2933c): Mirror 1: 0x67f35ae5 Mirror 5: 0x67f35ae5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2933c: skipped 000912 1787848712.776764 write 0x03c647 thru 0x03ffff (0x39b9 bytes) 000913 1787848712.904271 read 0x0072d8 thru 0x0109ed (0x9716 bytes) 000914 1787848712.928252 write 0x005aa0 thru 0x0074c6 (0x1a27 bytes) 000915 1787848712.944395 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x497339a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x497339a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000917 1787848713.326218 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x497339a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc58f4729 Mirror 5: 0xc58f4729 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000918 1787848714.799066 trunc from 0x040000 to 0x018d13 (0x272ec bytes) 000920 1787848714.937620 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18d14): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18d14): Mirror 4: 0x48c28a37 (stale) Mirror 5: 0x7285dd64 (stale) CRC-32 checksum value for chunk [0, 0x18d14): Mirror 1: 0x7285dd64 Mirror 6: 0x7285dd64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18d14: skipped 000921 1787848716.108168 mapwrite 0x03c43b thru 0x03ffff (0x3bc5 bytes) 000922 1787848716.289598 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x497339a2 (stale) Mirror 5: 0xc58f4729 (stale) Mirror 6: 0xb044ab0d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe2054a57 Mirror 7: 0xe2054a57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000923 1787848717.489812 write 0x03f2fa thru 0x03f660 (0x367 bytes) 000925 1787848717.570893 mapread 0x033034 thru 0x03cbc4 (0x9b91 bytes) 000927 1787848717.592867 read 0x002352 thru 0x010fdd (0xec8c bytes) 000928 1787848717.603526 mapread 0x03dd33 thru 0x03ffff (0x22cd bytes) 000931 1787848717.619094 MIRROR_RESYNC 5 mirrors 000932 1787848719.106940 mapread 0x032dc1 thru 0x033a71 (0xcb1 bytes) 000934 1787848719.119271 read 0x0252e3 thru 0x0332c5 (0xdfe3 bytes) 000935 1787848719.131288 trunc from 0x040000 to 0x009721 (0x368de bytes) 000936 1787848719.236083 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9722): Mirror 4: 0x74c112aa (stale) Mirror 6: 0x74c112aa (stale) Mirror 7: 0x74c112aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9722): Mirror 4: 0x74c112aa (stale) Mirror 6: 0x74c112aa (stale) Mirror 7: 0x74c112aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9722: skipped 000938 1787848719.675210 mapread 0x004973 thru 0x009721 (0x4daf bytes) 000939 1787848719.694632 mapwrite 0x03354c thru 0x03a902 (0x73b7 bytes) 000940 1787848719.780169 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a903): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a903): Mirror 4: 0x6a9a15a1 (stale) Mirror 6: 0x6a9a15a1 (stale) Mirror 7: 0x6a9a15a1 (stale) CRC-32 checksum value for chunk [0, 0x3a903): Mirror 1: 0x65b2959 Mirror 8: 0x65b2959 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a903: skipped 000942 1787848720.782939 read 0x01c677 thru 0x01ec77 (0x2601 bytes) 000943 1787848720.790448 trunc from 0x03a903 to 0x03cce3 (0x23e1 bytes) 000944 1787848720.889537 read 0x01a7b1 thru 0x01fbc6 (0x5416 bytes) 000945 1787848720.904014 mapwrite 0x0105a7 thru 0x01bfd0 (0xba2a bytes) 000946 1787848720.957518 mapwrite 0x01082f thru 0x01eeac (0xe67e bytes) 000949 1787848721.012185 write 0x023b8a thru 0x02d521 (0x9998 bytes) 000950 1787848721.014586 read 0x02e408 thru 0x03ccc8 (0xe8c1 bytes) 000951 1787848721.017762 trunc from 0x03cce4 to 0x00e80e (0x2e4d5 bytes) 000952 1787848721.042300 write 0x004ce3 thru 0x00d953 (0x8c71 bytes) 000954 1787848721.045165 read 0x001b1d thru 0x00e158 (0xc63c bytes) 000959 1787848721.052178 MIRROR_RESYNC 5 mirrors 000960 1787848723.049178 trunc from 0x00e80f to 0x03aa1e (0x2c210 bytes) 000961 1787848723.131301 read 0x008233 thru 0x00a7dc (0x25aa bytes) 000962 1787848723.142597 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x196:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aa1f): Mirror 4: 0xe54caede (stale) Mirror 6: 0xe54caede (stale) Mirror 8: 0xe54caede (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa1f): Mirror 4: 0xe54caede (stale) Mirror 6: 0xe54caede (stale) Mirror 8: 0xe54caede (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa1f: skipped 000963 1787848723.475328 mapwrite 0x014402 thru 0x023c28 (0xf827 bytes) 000965 1787848723.563531 mapwrite 0x031c30 thru 0x032a16 (0xde7 bytes) 000966 1787848723.605229 trunc from 0x03aa1f to 0x0071f5 (0x33829 bytes) 000969 1787848723.627106 write 0x00ebc2 thru 0x010ec6 (0x2305 bytes) 000970 1787848723.634974 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x10ec7): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10ec7): Mirror 4: 0x5fc8d2db (stale) Mirror 6: 0x5fc8d2db (stale) Mirror 8: 0x5fc8d2db (stale) CRC-32 checksum value for chunk [0, 0x10ec7): Mirror 1: 0x700d3e07 Mirror 9: 0x700d3e07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10ec7: skipped 000972 1787848724.900203 read 0x00df39 thru 0x010ec6 (0x2f8e bytes) 000975 1787848724.906564 write 0x01eeee thru 0x0270ff (0x8212 bytes) 000976 1787848724.972948 read 0x010228 thru 0x01a522 (0xa2fb bytes) 000977 1787848724.985540 write 0x020ac6 thru 0x024c28 (0x4163 bytes) 000978 1787848724.987728 mapread 0x00703d thru 0x015b08 (0xeacc bytes) 000979 1787848724.989597 read 0x020ca2 thru 0x0270ff (0x645e bytes) 000983 1787848724.994031 write 0x017f46 thru 0x01ebeb (0x6ca6 bytes) 000984 1787848724.996399 mapwrite 0x01dd75 thru 0x025c11 (0x7e9d bytes) 000985 1787848725.048513 mapread 0x023a75 thru 0x0256ed (0x1c79 bytes) 000988 1787848725.052310 trunc from 0x027100 to 0x010f6a (0x16195 bytes) 000991 1787848725.079011 read 0x010166 thru 0x010f6a (0xe05 bytes) 000992 1787848725.080484 read 0x0041a7 thru 0x007d87 (0x3be1 bytes) 000995 1787848725.083759 trunc from 0x010f6b to 0x019a1b (0x8ab1 bytes) 000996 1787848725.107007 trunc from 0x019a1c to 0x0270ba (0xd69f bytes) 000997 1787848725.135958 read 0x015782 thru 0x016685 (0xf04 bytes) 000998 1787848725.166632 MIRROR_RESYNC 5 mirrors All operations completed A-OK! PASS 70a (113s) == sanity-flr test 71: check mirror extend parent fid ==== 12:38:52 (1787848732) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 12:38:56 (1787848736) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259851 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162353 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 12:39:07 (1787848747) Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre2 Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5583264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5557696 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8394720 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7896128 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8031776 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8835008 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8978048 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8470592 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8888672 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8985888 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58729 58730 58731 58733 58735 done 192.168.206.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg615-client.virtnet /mnt/lustre2 (opts:) 192.168.206.115@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg615-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:40:31 (1787848831) Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre2 Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2346560 .. Extending file size to 3034848 .. Extending file size to 4766400 .. Extending file size to 2774848 .. Extending file size to 7919840 .. Extending file size to 4517088 .. Extending file size to 5439296 .. Extending file size to 7800064 .. Extending file size to 8166848 .. Extending file size to 8347872 .. Extending file size to 8509824 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9344320 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8229408 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8675328 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8878112 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 60416 60417 60418 60421 60423 failed 192.168.206.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg615-client.virtnet /mnt/lustre2 (opts:) 192.168.206.115@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg615-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:41:52 (1787848912) Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178913 s, 11.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.296212 s, 7.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x62e7a051 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg615-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:42:11 (1787848931) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131882 s, 15.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 12:42:19 (1787848939) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e9:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:42:28 (1787848948) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274795 s, 7.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:42:37 (1787848957) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182367 s, 11.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:42:46 (1787848966) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.281046 s, 7.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:43:19 (1787848999) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg615-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181631 s, 5.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:44:18 (1787849058) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg615-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.423555 s, 5.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:45:16 (1787849116) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg615-server mds-ost sync done. Creating new pool test_204f oleg615-server: Pool lustre.test_204f created Adding targets to pool oleg615-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg615-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255655 s, 8.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg615-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg615-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg615-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:46:36 (1787849196) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:46:46 (1787849206) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:46:55 (1787849215) oleg615-server: Pool lustre.test_206 created Adding targets to pool oleg615-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg615-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg615-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg615-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg615-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:47:19 (1787849239) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824495 s, 12.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544173 s, 19.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x64502d20 Mirror 3: 0x64502d20 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:47:30 (1787849250) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:47:35 (1787849255) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:47:39 (1787849259) Starting client: oleg615-client.virtnet: -o user_xattr,flock 192.168.206.115@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.123414 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x19f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c00000 Read this (4 bytes): BBBB 192.168.206.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg615-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:48:23 (1787849303) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0421876 s, 194 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:48:29 (1787849309) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484326 s, 21.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20e lmm_fid: [0x200000404:0x20e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20e lmm_fid: [0x200000404:0x20e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834446 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 12:48:31.000000000 -0400 Modify: 2026-08-27 12:48:31.000000000 -0400 Change: 2026-08-27 12:48:32.000000000 -0400 Birth: 2026-08-27 12:48:31.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:48:37 (1787849317) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:48:40 (1787849320) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.461726 s, 88.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0603045 s, 67.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 12:48 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 12:48 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:48:48 (1787849328) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:48:55 (1787849335) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1809 sec =========== 12:49:03 (1787849343) === sanity-flr: start cleanup 12:49:05 (1787849345) === === sanity-flr: finish cleanup 12:49:12 (1787849352) ===