-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 18:45:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg401-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:45:38 (1787784338) === oleg401-client.virtnet: executing check_config_client /mnt/lustre oleg401-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg401-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93da5037e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93da5037e000.idle_timeout=debug disable quota as required oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:46:21 (1787784381) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7640] == sanity-flr test 0a: lfs mirror create with -N option == 18:46:24 (1787784384) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:46:46 (1787784406) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:46:50 (1787784410) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:46:54 (1787784414) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:47:26 (1787784446) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:47:31 (1787784451) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 18:47:35 (1787784455) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93da5037e000.stats=clear osc.lustre-OST0001-osc-ffff93da5037e000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:47:44 (1787784464) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:47:58 (1787784478) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:48:08 (1787784488) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:48:12 (1787784492) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:48:21 (1787784501) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 18:48:32 (1787784512) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:48:40 (1787784520) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:48:43 (1787784523) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:48:51 (1787784531) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:49:03 (1787784543) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:49:11 (1787784551) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.0006 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12014 s, 8.7 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:49:23 (1787784563) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.414085 s, 38.0 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:49:34 (1787784574) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:49:47 (1787784587) Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93da5037e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93da5037e000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93da5037e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93da5037e000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (88s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:51:15 (1787784675) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146242 s, 14.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93da5037e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93da5037e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:51:57 (1787784717) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93da5037e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93da5037e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93da5037e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93da5037e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:53:24 (1787784804) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.97772 s, 10.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 18:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.099s user 0m0.000s sys 0m0.165s Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93da5037e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93da5037e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.252s user 0m0.001s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93da5037e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93da5037e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.425s user 0m0.000s sys 0m0.220s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93da5037e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93da5037e000.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:55:09 (1787784909) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:55:13 (1787784913) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:55:16 (1787784916) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 18:55:20 (1787784920) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 18:55:28 (1787784928) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.540245 s, 23.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:55:42 (1787784942) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442429 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567077 s, 18.5 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:55:49 (1787784949) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:55:57 (1787784957) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00161907 s, 632 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00166396 s, 615 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111049 s, 922 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.001375 s, 745 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 18:56:08 (1787784968) create test files with size 15737618 .. 2fea967a10102a5f4a9975709a61796b - 213dac5094d797bf7570fb6e641d935f - 798361dfae2e5274944bc0f57c679113 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 18:56:34 (1787784994) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:56:50 (1787785010) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:56:58 (1787785018) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131913 s, 7.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272551 s, 38.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 18:57:07 (1787785027) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141992 s, 14.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.291628 s, 14.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193792 s, 10.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 18:57:21 (1787785041) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 18:57:25 (1787785045) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:57:28 (1787785048) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239641 s, 43.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 18:57:35 (1787785055) llite.lustre-ffff93da4318a800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 18:57:42 (1787785062) llite.lustre-ffff93da4318a800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284681 s, 36.8 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333095 s, 31.5 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 18:57:52 (1787785072) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:57:56 (1787785076) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:58:06 (1787785086) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.378708 s, 27.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:58:20 (1787785100) llite.lustre-ffff93da47ade000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.498967 s, 21.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff93da47143000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:58:39 (1787785119) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:59:24 (1787785164) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:00:02 (1787785202) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 19:00:06 (1787785206) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.203385 s, 15.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.202883 s, 15.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x69f7865f (nosync,stale) Mirror 2: 0x69f7865f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 9b94c0d78c7094a6812e4c31e893fe0f - mirror 1 checksum: 69536c4e72cd8c2d84c4f48496388205 - mirror 2 checksum: 9b94c0d78c7094a6812e4c31e893fe0f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T19:00:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T19:00:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9b94c0d78c7094a6812e4c31e893fe0f - mirror 1 checksum: 69536c4e72cd8c2d84c4f48496388205 - mirror 2 checksum: 69536c4e72cd8c2d84c4f48496388205 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T19:00:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T19:00:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:00:20 (1787785220) Failing mds1 on oleg401-server Stopping /mnt/lustre-mds1 (opts:) on oleg401-server 19:00:27 (1787785227) shut down facet: mds1 facet_host: oleg401-server facet_failover_host: oleg401-server Failover mds1 to oleg401-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0000 19:01:02 (1787785262) targets are mounted 19:01:02 (1787785262) facet_failover done oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535412 s, 19.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:01:26 (1787785286) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:01:30 (1787785290) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:01:34 (1787785294) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0101822 s, 1.6 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:01:43 (1787785303) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:01:47 (1787785307) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0141565 s, 289 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0549939 s, 74.5 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:01:54 (1787785314) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:02:21 (1787785341) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:02:55 (1787785375) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:03:23 (1787785403) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:03:50 (1787785430) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 19:03:59 (1787785439) mirror create pid 55612 mirror split pid 55613 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55612 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55613 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 19:05:08 (1787785508) Seed set to 5510 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004 1787785511.181648 mapwrite 0x00e9d1 thru 0x01260d (0x3c3d bytes) truncating to largest ever: 0x39234 000008 1787785511.376986 trunc from 0x01260e to 0x039233 (0x26c26 bytes) 000010 1787785511.414575 mapread 0x007ece thru 0x00d356 (0x5489 bytes) 000011 1787785511.447693 MIRROR_EXTEND to 2 mirrors 000014 1787785512.491186 MIRROR_EXTEND to 3 mirrors 000015 1787785513.923110 mapread 0x013245 thru 0x01dabe (0xa87a bytes) 000016 1787785513.950618 mapread 0x0015a3 thru 0x008123 (0x6b81 bytes) 000017 1787785513.962034 write 0x028e57 thru 0x0291e4 (0x38e bytes) 000019 1787785514.120008 MIRROR_RESYNC 3 mirrors 000021 1787785516.039789 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) 000022 1787785516.545004 write 0x02407e thru 0x032e5a (0xeddd bytes) 000023 1787785516.772239 MIRROR_RESYNC 2 mirrors 000024 1787785518.186152 MIRROR_EXTEND to 3 mirrors 000027 1787785519.463985 mapread 0x035c6c thru 0x039233 (0x35c8 bytes) 000029 1787785519.521048 write 0x03b4f6 thru 0x03ffff (0x4b0a bytes) 000031 1787785519.752243 mapwrite 0x0191a3 thru 0x021fca (0x8e28 bytes) 000033 1787785519.867493 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf4cf1d57 (stale) Mirror 4: 0xf4cf1d57 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8b57a04b Mirror 5: 0x8b57a04b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000034 1787785521.558256 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf4cf1d57 (stale) Mirror 4: 0xf4cf1d57 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8b57a04b Mirror 5: 0x8b57a04b Mirror 6: 0x8b57a04b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000036 1787785523.911287 trunc from 0x040000 to 0x02fc9a (0x10365 bytes) 000037 1787785524.059226 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2fc9b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fc9b): Mirror 3: 0xef9e8c25 (stale) Mirror 4: 0xef9e8c25 (stale) Mirror 5: 0xcdfbb957 (stale) Mirror 6: 0xcdfbb957 (stale) CRC-32 checksum value for chunk [0, 0x2fc9b): Mirror 1: 0xcdfbb957 Mirror 7: 0xcdfbb957 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fc9b: skipped 000038 1787785525.143626 mapread 0x028981 thru 0x02fc9a (0x731a bytes) 000039 1787785525.165421 mapread 0x019453 thru 0x01fc32 (0x67e0 bytes) 000041 1787785525.180286 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fc9b): Mirror 3: 0xef9e8c25 (stale) Mirror 5: 0xcdfbb957 (stale) Mirror 6: 0xcdfbb957 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fc9b): Mirror 3: 0xef9e8c25 (stale) Mirror 5: 0xcdfbb957 (stale) Mirror 6: 0xcdfbb957 (stale) CRC-32 checksum value for chunk [0, 0x2fc9b): Mirror 1: 0xcdfbb957 Mirror 7: 0xcdfbb957 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fc9b: skipped 000042 1787785525.903038 mapwrite 0x01755a thru 0x02613d (0xebe4 bytes) 000043 1787785526.160004 MIRROR_RESYNC 5 mirrors 000044 1787785527.906373 read 0x02196d thru 0x0262a6 (0x493a bytes) 000045 1787785527.917771 mapread 0x00bbb4 thru 0x00edff (0x324c bytes) 000046 1787785527.938486 write 0x009796 thru 0x00c820 (0x308b bytes) 000047 1787785528.024159 mapread 0x00de18 thru 0x01dcf9 (0xfee2 bytes) 000048 1787785528.049210 read 0x012d12 thru 0x012ee0 (0x1cf bytes) 000049 1787785528.050054 mapread 0x01418e thru 0x01948b (0x52fe bytes) 000050 1787785528.051327 mapwrite 0x00fd03 thru 0x0165e1 (0x68df bytes) 000051 1787785528.122516 mapwrite 0x0341ba thru 0x03ffff (0xbe46 bytes) 000052 1787785528.236548 mapread 0x03ab11 thru 0x03bb15 (0x1005 bytes) 000053 1787785528.238589 write 0x025337 thru 0x02a613 (0x52dd bytes) 000055 1787785528.244043 write 0x011adc thru 0x0210d0 (0xf5f5 bytes) 000056 1787785528.251924 read 0x0052f9 thru 0x010ddb (0xbae3 bytes) 000057 1787785528.253280 write 0x00a6c4 thru 0x00c697 (0x1fd4 bytes) 000059 1787785528.256320 mapread 0x024eb2 thru 0x0256d8 (0x827 bytes) 000060 1787785528.264773 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x71b09362 (stale) Mirror 6: 0x71b09362 (stale) Mirror 7: 0x71b09362 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x71b09362 (stale) Mirror 6: 0x71b09362 (stale) Mirror 7: 0x71b09362 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000062 1787785528.723846 mapwrite 0x0081ad thru 0x00c17c (0x3fd0 bytes) 000065 1787785528.808900 write 0x01f0c5 thru 0x026161 (0x709d bytes) 000066 1787785528.820859 trunc from 0x040000 to 0x027a69 (0x18596 bytes) 000071 1787785528.849061 mapwrite 0x01b849 thru 0x01b96c (0x124 bytes) 000072 1787785529.008867 write 0x01c4fb thru 0x0280d9 (0xbbdf bytes) 000075 1787785529.015255 mapwrite 0x01b609 thru 0x021140 (0x5b38 bytes) 000076 1787785529.160493 mapread 0x00f7f5 thru 0x01de21 (0xe62d bytes) 000077 1787785529.184360 trunc from 0x0280da to 0x02363e (0x4a9b bytes) 000078 1787785529.217890 mapread 0x020b17 thru 0x02363e (0x2b28 bytes) 000080 1787785529.219278 write 0x00d176 thru 0x01114e (0x3fd9 bytes) 000081 1787785529.225715 mapwrite 0x03517b thru 0x03e460 (0x92e6 bytes) 000084 1787785529.394753 trunc from 0x03e461 to 0x01d073 (0x213ed bytes) 000085 1787785529.440905 write 0x02ce6b thru 0x02e1ab (0x1341 bytes) truncating to largest ever: 0x3c243 000088 1787785529.448561 trunc from 0x02e1ac to 0x03c242 (0xe097 bytes) 000091 1787785529.500770 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c243): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c243): Mirror 5: 0xeed9e37b (stale) Mirror 6: 0xeed9e37b (stale) Mirror 7: 0xeed9e37b (stale) CRC-32 checksum value for chunk [0, 0x3c243): Mirror 1: 0xb2731587 Mirror 8: 0xb2731587 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c243: skipped truncating to largest ever: 0x3caee 000092 1787785531.345946 trunc from 0x03c243 to 0x03caed (0x8ab bytes) 000093 1787785531.468828 write 0x023ff0 thru 0x0323b7 (0xe3c8 bytes) 000094 1787785531.482025 read 0x0114b9 thru 0x01a134 (0x8c7c bytes) 000095 1787785531.493194 mapread 0x002b3e thru 0x00bb4c (0x900f bytes) 000096 1787785531.515757 read 0x02b292 thru 0x030d4a (0x5ab9 bytes) 000098 1787785531.517795 trunc from 0x03caee to 0x029322 (0x137cb bytes) 000100 1787785531.549033 read 0x00d72e thru 0x018be6 (0xb4b9 bytes) 000102 1787785531.556477 MIRROR_RESYNC 5 mirrors 000103 1787785532.964237 MIRROR_EXTEND to 6 mirrors 000104 1787785534.207379 read 0x01b376 thru 0x028c96 (0xd921 bytes) 000105 1787785534.230131 trunc from 0x029323 to 0x019a9c (0xf886 bytes) 000106 1787785534.416888 mapwrite 0x008544 thru 0x012821 (0xa2de bytes) 000107 1787785534.622549 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x19a9d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a9d): Mirror 5: 0x2848a5b (stale) Mirror 6: 0x2848a5b (stale) Mirror 7: 0x2848a5b (stale) Mirror 8: 0x2848a5b (stale) Mirror 9: 0x2848a5b (stale) CRC-32 checksum value for chunk [0, 0x19a9d): Mirror 1: 0xb595eb88 Mirror 10: 0xb595eb88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a9d: skipped 000108 1787785536.319024 mapwrite 0x00e11d thru 0x015b53 (0x7a37 bytes) 000110 1787785536.576765 mapread 0x011d21 thru 0x019a9c (0x7d7c bytes) 000111 1787785536.590914 trunc from 0x019a9d to 0x023ffe (0xa562 bytes) 000112 1787785536.635234 mapread 0x01be38 thru 0x023ffe (0x81c7 bytes) 000113 1787785536.650243 trunc from 0x023fff to 0x02f5bd (0xb5bf bytes) 000116 1787785536.679830 trunc from 0x02f5be to 0x022bab (0xca12 bytes) 000117 1787785536.703458 read 0x01b733 thru 0x01b879 (0x147 bytes) 000120 1787785536.705163 read 0x00fd3f thru 0x016c23 (0x6ee5 bytes) 000123 1787785536.710904 mapread 0x022233 thru 0x022bab (0x979 bytes) 000124 1787785536.712831 read 0x00cf28 thru 0x014ab0 (0x7b89 bytes) 000125 1787785536.714743 trunc from 0x022bac to 0x02a18f (0x75e4 bytes) 000127 1787785536.737676 trunc from 0x02a190 to 0x018d7c (0x11413 bytes) 000130 1787785536.759692 read 0x008c1f thru 0x012c98 (0xa07a bytes) 000131 1787785536.760525 mapread 0x00beee thru 0x015388 (0x949b bytes) 000132 1787785536.762801 mapread 0x00fa62 thru 0x018d7c (0x931b bytes) 000134 1787785536.764969 mapwrite 0x03b8bf thru 0x03dd26 (0x2468 bytes) 000136 1787785536.858733 mapread 0x0185c7 thru 0x02394c (0xb386 bytes) 000140 1787785536.870945 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd27): Mirror 5: 0x9aa347c7 (stale) Mirror 6: 0x9aa347c7 (stale) Mirror 7: 0x9aa347c7 (stale) Mirror 9: 0x9aa347c7 (stale) Mirror 10: 0x54c02465 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd27): Mirror 5: 0x9aa347c7 (stale) Mirror 6: 0x9aa347c7 (stale) Mirror 7: 0x9aa347c7 (stale) Mirror 9: 0x9aa347c7 (stale) Mirror 10: 0x54c02465 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd27: skipped 000141 1787785537.497832 mapread 0x0032b6 thru 0x00d360 (0xa0ab bytes) 000142 1787785537.531823 mapwrite 0x01da34 thru 0x024a36 (0x7003 bytes) 000145 1787785537.663560 trunc from 0x03dd27 to 0x014d91 (0x28f95 bytes) 000146 1787785537.694734 read 0x00d75e thru 0x014d91 (0x7634 bytes) 000147 1787785537.705815 read 0x002a98 thru 0x007344 (0x48ad bytes) 000149 1787785537.707615 write 0x00dfe7 thru 0x019c9e (0xbcb8 bytes) 000151 1787785537.712557 mapread 0x012ed7 thru 0x017a56 (0x4b80 bytes) 000152 1787785537.714373 read 0x006799 thru 0x01089f (0xa107 bytes) 000154 1787785537.717013 trunc from 0x019c9f to 0x02035e (0x66c0 bytes) 000155 1787785537.750393 write 0x00252b thru 0x011e2e (0xf904 bytes) 000156 1787785537.754375 read 0x01cbcb thru 0x02035e (0x3794 bytes) 000158 1787785537.771889 mapread 0x00afb2 thru 0x01a8b8 (0xf907 bytes) 000160 1787785537.773438 mapwrite 0x008d12 thru 0x00a947 (0x1c36 bytes) 000161 1787785537.908600 mapwrite 0x03eeef thru 0x03ffff (0x1111 bytes) 000163 1787785538.085154 mapwrite 0x003345 thru 0x009b29 (0x67e5 bytes) 000165 1787785538.207036 trunc from 0x040000 to 0x022cf5 (0x1d30a bytes) 000167 1787785538.253833 mapread 0x0133bd thru 0x019df9 (0x6a3d bytes) 000168 1787785538.254947 write 0x02d0ae thru 0x03263d (0x5590 bytes) 000172 1787785538.258003 mapread 0x029f18 thru 0x02b9ff (0x1ae8 bytes) 000173 1787785538.272751 trunc from 0x03263e to 0x004fb2 (0x2d68b bytes) 000175 1787785538.307432 mapwrite 0x008b2e thru 0x0121fa (0x96cd bytes) 000180 1787785538.492566 write 0x01cd86 thru 0x022510 (0x578b bytes) 000183 1787785538.500587 trunc from 0x022511 to 0x013a09 (0xeb07 bytes) 000184 1787785538.526864 mapread 0x007d46 thru 0x009082 (0x133d bytes) 000186 1787785538.539042 mapread 0x006e30 thru 0x00a664 (0x3835 bytes) 000187 1787785538.545001 write 0x01c90d thru 0x01f30c (0x2a00 bytes) 000189 1787785538.556711 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f30d): Mirror 5: 0x732726a3 (stale) Mirror 6: 0x732726a3 (stale) Mirror 7: 0x732726a3 (stale) Mirror 10: 0x292c85a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f30d): Mirror 5: 0x732726a3 (stale) Mirror 6: 0x732726a3 (stale) Mirror 7: 0x732726a3 (stale) Mirror 10: 0x292c85a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f30d: skipped 000192 1787785539.201463 mapread 0x00ca6b thru 0x0185cb (0xbb61 bytes) 000194 1787785539.271250 trunc from 0x01f30d to 0x0256f3 (0x63e7 bytes) 000197 1787785539.335596 MIRROR_RESYNC 5 mirrors 000198 1787785541.508495 mapread 0x00afff thru 0x014cdf (0x9ce1 bytes) 000200 1787785541.543283 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) 000203 1787785542.001210 trunc from 0x0256f4 to 0x0019f2 (0x23d01 bytes) 000204 1787785542.155829 trunc from 0x0019f3 to 0x0168bf (0x14ecd bytes) 000205 1787785542.191783 MIRROR_RESYNC 4 mirrors 000206 1787785543.749161 mapread 0x0016f5 thru 0x008180 (0x6a8c bytes) 000209 1787785543.786554 write 0x006163 thru 0x00ea35 (0x88d3 bytes) 000210 1787785543.943988 mapwrite 0x0068bb thru 0x00c170 (0x58b6 bytes) 000211 1787785544.064671 write 0x00e971 thru 0x011a7c (0x310c bytes) 000214 1787785544.077269 mapread 0x001b60 thru 0x0083a0 (0x6841 bytes) 000217 1787785544.112708 mapwrite 0x03d21d thru 0x03ffff (0x2de3 bytes) truncating to largest ever: 0x3f5f6 000218 1787785544.241768 trunc from 0x040000 to 0x03f5f5 (0xa0a bytes) 000221 1787785544.269621 write 0x01c976 thru 0x029abf (0xd14a bytes) 000223 1787785544.294525 trunc from 0x03f5f6 to 0x026dfa (0x187fb bytes) 000224 1787785544.343391 mapread 0x01f304 thru 0x026dfa (0x7af7 bytes) 000225 1787785544.346790 mapread 0x003679 thru 0x00ab13 (0x749b bytes) 000227 1787785544.349946 mapread 0x000aaf thru 0x00986c (0x8dbe bytes) 000228 1787785544.361267 mapwrite 0x036ce7 thru 0x03ffff (0x9319 bytes) 000229 1787785544.484738 read 0x02a4a3 thru 0x03904e (0xebac bytes) 000230 1787785544.501476 mapwrite 0x0032e6 thru 0x007792 (0x44ad bytes) 000236 1787785544.562693 read 0x005ac2 thru 0x0095c7 (0x3b06 bytes) 000240 1787785544.570841 MIRROR_RESYNC 4 mirrors 000242 1787785546.452608 MIRROR_RESYNC 4 mirrors 000243 1787785547.123865 MIRROR_EXTEND to 5 mirrors 000248 1787785548.400852 mapread 0x01f133 thru 0x02c516 (0xd3e4 bytes) 000250 1787785548.453970 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) 000252 1787785548.856955 mapwrite 0x0264df thru 0x02e6de (0x8200 bytes) 000253 1787785549.013856 trunc from 0x040000 to 0x0151b7 (0x2ae48 bytes) 000254 1787785549.040663 read 0x011944 thru 0x014dbc (0x3479 bytes) 000255 1787785549.049554 mapwrite 0x0278df thru 0x030b9a (0x92bc bytes) 000257 1787785549.146966 mapread 0x006142 thru 0x00b43d (0x52fc bytes) 000262 1787785549.159501 mapwrite 0x020cbb thru 0x02e117 (0xd45d bytes) 000263 1787785549.217416 MIRROR_RESYNC 4 mirrors 000265 1787785550.258606 write 0x01c4f3 thru 0x028e98 (0xc9a6 bytes) 000267 1787785550.347804 trunc from 0x030b9b to 0x035630 (0x4a96 bytes) 000268 1787785550.409750 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35631): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35631): Mirror 7: 0x4935de3 (stale) Mirror 10: 0x4935de3 (stale) Mirror 11: 0x4935de3 (stale) CRC-32 checksum value for chunk [0, 0x35631): Mirror 1: 0xac2b214a Mirror 12: 0xac2b214a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35631: skipped 000270 1787785551.615852 write 0x005c50 thru 0x013430 (0xd7e1 bytes) 000272 1787785551.779812 trunc from 0x035631 to 0x00afb0 (0x2a680 bytes) 000273 1787785551.814117 mapwrite 0x035277 thru 0x03ffff (0xad89 bytes) 000276 1787785552.005276 mapread 0x00b14c thru 0x0134b5 (0x836a bytes) 000279 1787785552.040025 read 0x01191f thru 0x01fa4c (0xe12e bytes) 000280 1787785552.052517 trunc from 0x040000 to 0x00fdad (0x30252 bytes) 000282 1787785552.085672 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xfdae): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfdae): Mirror 7: 0x4ddc9b3 (stale) Mirror 10: 0x4ddc9b3 (stale) Mirror 11: 0x4ddc9b3 (stale) Mirror 12: 0x4ddc9b3 (stale) CRC-32 checksum value for chunk [0, 0xfdae): Mirror 1: 0xf8bbc671 Mirror 13: 0xf8bbc671 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfdae: skipped 000284 1787785553.402858 read 0x00d287 thru 0x00fdad (0x2b27 bytes) 000285 1787785553.432718 read 0x001aca thru 0x0071ee (0x5725 bytes) 000286 1787785553.461463 read 0x00b62f thru 0x00c94e (0x1320 bytes) 000288 1787785553.462272 mapwrite 0x00270d thru 0x00a8c2 (0x81b6 bytes) 000289 1787785553.761739 trunc from 0x00fdae to 0x00d10e (0x2c9f bytes) 000291 1787785553.849862 MIRROR_RESYNC 6 mirrors 000293 1787785555.627363 mapread 0x0057e7 thru 0x006a35 (0x124f bytes) 000297 1787785555.641436 read 0x005d79 thru 0x00b712 (0x599a bytes) 000298 1787785555.664887 MIRROR_EXTEND to 7 mirrors 000299 1787785556.665402 read 0x000d73 thru 0x00d10e (0xc39c bytes) 000300 1787785556.680459 read 0x002243 thru 0x002dd7 (0xb95 bytes) 000301 1787785556.681264 write 0x013d58 thru 0x014f32 (0x11db bytes) 000302 1787785556.798283 mapwrite 0x012e5b thru 0x020180 (0xd326 bytes) 000304 1787785556.889580 mapwrite 0x02e6ab thru 0x0399a3 (0xb2f9 bytes) 000305 1787785556.983167 read 0x0132aa thru 0x02029a (0xcff1 bytes) 000307 1787785556.984391 trunc from 0x0399a4 to 0x002a72 (0x36f31 bytes) 000309 1787785557.008899 mapwrite 0x004f8c thru 0x01320e (0xe283 bytes) 000310 1787785557.119943 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1320f): Mirror 7: 0xb818fd7c (stale) Mirror 10: 0xb818fd7c (stale) Mirror 11: 0xb818fd7c (stale) Mirror 13: 0xb818fd7c (stale) Mirror 14: 0xb818fd7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1320f): Mirror 7: 0xb818fd7c (stale) Mirror 10: 0xb818fd7c (stale) Mirror 11: 0xb818fd7c (stale) Mirror 13: 0xb818fd7c (stale) Mirror 14: 0xb818fd7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1320f: skipped 000312 1787785557.641442 mapread 0x012ed3 thru 0x01320e (0x33c bytes) 000313 1787785557.656681 mapread 0x0117bd thru 0x01320e (0x1a52 bytes) 000314 1787785557.682018 mapwrite 0x03d6e6 thru 0x03ffff (0x291a bytes) 000316 1787785557.822214 write 0x03ba5a thru 0x03ffff (0x45a6 bytes) 000317 1787785557.824038 mapwrite 0x0110d0 thru 0x01e3b6 (0xd2e7 bytes) 000318 1787785558.041477 write 0x00c871 thru 0x016934 (0xa0c4 bytes) 000319 1787785558.050905 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xc8c79d15 (stale) Mirror 10: 0xc8c79d15 (stale) Mirror 11: 0xc8c79d15 (stale) Mirror 13: 0xc8c79d15 (stale) Mirror 14: 0xc8c79d15 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8e134558 Mirror 15: 0x8e134558 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000321 1787785559.739718 write 0x02c836 thru 0x03836a (0xbb35 bytes) 000323 1787785559.915312 mapwrite 0x00c35e thru 0x00d351 (0xff4 bytes) 000324 1787785560.007218 write 0x033c53 thru 0x03aed4 (0x7282 bytes) 000325 1787785560.038191 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xc8c79d15 (stale) Mirror 10: 0xc8c79d15 (stale) Mirror 11: 0xc8c79d15 (stale) Mirror 13: 0xc8c79d15 (stale) Mirror 14: 0xc8c79d15 (stale) Mirror 15: 0x8e134558 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x35ea9ef7 Mirror 16: 0x35ea9ef7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000326 1787785561.822748 write 0x006d27 thru 0x016047 (0xf321 bytes) 000328 1787785561.952352 mapwrite 0x01f166 thru 0x02dbbc (0xea57 bytes) 000329 1787785562.101381 MIRROR_RESYNC 8 mirrors 000330 1787785563.543012 write 0x012b95 thru 0x01465d (0x1ac9 bytes) 000331 1787785563.623557 mapwrite 0x00666a thru 0x010d6c (0xa703 bytes) 000332 1787785563.706390 write 0x0102d7 thru 0x012fcb (0x2cf5 bytes) 000334 1787785563.710136 mapwrite 0x01e38f thru 0x02aefa (0xcb6c bytes) 000336 1787785563.775214 write 0x0384f9 thru 0x039de4 (0x18ec bytes) 000339 1787785563.786021 mapread 0x03db84 thru 0x03ffff (0x247c bytes) 000341 1787785563.797811 mapread 0x007a62 thru 0x0159e3 (0xdf82 bytes) 000342 1787785563.807237 read 0x02be7f thru 0x03449f (0x8621 bytes) 000345 1787785563.818636 read 0x03c9ed thru 0x03ffff (0x3613 bytes) 000350 1787785563.822563 mapwrite 0x014f7b thru 0x01cd4d (0x7dd3 bytes) 000352 1787785563.874902 mapread 0x0388a4 thru 0x03ffff (0x775c bytes) 000353 1787785563.876151 mapread 0x0238e9 thru 0x0269c0 (0x30d8 bytes) 000354 1787785563.884309 mapread 0x022158 thru 0x0297d8 (0x7681 bytes) 000355 1787785563.887333 write 0x01a5ae thru 0x021729 (0x717c bytes) 000357 1787785563.890428 mapread 0x022fd9 thru 0x0280b3 (0x50db bytes) 000359 1787785563.892638 mapwrite 0x02a4e6 thru 0x033d97 (0x98b2 bytes) 000360 1787785563.969526 write 0x00915b thru 0x0129a5 (0x984b bytes) 000364 1787785563.974587 mapread 0x026de5 thru 0x03071d (0x9939 bytes) 000365 1787785563.976396 mapread 0x03c8fd thru 0x03cfdd (0x6e1 bytes) 000366 1787785563.982066 MIRROR_RESYNC 8 mirrors 000367 1787785565.615054 mapread 0x00af58 thru 0x01219c (0x7245 bytes) 000368 1787785565.656142 MIRROR_EXTEND to 9 mirrors 000369 1787785566.434059 read 0x003716 thru 0x00ed3e (0xb629 bytes) 000370 1787785566.445760 mapwrite 0x032a65 thru 0x032ac8 (0x64 bytes) 000373 1787785566.595552 mapread 0x008384 thru 0x01057e (0x81fb bytes) 000377 1787785566.610421 mapread 0x0213da thru 0x021c98 (0x8bf bytes) 000380 1787785566.632248 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x899e21dd (stale) Mirror 10: 0x899e21dd (stale) Mirror 11: 0x899e21dd (stale) Mirror 13: 0x899e21dd (stale) Mirror 15: 0x899e21dd (stale) Mirror 16: 0x899e21dd (stale) Mirror 17: 0x899e21dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x899e21dd (stale) Mirror 10: 0x899e21dd (stale) Mirror 11: 0x899e21dd (stale) Mirror 13: 0x899e21dd (stale) Mirror 15: 0x899e21dd (stale) Mirror 16: 0x899e21dd (stale) Mirror 17: 0x899e21dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000386 1787785567.211138 read 0x01b573 thru 0x01dbd3 (0x2661 bytes) 000389 1787785567.219039 mapwrite 0x011e25 thru 0x02182b (0xfa07 bytes) 000390 1787785567.305249 trunc from 0x040000 to 0x006afc (0x39503 bytes) 000392 1787785567.322671 trunc from 0x006afd to 0x023fb1 (0x1d4b5 bytes) 000393 1787785567.344733 read 0x01f983 thru 0x023fb1 (0x462f bytes) 000395 1787785567.356390 mapwrite 0x02054c thru 0x024816 (0x42cb bytes) 000398 1787785567.423628 trunc from 0x024817 to 0x03ccea (0x184d4 bytes) 000399 1787785567.452688 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cceb): Mirror 7: 0x7e54fb1c (stale) Mirror 10: 0x7e54fb1c (stale) Mirror 11: 0x7e54fb1c (stale) Mirror 13: 0x7e54fb1c (stale) Mirror 15: 0x7e54fb1c (stale) Mirror 17: 0x7e54fb1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cceb): Mirror 7: 0x7e54fb1c (stale) Mirror 10: 0x7e54fb1c (stale) Mirror 11: 0x7e54fb1c (stale) Mirror 13: 0x7e54fb1c (stale) Mirror 15: 0x7e54fb1c (stale) Mirror 17: 0x7e54fb1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cceb: skipped 000400 1787785568.040495 mapread 0x016b6c thru 0x023bc1 (0xd056 bytes) 000401 1787785568.070395 write 0x035545 thru 0x03ffff (0xaabb bytes) 000402 1787785568.077039 write 0x026efe thru 0x02fe3c (0x8f3f bytes) 000403 1787785568.084382 read 0x02584b thru 0x028505 (0x2cbb bytes) 000404 1787785568.085263 write 0x00f881 thru 0x01bb01 (0xc281 bytes) 000405 1787785568.097711 MIRROR_RESYNC 7 mirrors 000406 1787785569.767450 read 0x03029e thru 0x038cfa (0x8a5d bytes) 000407 1787785569.782033 read 0x00a437 thru 0x015826 (0xb3f0 bytes) 000408 1787785569.805187 trunc from 0x040000 to 0x0096fe (0x36901 bytes) 000409 1787785569.916719 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x96ff): Mirror 7: 0x34de1392 (stale) Mirror 11: 0x34de1392 (stale) Mirror 13: 0x34de1392 (stale) Mirror 15: 0x34de1392 (stale) Mirror 17: 0x34de1392 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x96ff): Mirror 7: 0x34de1392 (stale) Mirror 11: 0x34de1392 (stale) Mirror 13: 0x34de1392 (stale) Mirror 15: 0x34de1392 (stale) Mirror 17: 0x34de1392 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x96ff: skipped 000410 1787785570.431890 write 0x00898c thru 0x00a335 (0x19aa bytes) 000412 1787785570.440415 trunc from 0x00a336 to 0x03ea3f (0x3470a bytes) 000414 1787785570.462612 write 0x01365c thru 0x02328b (0xfc30 bytes) 000415 1787785570.476044 write 0x017c3b thru 0x022333 (0xa6f9 bytes) 000416 1787785570.481709 read 0x030f4b thru 0x039469 (0x851f bytes) 000417 1787785570.491570 mapwrite 0x021c74 thru 0x030fcb (0xf358 bytes) 000418 1787785570.577944 trunc from 0x03ea40 to 0x03da78 (0xfc7 bytes) 000419 1787785570.598386 mapwrite 0x009d6b thru 0x012282 (0x8518 bytes) 000420 1787785570.667699 mapwrite 0x0304dd thru 0x031098 (0xbbc bytes) 000422 1787785570.721368 trunc from 0x03da79 to 0x02ed14 (0xed64 bytes) 000423 1787785570.742520 read 0x008d0d thru 0x012fef (0xa2e3 bytes) 000424 1787785570.749462 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ed15): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed15): Mirror 7: 0x449e6a03 (stale) Mirror 11: 0x449e6a03 (stale) Mirror 13: 0x449e6a03 (stale) Mirror 15: 0x449e6a03 (stale) Mirror 17: 0x449e6a03 (stale) CRC-32 checksum value for chunk [0, 0x2ed15): Mirror 1: 0x4ecbb699 Mirror 18: 0x4ecbb699 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed15: skipped 000425 1787785571.905474 write 0x026d0f thru 0x028a08 (0x1cfa bytes) 000427 1787785572.052364 trunc from 0x02ed15 to 0x018d81 (0x15f93 bytes) 000428 1787785572.093571 MIRROR_RESYNC 7 mirrors 000429 1787785573.915000 write 0x01cbc3 thru 0x02c14e (0xf58c bytes) 000430 1787785574.007945 mapwrite 0x024b3e thru 0x02e25b (0x971e bytes) 000435 1787785574.188692 trunc from 0x02e25c to 0x03cd57 (0xeafc bytes) 000436 1787785574.215887 mapwrite 0x003955 thru 0x010e51 (0xd4fd bytes) 000438 1787785574.311134 write 0x038b6e thru 0x03ffff (0x7492 bytes) 000439 1787785574.322981 mapwrite 0x02ba76 thru 0x034f9b (0x9526 bytes) 000442 1787785574.436803 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x2adc2d4 (stale) Mirror 11: 0x2adc2d4 (stale) Mirror 13: 0x2adc2d4 (stale) Mirror 15: 0x2adc2d4 (stale) Mirror 17: 0x2adc2d4 (stale) Mirror 18: 0x2adc2d4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcc899110 Mirror 19: 0xcc899110 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000443 1787785575.972992 read 0x001298 thru 0x00e315 (0xd07e bytes) 000444 1787785576.009929 mapwrite 0x013bf6 thru 0x018c6b (0x5076 bytes) 000449 1787785576.251191 read 0x0105c9 thru 0x01ac17 (0xa64f bytes) 000450 1787785576.252237 write 0x01b69d thru 0x0268f0 (0xb254 bytes) 000452 1787785576.277678 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x2adc2d4 (stale) Mirror 11: 0x2adc2d4 (stale) Mirror 13: 0x2adc2d4 (stale) Mirror 15: 0x2adc2d4 (stale) Mirror 17: 0x2adc2d4 (stale) Mirror 18: 0x2adc2d4 (stale) Mirror 19: 0xcc899110 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1b70cec7 Mirror 20: 0x1b70cec7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000453 1787785577.808604 write 0x025794 thru 0x02c965 (0x71d2 bytes) 000454 1787785577.903787 trunc from 0x040000 to 0x0262ae (0x19d51 bytes) 000457 1787785577.963936 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x262af): Mirror 7: 0x4ebf7211 (stale) Mirror 11: 0x4ebf7211 (stale) Mirror 13: 0x4ebf7211 (stale) Mirror 15: 0x4ebf7211 (stale) Mirror 17: 0x4ebf7211 (stale) Mirror 19: 0x2dedf5e0 (stale) Mirror 20: 0x8a65ec0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x262af): Mirror 7: 0x4ebf7211 (stale) Mirror 11: 0x4ebf7211 (stale) Mirror 13: 0x4ebf7211 (stale) Mirror 15: 0x4ebf7211 (stale) Mirror 17: 0x4ebf7211 (stale) Mirror 19: 0x2dedf5e0 (stale) Mirror 20: 0x8a65ec0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x262af: skipped 000459 1787785578.659590 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x262af): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x262af): Mirror 7: 0x4ebf7211 (stale) Mirror 11: 0x4ebf7211 (stale) Mirror 13: 0x4ebf7211 (stale) Mirror 15: 0x4ebf7211 (stale) Mirror 17: 0x4ebf7211 (stale) Mirror 19: 0x2dedf5e0 (stale) Mirror 20: 0x8a65ec0b (stale) CRC-32 checksum value for chunk [0, 0x262af): Mirror 1: 0x4556cece Mirror 21: 0x4556cece lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x262af: skipped 000461 1787785579.972029 mapwrite 0x01c50f thru 0x02b302 (0xedf4 bytes) 000463 1787785580.256333 trunc from 0x02b303 to 0x03e4c4 (0x131c2 bytes) 000464 1787785580.282565 trunc from 0x03e4c5 to 0x018794 (0x25d30 bytes) 000465 1787785580.346438 write 0x01fa60 thru 0x023840 (0x3de1 bytes) 000466 1787785580.354745 mapwrite 0x020106 thru 0x02def5 (0xddf0 bytes) 000468 1787785580.522356 trunc from 0x02def6 to 0x0288d8 (0x561d bytes) 000469 1787785580.567854 MIRROR_RESYNC 9 mirrors 000470 1787785582.428256 read 0x00c5ef thru 0x016641 (0xa053 bytes) 000471 1787785582.440919 read 0x01b89c thru 0x0228b6 (0x701b bytes) 000472 1787785582.451298 write 0x014d81 thru 0x01d65e (0x88de bytes) 000474 1787785582.579589 write 0x001c12 thru 0x007f63 (0x6352 bytes) 000475 1787785582.590895 trunc from 0x0288d9 to 0x014c68 (0x13c70 bytes) 000476 1787785582.628860 write 0x00818c thru 0x013200 (0xb075 bytes) 000482 1787785582.668204 read 0x000eaf thru 0x00e873 (0xd9c5 bytes) 000483 1787785582.678894 mapread 0x00ed18 thru 0x01318c (0x4475 bytes) 000487 1787785582.701848 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x14c69): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14c69): Mirror 7: 0x64094a1a (stale) Mirror 11: 0x64094a1a (stale) Mirror 13: 0x64094a1a (stale) Mirror 15: 0x64094a1a (stale) Mirror 17: 0x64094a1a (stale) Mirror 19: 0x64094a1a (stale) Mirror 20: 0x64094a1a (stale) Mirror 21: 0x64094a1a (stale) CRC-32 checksum value for chunk [0, 0x14c69): Mirror 1: 0xdd935d2d Mirror 22: 0xdd935d2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14c69: skipped 000489 1787785584.142638 trunc from 0x014c69 to 0x0317d7 (0x1cb6f bytes) 000491 1787785584.279337 MIRROR_RESYNC 10 mirrors 000494 1787785586.946402 read 0x018bd9 thru 0x025e93 (0xd2bb bytes) 000496 1787785586.959388 read 0x02ab79 thru 0x0317d7 (0x6c5f bytes) 000499 1787785586.991140 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) 000501 1787785587.502666 mapwrite 0x035307 thru 0x03f123 (0x9e1d bytes) 000503 1787785587.726384 write 0x012c94 thru 0x015540 (0x28ad bytes) 000508 1787785587.743734 mapwrite 0x0005b2 thru 0x00d88c (0xd2db bytes) 000510 1787785587.826492 mapread 0x03b006 thru 0x03f123 (0x411e bytes) 000512 1787785587.829560 write 0x014808 thru 0x021df9 (0xd5f2 bytes) 000513 1787785587.836546 write 0x006592 thru 0x012f73 (0xc9e2 bytes) 000517 1787785587.841364 write 0x03f9eb thru 0x03ffff (0x615 bytes) 000518 1787785587.842796 read 0x033fda thru 0x03b22e (0x7255 bytes) 000519 1787785587.843673 mapread 0x00b9a9 thru 0x01856c (0xcbc4 bytes) 000521 1787785587.848000 mapread 0x03ea85 thru 0x03ffff (0x157b bytes) 000523 1787785587.852889 trunc from 0x040000 to 0x002211 (0x3ddee bytes) 000524 1787785587.890832 mapread 0x000bd4 thru 0x002211 (0x163e bytes) 000526 1787785587.893547 write 0x038159 thru 0x03deec (0x5d94 bytes) 000527 1787785587.895717 mapwrite 0x016ca7 thru 0x017311 (0x66b bytes) 000528 1787785587.969333 mapread 0x03ac3e thru 0x03b0c7 (0x48a bytes) 000532 1787785587.976807 trunc from 0x03deed to 0x01ab85 (0x23367 bytes) 000533 1787785588.015383 trunc from 0x01ab86 to 0x0121fa (0x898b bytes) 000534 1787785588.041799 read 0x009ffb thru 0x0121fa (0x8200 bytes) 000535 1787785588.070369 MIRROR_RESYNC 9 mirrors 000536 1787785590.639986 read 0x0065d7 thru 0x008199 (0x1bc3 bytes) 000540 1787785590.648345 mapwrite 0x01fdd8 thru 0x02c009 (0xc232 bytes) 000543 1787785590.926931 mapwrite 0x038d09 thru 0x03ffff (0x72f7 bytes) 000545 1787785591.060902 mapwrite 0x0033d4 thru 0x01242a (0xf057 bytes) 000546 1787785591.152881 write 0x01213a thru 0x0193a7 (0x726e bytes) 000548 1787785591.154747 mapread 0x02aacb thru 0x0317d1 (0x6d07 bytes) 000549 1787785591.161149 mapwrite 0x007895 thru 0x009e64 (0x25d0 bytes) 000550 1787785591.227753 write 0x0286e3 thru 0x02a92e (0x224c bytes) 000552 1787785591.233982 mapwrite 0x00bf47 thru 0x00df0e (0x1fc8 bytes) 000553 1787785591.283458 read 0x019554 thru 0x019eb1 (0x95e bytes) 000554 1787785591.284188 mapwrite 0x00b3de thru 0x015b60 (0xa783 bytes) 000557 1787785591.378998 trunc from 0x040000 to 0x009667 (0x36998 bytes) 000558 1787785591.405209 trunc from 0x009668 to 0x019ad6 (0x1046f bytes) 000559 1787785591.442869 mapwrite 0x02fb60 thru 0x036dc2 (0x7263 bytes) 000560 1787785591.577367 mapread 0x022599 thru 0x02cfd0 (0xaa38 bytes) 000561 1787785591.584211 read 0x02cf4e thru 0x0346d3 (0x7786 bytes) 000562 1787785591.586753 trunc from 0x036dc3 to 0x039972 (0x2bb0 bytes) 000563 1787785591.660207 mapread 0x033d00 thru 0x039972 (0x5c73 bytes) 000565 1787785591.675581 MIRROR_RESYNC 9 mirrors 000567 1787785594.828225 trunc from 0x039973 to 0x003e50 (0x35b22 bytes) 000568 1787785594.932562 mapread 0x003134 thru 0x003e50 (0xd1d bytes) 000569 1787785594.942066 write 0x01b334 thru 0x029170 (0xde3d bytes) 000572 1787785594.947273 mapwrite 0x01eedf thru 0x02568b (0x67ad bytes) 000576 1787785595.017006 mapwrite 0x030202 thru 0x0344d1 (0x42d0 bytes) 000578 1787785595.124891 mapread 0x010a9f thru 0x0209e4 (0xff46 bytes) 000579 1787785595.151423 MIRROR_RESYNC 9 mirrors 000582 1787785597.727564 trunc from 0x0344d2 to 0x027ac9 (0xca08 bytes) 000584 1787785597.828311 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x27aca): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27aca): Mirror 7: 0x7c2c23b3 (stale) Mirror 11: 0x7c2c23b3 (stale) Mirror 13: 0x7c2c23b3 (stale) Mirror 15: 0x7c2c23b3 (stale) Mirror 17: 0x7c2c23b3 (stale) Mirror 19: 0x7c2c23b3 (stale) Mirror 20: 0x7c2c23b3 (stale) Mirror 21: 0x7c2c23b3 (stale) CRC-32 checksum value for chunk [0, 0x27aca): Mirror 1: 0x7c2c23b3 Mirror 22: 0x7c2c23b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27aca: skipped 000587 1787785599.317862 trunc from 0x027aca to 0x011106 (0x169c3 bytes) 000588 1787785599.432846 write 0x01fb54 thru 0x02d23c (0xd6e9 bytes) 000589 1787785599.440925 write 0x00f1df thru 0x015880 (0x66a2 bytes) 000592 1787785599.453543 mapread 0x022486 thru 0x02803d (0x5bb8 bytes) 000594 1787785599.455850 read 0x01e26c thru 0x01e5d2 (0x367 bytes) 000595 1787785599.461622 trunc from 0x02d23d to 0x028c73 (0x45c9 bytes) 000596 1787785599.488936 mapread 0x01b089 thru 0x0281ac (0xd124 bytes) 000597 1787785599.508915 trunc from 0x028c74 to 0x009188 (0x1faeb bytes) 000600 1787785599.532844 read 0x002f90 thru 0x009188 (0x61f9 bytes) 000602 1787785599.550229 mapread 0x008256 thru 0x009188 (0xf33 bytes) 000603 1787785599.562895 MIRROR_RESYNC 10 mirrors 000605 1787785601.968440 mapwrite 0x01812c thru 0x025158 (0xd02d bytes) 000606 1787785602.238104 mapread 0x000034 thru 0x00b342 (0xb30f bytes) 000607 1787785602.269894 trunc from 0x025159 to 0x031d2d (0xcbd5 bytes) 000609 1787785602.308901 read 0x00e1b2 thru 0x01b3bc (0xd20b bytes) 000610 1787785602.320620 read 0x01eeab thru 0x028335 (0x948b bytes) 000613 1787785602.336270 read 0x0004a5 thru 0x00c520 (0xc07c bytes) 000614 1787785602.337637 trunc from 0x031d2e to 0x03272e (0xa01 bytes) 000615 1787785602.385126 write 0x013e16 thru 0x01d3e9 (0x95d4 bytes) 000616 1787785602.400273 write 0x017bfa thru 0x021be7 (0x9fee bytes) 000617 1787785602.407951 write 0x010bbb thru 0x016c2d (0x6073 bytes) 000618 1787785602.411403 mapwrite 0x01499e thru 0x02109c (0xc6ff bytes) 000622 1787785602.525518 mapread 0x0278a9 thru 0x03272e (0xae86 bytes) 000629 1787785602.541956 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3272f): Mirror 7: 0x568b4097 (stale) Mirror 11: 0x568b4097 (stale) Mirror 15: 0x568b4097 (stale) Mirror 17: 0x568b4097 (stale) Mirror 19: 0x568b4097 (stale) Mirror 20: 0x568b4097 (stale) Mirror 21: 0x568b4097 (stale) Mirror 22: 0x568b4097 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3272f): Mirror 7: 0x568b4097 (stale) Mirror 11: 0x568b4097 (stale) Mirror 15: 0x568b4097 (stale) Mirror 17: 0x568b4097 (stale) Mirror 19: 0x568b4097 (stale) Mirror 20: 0x568b4097 (stale) Mirror 21: 0x568b4097 (stale) Mirror 22: 0x568b4097 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3272f: skipped 000630 1787785603.391021 mapwrite 0x0037d0 thru 0x00f54e (0xbd7f bytes) 000632 1787785603.517664 MIRROR_RESYNC 9 mirrors 000633 1787785605.543490 mapread 0x004fec thru 0x005503 (0x518 bytes) 000637 1787785605.558800 MIRROR_EXTEND to 10 mirrors 000638 1787785606.971845 read 0x03134c thru 0x03272e (0x13e3 bytes) 000641 1787785606.983366 MIRROR_EXTEND to 11 mirrors 000642 1787785608.259964 write 0x0231d8 thru 0x031e37 (0xec60 bytes) 000643 1787785608.356670 write 0x03c702 thru 0x03ffff (0x38fe bytes) 000644 1787785608.358036 mapread 0x003676 thru 0x00e2cf (0xac5a bytes) 000645 1787785608.382528 mapread 0x01210b thru 0x01bcc1 (0x9bb7 bytes) 000646 1787785608.404029 write 0x0078d0 thru 0x013c9a (0xc3cb bytes) 000647 1787785608.417908 MIRROR_RESYNC 11 mirrors 000648 1787785610.539426 mapwrite 0x028737 thru 0x02d931 (0x51fb bytes) 000649 1787785610.704707 trunc from 0x040000 to 0x023d79 (0x1c286 bytes) 000651 1787785610.731128 mapread 0x01f163 thru 0x0221fb (0x3099 bytes) 000652 1787785610.749309 mapread 0x00506e thru 0x012d47 (0xdcda bytes) 000653 1787785610.769350 mapread 0x005e53 thru 0x014ddc (0xef8a bytes) 000655 1787785610.776949 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23d7a): Mirror 7: 0x3e92e372 (stale) Mirror 11: 0x3e92e372 (stale) Mirror 15: 0x3e92e372 (stale) Mirror 19: 0x3e92e372 (stale) Mirror 20: 0x3e92e372 (stale) Mirror 21: 0x3e92e372 (stale) Mirror 22: 0x3e92e372 (stale) Mirror 23: 0x3e92e372 (stale) Mirror 24: 0x3e92e372 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d7a): Mirror 7: 0x3e92e372 (stale) Mirror 11: 0x3e92e372 (stale) Mirror 15: 0x3e92e372 (stale) Mirror 19: 0x3e92e372 (stale) Mirror 20: 0x3e92e372 (stale) Mirror 21: 0x3e92e372 (stale) Mirror 22: 0x3e92e372 (stale) Mirror 23: 0x3e92e372 (stale) Mirror 24: 0x3e92e372 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d7a: skipped 000657 1787785611.663651 write 0x02f5db thru 0x03db84 (0xe5aa bytes) 000658 1787785611.669870 write 0x00f5ff thru 0x01a9f4 (0xb3f6 bytes) 000660 1787785611.689475 MIRROR_RESYNC 10 mirrors 000661 1787785613.556056 read 0x03d530 thru 0x03db84 (0x655 bytes) 000673 1787785613.579362 mapread 0x02e965 thru 0x03c289 (0xd925 bytes) 000674 1787785613.639599 MIRROR_EXTEND to 11 mirrors 000678 1787785614.657484 mapread 0x00a84b thru 0x0191ab (0xe961 bytes) 000680 1787785614.700123 write 0x0150ec thru 0x022f6c (0xde81 bytes) 000681 1787785614.845851 mapwrite 0x017d3c thru 0x021871 (0x9b36 bytes) 000682 1787785614.914664 mapread 0x024268 thru 0x028b6f (0x4908 bytes) 000684 1787785614.944643 trunc from 0x03db85 to 0x03d1cb (0x9b9 bytes) 000685 1787785614.974798 mapwrite 0x03d339 thru 0x03ffff (0x2cc7 bytes) 000687 1787785615.074045 mapread 0x017065 thru 0x01a86a (0x3806 bytes) 000690 1787785615.077239 trunc from 0x040000 to 0x001eed (0x3e112 bytes) 000692 1787785615.122361 MIRROR_RESYNC 11 mirrors 000696 1787785616.824767 mapread 0x000ec9 thru 0x001eed (0x1025 bytes) 000698 1787785616.839943 trunc from 0x001eee to 0x0381c7 (0x362da bytes) 000701 1787785616.953421 mapwrite 0x02bd79 thru 0x038d4c (0xcfd4 bytes) 000703 1787785617.049138 read 0x01adbe thru 0x01c570 (0x17b3 bytes) 000704 1787785617.057406 trunc from 0x038d4d to 0x00ddeb (0x2af61 bytes) 000705 1787785617.076011 mapread 0x003a63 thru 0x006f6d (0x350b bytes) 000706 1787785617.091561 write 0x0381c5 thru 0x03bb43 (0x397f bytes) 000707 1787785617.093463 trunc from 0x03bb44 to 0x03285d (0x92e6 bytes) 000709 1787785617.116877 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3285e): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3285e): Mirror 7: 0x272025f3 (stale) Mirror 11: 0x272025f3 (stale) Mirror 15: 0x272025f3 (stale) Mirror 19: 0x272025f3 (stale) Mirror 20: 0x272025f3 (stale) Mirror 21: 0x272025f3 (stale) Mirror 22: 0x272025f3 (stale) Mirror 23: 0x272025f3 (stale) Mirror 24: 0x272025f3 (stale) Mirror 25: 0x272025f3 (stale) CRC-32 checksum value for chunk [0, 0x3285e): Mirror 1: 0x272025f3 Mirror 26: 0x272025f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3285e: skipped 000710 1787785618.484446 write 0x02a68a thru 0x03a23b (0xfbb2 bytes) 000712 1787785618.601646 mapwrite 0x0007ca thru 0x005131 (0x4968 bytes) 000713 1787785618.691143 write 0x00c2e4 thru 0x013d6f (0x7a8c bytes) 000715 1787785618.705328 MIRROR_RESYNC 12 mirrors 000718 1787785621.746731 MIRROR_EXTEND to 13 mirrors 000719 1787785622.947662 write 0x030c14 thru 0x032a37 (0x1e24 bytes) 000721 1787785623.095269 mapread 0x03867d thru 0x03a23b (0x1bbf bytes) 000723 1787785623.136056 write 0x016d4d thru 0x01c635 (0x58e9 bytes) 000724 1787785623.149669 trunc from 0x03a23c to 0x003d7d (0x364be bytes) 000726 1787785623.205721 trunc from 0x003d7e to 0x015059 (0x112dc bytes) 000727 1787785623.239043 read 0x00215d thru 0x00751b (0x53bf bytes) 000728 1787785623.252403 write 0x030df4 thru 0x03487f (0x3a8c bytes) 000729 1787785623.253760 mapread 0x00f24b thru 0x01f1a3 (0xff59 bytes) 000733 1787785623.273667 mapread 0x031a16 thru 0x03465b (0x2c46 bytes) 000736 1787785623.276636 mapwrite 0x0324d9 thru 0x03af5e (0x8a86 bytes) 000737 1787785623.397137 mapread 0x028281 thru 0x02d3c8 (0x5148 bytes) 000738 1787785623.420603 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3af5f): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af5f): Mirror 7: 0x384c3266 (stale) Mirror 11: 0x384c3266 (stale) Mirror 15: 0x384c3266 (stale) Mirror 19: 0x384c3266 (stale) Mirror 20: 0x384c3266 (stale) Mirror 21: 0x384c3266 (stale) Mirror 22: 0x384c3266 (stale) Mirror 23: 0x384c3266 (stale) Mirror 24: 0x384c3266 (stale) Mirror 25: 0x384c3266 (stale) Mirror 26: 0x384c3266 (stale) Mirror 27: 0x384c3266 (stale) CRC-32 checksum value for chunk [0, 0x3af5f): Mirror 1: 0x3e354c8b Mirror 28: 0x3e354c8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af5f: skipped 000739 1787785624.524362 trunc from 0x03af5f to 0x01b1f1 (0x1fd6d bytes) 000740 1787785624.653121 mapwrite 0x0188c5 thru 0x0256ca (0xce06 bytes) 000741 1787785624.770742 read 0x0191ef thru 0x01da1c (0x482e bytes) 000742 1787785624.771502 mapread 0x019023 thru 0x01e143 (0x5121 bytes) 000744 1787785624.774962 mapread 0x01bdf6 thru 0x023ee9 (0x80f4 bytes) 000745 1787785624.778423 write 0x01b421 thru 0x028ede (0xdabe bytes) 000746 1787785624.780431 trunc from 0x028edf to 0x03cc5e (0x13d80 bytes) 000747 1787785624.798728 trunc from 0x03cc5f to 0x03d8bc (0xc5e bytes) 000748 1787785624.821107 write 0x00a0ea thru 0x016f05 (0xce1c bytes) 000751 1787785624.829474 mapwrite 0x01b607 thru 0x020938 (0x5332 bytes) 000752 1787785624.879479 mapwrite 0x02167d thru 0x0295dc (0x7f60 bytes) 000753 1787785624.944559 mapread 0x0013dd thru 0x00b8c5 (0xa4e9 bytes) 000754 1787785624.959145 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3d8bd): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8bd): Mirror 7: 0x43863b2c (stale) Mirror 11: 0x43863b2c (stale) Mirror 15: 0x43863b2c (stale) Mirror 19: 0x43863b2c (stale) Mirror 20: 0x43863b2c (stale) Mirror 21: 0x43863b2c (stale) Mirror 22: 0x43863b2c (stale) Mirror 23: 0x43863b2c (stale) Mirror 24: 0x43863b2c (stale) Mirror 25: 0x43863b2c (stale) Mirror 26: 0x43863b2c (stale) Mirror 27: 0x43863b2c (stale) Mirror 28: 0x5fcd1ac5 (stale) CRC-32 checksum value for chunk [0, 0x3d8bd): Mirror 1: 0x7855106b Mirror 29: 0x7855106b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8bd: skipped 000755 1787785626.260767 trunc from 0x03d8bd to 0x016d2d (0x26b8f bytes) 000756 1787785626.381486 read 0x014cde thru 0x01584f (0xb72 bytes) 000757 1787785626.397594 MIRROR_RESYNC 15 mirrors 000760 1787785629.188615 mapwrite 0x018d6f thru 0x02130b (0x859d bytes) 000762 1787785629.521949 trunc from 0x02130c to 0x00a319 (0x16ff2 bytes) 000763 1787785629.549332 mapread 0x007310 thru 0x00a319 (0x300a bytes) 000764 1787785629.558388 mapread 0x001900 thru 0x004d85 (0x3486 bytes) 000766 1787785629.561828 trunc from 0x00a31a to 0x038b34 (0x2e81b bytes) 000767 1787785629.592797 trunc from 0x038b35 to 0x0035ed (0x35547 bytes) 000768 1787785629.620063 read 0x002477 thru 0x0035ed (0x1177 bytes) 000770 1787785629.634849 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ee): Mirror 7: 0xf2068f46 (stale) Mirror 11: 0xf2068f46 (stale) Mirror 15: 0xf2068f46 (stale) Mirror 20: 0xf2068f46 (stale) Mirror 21: 0xf2068f46 (stale) Mirror 22: 0xf2068f46 (stale) Mirror 23: 0xf2068f46 (stale) Mirror 24: 0xf2068f46 (stale) Mirror 25: 0xf2068f46 (stale) Mirror 26: 0xf2068f46 (stale) Mirror 27: 0xf2068f46 (stale) Mirror 28: 0xf2068f46 (stale) Mirror 29: 0xf2068f46 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ee): Mirror 7: 0xf2068f46 (stale) Mirror 11: 0xf2068f46 (stale) Mirror 15: 0xf2068f46 (stale) Mirror 20: 0xf2068f46 (stale) Mirror 21: 0xf2068f46 (stale) Mirror 22: 0xf2068f46 (stale) Mirror 23: 0xf2068f46 (stale) Mirror 24: 0xf2068f46 (stale) Mirror 25: 0xf2068f46 (stale) Mirror 26: 0xf2068f46 (stale) Mirror 27: 0xf2068f46 (stale) Mirror 28: 0xf2068f46 (stale) Mirror 29: 0xf2068f46 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ee: skipped 000771 1787785630.263160 mapwrite 0x0381d5 thru 0x03b787 (0x35b3 bytes) 000772 1787785630.364753 mapwrite 0x038ec7 thru 0x03ffff (0x7139 bytes) 000773 1787785630.431270 read 0x006f5a thru 0x00a7e2 (0x3889 bytes) 000775 1787785630.442355 trunc from 0x040000 to 0x003950 (0x3c6af bytes) 000777 1787785630.459922 write 0x014f6e thru 0x023641 (0xe6d4 bytes) 000778 1787785630.463388 mapwrite 0x027ee4 thru 0x030bf5 (0x8d12 bytes) 000779 1787785630.557650 trunc from 0x030bf6 to 0x0003ac (0x30849 bytes) 000780 1787785630.579670 mapwrite 0x03b53a thru 0x03ffff (0x4ac6 bytes) 000781 1787785630.655067 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x17e23590 (stale) Mirror 11: 0x17e23590 (stale) Mirror 15: 0x17e23590 (stale) Mirror 20: 0x17e23590 (stale) Mirror 21: 0x17e23590 (stale) Mirror 22: 0x17e23590 (stale) Mirror 23: 0x17e23590 (stale) Mirror 24: 0x17e23590 (stale) Mirror 25: 0x17e23590 (stale) Mirror 26: 0x17e23590 (stale) Mirror 27: 0x17e23590 (stale) Mirror 29: 0x17e23590 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x17e23590 (stale) Mirror 11: 0x17e23590 (stale) Mirror 15: 0x17e23590 (stale) Mirror 20: 0x17e23590 (stale) Mirror 21: 0x17e23590 (stale) Mirror 22: 0x17e23590 (stale) Mirror 23: 0x17e23590 (stale) Mirror 24: 0x17e23590 (stale) Mirror 25: 0x17e23590 (stale) Mirror 26: 0x17e23590 (stale) Mirror 27: 0x17e23590 (stale) Mirror 29: 0x17e23590 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1787785631.315512 mapwrite 0x0294ef thru 0x034c88 (0xb79a bytes) 000784 1787785631.453537 MIRROR_RESYNC 13 mirrors 000786 1787785634.856417 mapwrite 0x006b5d thru 0x0111a8 (0xa64c bytes) 000788 1787785635.099163 trunc from 0x040000 to 0x03406e (0xbf91 bytes) 000789 1787785635.144214 mapread 0x016fc1 thru 0x026300 (0xf340 bytes) 000791 1787785635.206357 read 0x00d90e thru 0x01cd14 (0xf407 bytes) 000792 1787785635.207600 write 0x035efb thru 0x03e9f4 (0x8afa bytes) 000793 1787785635.211929 mapread 0x03b362 thru 0x03e9f4 (0x3693 bytes) 000794 1787785635.224402 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9f5): Mirror 7: 0xd54a8ed7 (stale) Mirror 11: 0xd54a8ed7 (stale) Mirror 15: 0xd54a8ed7 (stale) Mirror 20: 0xd54a8ed7 (stale) Mirror 21: 0xd54a8ed7 (stale) Mirror 22: 0xd54a8ed7 (stale) Mirror 24: 0xd54a8ed7 (stale) Mirror 25: 0xd54a8ed7 (stale) Mirror 26: 0xd54a8ed7 (stale) Mirror 27: 0xd54a8ed7 (stale) Mirror 29: 0xd54a8ed7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9f5): Mirror 7: 0xd54a8ed7 (stale) Mirror 11: 0xd54a8ed7 (stale) Mirror 15: 0xd54a8ed7 (stale) Mirror 20: 0xd54a8ed7 (stale) Mirror 21: 0xd54a8ed7 (stale) Mirror 22: 0xd54a8ed7 (stale) Mirror 24: 0xd54a8ed7 (stale) Mirror 25: 0xd54a8ed7 (stale) Mirror 26: 0xd54a8ed7 (stale) Mirror 27: 0xd54a8ed7 (stale) Mirror 29: 0xd54a8ed7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9f5: skipped 000795 1787785635.999447 mapread 0x0032c9 thru 0x007bce (0x4906 bytes) 000797 1787785636.037008 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9f5): Mirror 7: 0xd54a8ed7 (stale) Mirror 15: 0xd54a8ed7 (stale) Mirror 20: 0xd54a8ed7 (stale) Mirror 21: 0xd54a8ed7 (stale) Mirror 22: 0xd54a8ed7 (stale) Mirror 24: 0xd54a8ed7 (stale) Mirror 25: 0xd54a8ed7 (stale) Mirror 26: 0xd54a8ed7 (stale) Mirror 27: 0xd54a8ed7 (stale) Mirror 29: 0xd54a8ed7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9f5): Mirror 7: 0xd54a8ed7 (stale) Mirror 15: 0xd54a8ed7 (stale) Mirror 20: 0xd54a8ed7 (stale) Mirror 21: 0xd54a8ed7 (stale) Mirror 22: 0xd54a8ed7 (stale) Mirror 24: 0xd54a8ed7 (stale) Mirror 25: 0xd54a8ed7 (stale) Mirror 26: 0xd54a8ed7 (stale) Mirror 27: 0xd54a8ed7 (stale) Mirror 29: 0xd54a8ed7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9f5: skipped 000798 1787785636.747831 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3e9f5): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9f5): Mirror 7: 0xd54a8ed7 (stale) Mirror 15: 0xd54a8ed7 (stale) Mirror 20: 0xd54a8ed7 (stale) Mirror 21: 0xd54a8ed7 (stale) Mirror 22: 0xd54a8ed7 (stale) Mirror 24: 0xd54a8ed7 (stale) Mirror 25: 0xd54a8ed7 (stale) Mirror 26: 0xd54a8ed7 (stale) Mirror 27: 0xd54a8ed7 (stale) Mirror 29: 0xd54a8ed7 (stale) CRC-32 checksum value for chunk [0, 0x3e9f5): Mirror 1: 0xe7725109 Mirror 30: 0xe7725109 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9f5: skipped 000799 1787785638.477805 mapwrite 0x011741 thru 0x01cb62 (0xb422 bytes) 000801 1787785638.689723 mapread 0x013651 thru 0x01dea1 (0xa851 bytes) 000803 1787785638.692861 write 0x003c60 thru 0x013318 (0xf6b9 bytes) 000805 1787785638.705795 read 0x027a29 thru 0x02af17 (0x34ef bytes) 000806 1787785638.713420 mapwrite 0x0364ed thru 0x03ffff (0x9b13 bytes) 000807 1787785638.804233 mapread 0x033bad thru 0x03ffff (0xc453 bytes) 000808 1787785638.813193 trunc from 0x040000 to 0x00a79f (0x35860 bytes) 000809 1787785638.842754 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0xa7a0): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa7a0): Mirror 7: 0x8a2189e9 (stale) Mirror 15: 0x8a2189e9 (stale) Mirror 20: 0x8a2189e9 (stale) Mirror 21: 0x8a2189e9 (stale) Mirror 22: 0x8a2189e9 (stale) Mirror 24: 0x8a2189e9 (stale) Mirror 25: 0x8a2189e9 (stale) Mirror 26: 0x8a2189e9 (stale) Mirror 27: 0x8a2189e9 (stale) Mirror 29: 0x8a2189e9 (stale) Mirror 30: 0x6282e36a (stale) CRC-32 checksum value for chunk [0, 0xa7a0): Mirror 1: 0x3ad6b26 Mirror 31: 0x3ad6b26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa7a0: skipped 000810 1787785640.418989 write 0x02f01e thru 0x03ad37 (0xbd1a bytes) 000813 1787785640.545142 read 0x035e5e thru 0x03ad37 (0x4eda bytes) 000814 1787785640.547853 mapwrite 0x0354fa thru 0x036750 (0x1257 bytes) 000815 1787785640.621871 trunc from 0x03ad38 to 0x01619e (0x24b99 bytes) 000816 1787785640.696179 read 0x000bad thru 0x008ec1 (0x8315 bytes) 000817 1787785640.720343 write 0x013d41 thru 0x01a841 (0x6b01 bytes) 000818 1787785640.727362 trunc from 0x01a842 to 0x02bf9c (0x1175b bytes) 000821 1787785640.784774 mapread 0x025a91 thru 0x02bf9c (0x650c bytes) 000822 1787785640.818912 trunc from 0x02bf9d to 0x01662a (0x15972 bytes) 000823 1787785640.876784 MIRROR_RESYNC 13 mirrors 000824 1787785643.352000 mapread 0x00846f thru 0x00a780 (0x2312 bytes) 000825 1787785643.399657 trunc from 0x01662b to 0x037d81 (0x21757 bytes) 000826 1787785643.570705 write 0x03cbd6 thru 0x03ffff (0x342a bytes) 000827 1787785643.575920 mapwrite 0x013328 thru 0x0192ca (0x5fa3 bytes) 000828 1787785643.718870 read 0x00bd34 thru 0x0198b8 (0xdb85 bytes) 000829 1787785643.730741 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x589ff3a1 (stale) Mirror 15: 0x589ff3a1 (stale) Mirror 20: 0x589ff3a1 (stale) Mirror 21: 0x589ff3a1 (stale) Mirror 22: 0x589ff3a1 (stale) Mirror 24: 0x589ff3a1 (stale) Mirror 25: 0x589ff3a1 (stale) Mirror 26: 0x589ff3a1 (stale) Mirror 27: 0x589ff3a1 (stale) Mirror 29: 0x589ff3a1 (stale) Mirror 31: 0x589ff3a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x589ff3a1 (stale) Mirror 15: 0x589ff3a1 (stale) Mirror 20: 0x589ff3a1 (stale) Mirror 21: 0x589ff3a1 (stale) Mirror 22: 0x589ff3a1 (stale) Mirror 24: 0x589ff3a1 (stale) Mirror 25: 0x589ff3a1 (stale) Mirror 26: 0x589ff3a1 (stale) Mirror 27: 0x589ff3a1 (stale) Mirror 29: 0x589ff3a1 (stale) Mirror 31: 0x589ff3a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000832 1787785644.618639 mapread 0x00f260 thru 0x017e87 (0x8c28 bytes) 000833 1787785644.641765 write 0x035265 thru 0x03ffff (0xad9b bytes) 000834 1787785644.648608 write 0x00992f thru 0x012853 (0x8f25 bytes) 000837 1787785644.655476 trunc from 0x040000 to 0x0075c1 (0x38a3e bytes) 000840 1787785644.677762 write 0x00b99f thru 0x014f2e (0x9590 bytes) 000842 1787785644.684266 write 0x011c68 thru 0x016e7e (0x5217 bytes) 000843 1787785644.686258 write 0x03ea93 thru 0x03ffff (0x156d bytes) 000844 1787785644.693401 read 0x018b23 thru 0x01eb53 (0x6031 bytes) 000845 1787785644.703186 mapread 0x01e872 thru 0x022b65 (0x42f4 bytes) 000846 1787785644.719500 read 0x03eafb thru 0x03efc1 (0x4c7 bytes) 000847 1787785644.720317 read 0x00dbc3 thru 0x01674f (0x8b8d bytes) 000849 1787785644.722264 write 0x02fbd9 thru 0x03f1b7 (0xf5df bytes) 000851 1787785644.738814 mapread 0x0397f7 thru 0x03ffff (0x6809 bytes) 000853 1787785644.741235 write 0x007b32 thru 0x008b3e (0x100d bytes) 000854 1787785644.750649 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x589ff3a1 (stale) Mirror 15: 0x589ff3a1 (stale) Mirror 20: 0x589ff3a1 (stale) Mirror 21: 0x589ff3a1 (stale) Mirror 22: 0x589ff3a1 (stale) Mirror 24: 0x589ff3a1 (stale) Mirror 25: 0x589ff3a1 (stale) Mirror 26: 0x589ff3a1 (stale) Mirror 27: 0x589ff3a1 (stale) Mirror 29: 0x589ff3a1 (stale) Mirror 31: 0x589ff3a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8d6cdcf0 Mirror 32: 0x8d6cdcf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1787785646.083062 read 0x02fea9 thru 0x03f2a7 (0xf3ff bytes) 000856 1787785646.102489 write 0x03b3de thru 0x03ffff (0x4c22 bytes) 000857 1787785646.291755 trunc from 0x040000 to 0x01d33e (0x22cc1 bytes) 000858 1787785646.328240 mapwrite 0x0295a1 thru 0x033672 (0xa0d2 bytes) 000863 1787785646.552593 mapwrite 0x0217c6 thru 0x02b65d (0x9e98 bytes) 000869 1787785646.669891 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x33673): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33673): Mirror 7: 0xfa77c3ac (stale) Mirror 15: 0xfa77c3ac (stale) Mirror 20: 0xfa77c3ac (stale) Mirror 21: 0xfa77c3ac (stale) Mirror 22: 0xfa77c3ac (stale) Mirror 24: 0xfa77c3ac (stale) Mirror 25: 0xfa77c3ac (stale) Mirror 26: 0xfa77c3ac (stale) Mirror 27: 0xfa77c3ac (stale) Mirror 29: 0xfa77c3ac (stale) Mirror 31: 0xfa77c3ac (stale) Mirror 32: 0xf00ac4a1 (stale) CRC-32 checksum value for chunk [0, 0x33673): Mirror 1: 0xc61eb1ab Mirror 33: 0xc61eb1ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33673: skipped 000870 1787785648.123340 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33673): Mirror 7: 0xfa77c3ac (stale) Mirror 15: 0xfa77c3ac (stale) Mirror 21: 0xfa77c3ac (stale) Mirror 22: 0xfa77c3ac (stale) Mirror 24: 0xfa77c3ac (stale) Mirror 25: 0xfa77c3ac (stale) Mirror 26: 0xfa77c3ac (stale) Mirror 27: 0xfa77c3ac (stale) Mirror 29: 0xfa77c3ac (stale) Mirror 31: 0xfa77c3ac (stale) Mirror 32: 0xf00ac4a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33673): Mirror 7: 0xfa77c3ac (stale) Mirror 15: 0xfa77c3ac (stale) Mirror 21: 0xfa77c3ac (stale) Mirror 22: 0xfa77c3ac (stale) Mirror 24: 0xfa77c3ac (stale) Mirror 25: 0xfa77c3ac (stale) Mirror 26: 0xfa77c3ac (stale) Mirror 27: 0xfa77c3ac (stale) Mirror 29: 0xfa77c3ac (stale) Mirror 31: 0xfa77c3ac (stale) Mirror 32: 0xf00ac4a1 (stale) CRC-32 checksum value for chunk [0, 0x33673): Mirror 1: 0xc61eb1ab Mirror 33: 0xc61eb1ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33673: skipped 000871 1787785649.251839 trunc from 0x033673 to 0x038e9f (0x582d bytes) 000872 1787785649.362914 mapwrite 0x02e563 thru 0x03de08 (0xf8a6 bytes) 000873 1787785649.483372 read 0x01b06d thru 0x029823 (0xe7b7 bytes) 000874 1787785649.497648 trunc from 0x03de09 to 0x03f297 (0x148f bytes) 000875 1787785649.527277 MIRROR_RESYNC 13 mirrors 000876 1787785653.136788 read 0x010492 thru 0x01bdcc (0xb93b bytes) 000881 1787785653.149720 write 0x003e40 thru 0x009239 (0x53fa bytes) 000882 1787785653.306411 write 0x015e43 thru 0x025cf0 (0xfeae bytes) 000887 1787785653.337562 mapwrite 0x03518d thru 0x03ffff (0xae73 bytes) 000889 1787785653.533124 write 0x025ee9 thru 0x02f3b2 (0x94ca bytes) 000890 1787785653.551529 trunc from 0x040000 to 0x035873 (0xa78c bytes) 000892 1787785653.589972 write 0x028f26 thru 0x0361b7 (0xd292 bytes) 000896 1787785653.601589 trunc from 0x0361b8 to 0x01497e (0x21839 bytes) 000897 1787785653.636687 mapread 0x0028d7 thru 0x00c9e7 (0xa111 bytes) 000899 1787785653.653811 read 0x0019dc thru 0x007d9d (0x63c2 bytes) 000900 1787785653.656780 trunc from 0x01497f to 0x038917 (0x23f99 bytes) 000901 1787785653.683153 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38918): Mirror 7: 0x92b4b46 (stale) Mirror 15: 0x92b4b46 (stale) Mirror 21: 0x92b4b46 (stale) Mirror 22: 0x92b4b46 (stale) Mirror 25: 0x92b4b46 (stale) Mirror 26: 0x92b4b46 (stale) Mirror 27: 0x92b4b46 (stale) Mirror 29: 0x92b4b46 (stale) Mirror 31: 0x92b4b46 (stale) Mirror 32: 0x92b4b46 (stale) Mirror 33: 0x92b4b46 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38918): Mirror 7: 0x92b4b46 (stale) Mirror 15: 0x92b4b46 (stale) Mirror 21: 0x92b4b46 (stale) Mirror 22: 0x92b4b46 (stale) Mirror 25: 0x92b4b46 (stale) Mirror 26: 0x92b4b46 (stale) Mirror 27: 0x92b4b46 (stale) Mirror 29: 0x92b4b46 (stale) Mirror 31: 0x92b4b46 (stale) Mirror 32: 0x92b4b46 (stale) Mirror 33: 0x92b4b46 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38918: skipped 000902 1787785654.450948 mapread 0x0212f0 thru 0x03080a (0xf51b bytes) 000903 1787785654.501670 mapread 0x014c05 thru 0x014f16 (0x312 bytes) 000906 1787785654.525109 mapwrite 0x0110f8 thru 0x014c49 (0x3b52 bytes) 000908 1787785654.606633 mapwrite 0x03bbbd thru 0x03ffff (0x4443 bytes) 000911 1787785654.754770 mapread 0x018ca8 thru 0x028572 (0xf8cb bytes) 000913 1787785654.757391 trunc from 0x040000 to 0x002b7a (0x3d485 bytes) 000916 1787785654.780183 trunc from 0x002b7b to 0x02ea2b (0x2beb1 bytes) 000918 1787785654.806781 write 0x02def9 thru 0x036dd1 (0x8ed9 bytes) 000919 1787785654.813846 trunc from 0x036dd2 to 0x039c94 (0x2ec3 bytes) 000920 1787785654.846010 mapwrite 0x03e0b9 thru 0x03ffff (0x1f47 bytes) 000921 1787785654.928124 MIRROR_RESYNC 12 mirrors 000924 1787785658.077642 mapwrite 0x002e34 thru 0x007013 (0x41e0 bytes) 000925 1787785658.362254 trunc from 0x040000 to 0x02144b (0x1ebb4 bytes) 000926 1787785658.407804 mapread 0x01413b thru 0x01f4cb (0xb391 bytes) 000927 1787785658.442448 write 0x02ef38 thru 0x036b76 (0x7c3f bytes) 000928 1787785658.445386 mapwrite 0x027a92 thru 0x02e3ff (0x696e bytes) 000930 1787785658.553645 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36b77): Mirror 7: 0xe28070a (stale) Mirror 15: 0xe28070a (stale) Mirror 22: 0xe28070a (stale) Mirror 25: 0xe28070a (stale) Mirror 26: 0xe28070a (stale) Mirror 27: 0xe28070a (stale) Mirror 29: 0xe28070a (stale) Mirror 31: 0xe28070a (stale) Mirror 32: 0xe28070a (stale) Mirror 33: 0xe28070a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b77): Mirror 7: 0xe28070a (stale) Mirror 15: 0xe28070a (stale) Mirror 22: 0xe28070a (stale) Mirror 25: 0xe28070a (stale) Mirror 26: 0xe28070a (stale) Mirror 27: 0xe28070a (stale) Mirror 29: 0xe28070a (stale) Mirror 31: 0xe28070a (stale) Mirror 32: 0xe28070a (stale) Mirror 33: 0xe28070a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b77: skipped 000931 1787785659.430658 mapwrite 0x026794 thru 0x026f85 (0x7f2 bytes) 000934 1787785659.539825 read 0x02d93d thru 0x033da0 (0x6464 bytes) 000936 1787785659.569697 mapwrite 0x022237 thru 0x02e784 (0xc54e bytes) 000938 1787785659.714600 mapwrite 0x00a241 thru 0x00b283 (0x1043 bytes) 000940 1787785659.817905 trunc from 0x036b77 to 0x027731 (0xf445 bytes) 000941 1787785659.856972 trunc from 0x027732 to 0x01fcfe (0x7a33 bytes) 000942 1787785659.904452 mapread 0x0090ab thru 0x009ff7 (0xf4d bytes) 000943 1787785659.910782 read 0x014fca thru 0x01fcfe (0xad35 bytes) 000945 1787785659.965836 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1fcff): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fcff): Mirror 7: 0xaa2a9f52 (stale) Mirror 15: 0xaa2a9f52 (stale) Mirror 22: 0xaa2a9f52 (stale) Mirror 25: 0xaa2a9f52 (stale) Mirror 26: 0xaa2a9f52 (stale) Mirror 27: 0xaa2a9f52 (stale) Mirror 29: 0xaa2a9f52 (stale) Mirror 31: 0xaa2a9f52 (stale) Mirror 32: 0xaa2a9f52 (stale) Mirror 33: 0xaa2a9f52 (stale) CRC-32 checksum value for chunk [0, 0x1fcff): Mirror 1: 0xd3d7f31 Mirror 34: 0xd3d7f31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fcff: skipped 000946 1787785661.820604 read 0x00d7fd thru 0x01c2d2 (0xead6 bytes) 000947 1787785661.844180 mapwrite 0x00603b thru 0x012d7d (0xcd43 bytes) 000948 1787785662.100352 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1fcff): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fcff): Mirror 7: 0xaa2a9f52 (stale) Mirror 15: 0xaa2a9f52 (stale) Mirror 22: 0xaa2a9f52 (stale) Mirror 25: 0xaa2a9f52 (stale) Mirror 26: 0xaa2a9f52 (stale) Mirror 27: 0xaa2a9f52 (stale) Mirror 29: 0xaa2a9f52 (stale) Mirror 31: 0xaa2a9f52 (stale) Mirror 32: 0xaa2a9f52 (stale) Mirror 33: 0xaa2a9f52 (stale) Mirror 34: 0xd3d7f31 (stale) CRC-32 checksum value for chunk [0, 0x1fcff): Mirror 1: 0x8b65a23a Mirror 35: 0x8b65a23a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fcff: skipped 000949 1787785663.538902 write 0x001f7d thru 0x00514f (0x31d3 bytes) 000952 1787785663.639701 trunc from 0x01fcff to 0x028217 (0x8519 bytes) 000954 1787785663.661456 mapread 0x0223b1 thru 0x028217 (0x5e67 bytes) 000955 1787785663.709669 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x28218): Mirror 7: (stale - read error) Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28218): Mirror 7: 0x28c5f84c (stale) Mirror 15: 0x28c5f84c (stale) Mirror 22: 0x28c5f84c (stale) Mirror 25: 0x28c5f84c (stale) Mirror 26: 0x28c5f84c (stale) Mirror 27: 0x28c5f84c (stale) Mirror 29: 0x28c5f84c (stale) Mirror 31: 0x28c5f84c (stale) Mirror 32: 0x28c5f84c (stale) Mirror 33: 0x28c5f84c (stale) Mirror 34: 0xe30ba554 (stale) Mirror 35: 0xc3e31e44 (stale) CRC-32 checksum value for chunk [0, 0x28218): Mirror 1: 0x1dd309b5 Mirror 36: 0x1dd309b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28218: skipped 000958 1787785665.295563 mapwrite 0x023c0d thru 0x030c33 (0xd027 bytes) 000961 1787785665.497880 write 0x018bca thru 0x024b32 (0xbf69 bytes) 000962 1787785665.506282 mapread 0x009c01 thru 0x0173d1 (0xd7d1 bytes) 000966 1787785665.537708 MIRROR_RESYNC 14 mirrors 000969 1787785668.345587 MIRROR_RESYNC 14 mirrors 000971 1787785669.443588 trunc from 0x030c34 to 0x01a3bc (0x16877 bytes) 000973 1787785669.605938 mapread 0x018e19 thru 0x01a3bc (0x15a4 bytes) 000975 1787785669.652679 mapwrite 0x004c90 thru 0x006ac2 (0x1e33 bytes) 000977 1787785669.834971 mapwrite 0x022c6e thru 0x03137e (0xe711 bytes) 000978 1787785670.091726 MIRROR_RESYNC 14 mirrors 000979 1787785672.135733 read 0x002443 thru 0x0117e6 (0xf3a4 bytes) 000982 1787785672.149761 mapwrite 0x03cf02 thru 0x03ffff (0x30fe bytes) 000983 1787785672.329607 read 0x00cb7a thru 0x01366c (0x6af3 bytes) 000984 1787785672.343425 mapread 0x03da2d thru 0x03ffff (0x25d3 bytes) 000986 1787785672.345065 read 0x021b27 thru 0x028606 (0x6ae0 bytes) 000987 1787785672.351562 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3ae0e91 (stale) Mirror 15: 0x3ae0e91 (stale) Mirror 22: 0x3ae0e91 (stale) Mirror 26: 0x3ae0e91 (stale) Mirror 27: 0x3ae0e91 (stale) Mirror 29: 0x3ae0e91 (stale) Mirror 31: 0x3ae0e91 (stale) Mirror 32: 0x3ae0e91 (stale) Mirror 33: 0x3ae0e91 (stale) Mirror 34: 0x3ae0e91 (stale) Mirror 35: 0x3ae0e91 (stale) Mirror 36: 0x3ae0e91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3ae0e91 (stale) Mirror 15: 0x3ae0e91 (stale) Mirror 22: 0x3ae0e91 (stale) Mirror 26: 0x3ae0e91 (stale) Mirror 27: 0x3ae0e91 (stale) Mirror 29: 0x3ae0e91 (stale) Mirror 31: 0x3ae0e91 (stale) Mirror 32: 0x3ae0e91 (stale) Mirror 33: 0x3ae0e91 (stale) Mirror 34: 0x3ae0e91 (stale) Mirror 35: 0x3ae0e91 (stale) Mirror 36: 0x3ae0e91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1787785673.522359 write 0x017a91 thru 0x01f400 (0x7970 bytes) 000991 1787785673.542512 read 0x0207f0 thru 0x025b3d (0x534e bytes) 000993 1787785673.556428 mapwrite 0x01c3fa thru 0x02943d (0xd044 bytes) 000994 1787785673.654783 write 0x02472a thru 0x03213c (0xda13 bytes) 000995 1787785673.666202 read 0x02c69e thru 0x03b3f8 (0xed5b bytes) 000996 1787785673.706826 MIRROR_RESYNC 13 mirrors 000997 1787785676.201010 mapread 0x03e28b thru 0x03ffff (0x1d75 bytes) 000998 1787785676.227960 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17a:0x0]: Invalid argument (22) 001000 1787785676.813853 read 0x0074d5 thru 0x01564a (0xe176 bytes) All operations completed A-OK! PASS 70a (175s) == sanity-flr test 71: check mirror extend parent fid ==== 19:08:03 (1787785683) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 19:08:08 (1787785688) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205951 s, 10.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277673 s, 3.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 19:08:20 (1787785700) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8340416 .. Extending file size to 4261984 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8556736 .. Extending file size to 4625408 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6505344 .. Extending file size to 8336960 .. Extending file size to 8379424 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8455648 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8685824 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8963584 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8791808 .. Waiting 58679 58680 58681 58683 58685 done 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) 192.168.204.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:09:44 (1787785784) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3993056 .. Extending file size to 8831392 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5348448 .. Extending file size to 8247744 .. Extending file size to 9306592 .. Extending file size to 8551904 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 60356 60357 60358 60361 60363 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) 192.168.204.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:11:02 (1787785862) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199728 s, 10.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.392802 s, 5.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x321a9374 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:11:25 (1787785885) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122295 s, 17.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 19:11:33 (1787785893) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:11:40 (1787785900) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173983 s, 12.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:11:48 (1787785908) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165502 s, 12.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:11:57 (1787785917) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22206 s, 9.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:12:31 (1787785951) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295321 s, 3.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:13:36 (1787786016) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284383 s, 7.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:14:34 (1787786074) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. Creating new pool test_204f oleg401-server: Pool lustre.test_204f created Adding targets to pool oleg401-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg401-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.359841 s, 5.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg401-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg401-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg401-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:15:59 (1787786159) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:16:06 (1787786166) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:16:13 (1787786173) oleg401-server: Pool lustre.test_206 created Adding targets to pool oleg401-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg401-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg401-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg401-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg401-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:16:39 (1787786199) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101726 s, 10.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680027 s, 15.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa3bdeb1e Mirror 3: 0xa3bdeb1e PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:16:51 (1787786211) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:16:55 (1787786215) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:17:00 (1787786220) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0869241 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xf60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xf60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xf60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x19a0000 Read this (4 bytes): BBBB 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:17:42 (1787786262) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0140763 s, 582 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:17:49 (1787786269) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482775 s, 21.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ed lmm_fid: [0x200000404:0x1ed:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ed lmm_fid: [0x200000404:0x1ed:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834413 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 19:17:51.000000000 -0400 Modify: 2026-08-26 19:17:51.000000000 -0400 Change: 2026-08-26 19:17:52.000000000 -0400 Birth: 2026-08-26 19:17:52.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:17:57 (1787786277) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:18:01 (1787786281) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.484797 s, 84.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.097119 s, 42.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 19:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 19:18 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:18:10 (1787786290) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:18:17 (1787786297) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1989 sec =========== 19:18:26 (1787786306) === sanity-flr: start cleanup 19:18:28 (1787786308) === === sanity-flr: finish cleanup 19:18:34 (1787786314) ===