-----============= acceptance-small: sanity-flr ============----- Thu Aug 6 03:50:01 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg428-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 03:50:25 (1786002625) === oleg428-client.virtnet: executing check_config_client /mnt/lustre oleg428-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg428-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99985105b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff99985105b000.idle_timeout=debug disable quota as required oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 03:51:06 (1786002666) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7648] == sanity-flr test 0a: lfs mirror create with -N option == 03:51:09 (1786002669) PASS 0a (30s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:51:40 (1786002700) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:51:45 (1786002705) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:51:50 (1786002710) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:52:16 (1786002736) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:52:20 (1786002740) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 03:52:24 (1786002744) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99985105b000.stats=clear osc.lustre-OST0001-osc-ffff99985105b000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:52:32 (1786002752) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (15s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:52:47 (1786002767) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:52:58 (1786002778) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:53:03 (1786002783) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:53:12 (1786002792) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 03:53:25 (1786002805) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:53:33 (1786002813) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:53:37 (1786002817) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:53:48 (1786002828) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:54:01 (1786002841) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:54:10 (1786002850) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 2.77232 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302168 s, 3.5 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:54:24 (1786002864) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.206235 s, 25.4 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:54:35 (1786002875) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:54:50 (1786002890) Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99985105b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99985105b000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99985105b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99985105b000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (100s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:56:31 (1786002991) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.155562 s, 20.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99985105b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99985105b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (55s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:57:25 (1786003045) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99985105b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99985105b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99985105b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99985105b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (104s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:59:09 (1786003149) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.68033 s, 6.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 6 03:59 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.931s user 0m0.002s sys 0m0.130s Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99985105b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99985105b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.995s user 0m0.000s sys 0m0.147s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99985105b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99985105b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m2.189s user 0m0.001s sys 0m0.182s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99985105b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99985105b000.ost_server_uuid in FULL state after 0 sec PASS 33b (136s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:01:25 (1786003285) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:01:30 (1786003290) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:01:34 (1786003294) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 04:01:38 (1786003298) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 04:01:47 (1786003307) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.775095 s, 16.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:02:01 (1786003321) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091242 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437076 s, 24.0 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:02:10 (1786003330) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:02:19 (1786003339) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121587 s, 842 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000932684 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00204709 s, 500 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00190576 s, 537 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 04:02:33 (1786003353) create test files with size 15732785 .. 3e5dd51bc29a7422309c8885a9fc1b62 - d219e559eb4421ce4d920c1fcc4449e0 - 31e0699df7b7dea3eda7a6a722507358 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 04:03:01 (1786003381) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (30s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:03:31 (1786003411) PASS 39 (11s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:03:42 (1786003422) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238552 s, 4.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628603 s, 16.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 04:03:56 (1786003436) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0961215 s, 21.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.269209 s, 15.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218948 s, 9.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 04:04:15 (1786003455) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 04:04:19 (1786003459) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:04:23 (1786003463) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333471 s, 31.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 04:04:32 (1786003472) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:04:37 (1786003477) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:04:49 (1786003489) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.377108 s, 27.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:05:02 (1786003502) llite.lustre-ffff9998604cb800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.500217 s, 21.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff999848d35800.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:05:23 (1786003523) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (53s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:06:15 (1786003575) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (53s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:07:08 (1786003628) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 04:07:12 (1786003632) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.319185 s, 9.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.130525 s, 24.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4cb49af0 (nosync,stale) Mirror 2: 0x4cb49af0 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5fb9ac09a132e9b8c52ec2295728c09b - mirror 1 checksum: 20beaf286d43c75d3f54399840544fbf - mirror 2 checksum: 5fb9ac09a132e9b8c52ec2295728c09b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-06T04:07:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xfc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-06T04:07:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xfd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5fb9ac09a132e9b8c52ec2295728c09b - mirror 1 checksum: 20beaf286d43c75d3f54399840544fbf - mirror 2 checksum: 20beaf286d43c75d3f54399840544fbf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-06T04:07:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xfc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-06T04:07:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xfd:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:07:29 (1786003649) Failing mds1 on oleg428-server Stopping /mnt/lustre-mds1 (opts:) on oleg428-server 04:07:40 (1786003660) shut down facet: mds1 facet_host: oleg428-server facet_failover_host: oleg428-server Failover mds1 to oleg428-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 04:08:15 (1786003695) targets are mounted 04:08:15 (1786003695) facet_failover done oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067905 s, 15.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xfe:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:08:35 (1786003715) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:08:40 (1786003720) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:08:45 (1786003725) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0127256 s, 1.3 MB/s PASS 50c (13s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:08:58 (1786003738) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:09:03 (1786003743) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00497616 s, 823 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.124446 s, 32.9 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:09:13 (1786003753) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (31s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:09:45 (1786003785) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (36s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:10:22 (1786003822) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (30s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:10:52 (1786003852) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (30s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:11:23 (1786003883) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 04:11:32 (1786003892) mirror create pid 54374 mirror split pid 54375 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54374 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54375 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (71s) == sanity-flr test 70a: flr mode fsx test ================ 04:12:43 (1786003963) Seed set to 3965 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1786003966.285864 MIRROR_EXTEND to 2 mirrors skipping zero size read skipping zero size read 000009 1786003967.349147 mapwrite 0x014467 thru 0x01c6df (0x8279 bytes) 000010 1786003967.635200 read 0x0154c2 thru 0x01c6df (0x721e bytes) 000012 1786003967.653155 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x16d15 000013 1786003968.937803 trunc from 0x01c6e0 to 0x016d14 (0x59cb bytes) 000017 1786003969.168476 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000025 1786003969.504137 mapread 0x006285 thru 0x00b0f0 (0x4e6c bytes) 000026 1786003969.568690 write 0x02d9d5 thru 0x0386da (0xad06 bytes) 000027 1786003969.575186 write 0x033c55 thru 0x03d4eb (0x9897 bytes) 000028 1786003969.634267 MIRROR_EXTEND to 2 mirrors 000030 1786003971.614336 mapwrite 0x032fd1 thru 0x03ffff (0xd02f bytes) 000031 1786003972.089504 read 0x034162 thru 0x037308 (0x31a7 bytes) 000035 1786003972.099390 mapwrite 0x034abe thru 0x03f0f7 (0xa63a bytes) 000036 1786003972.208698 read 0x011895 thru 0x01a7c3 (0x8f2f bytes) truncating to largest ever: 0x3439c 000037 1786003972.238959 trunc from 0x040000 to 0x03439b (0xbc64 bytes) 000039 1786003972.279474 mapread 0x030c21 thru 0x033945 (0x2d25 bytes) 000040 1786003972.280594 read 0x02fd8e thru 0x03439b (0x460e bytes) 000041 1786003972.289969 trunc from 0x03439c to 0x01cf77 (0x17424 bytes) 000042 1786003972.344021 trunc from 0x01cf78 to 0x020d51 (0x3dda bytes) 000045 1786003972.383373 mapwrite 0x03a0f6 thru 0x03ffff (0x5f0a bytes) 000046 1786003972.504256 trunc from 0x040000 to 0x002aa0 (0x3d55f bytes) 000047 1786003972.528806 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2aa1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa1): Mirror 2: 0x822865b0 (stale) CRC-32 checksum value for chunk [0, 0x2aa1): Mirror 1: 0x822865b0 Mirror 3: 0x822865b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa1: skipped 000050 1786003974.452700 mapwrite 0x005be3 thru 0x00dcbc (0x80da bytes) 000051 1786003974.852931 mapwrite 0x01b699 thru 0x029859 (0xe1c1 bytes) 000052 1786003975.138127 read 0x021b5e thru 0x026dc9 (0x526c bytes) 000054 1786003975.141829 mapwrite 0x001466 thru 0x005d32 (0x48cd bytes) 000056 1786003975.288438 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2985a): Mirror 3: 0xbebd92b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2985a): Mirror 3: 0xbebd92b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2985a: skipped 000058 1786003975.880413 trunc from 0x02985a to 0x01c062 (0xd7f7 bytes) 000059 1786003975.938496 mapread 0x000d00 thru 0x008d38 (0x8039 bytes) 000061 1786003976.026397 mapwrite 0x012ed2 thru 0x0209f1 (0xdb20 bytes) 000063 1786003976.378603 mapwrite 0x012a70 thru 0x01c855 (0x9de6 bytes) 000064 1786003976.479237 mapread 0x00e08b thru 0x01c447 (0xe3bd bytes) 000065 1786003976.480421 mapread 0x00b6ca thru 0x018afd (0xd434 bytes) 000067 1786003976.482694 read 0x0065f6 thru 0x007b0b (0x1516 bytes) 000068 1786003976.483565 trunc from 0x0209f2 to 0x009ed2 (0x16b1f bytes) 000070 1786003976.505378 trunc from 0x009ed3 to 0x02cdce (0x22efc bytes) 000073 1786003976.571492 write 0x02a996 thru 0x034d73 (0xa3de bytes) 000074 1786003976.596543 mapwrite 0x017b2e thru 0x02783b (0xfd0e bytes) 000077 1786003976.810306 mapwrite 0x00ca57 thru 0x01bc00 (0xf1aa bytes) 000078 1786003976.870019 MIRROR_RESYNC 2 mirrors 000079 1786003977.569668 mapread 0x015268 thru 0x01bd5d (0x6af6 bytes) 000081 1786003977.630263 read 0x025ce7 thru 0x02e41b (0x8735 bytes) 000083 1786003977.646281 read 0x024ca8 thru 0x02fa9a (0xadf3 bytes) 000084 1786003977.657211 read 0x005abd thru 0x00f81b (0x9d5f bytes) 000085 1786003977.662582 trunc from 0x034d74 to 0x030ca2 (0x40d1 bytes) 000086 1786003977.893643 trunc from 0x030ca3 to 0x023cff (0xcfa3 bytes) 000087 1786003977.943860 read 0x00e0d9 thru 0x00fb28 (0x1a50 bytes) 000088 1786003977.960132 read 0x017191 thru 0x018698 (0x1508 bytes) 000093 1786003977.974476 read 0x011390 thru 0x01d987 (0xc5f8 bytes) 000094 1786003977.976802 mapread 0x0105dc thru 0x01913b (0x8b60 bytes) 000095 1786003978.005182 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000096 1786003978.164433 mapread 0x01cc8f thru 0x022877 (0x5be9 bytes) 000097 1786003978.182782 mapread 0x00a11c thru 0x013517 (0x93fc bytes) 000098 1786003978.203985 write 0x039d53 thru 0x03ffff (0x62ad bytes) 000100 1786003978.208273 mapread 0x012776 thru 0x01cb54 (0xa3df bytes) 000105 1786003978.213838 mapwrite 0x02168e thru 0x02f3fe (0xdd71 bytes) 000106 1786003978.290915 trunc from 0x040000 to 0x001ccc (0x3e333 bytes) 000108 1786003978.313320 trunc from 0x001ccd to 0x00f154 (0xd488 bytes) 000110 1786003978.329269 trunc from 0x00f155 to 0x02fc4c (0x20af8 bytes) 000111 1786003978.343241 write 0x0287fa thru 0x0316b0 (0x8eb7 bytes) 000117 1786003978.355283 read 0x00f5f0 thru 0x01cc05 (0xd616 bytes) 000118 1786003978.364272 read 0x007cef thru 0x014817 (0xcb29 bytes) 000120 1786003978.376929 mapwrite 0x00a7b4 thru 0x00a9f9 (0x246 bytes) 000121 1786003978.424566 write 0x03d9d8 thru 0x03ffff (0x2628 bytes) 000122 1786003978.426913 mapwrite 0x0184c4 thru 0x027465 (0xefa2 bytes) 000123 1786003978.497189 read 0x02a218 thru 0x02fda9 (0x5b92 bytes) 000125 1786003978.502181 read 0x015688 thru 0x01f352 (0x9ccb bytes) 000127 1786003978.503630 read 0x00ffb7 thru 0x01d7bb (0xd805 bytes) 000130 1786003978.507242 write 0x036c0c thru 0x03cd67 (0x615c bytes) 000131 1786003978.520623 mapread 0x0039d1 thru 0x009104 (0x5734 bytes) 000137 1786003978.529006 trunc from 0x040000 to 0x01125a (0x2eda5 bytes) 000139 1786003978.557752 mapread 0x0021d6 thru 0x0024e9 (0x314 bytes) 000140 1786003978.559545 mapread 0x004f2e thru 0x01125a (0xc32d bytes) 000141 1786003978.562776 trunc from 0x01125b to 0x01cdf2 (0xbb98 bytes) 000143 1786003978.596009 mapread 0x0044ab thru 0x012985 (0xe4db bytes) 000144 1786003978.606992 write 0x023c9f thru 0x02402b (0x38d bytes) 000145 1786003978.609747 mapread 0x0107bf thru 0x0151b2 (0x49f4 bytes) 000146 1786003978.638155 mapread 0x008a44 thru 0x015b34 (0xd0f1 bytes) 000148 1786003978.647291 mapread 0x01b83b thru 0x02402b (0x87f1 bytes) 000152 1786003978.728022 read 0x003b4b thru 0x0108b2 (0xcd68 bytes) truncating to largest ever: 0x3dd4f 000154 1786003978.732622 trunc from 0x02402c to 0x03dd4e (0x19d23 bytes) 000156 1786003978.750010 mapwrite 0x03041f thru 0x033645 (0x3227 bytes) 000158 1786003978.828807 read 0x03ad06 thru 0x03dd4e (0x3049 bytes) 000159 1786003978.830273 trunc from 0x03dd4f to 0x0009d8 (0x3d376 bytes) 000160 1786003978.866846 mapread 0x0005a0 thru 0x0009d8 (0x439 bytes) 000161 1786003978.867931 mapwrite 0x01b470 thru 0x02790e (0xc49f bytes) 000162 1786003978.941479 write 0x01d086 thru 0x02245d (0x53d8 bytes) 000163 1786003978.943006 write 0x01d740 thru 0x02a4c9 (0xcd8a bytes) 000167 1786003978.948250 read 0x01246d thru 0x01e864 (0xc3f8 bytes) 000168 1786003978.949129 mapread 0x0221cf thru 0x02a4c9 (0x82fb bytes) 000169 1786003978.950508 mapwrite 0x03edc1 thru 0x03ffff (0x123f bytes) 000170 1786003979.034731 mapread 0x002a2f thru 0x00b26c (0x883e bytes) 000172 1786003979.036386 mapread 0x01f6f9 thru 0x029053 (0x995b bytes) 000174 1786003979.045983 mapread 0x01c7c0 thru 0x0283ae (0xbbef bytes) 000178 1786003979.056362 mapwrite 0x016547 thru 0x018403 (0x1ebd bytes) 000180 1786003979.154589 read 0x02538b thru 0x025fa4 (0xc1a bytes) 000181 1786003979.155250 write 0x00585e thru 0x009638 (0x3ddb bytes) 000183 1786003979.158180 trunc from 0x040000 to 0x00cb73 (0x3348c bytes) 000188 1786003979.197947 trunc from 0x00cb74 to 0x006e42 (0x5d31 bytes) 000190 1786003979.221589 trunc from 0x006e43 to 0x019ce1 (0x12e9f bytes) 000191 1786003979.249242 write 0x01068f thru 0x011125 (0xa97 bytes) 000193 1786003979.272345 mapwrite 0x01443f thru 0x016b6d (0x272f bytes) 000196 1786003979.389007 write 0x0129ab thru 0x01b1e2 (0x8838 bytes) 000198 1786003979.407232 MIRROR_EXTEND to 2 mirrors 000200 1786003980.238444 mapwrite 0x027030 thru 0x0337a8 (0xc779 bytes) 000203 1786003980.387557 trunc from 0x0337a9 to 0x01a77e (0x1902a bytes) 000209 1786003980.407458 write 0x00cce1 thru 0x00fd38 (0x3058 bytes) 000210 1786003980.419513 mapread 0x00463d thru 0x00892f (0x42f3 bytes) 000211 1786003980.426350 read 0x00273e thru 0x00ec3f (0xc502 bytes) 000212 1786003980.434977 write 0x01b75f thru 0x024d64 (0x9606 bytes) 000215 1786003980.439015 trunc from 0x024d65 to 0x02303d (0x1d27 bytes) 000216 1786003980.454009 write 0x013752 thru 0x020abe (0xd36d bytes) 000217 1786003980.461168 trunc from 0x02303e to 0x01f3e0 (0x3c5d bytes) 000218 1786003980.476738 trunc from 0x01f3e1 to 0x0334da (0x140fa bytes) 000221 1786003980.495738 mapread 0x020a1e thru 0x029ea5 (0x9488 bytes) 000222 1786003980.511147 mapread 0x02d4ef thru 0x031050 (0x3b62 bytes) 000224 1786003980.513532 write 0x00f53d thru 0x014f33 (0x59f7 bytes) 000227 1786003980.516800 write 0x0084fa thru 0x009715 (0x121c bytes) 000228 1786003980.518143 write 0x000bff thru 0x0015a8 (0x9aa bytes) 000229 1786003980.522836 write 0x039855 thru 0x03bb65 (0x2311 bytes) 000230 1786003980.523850 mapread 0x038ac8 thru 0x03bb65 (0x309e bytes) 000231 1786003980.534291 MIRROR_RESYNC 2 mirrors 000232 1786003981.458341 mapread 0x0320ce thru 0x03b20a (0x913d bytes) 000234 1786003981.475485 trunc from 0x03bb66 to 0x038c80 (0x2ee5 bytes) 000236 1786003981.532428 write 0x029ab9 thru 0x02ce9b (0x33e3 bytes) 000237 1786003981.541358 read 0x02b0fb thru 0x038c80 (0xdb86 bytes) 000238 1786003981.551462 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000240 1786003981.778156 write 0x020c78 thru 0x021498 (0x821 bytes) truncating to largest ever: 0x3dfc5 000241 1786003981.795116 trunc from 0x038c81 to 0x03dfc4 (0x5344 bytes) 000242 1786003981.848307 mapwrite 0x0321d3 thru 0x0359d3 (0x3801 bytes) 000244 1786003982.084548 read 0x02066e thru 0x030478 (0xfe0b bytes) 000247 1786003982.110006 trunc from 0x03dfc5 to 0x013143 (0x2ae81 bytes) 000248 1786003982.125067 mapwrite 0x01e963 thru 0x0244cf (0x5b6d bytes) 000250 1786003982.258966 trunc from 0x0244d0 to 0x02b599 (0x70ca bytes) 000251 1786003982.312691 write 0x03e899 thru 0x03ffff (0x1767 bytes) 000252 1786003982.317176 mapwrite 0x00b680 thru 0x00ec91 (0x3612 bytes) 000253 1786003982.497650 write 0x00397c thru 0x00c299 (0x891e bytes) 000256 1786003982.512212 MIRROR_EXTEND to 2 mirrors 000259 1786003983.713433 mapwrite 0x03e4de thru 0x03ffff (0x1b22 bytes) 000260 1786003984.017303 read 0x031a47 thru 0x03fa0c (0xdfc6 bytes) 000261 1786003984.023665 mapread 0x03b98d thru 0x03ffff (0x4673 bytes) 000263 1786003984.025727 trunc from 0x040000 to 0x026346 (0x19cb9 bytes) 000267 1786003984.055013 write 0x00bf76 thru 0x01097c (0x4a07 bytes) 000268 1786003984.078451 write 0x02805a thru 0x0351e3 (0xd18a bytes) 000269 1786003984.081526 trunc from 0x0351e4 to 0x0247d2 (0x10a11 bytes) 000270 1786003984.121610 mapread 0x0195d0 thru 0x0198de (0x30f bytes) 000272 1786003984.135959 mapwrite 0x02281c thru 0x02fac9 (0xd2ae bytes) 000273 1786003984.349279 read 0x00b26a thru 0x018387 (0xd11e bytes) 000274 1786003984.366114 mapwrite 0x01bca5 thru 0x01f425 (0x3781 bytes) 000277 1786003984.492717 mapread 0x02990a thru 0x02fac9 (0x61c0 bytes) 000280 1786003984.498547 mapread 0x004c69 thru 0x011c5b (0xcff3 bytes) 000281 1786003984.501499 trunc from 0x02faca to 0x039ad3 (0xa00a bytes) 000284 1786003984.542703 mapwrite 0x021231 thru 0x027a44 (0x6814 bytes) 000286 1786003984.710611 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000287 1786003985.000518 mapwrite 0x0066fe thru 0x015db0 (0xf6b3 bytes) 000288 1786003985.188307 MIRROR_EXTEND to 2 mirrors 000289 1786003987.547321 trunc from 0x039ad4 to 0x031ff7 (0x7adc bytes) 000290 1786003987.742012 trunc from 0x031ff8 to 0x0316a9 (0x94e bytes) 000292 1786003987.819534 write 0x0093eb thru 0x00e45e (0x5074 bytes) 000293 1786003987.853640 read 0x029e6b thru 0x0316a9 (0x783f bytes) 000294 1786003987.886632 mapread 0x002360 thru 0x0115ad (0xf24e bytes) 000295 1786003987.992840 MIRROR_RESYNC 2 mirrors 000298 1786003989.990825 write 0x004331 thru 0x00a847 (0x6517 bytes) 000301 1786003990.229593 trunc from 0x0316aa to 0x00636b (0x2b33e bytes) 000302 1786003990.326009 write 0x02d4d2 thru 0x036b45 (0x9674 bytes) 000303 1786003990.327634 read 0x0217b4 thru 0x02791b (0x6168 bytes) 000305 1786003990.339382 mapwrite 0x020384 thru 0x023543 (0x31c0 bytes) 000306 1786003990.471745 write 0x016e02 thru 0x01ee11 (0x8010 bytes) 000311 1786003990.497416 trunc from 0x036b46 to 0x01816b (0x1e9da bytes) 000313 1786003990.550301 write 0x002c52 thru 0x0075b8 (0x4967 bytes) 000315 1786003990.613736 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1816c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1816c): Mirror 2: 0x452bcf72 (stale) CRC-32 checksum value for chunk [0, 0x1816c): Mirror 1: 0x787564c2 Mirror 3: 0x787564c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1816c: skipped 000317 1786003992.576387 trunc from 0x01816c to 0x0236f1 (0xb586 bytes) 000318 1786003992.872966 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x236f2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x236f2): Mirror 2: 0xd1c4cbc4 (stale) Mirror 3: 0x60ae73dd (stale) CRC-32 checksum value for chunk [0, 0x236f2): Mirror 1: 0x60ae73dd Mirror 4: 0x60ae73dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x236f2: skipped 000319 1786003995.348034 mapread 0x00497c thru 0x009d43 (0x53c8 bytes) 000320 1786003995.446544 trunc from 0x0236f2 to 0x030c8d (0xd59c bytes) 000321 1786003995.634670 mapread 0x006d8a thru 0x0085f3 (0x186a bytes) 000322 1786003995.646002 write 0x0236b6 thru 0x02afe4 (0x792f bytes) 000324 1786003995.658236 mapread 0x01fd85 thru 0x024029 (0x42a5 bytes) 000325 1786003995.672421 mapread 0x025481 thru 0x029f50 (0x4ad0 bytes) 000326 1786003995.679905 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 2: 0x4ecffba3 (stale) Mirror 3: 0x844c6a23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 2: 0x4ecffba3 (stale) Mirror 3: 0x844c6a23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c8e: skipped 000327 1786003996.302278 read 0x005a21 thru 0x013b89 (0xe169 bytes) 000328 1786003996.362120 mapread 0x02bb08 thru 0x030104 (0x45fd bytes) 000329 1786003996.450962 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 2: 0x4ecffba3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c8e): Mirror 2: 0x4ecffba3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c8e: skipped 000331 1786003997.098962 mapread 0x016490 thru 0x0190d4 (0x2c45 bytes) 000332 1786003997.187633 trunc from 0x030c8e to 0x036f11 (0x6284 bytes) 000334 1786003997.219018 read 0x01bc59 thru 0x02497e (0x8d26 bytes) 000336 1786003997.255849 write 0x001253 thru 0x003f5a (0x2d08 bytes) 000337 1786003997.279121 trunc from 0x036f12 to 0x008ba6 (0x2e36b bytes) 000339 1786003997.352507 MIRROR_RESYNC 2 mirrors 000340 1786003998.977571 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000342 1786003999.223891 read 0x007438 thru 0x008ba6 (0x176f bytes) 000345 1786003999.233381 read 0x001d86 thru 0x008ba6 (0x6e21 bytes) 000346 1786003999.252759 mapread 0x002100 thru 0x006a0f (0x4910 bytes) 000347 1786003999.253879 read 0x005f03 thru 0x00702f (0x112d bytes) 000350 1786003999.263455 trunc from 0x008ba7 to 0x008807 (0x39f bytes) 000352 1786003999.321518 MIRROR_EXTEND to 2 mirrors 000353 1786004000.799078 write 0x00798e thru 0x01674d (0xedc0 bytes) 000354 1786004000.917376 mapwrite 0x028bf3 thru 0x038533 (0xf941 bytes) 000355 1786004001.216791 mapwrite 0x03061d thru 0x036b89 (0x656d bytes) 000356 1786004001.391998 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000357 1786004001.625890 mapread 0x01dff3 thru 0x022bff (0x4c0d bytes) 000358 1786004001.684643 mapwrite 0x003a66 thru 0x013a31 (0xffcc bytes) 000360 1786004001.895498 mapread 0x020d41 thru 0x029a79 (0x8d39 bytes) 000362 1786004001.914879 trunc from 0x038534 to 0x02f377 (0x91bc bytes) 000363 1786004001.958373 trunc from 0x02f378 to 0x02ecb4 (0x6c3 bytes) 000365 1786004002.002874 write 0x023c1f thru 0x02a77f (0x6b61 bytes) 000367 1786004002.009363 read 0x01a7c3 thru 0x01f66b (0x4ea9 bytes) 000368 1786004002.010173 read 0x010611 thru 0x0142e4 (0x3cd4 bytes) 000369 1786004002.018867 mapread 0x006be0 thru 0x0095d4 (0x29f5 bytes) 000373 1786004002.030339 read 0x00b6a4 thru 0x00dfaa (0x2907 bytes) 000375 1786004002.033157 trunc from 0x02ecb5 to 0x01be80 (0x12e34 bytes) 000377 1786004002.074764 mapwrite 0x012502 thru 0x014503 (0x2002 bytes) 000379 1786004002.280826 mapwrite 0x033217 thru 0x03ffff (0xcde9 bytes) 000382 1786004002.509895 mapread 0x035df1 thru 0x03ffff (0xa20f bytes) 000383 1786004002.511179 write 0x0075d5 thru 0x00dcd4 (0x6700 bytes) 000385 1786004002.526878 mapread 0x0093ba thru 0x00f180 (0x5dc7 bytes) 000386 1786004002.536072 write 0x018d6a thru 0x026673 (0xd90a bytes) 000387 1786004002.570492 MIRROR_EXTEND to 2 mirrors 000389 1786004004.810005 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000391 1786004005.013879 write 0x00709b thru 0x01643b (0xf3a1 bytes) 000392 1786004005.054985 write 0x026d75 thru 0x02d2b2 (0x653e bytes) 000393 1786004005.077191 MIRROR_EXTEND to 2 mirrors 000395 1786004006.395921 mapread 0x016a52 thru 0x019713 (0x2cc2 bytes) 000397 1786004006.472594 trunc from 0x040000 to 0x025393 (0x1ac6c bytes) 000398 1786004006.684557 write 0x00bbda thru 0x00fe07 (0x422e bytes) 000400 1786004006.708213 trunc from 0x025394 to 0x0380cc (0x12d39 bytes) 000401 1786004006.742887 read 0x0140d5 thru 0x018222 (0x414e bytes) 000402 1786004006.763423 read 0x032e4e thru 0x033767 (0x91a bytes) 000403 1786004006.775645 MIRROR_RESYNC 2 mirrors 000404 1786004008.337796 mapread 0x00da7c thru 0x00f0c0 (0x1645 bytes) 000407 1786004008.397326 mapwrite 0x0154a8 thru 0x0241af (0xed08 bytes) 000411 1786004008.740166 read 0x006e41 thru 0x00e164 (0x7324 bytes) 000412 1786004008.768989 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x380cd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380cd): Mirror 2: 0x12554c1b (stale) CRC-32 checksum value for chunk [0, 0x380cd): Mirror 1: 0xff615774 Mirror 3: 0xff615774 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380cd: skipped 000415 1786004010.927049 mapread 0x024146 thru 0x02f3ea (0xb2a5 bytes) 000416 1786004011.026621 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x380cd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380cd): Mirror 2: 0x12554c1b (stale) CRC-32 checksum value for chunk [0, 0x380cd): Mirror 1: 0xff615774 Mirror 3: 0xff615774 Mirror 4: 0xff615774 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380cd: skipped 000417 1786004013.093323 mapwrite 0x007b67 thru 0x011617 (0x9ab1 bytes) 000419 1786004013.438607 write 0x0043b5 thru 0x0124fd (0xe149 bytes) 000420 1786004013.441756 mapread 0x00510a thru 0x00c796 (0x768d bytes) 000421 1786004013.451037 mapread 0x00529e thru 0x01490a (0xf66d bytes) 000422 1786004013.452134 mapwrite 0x03dd67 thru 0x03f3cb (0x1665 bytes) 000423 1786004013.693775 mapwrite 0x0115f8 thru 0x020547 (0xef50 bytes) 000424 1786004013.884878 write 0x0093ab thru 0x00f34e (0x5fa4 bytes) 000427 1786004013.894502 mapwrite 0x02961b thru 0x036ace (0xd4b4 bytes) 000428 1786004014.087903 write 0x01d9fb thru 0x01e08b (0x691 bytes) 000430 1786004014.091628 write 0x01cfa8 thru 0x02206a (0x50c3 bytes) 000431 1786004014.094491 trunc from 0x03f3cc to 0x032b1d (0xc8ae bytes) 000432 1786004014.119143 write 0x02ba50 thru 0x0306be (0x4c6f bytes) 000433 1786004014.120349 mapread 0x025bb7 thru 0x0321e5 (0xc62f bytes) 000434 1786004014.130920 mapwrite 0x032e50 thru 0x0380df (0x5290 bytes) 000435 1786004014.356226 read 0x01fe73 thru 0x021aa6 (0x1c34 bytes) 000436 1786004014.356875 mapwrite 0x0189f2 thru 0x01be63 (0x3472 bytes) 000439 1786004014.453266 mapread 0x02d2b5 thru 0x0380df (0xae2b bytes) 000440 1786004014.454904 mapwrite 0x026714 thru 0x02bd55 (0x5642 bytes) 000442 1786004014.587855 trunc from 0x0380e0 to 0x016755 (0x2198a bytes) 000448 1786004014.639965 trunc from 0x016756 to 0x03bc83 (0x2552e bytes) 000449 1786004014.686283 read 0x024f9f thru 0x027459 (0x24bb bytes) 000450 1786004014.711145 write 0x00efdd thru 0x0162a8 (0x72cc bytes) 000451 1786004014.732247 MIRROR_RESYNC 4 mirrors 000452 1786004017.047237 read 0x0232d6 thru 0x0247eb (0x1516 bytes) 000454 1786004017.054032 read 0x00a849 thru 0x00b033 (0x7eb bytes) 000455 1786004017.062034 read 0x0084e3 thru 0x0177f0 (0xf30e bytes) 000457 1786004017.065612 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000459 1786004017.329546 mapwrite 0x00901d thru 0x0171a7 (0xe18b bytes) 000460 1786004017.523544 mapread 0x0113ab thru 0x017f8c (0x6be2 bytes) 000463 1786004017.525465 read 0x03094e thru 0x03604e (0x5701 bytes) 000464 1786004017.578618 mapread 0x011785 thru 0x013008 (0x1884 bytes) 000466 1786004017.585196 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bc84): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc84): Mirror 2: 0x5fba3589 (stale) Mirror 4: 0x5fba3589 (stale) CRC-32 checksum value for chunk [0, 0x3bc84): Mirror 1: 0x9c3ad055 Mirror 5: 0x9c3ad055 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc84: skipped 000468 1786004019.143842 read 0x02ec78 thru 0x03bc83 (0xd00c bytes) 000469 1786004019.175294 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bc84): Mirror 2: 0x5fba3589 (stale) Mirror 4: 0x5fba3589 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc84): Mirror 2: 0x5fba3589 (stale) Mirror 4: 0x5fba3589 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc84: skipped 000471 1786004019.858039 read 0x009f5c thru 0x011368 (0x740d bytes) 000472 1786004019.866141 mapwrite 0x025d5a thru 0x02ad2d (0x4fd4 bytes) 000480 1786004020.150929 mapread 0x02acab thru 0x03592c (0xac82 bytes) 000483 1786004020.174717 write 0x01fd3d thru 0x02bf08 (0xc1cc bytes) 000484 1786004020.184369 write 0x021caa thru 0x027451 (0x57a8 bytes) 000485 1786004020.191064 mapwrite 0x017db7 thru 0x01faef (0x7d39 bytes) 000486 1786004020.312842 read 0x01d3de thru 0x021053 (0x3c76 bytes) 000488 1786004020.314727 write 0x03d587 thru 0x03e895 (0x130f bytes) 000489 1786004020.316390 mapwrite 0x01684f thru 0x023cbb (0xd46d bytes) truncating to largest ever: 0x3ed42 000491 1786004020.390671 trunc from 0x03e896 to 0x03ed41 (0x4ac bytes) 000492 1786004020.408751 read 0x01ddef thru 0x028282 (0xa494 bytes) 000493 1786004020.410030 write 0x00c8a8 thru 0x01a2e5 (0xda3e bytes) 000494 1786004020.421992 trunc from 0x03ed42 to 0x036341 (0x8a00 bytes) 000495 1786004020.441057 mapread 0x03154e thru 0x036341 (0x4df4 bytes) 000496 1786004020.443378 read 0x02c722 thru 0x02d54a (0xe29 bytes) 000498 1786004020.450785 MIRROR_RESYNC 3 mirrors 000499 1786004021.637748 MIRROR_EXTEND to 4 mirrors 000500 1786004022.527440 write 0x035722 thru 0x0382e4 (0x2bc3 bytes) 000501 1786004022.782426 mapread 0x025129 thru 0x02ba2c (0x6904 bytes) 000504 1786004022.811419 MIRROR_RESYNC 4 mirrors 000506 1786004026.246333 mapread 0x03101c thru 0x0316fd (0x6e2 bytes) 000508 1786004026.282400 read 0x0095fc thru 0x01255b (0x8f60 bytes) 000509 1786004026.368032 mapwrite 0x001ae2 thru 0x00f3cb (0xd8ea bytes) 000514 1786004027.129205 trunc from 0x0382e5 to 0x011eee (0x263f6 bytes) 000515 1786004027.182494 trunc from 0x011eef to 0x01abe6 (0x8cf8 bytes) 000518 1786004027.220290 read 0x019001 thru 0x01abe6 (0x1be6 bytes) 000520 1786004027.268337 trunc from 0x01abe7 to 0x03dc45 (0x2305f bytes) 000521 1786004027.307854 mapread 0x00e578 thru 0x01834b (0x9dd4 bytes) 000522 1786004027.313564 write 0x02f7fe thru 0x03721c (0x7a1f bytes) 000523 1786004027.363394 mapwrite 0x00b2a5 thru 0x00c230 (0xf8c bytes) 000525 1786004027.585699 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3dc46): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc46): Mirror 2: 0xa6e306d6 (stale) Mirror 4: 0xa6e306d6 (stale) Mirror 5: 0xa6e306d6 (stale) CRC-32 checksum value for chunk [0, 0x3dc46): Mirror 1: 0x46adf895 Mirror 6: 0x46adf895 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc46: skipped 000526 1786004029.391459 mapwrite 0x022738 thru 0x028767 (0x6030 bytes) 000530 1786004029.710334 mapwrite 0x0309fd thru 0x03c5e2 (0xbbe6 bytes) 000531 1786004029.884313 trunc from 0x03dc46 to 0x0091e2 (0x34a63 bytes) 000532 1786004029.925662 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x91e3): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x91e3): Mirror 2: 0xa95896bd (stale) Mirror 4: 0xa95896bd (stale) Mirror 5: 0xa95896bd (stale) Mirror 6: 0x5693dd42 (stale) CRC-32 checksum value for chunk [0, 0x91e3): Mirror 1: 0x5693dd42 Mirror 7: 0x5693dd42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x91e3: skipped 000533 1786004031.500831 trunc from 0x0091e3 to 0x00f42e (0x624c bytes) 000534 1786004031.584906 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf42f): Mirror 2: 0xbdf8f301 (stale) Mirror 4: 0xbdf8f301 (stale) Mirror 5: 0xbdf8f301 (stale) Mirror 6: 0x286c57d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf42f): Mirror 2: 0xbdf8f301 (stale) Mirror 4: 0xbdf8f301 (stale) Mirror 5: 0xbdf8f301 (stale) Mirror 6: 0x286c57d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf42f: skipped 000535 1786004031.821621 trunc from 0x00f42f to 0x00d5c6 (0x1e68 bytes) 000536 1786004031.836247 mapread 0x00c850 thru 0x00d5c6 (0xd77 bytes) 000537 1786004031.843571 write 0x036b5a thru 0x03aa73 (0x3f1a bytes) 000539 1786004031.845862 trunc from 0x03aa74 to 0x01d4df (0x1d594 bytes) 000543 1786004031.862429 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1d4e0): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d4e0): Mirror 2: 0x653f690 (stale) Mirror 4: 0x653f690 (stale) Mirror 5: 0x653f690 (stale) Mirror 6: 0xb57746e4 (stale) CRC-32 checksum value for chunk [0, 0x1d4e0): Mirror 1: 0xa9a645cc Mirror 7: 0xa9a645cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d4e0: skipped 000544 1786004032.788303 write 0x02a078 thru 0x02f8e3 (0x586c bytes) 000545 1786004032.854264 trunc from 0x02f8e4 to 0x0396d9 (0x9df6 bytes) 000546 1786004032.875126 mapwrite 0x03fa46 thru 0x03ffff (0x5ba bytes) 000548 1786004032.930027 mapread 0x02627d thru 0x02c1b0 (0x5f34 bytes) 000550 1786004032.941689 mapwrite 0x01e2a4 thru 0x028a3c (0xa799 bytes) 000551 1786004032.991331 mapwrite 0x03ebd9 thru 0x03ffff (0x1427 bytes) 000552 1786004033.025702 read 0x022775 thru 0x0234e5 (0xd71 bytes) 000556 1786004033.029472 mapwrite 0x03f982 thru 0x03ffff (0x67e bytes) 000557 1786004033.060841 read 0x0267bc thru 0x030ff3 (0xa838 bytes) 000559 1786004033.108344 read 0x036dde thru 0x03802c (0x124f bytes) 000560 1786004033.109002 trunc from 0x040000 to 0x00e9c3 (0x3163c bytes) 000561 1786004033.177800 read 0x0054a1 thru 0x00a3e0 (0x4f40 bytes) 000562 1786004033.180941 mapread 0x006f39 thru 0x00c320 (0x53e8 bytes) 000564 1786004033.201851 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xe9c4): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe9c4): Mirror 2: 0x860b101f (stale) Mirror 4: 0x860b101f (stale) Mirror 5: 0x860b101f (stale) Mirror 6: 0xf05395c6 (stale) Mirror 7: 0x106b6829 (stale) CRC-32 checksum value for chunk [0, 0xe9c4): Mirror 1: 0x106b6829 Mirror 8: 0x106b6829 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe9c4: skipped 000566 1786004035.779618 trunc from 0x00e9c4 to 0x02f2f0 (0x2092d bytes) 000569 1786004036.058158 trunc from 0x02f2f1 to 0x02bc0e (0x36e2 bytes) 000571 1786004036.095320 write 0x010987 thru 0x012904 (0x1f7e bytes) 000572 1786004036.117264 trunc from 0x02bc0f to 0x0336e6 (0x7ad8 bytes) 000573 1786004036.182561 write 0x02ab94 thru 0x02ead2 (0x3f3f bytes) 000574 1786004036.190405 read 0x00a649 thru 0x0142a1 (0x9c59 bytes) 000576 1786004036.206823 read 0x00505e thru 0x011f92 (0xcf35 bytes) 000577 1786004036.226507 read 0x00b1be thru 0x00bd05 (0xb48 bytes) 000581 1786004036.238645 mapread 0x01765f thru 0x01c468 (0x4e0a bytes) 000582 1786004036.250160 mapwrite 0x0213c9 thru 0x025b5d (0x4795 bytes) 000584 1786004036.427588 MIRROR_RESYNC 7 mirrors 000585 1786004042.044655 mapwrite 0x03c633 thru 0x03d7d9 (0x11a7 bytes) 000586 1786004042.305806 MIRROR_RESYNC 7 mirrors 000587 1786004048.451875 trunc from 0x03d7da to 0x020088 (0x1d751 bytes) 000588 1786004048.620847 trunc from 0x020089 to 0x00fe03 (0x10285 bytes) 000589 1786004048.655524 mapwrite 0x0058c7 thru 0x013f31 (0xe66b bytes) 000590 1786004048.788281 write 0x030450 thru 0x03ba7b (0xb62c bytes) 000591 1786004048.794493 mapread 0x000f42 thru 0x005ebd (0x4f7c bytes) 000593 1786004048.829733 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba7c): Mirror 2: 0x4ecd2e2e (stale) Mirror 4: 0x4ecd2e2e (stale) Mirror 5: 0x4ecd2e2e (stale) Mirror 7: 0x4ecd2e2e (stale) Mirror 8: 0x4ecd2e2e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba7c): Mirror 2: 0x4ecd2e2e (stale) Mirror 4: 0x4ecd2e2e (stale) Mirror 5: 0x4ecd2e2e (stale) Mirror 7: 0x4ecd2e2e (stale) Mirror 8: 0x4ecd2e2e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba7c: skipped 000597 1786004049.334874 read 0x0033c7 thru 0x00759c (0x41d6 bytes) 000599 1786004049.344505 mapwrite 0x020958 thru 0x02926b (0x8914 bytes) 000600 1786004049.431514 write 0x0062ac thru 0x0123ed (0xc142 bytes) 000601 1786004049.438410 trunc from 0x03ba7c to 0x0211d9 (0x1a8a2 bytes) 000602 1786004049.474764 write 0x03b631 thru 0x03ffff (0x49cf bytes) 000603 1786004049.476053 trunc from 0x040000 to 0x024ae3 (0x1b51c bytes) 000604 1786004049.504387 mapwrite 0x025897 thru 0x02ed41 (0x94ab bytes) 000605 1786004049.716851 mapread 0x029b7a thru 0x02e3e2 (0x4869 bytes) 000606 1786004049.720558 read 0x018a17 thru 0x021d8c (0x9376 bytes) 000607 1786004049.732167 read 0x0105bc thru 0x0188e1 (0x8326 bytes) 000608 1786004049.733867 trunc from 0x02ed42 to 0x004610 (0x2a731 bytes) 000610 1786004049.751533 mapwrite 0x01b7f9 thru 0x021e25 (0x662d bytes) 000611 1786004049.840475 mapwrite 0x02b637 thru 0x02e19e (0x2b68 bytes) 000613 1786004049.932080 write 0x0142ad thru 0x0219d5 (0xd729 bytes) 000614 1786004049.934124 read 0x02a013 thru 0x02e19e (0x418c bytes) 000615 1786004049.934985 mapread 0x0232d2 thru 0x02d843 (0xa572 bytes) 000616 1786004049.936558 write 0x01a95c thru 0x02121c (0x68c1 bytes) 000617 1786004049.938106 mapwrite 0x035ee6 thru 0x03ffff (0xa11a bytes) 000618 1786004050.089047 mapread 0x0047b6 thru 0x01310d (0xe958 bytes) 000620 1786004050.105661 MIRROR_RESYNC 6 mirrors 000621 1786004052.776857 mapwrite 0x0091cc thru 0x00feec (0x6d21 bytes) 000622 1786004053.114820 trunc from 0x040000 to 0x024b46 (0x1b4b9 bytes) 000623 1786004053.167235 trunc from 0x024b47 to 0x01bb47 (0x8fff bytes) 000624 1786004053.193966 mapwrite 0x01b473 thru 0x01dc34 (0x27c2 bytes) 000625 1786004053.291326 trunc from 0x01dc35 to 0x00d28c (0x109a8 bytes) 000626 1786004053.304446 mapread 0x001cd0 thru 0x00ceb9 (0xb1ea bytes) 000627 1786004053.310436 write 0x02372c thru 0x028e57 (0x572c bytes) 000628 1786004053.312067 mapread 0x01738e thru 0x01f16d (0x7de0 bytes) 000629 1786004053.320799 mapwrite 0x0151e7 thru 0x01b63b (0x6455 bytes) 000630 1786004053.478881 MIRROR_RESYNC 6 mirrors 000633 1786004056.046985 trunc from 0x028e58 to 0x01a97a (0xe4dd bytes) 000638 1786004056.136613 trunc from 0x01a97b to 0x03afc4 (0x2064a bytes) 000639 1786004056.148713 mapwrite 0x01ab22 thru 0x01d332 (0x2811 bytes) 000640 1786004056.356696 mapread 0x01952f thru 0x027e54 (0xe926 bytes) 000641 1786004056.386986 write 0x002c4e thru 0x0126cd (0xfa80 bytes) 000643 1786004056.444077 trunc from 0x03afc5 to 0x012735 (0x2888f bytes) 000644 1786004056.530505 mapwrite 0x016fcb thru 0x01a048 (0x307e bytes) 000645 1786004056.730689 write 0x034deb thru 0x03ffff (0xb215 bytes) 000647 1786004056.744500 mapwrite 0x008d2f thru 0x0160ec (0xd3be bytes) 000648 1786004056.982385 trunc from 0x040000 to 0x01baf4 (0x2450b bytes) 000649 1786004057.059385 write 0x02b386 thru 0x0334fd (0x8178 bytes) 000650 1786004057.067427 trunc from 0x0334fe to 0x032c6c (0x891 bytes) 000651 1786004057.148755 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32c6d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c6d): Mirror 2: 0xc0fc5c62 (stale) Mirror 4: 0xc0fc5c62 (stale) Mirror 5: 0xc0fc5c62 (stale) Mirror 7: 0xc0fc5c62 (stale) Mirror 8: 0xc0fc5c62 (stale) CRC-32 checksum value for chunk [0, 0x32c6d): Mirror 1: 0xd96fcc3b Mirror 9: 0xd96fcc3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c6d: skipped 000652 1786004060.042953 MIRROR_RESYNC 7 mirrors 000654 1786004063.025905 trunc from 0x032c6d to 0x02fe85 (0x2de7 bytes) 000657 1786004063.159374 write 0x013c27 thru 0x01d7aa (0x9b84 bytes) 000661 1786004063.207798 write 0x006718 thru 0x009c89 (0x3572 bytes) 000663 1786004063.231880 write 0x02bbfa thru 0x037558 (0xb95f bytes) 000666 1786004063.253373 trunc from 0x037559 to 0x00d7ac (0x29dac bytes) 000667 1786004063.289373 write 0x03b1fc thru 0x03bc95 (0xa9a bytes) 000668 1786004063.323246 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3bc96): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc96): Mirror 2: 0x6c876d4a (stale) Mirror 4: 0x6c876d4a (stale) Mirror 5: 0x6c876d4a (stale) Mirror 7: 0x6c876d4a (stale) Mirror 8: 0x6c876d4a (stale) Mirror 9: 0x6c876d4a (stale) CRC-32 checksum value for chunk [0, 0x3bc96): Mirror 1: 0x8a378f0c Mirror 10: 0x8a378f0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc96: skipped 000669 1786004065.067937 read 0x0136f5 thru 0x018140 (0x4a4c bytes) 000670 1786004065.082456 trunc from 0x03bc96 to 0x001e52 (0x39e43 bytes) 000671 1786004065.167371 trunc from 0x001e53 to 0x01d562 (0x1b710 bytes) 000673 1786004065.187942 read 0x00af83 thru 0x010f60 (0x5fde bytes) 000674 1786004065.196766 mapwrite 0x0349a1 thru 0x03d7e7 (0x8e47 bytes) truncating to largest ever: 0x3f3f1 000676 1786004065.262856 trunc from 0x03d7e8 to 0x03f3f0 (0x1c09 bytes) 000678 1786004065.280037 trunc from 0x03f3f1 to 0x029435 (0x15fbb bytes) 000679 1786004065.323171 read 0x019c6d thru 0x02867f (0xea13 bytes) 000682 1786004065.329473 read 0x00e11d thru 0x013195 (0x5079 bytes) 000684 1786004065.330782 mapread 0x0204d8 thru 0x0214eb (0x1014 bytes) 000686 1786004065.339648 trunc from 0x029436 to 0x01cf88 (0xc4ad bytes) 000692 1786004065.398229 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1cf89): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf89): Mirror 2: 0xe0e2495b (stale) Mirror 4: 0xe0e2495b (stale) Mirror 5: 0xe0e2495b (stale) Mirror 7: 0xe0e2495b (stale) Mirror 8: 0xe0e2495b (stale) Mirror 9: 0xe0e2495b (stale) Mirror 10: 0x28a2b2fd (stale) CRC-32 checksum value for chunk [0, 0x1cf89): Mirror 1: 0x949d3d3f Mirror 11: 0x949d3d3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf89: skipped 000694 1786004067.039813 MIRROR_RESYNC 9 mirrors 000695 1786004069.183527 mapread 0x0117ae thru 0x01ab8a (0x93dd bytes) 000698 1786004069.236135 MIRROR_EXTEND to 10 mirrors 000700 1786004070.454943 MIRROR_EXTEND to 11 mirrors 000703 1786004071.264564 trunc from 0x01cf89 to 0x0265c2 (0x963a bytes) 000704 1786004071.385287 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x265c3): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x265c3): Mirror 2: 0x9a3a888c (stale) Mirror 4: 0x9a3a888c (stale) Mirror 5: 0x9a3a888c (stale) Mirror 7: 0x9a3a888c (stale) Mirror 8: 0x9a3a888c (stale) Mirror 9: 0x9a3a888c (stale) Mirror 10: 0x9a3a888c (stale) Mirror 11: 0x9a3a888c (stale) Mirror 12: 0x9a3a888c (stale) Mirror 13: 0x9a3a888c (stale) CRC-32 checksum value for chunk [0, 0x265c3): Mirror 1: 0x9a3a888c Mirror 14: 0x9a3a888c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x265c3: skipped 000705 1786004073.210240 MIRROR_RESYNC 12 mirrors 000706 1786004076.370894 write 0x01f2d1 thru 0x02c712 (0xd442 bytes) 000707 1786004076.443752 write 0x0200a7 thru 0x02d662 (0xd5bc bytes) 000708 1786004076.447017 trunc from 0x02d663 to 0x01ae83 (0x127df bytes) 000711 1786004076.468665 write 0x017809 thru 0x01864b (0xe43 bytes) 000713 1786004076.479206 trunc from 0x01ae84 to 0x019cb2 (0x11d1 bytes) 000715 1786004076.494626 read 0x00c1d0 thru 0x010dfc (0x4c2d bytes) 000716 1786004076.503431 mapread 0x016905 thru 0x019cb2 (0x33ae bytes) 000717 1786004076.512829 write 0x000009 thru 0x003f15 (0x3f0d bytes) 000718 1786004076.521694 mapwrite 0x00d740 thru 0x01d614 (0xfed5 bytes) 000719 1786004076.635422 write 0x009f1f thru 0x0101b7 (0x6299 bytes) 000720 1786004076.641034 mapwrite 0x000755 thru 0x0086b6 (0x7f62 bytes) 000722 1786004076.694522 MIRROR_RESYNC 12 mirrors 000723 1786004079.748894 read 0x006bab thru 0x00dc3f (0x7095 bytes) 000728 1786004079.787425 mapwrite 0x01b547 thru 0x01dca3 (0x275d bytes) 000729 1786004080.225338 mapwrite 0x03479b thru 0x03ffff (0xb865 bytes) 000730 1786004080.502461 read 0x002727 thru 0x00851c (0x5df6 bytes) 000733 1786004080.558521 trunc from 0x040000 to 0x0193ea (0x26c15 bytes) 000735 1786004080.622203 MIRROR_RESYNC 12 mirrors 000736 1786004084.398394 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000738 1786004084.767952 read 0x010288 thru 0x0193ea (0x9163 bytes) 000740 1786004084.781694 mapwrite 0x012c50 thru 0x021b21 (0xeed2 bytes) 000741 1786004084.995077 MIRROR_RESYNC 11 mirrors 000742 1786004087.825167 MIRROR_RESYNC 11 mirrors 000746 1786004088.769965 write 0x02439d thru 0x0289df (0x4643 bytes) 000747 1786004088.983543 mapread 0x02490f thru 0x025c6f (0x1361 bytes) 000748 1786004088.988777 trunc from 0x0289e0 to 0x01894c (0x10093 bytes) 000752 1786004089.019201 trunc from 0x01894d to 0x0033d3 (0x15579 bytes) 000753 1786004089.061539 MIRROR_RESYNC 11 mirrors 000754 1786004091.864376 MIRROR_RESYNC 11 mirrors 000755 1786004092.710641 trunc from 0x0033d4 to 0x0373ef (0x3401c bytes) 000756 1786004092.948124 trunc from 0x0373f0 to 0x020d72 (0x1667d bytes) 000757 1786004092.999060 read 0x00af50 thru 0x01138a (0x643b bytes) 000759 1786004093.013999 read 0x019ec6 thru 0x01f13d (0x5278 bytes) 000761 1786004093.042990 MIRROR_RESYNC 11 mirrors 000762 1786004096.965739 mapwrite 0x038936 thru 0x03ffff (0x76ca bytes) 000763 1786004097.366022 write 0x01f331 thru 0x026b81 (0x7851 bytes) 000764 1786004097.395858 write 0x032cbf thru 0x03ffff (0xd341 bytes) 000766 1786004097.405182 write 0x000ad3 thru 0x00cd02 (0xc230 bytes) 000767 1786004097.440612 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x47aa5dbd (stale) Mirror 5: 0x47aa5dbd (stale) Mirror 7: 0x47aa5dbd (stale) Mirror 8: 0x47aa5dbd (stale) Mirror 9: 0x47aa5dbd (stale) Mirror 10: 0x47aa5dbd (stale) Mirror 11: 0x47aa5dbd (stale) Mirror 12: 0x47aa5dbd (stale) Mirror 14: 0x47aa5dbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x47aa5dbd (stale) Mirror 5: 0x47aa5dbd (stale) Mirror 7: 0x47aa5dbd (stale) Mirror 8: 0x47aa5dbd (stale) Mirror 9: 0x47aa5dbd (stale) Mirror 10: 0x47aa5dbd (stale) Mirror 11: 0x47aa5dbd (stale) Mirror 12: 0x47aa5dbd (stale) Mirror 14: 0x47aa5dbd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000769 1786004098.349845 mapread 0x01214d thru 0x012773 (0x627 bytes) 000770 1786004098.363845 trunc from 0x040000 to 0x000bbc (0x3f443 bytes) 000772 1786004098.382074 mapwrite 0x01d173 thru 0x021004 (0x3e92 bytes) 000773 1786004098.536180 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21005): Mirror 4: 0x346dfcd3 (stale) Mirror 5: 0x346dfcd3 (stale) Mirror 7: 0x346dfcd3 (stale) Mirror 8: 0x346dfcd3 (stale) Mirror 9: 0x346dfcd3 (stale) Mirror 11: 0x346dfcd3 (stale) Mirror 12: 0x346dfcd3 (stale) Mirror 14: 0x346dfcd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21005): Mirror 4: 0x346dfcd3 (stale) Mirror 5: 0x346dfcd3 (stale) Mirror 7: 0x346dfcd3 (stale) Mirror 8: 0x346dfcd3 (stale) Mirror 9: 0x346dfcd3 (stale) Mirror 11: 0x346dfcd3 (stale) Mirror 12: 0x346dfcd3 (stale) Mirror 14: 0x346dfcd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21005: skipped 000774 1786004099.510116 mapread 0x0074d6 thru 0x00e9a3 (0x74ce bytes) 000775 1786004099.555040 write 0x017b4c thru 0x018ecf (0x1384 bytes) 000778 1786004099.578084 mapwrite 0x009e25 thru 0x0195f9 (0xf7d5 bytes) 000781 1786004099.766360 read 0x006435 thru 0x01302b (0xcbf7 bytes) 000782 1786004099.767068 read 0x014a81 thru 0x021004 (0xc584 bytes) 000785 1786004099.807979 write 0x03614e thru 0x03b044 (0x4ef7 bytes) 000786 1786004099.830681 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b045): Mirror 4: 0xde088d7c (stale) Mirror 5: 0xde088d7c (stale) Mirror 7: 0xde088d7c (stale) Mirror 8: 0xde088d7c (stale) Mirror 9: 0xde088d7c (stale) Mirror 12: 0xde088d7c (stale) Mirror 14: 0xde088d7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b045): Mirror 4: 0xde088d7c (stale) Mirror 5: 0xde088d7c (stale) Mirror 7: 0xde088d7c (stale) Mirror 8: 0xde088d7c (stale) Mirror 9: 0xde088d7c (stale) Mirror 12: 0xde088d7c (stale) Mirror 14: 0xde088d7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b045: skipped 000788 1786004101.147813 MIRROR_RESYNC 8 mirrors 000789 1786004107.882310 read 0x021b5a thru 0x029013 (0x74ba bytes) 000791 1786004107.895638 MIRROR_RESYNC 8 mirrors 000792 1786004109.140504 trunc from 0x03b045 to 0x016489 (0x24bbb bytes) 000793 1786004109.386509 read 0x00fc2c thru 0x0138ba (0x3c8f bytes) 000795 1786004109.399853 mapwrite 0x02047e thru 0x029921 (0x94a4 bytes) 000797 1786004109.683475 read 0x00f196 thru 0x01e227 (0xf092 bytes) 000798 1786004109.720809 mapread 0x000db8 thru 0x006a51 (0x5c9a bytes) 000800 1786004109.725591 write 0x008f1d thru 0x018369 (0xf44d bytes) 000801 1786004109.735407 MIRROR_RESYNC 8 mirrors 000803 1786004115.680933 trunc from 0x029922 to 0x01434f (0x155d2 bytes) 000805 1786004115.913580 mapread 0x012500 thru 0x01434f (0x1e50 bytes) 000807 1786004115.947561 write 0x024d34 thru 0x0279d4 (0x2ca1 bytes) 000808 1786004115.948652 read 0x02254a thru 0x0279d4 (0x548b bytes) 000809 1786004115.975869 write 0x0212bd thru 0x029e92 (0x8bd6 bytes) 000811 1786004116.001605 mapwrite 0x03676d thru 0x03ffff (0x9893 bytes) 000813 1786004116.302996 trunc from 0x040000 to 0x0330e5 (0xcf1a bytes) 000815 1786004116.383025 write 0x0236f6 thru 0x02cc41 (0x954c bytes) 000816 1786004116.436993 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x330e6): Mirror 4: 0xf3731c88 (stale) Mirror 5: 0xf3731c88 (stale) Mirror 7: 0xf3731c88 (stale) Mirror 8: 0xf3731c88 (stale) Mirror 9: 0xf3731c88 (stale) Mirror 12: 0xf3731c88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x330e6): Mirror 4: 0xf3731c88 (stale) Mirror 5: 0xf3731c88 (stale) Mirror 7: 0xf3731c88 (stale) Mirror 8: 0xf3731c88 (stale) Mirror 9: 0xf3731c88 (stale) Mirror 12: 0xf3731c88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x330e6: skipped 000817 1786004117.484155 mapwrite 0x0051c5 thru 0x013c81 (0xeabd bytes) 000818 1786004117.713166 write 0x03fb04 thru 0x03ffff (0x4fc bytes) 000819 1786004117.717632 write 0x03ef56 thru 0x03ffff (0x10aa bytes) 000821 1786004117.740597 MIRROR_RESYNC 7 mirrors 000822 1786004122.842722 mapread 0x034f80 thru 0x038490 (0x3511 bytes) 000823 1786004122.862979 write 0x03e871 thru 0x03ffff (0x178f bytes) 000827 1786004122.952009 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x390043f9 (stale) Mirror 5: 0x390043f9 (stale) Mirror 7: 0x390043f9 (stale) Mirror 8: 0x390043f9 (stale) Mirror 9: 0x390043f9 (stale) Mirror 12: 0x390043f9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ec4c9a6 Mirror 13: 0x5ec4c9a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000829 1786004124.562153 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x390043f9 (stale) Mirror 5: 0x390043f9 (stale) Mirror 8: 0x390043f9 (stale) Mirror 9: 0x390043f9 (stale) Mirror 12: 0x390043f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x390043f9 (stale) Mirror 5: 0x390043f9 (stale) Mirror 8: 0x390043f9 (stale) Mirror 9: 0x390043f9 (stale) Mirror 12: 0x390043f9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ec4c9a6 Mirror 13: 0x5ec4c9a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000832 1786004125.465530 mapread 0x0010e6 thru 0x0050c3 (0x3fde bytes) 000833 1786004125.503282 mapwrite 0x01ecd8 thru 0x020af9 (0x1e22 bytes) 000834 1786004125.842636 mapread 0x00f8da thru 0x01c5ce (0xccf5 bytes) 000836 1786004125.856876 write 0x02a3c8 thru 0x0313d8 (0x7011 bytes) 000837 1786004125.881777 read 0x001d45 thru 0x00978f (0x7a4b bytes) 000838 1786004125.900321 write 0x01ff25 thru 0x02bec4 (0xbfa0 bytes) 000839 1786004125.901622 mapread 0x01a735 thru 0x024b97 (0xa463 bytes) 000846 1786004125.913921 trunc from 0x040000 to 0x0169a1 (0x2965e bytes) 000849 1786004125.980390 read 0x00e39b thru 0x0169a1 (0x8607 bytes) 000851 1786004125.985082 mapwrite 0x02cdf7 thru 0x0365fa (0x9804 bytes) 000852 1786004126.239135 mapread 0x028b71 thru 0x03416d (0xb5fd bytes) 000855 1786004126.247294 trunc from 0x0365fb to 0x019f3d (0x1c6bd bytes) 000857 1786004126.305057 trunc from 0x019f3e to 0x00021a (0x19d23 bytes) 000858 1786004126.324629 read 0x000097 thru 0x00021a (0x184 bytes) 000860 1786004126.337676 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21b): Mirror 4: 0xccfcb0fd (stale) Mirror 5: 0xccfcb0fd (stale) Mirror 8: 0xccfcb0fd (stale) Mirror 9: 0xccfcb0fd (stale) Mirror 12: 0xccfcb0fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b): Mirror 4: 0xccfcb0fd (stale) Mirror 5: 0xccfcb0fd (stale) Mirror 8: 0xccfcb0fd (stale) Mirror 9: 0xccfcb0fd (stale) Mirror 12: 0xccfcb0fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b: skipped 000861 1786004126.877625 mapread 0x00015a thru 0x00021a (0xc1 bytes) 000862 1786004126.899719 write 0x023743 thru 0x024e5e (0x171c bytes) 000863 1786004126.904452 trunc from 0x024e5f to 0x02a6ad (0x584f bytes) 000864 1786004126.945893 mapwrite 0x03b68e thru 0x03dddc (0x274f bytes) 000867 1786004127.140702 write 0x03745d thru 0x03ffff (0x8ba3 bytes) 000868 1786004127.158016 MIRROR_RESYNC 6 mirrors 000870 1786004131.559277 mapread 0x0362ca thru 0x03ffff (0x9d36 bytes) 000871 1786004131.646329 read 0x01faa1 thru 0x02b8d1 (0xbe31 bytes) 000872 1786004131.690640 mapwrite 0x02e6a2 thru 0x033f57 (0x58b6 bytes) 000873 1786004132.170212 mapread 0x00c474 thru 0x018967 (0xc4f4 bytes) 000874 1786004132.223912 MIRROR_RESYNC 6 mirrors 000875 1786004137.804419 mapwrite 0x02b4c1 thru 0x03b324 (0xfe64 bytes) 000876 1786004138.134546 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf5a55aa (stale) Mirror 8: 0xdf5a55aa (stale) Mirror 9: 0xdf5a55aa (stale) Mirror 12: 0xdf5a55aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf5a55aa (stale) Mirror 8: 0xdf5a55aa (stale) Mirror 9: 0xdf5a55aa (stale) Mirror 12: 0xdf5a55aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000877 1786004138.764085 mapwrite 0x032bff thru 0x03536c (0x276e bytes) 000880 1786004138.808916 write 0x01c97b thru 0x024006 (0x768c bytes) 000882 1786004138.819658 mapwrite 0x02d0f1 thru 0x02f288 (0x2198 bytes) 000887 1786004138.865544 write 0x003fac thru 0x0066ce (0x2723 bytes) 000888 1786004138.874305 read 0x00492a thru 0x00c7b3 (0x7e8a bytes) 000889 1786004138.884365 read 0x00abfa thru 0x01168a (0x6a91 bytes) 000890 1786004138.892230 trunc from 0x040000 to 0x016b15 (0x294ea bytes) 000891 1786004138.910291 trunc from 0x016b16 to 0x03dae9 (0x26fd4 bytes) 000892 1786004138.931323 mapread 0x0044a6 thru 0x0102de (0xbe39 bytes) 000893 1786004138.936689 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3daea): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3daea): Mirror 5: 0x3596a155 (stale) Mirror 8: 0x3596a155 (stale) Mirror 9: 0x3596a155 (stale) Mirror 12: 0x3596a155 (stale) CRC-32 checksum value for chunk [0, 0x3daea): Mirror 1: 0x9fc65bd5 Mirror 13: 0x9fc65bd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3daea: skipped 000895 1786004140.985360 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3daea): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3daea): Mirror 5: 0x3596a155 (stale) Mirror 8: 0x3596a155 (stale) Mirror 9: 0x3596a155 (stale) Mirror 12: 0x3596a155 (stale) CRC-32 checksum value for chunk [0, 0x3daea): Mirror 1: 0x9fc65bd5 Mirror 13: 0x9fc65bd5 Mirror 14: 0x9fc65bd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3daea: skipped 000899 1786004143.403414 trunc from 0x03daea to 0x012f64 (0x2ab85 bytes) 000900 1786004143.618760 mapwrite 0x032cae thru 0x03ffff (0xd352 bytes) 000902 1786004143.933147 mapread 0x0064bc thru 0x01284e (0xc393 bytes) 000903 1786004143.954641 write 0x001e58 thru 0x006795 (0x493e bytes) 000904 1786004143.955811 mapwrite 0x013562 thru 0x022714 (0xf1b3 bytes) 000907 1786004144.176320 write 0x01d709 thru 0x01e645 (0xf3d bytes) 000908 1786004144.201083 MIRROR_RESYNC 7 mirrors 000909 1786004149.604271 write 0x00b16b thru 0x017b71 (0xca07 bytes) 000911 1786004149.729402 mapread 0x035d88 thru 0x03da78 (0x7cf1 bytes) 000914 1786004149.756366 MIRROR_RESYNC 7 mirrors 000916 1786004153.266961 trunc from 0x040000 to 0x02ab48 (0x154b7 bytes) 000917 1786004153.345953 trunc from 0x02ab49 to 0x01af5c (0xfbec bytes) 000918 1786004153.364146 read 0x00046c thru 0x009271 (0x8e06 bytes) 000919 1786004153.371076 write 0x0396d2 thru 0x03c6b8 (0x2fe7 bytes) 000920 1786004153.372118 mapread 0x00a923 thru 0x0125c4 (0x7ca2 bytes) 000921 1786004153.395768 write 0x0055b0 thru 0x00bde8 (0x6839 bytes) 000922 1786004153.397564 read 0x019d28 thru 0x01f9b7 (0x5c90 bytes) 000925 1786004153.424653 mapwrite 0x0157a8 thru 0x022ef0 (0xd749 bytes) 000926 1786004153.591462 write 0x0215f6 thru 0x02e70c (0xd117 bytes) 000927 1786004153.603996 write 0x00e654 thru 0x014892 (0x623f bytes) 000928 1786004153.605619 trunc from 0x03c6b9 to 0x02723f (0x15479 bytes) 000929 1786004153.625574 write 0x005362 thru 0x00c8c2 (0x7561 bytes) 000931 1786004153.630548 read 0x01c019 thru 0x01d7be (0x17a6 bytes) 000933 1786004153.636406 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x27240): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27240): Mirror 5: 0x13ce44b7 (stale) Mirror 8: 0x13ce44b7 (stale) Mirror 9: 0x13ce44b7 (stale) Mirror 12: 0x13ce44b7 (stale) Mirror 13: 0x13ce44b7 (stale) Mirror 14: 0x13ce44b7 (stale) CRC-32 checksum value for chunk [0, 0x27240): Mirror 1: 0x8832f618 Mirror 15: 0x8832f618 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27240: skipped 000935 1786004155.921515 read 0x011e64 thru 0x01e05f (0xc1fc bytes) 000936 1786004155.965084 MIRROR_RESYNC 8 mirrors 000937 1786004158.964806 read 0x023258 thru 0x02723f (0x3fe8 bytes) 000939 1786004159.033600 trunc from 0x027240 to 0x026094 (0x11ab bytes) 000940 1786004159.313121 mapread 0x009e32 thru 0x00b285 (0x1454 bytes) 000943 1786004159.373967 mapwrite 0x023ccb thru 0x02f0e7 (0xb41d bytes) 000944 1786004159.710920 MIRROR_RESYNC 8 mirrors 000946 1786004162.414929 mapwrite 0x039736 thru 0x03e64b (0x4f16 bytes) 000948 1786004162.638426 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e64c): Mirror 5: 0x2d118434 (stale) Mirror 9: 0x2d118434 (stale) Mirror 12: 0x2d118434 (stale) Mirror 13: 0x2d118434 (stale) Mirror 14: 0x2d118434 (stale) Mirror 15: 0x2d118434 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e64c): Mirror 5: 0x2d118434 (stale) Mirror 9: 0x2d118434 (stale) Mirror 12: 0x2d118434 (stale) Mirror 13: 0x2d118434 (stale) Mirror 14: 0x2d118434 (stale) Mirror 15: 0x2d118434 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e64c: skipped 000949 1786004163.285611 mapwrite 0x003232 thru 0x003aa6 (0x875 bytes) 000950 1786004163.466334 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e64c): Mirror 9: 0x2d118434 (stale) Mirror 12: 0x2d118434 (stale) Mirror 13: 0x2d118434 (stale) Mirror 14: 0x2d118434 (stale) Mirror 15: 0x2d118434 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e64c): Mirror 9: 0x2d118434 (stale) Mirror 12: 0x2d118434 (stale) Mirror 13: 0x2d118434 (stale) Mirror 14: 0x2d118434 (stale) Mirror 15: 0x2d118434 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e64c: skipped 000952 1786004164.192347 MIRROR_RESYNC 6 mirrors 000957 1786004166.357541 trunc from 0x03e64c to 0x02328a (0x1b3c1 bytes) 000958 1786004166.464934 read 0x012e3e thru 0x0199c6 (0x6b89 bytes) 000959 1786004166.476504 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2328b): Mirror 12: 0xd6bd7b4f (stale) Mirror 13: 0xd6bd7b4f (stale) Mirror 14: 0xd6bd7b4f (stale) Mirror 15: 0xd6bd7b4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2328b): Mirror 12: 0xd6bd7b4f (stale) Mirror 13: 0xd6bd7b4f (stale) Mirror 14: 0xd6bd7b4f (stale) Mirror 15: 0xd6bd7b4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2328b: skipped 000961 1786004166.938440 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2328b): Mirror 13: 0xd6bd7b4f (stale) Mirror 14: 0xd6bd7b4f (stale) Mirror 15: 0xd6bd7b4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2328b): Mirror 13: 0xd6bd7b4f (stale) Mirror 14: 0xd6bd7b4f (stale) Mirror 15: 0xd6bd7b4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2328b: skipped 000964 1786004167.285586 read 0x010b0b thru 0x01e06e (0xd564 bytes) 000965 1786004167.292643 mapwrite 0x002764 thru 0x00ff26 (0xd7c3 bytes) 000966 1786004167.355578 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2328b): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2328b): Mirror 13: 0xd6bd7b4f (stale) Mirror 14: 0xd6bd7b4f (stale) Mirror 15: 0xd6bd7b4f (stale) CRC-32 checksum value for chunk [0, 0x2328b): Mirror 1: 0x3ce93f82 Mirror 16: 0x3ce93f82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2328b: skipped 000969 1786004167.873684 read 0x00ad83 thru 0x01a43b (0xf6b9 bytes) 000970 1786004167.883740 write 0x012a86 thru 0x01daa2 (0xb01d bytes) 000971 1786004167.951508 read 0x01dd42 thru 0x02328a (0x5549 bytes) 000973 1786004167.964654 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2328b): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2328b): Mirror 13: 0xd6bd7b4f (stale) Mirror 14: 0xd6bd7b4f (stale) Mirror 15: 0xd6bd7b4f (stale) Mirror 16: 0x3ce93f82 (stale) CRC-32 checksum value for chunk [0, 0x2328b): Mirror 1: 0x263c9ba0 Mirror 17: 0x263c9ba0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2328b: skipped 000975 1786004168.632522 read 0x000a25 thru 0x0017e3 (0xdbf bytes) 000976 1786004168.641547 mapread 0x013c0c thru 0x01561f (0x1a14 bytes) 000977 1786004168.664356 write 0x018f04 thru 0x024b83 (0xbc80 bytes) 000978 1786004168.731167 MIRROR_RESYNC 6 mirrors 000979 1786004169.758741 mapread 0x00b410 thru 0x00d51c (0x210d bytes) 000980 1786004169.837940 mapwrite 0x01c4b0 thru 0x02a1ac (0xdcfd bytes) 000981 1786004170.125275 trunc from 0x02a1ad to 0x01caa2 (0xd70a bytes) 000983 1786004170.183704 read 0x00555f thru 0x00e0cb (0x8b6d bytes) 000984 1786004170.198837 trunc from 0x01caa3 to 0x013a9b (0x9007 bytes) 000985 1786004170.253684 read 0x0126f5 thru 0x013a9b (0x13a7 bytes) 000986 1786004170.288138 mapread 0x0108f2 thru 0x013a9b (0x31aa bytes) 000987 1786004170.291325 mapwrite 0x03bbed thru 0x03ffff (0x4413 bytes) 000988 1786004170.470765 write 0x00651b thru 0x00e48f (0x7f75 bytes) 000989 1786004170.473701 write 0x02523e thru 0x029276 (0x4039 bytes) 000991 1786004170.480220 write 0x020da2 thru 0x02b8b9 (0xab18 bytes) 000994 1786004170.501781 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc7bc13d8 (stale) Mirror 14: 0xc7bc13d8 (stale) Mirror 15: 0xc7bc13d8 (stale) Mirror 17: 0xc7bc13d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc7bc13d8 (stale) Mirror 14: 0xc7bc13d8 (stale) Mirror 15: 0xc7bc13d8 (stale) Mirror 17: 0xc7bc13d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000995 1786004171.440305 read 0x0143ec thru 0x01da05 (0x961a bytes) 000996 1786004171.467495 mapwrite 0x019118 thru 0x027106 (0xdfef bytes) 000998 1786004171.801679 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc7bc13d8 (stale) Mirror 14: 0xc7bc13d8 (stale) Mirror 15: 0xc7bc13d8 (stale) Mirror 17: 0xc7bc13d8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf22738ea Mirror 18: 0xf22738ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 001000 1786004173.378304 trunc from 0x040000 to 0x03d809 (0x27f6 bytes) All operations completed A-OK! PASS 70a (217s) == sanity-flr test 71: check mirror extend parent fid ==== 04:16:20 (1786004180) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 04:16:26 (1786004186) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22753 s, 9.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180818 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 04:16:37 (1786004197) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8718240 .. Extending file size to 1681664 .. Extending file size to 8021504 .. Extending file size to 8176000 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8735936 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8652096 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57432 57434 57435 57437 57438 failed 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) 192.168.204.128@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:18:05 (1786004285) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8884096 .. Extending file size to 8399168 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8729152 .. Extending file size to 9325536 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58670 58671 58673 58674 58675 failed 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) 192.168.204.128@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (88s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:19:33 (1786004373) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.253025 s, 8.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.613334 s, 3.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x59bcc0c9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:19:57 (1786004397) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176045 s, 11.9 MB/s PASS 202 (11s) == sanity-flr test 203: mirror file preserve mirror ID === 04:20:08 (1786004408) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:20:18 (1786004418) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.413136 s, 5.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:20:30 (1786004430) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.455476 s, 4.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } PASS 204b (12s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:20:42 (1786004442) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193627 s, 10.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:21:19 (1786004479) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191851 s, 5.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } PASS 204d (66s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:22:25 (1786004545) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269211 s, 7.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (65s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:23:31 (1786004611) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. Creating new pool test_204f oleg428-server: Pool lustre.test_204f created Adding targets to pool oleg428-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg428-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170094 s, 12.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg428-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg428-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg428-server: Pool lustre.test_204f destroyed PASS 204f (90s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:25:01 (1786004701) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:25:11 (1786004711) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:25:23 (1786004723) oleg428-server: Pool lustre.test_206 created Adding targets to pool oleg428-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg428-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg428-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg428-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg428-server: Pool lustre.test_206 destroyed PASS 206 (36s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:25:59 (1786004759) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184548 s, 5.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168119 s, 6.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfc527f44 Mirror 3: 0xfc527f44 PASS 207 (19s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:26:18 (1786004778) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:26:23 (1786004783) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:26:28 (1786004788) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.172181 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x880000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x10d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x10d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x10d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x900000 Read this (4 bytes): BBBB 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) PASS 209a (49s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:27:17 (1786004837) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0140904 s, 581 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:27:27 (1786004847) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13527 s, 7.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f1 lmm_fid: [0x200000404:0x1f1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f1 lmm_fid: [0x200000404:0x1f1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834417 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-06 04:27:29.000000000 -0400 Modify: 2026-08-06 04:27:29.000000000 -0400 Change: 2026-08-06 04:27:31.000000000 -0400 Birth: 2026-08-06 04:27:30.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:27:38 (1786004858) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:27:42 (1786004862) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.17843 s, 34.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.081725 s, 50.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 6 04:27 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f3:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 6 04:27 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:27:55 (1786004875) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:28:03 (1786004883) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2292 sec =========== 04:28:15 (1786004895) === sanity-flr: start cleanup 04:28:18 (1786004898) === === sanity-flr: finish cleanup 04:28:27 (1786004907) ===