-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 19:44:34 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg452-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:44:55 (1788651895) === oleg452-client.virtnet: executing check_config_client /mnt/lustre oleg452-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg452-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c3c062f9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c3c062f9800.idle_timeout=debug disable quota as required oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:45:36 (1788651936) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7623] == sanity-flr test 0a: lfs mirror create with -N option == 19:45:39 (1788651939) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:45:57 (1788651957) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:46:01 (1788651961) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:46:05 (1788651965) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:46:34 (1788651994) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:46:38 (1788651998) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:46:42 (1788652002) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c3c062f9800.stats=clear osc.lustre-OST0001-osc-ffff8c3c062f9800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:46:52 (1788652012) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:47:06 (1788652026) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:47:15 (1788652035) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:47:20 (1788652040) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:47:29 (1788652049) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 19:47:39 (1788652059) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:47:47 (1788652067) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:47:51 (1788652071) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:48:00 (1788652080) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:48:11 (1788652091) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:48:18 (1788652098) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 2.04334 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161234 s, 6.5 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:48:31 (1788652111) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.432814 s, 33.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:48:40 (1788652120) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:48:50 (1788652130) Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3c062f9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3c062f9800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3c062f9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3c062f9800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:50:04 (1788652204) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.248083 s, 12.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3c062f9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3c062f9800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:50:48 (1788652248) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3c062f9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3c062f9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3c062f9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3c062f9800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:52:11 (1788652331) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.28835 s, 7.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 19:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.580s user 0m0.001s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3c062f9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3c062f9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.098s user 0m0.000s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg452-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg452-server oleg452-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3c062f9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3c062f9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.726s user 0m0.002s sys 0m0.181s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg452-server: oleg452-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg452-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c3c062f9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3c062f9800.ost_server_uuid in FULL state after 0 sec PASS 33b (112s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:54:04 (1788652444) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:54:07 (1788652447) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:54:10 (1788652450) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 19:54:14 (1788652454) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 19:54:22 (1788652462) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.623723 s, 20.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:54:35 (1788652475) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341366 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210703 s, 49.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:54:43 (1788652483) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:54:51 (1788652491) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108758 s, 942 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138729 s, 738 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000842296 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00257411 s, 398 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 19:55:03 (1788652503) create test files with size 15753003 .. 6f23fd0cccf39bf04fe8eaa24e7b89d8 - 81050c6fff5abf7082a8f019b2f9850b - aa03f42e02d18601067406adfaf9c6ab - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 19:55:29 (1788652529) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (20s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:55:49 (1788652549) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:55:56 (1788652556) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12294 s, 8.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061892 s, 16.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 19:56:06 (1788652566) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.212004 s, 9.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.291364 s, 14.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.278774 s, 7.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 19:56:25 (1788652585) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 19:56:30 (1788652590) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:56:34 (1788652594) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607472 s, 17.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 19:56:43 (1788652603) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:56:47 (1788652607) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:56:58 (1788652618) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.370943 s, 28.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:57:12 (1788652632) llite.lustre-ffff8c3c068f4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.346847 s, 30.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg452-client.virtnet /mnt/lustre (opts:) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8c3c08507000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:57:31 (1788652651) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:58:17 (1788652697) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:58:55 (1788652735) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 19:58:59 (1788652739) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.254899 s, 12.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.216161 s, 14.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4ca7aa0e (nosync,stale) Mirror 2: 0x4ca7aa0e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7db455d897df246eb634b03a668a971f - mirror 1 checksum: 12e206d5d49afa036bfbcce52f6ae604 - mirror 2 checksum: 7db455d897df246eb634b03a668a971f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T19:59:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T19:59:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7db455d897df246eb634b03a668a971f - mirror 1 checksum: 12e206d5d49afa036bfbcce52f6ae604 - mirror 2 checksum: 12e206d5d49afa036bfbcce52f6ae604 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T19:59:03-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T19:59:03-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:59:15 (1788652755) Failing mds1 on oleg452-server Stopping /mnt/lustre-mds1 (opts:) on oleg452-server 19:59:22 (1788652762) shut down facet: mds1 facet_host: oleg452-server facet_failover_host: oleg452-server Failover mds1 to oleg452-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 19:59:56 (1788652796) targets are mounted 19:59:56 (1788652796) facet_failover done oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203574 s, 51.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:00:15 (1788652815) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:00:19 (1788652819) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:00:23 (1788652823) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0136704 s, 1.2 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:00:33 (1788652833) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:00:36 (1788652836) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0134955 s, 304 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0884362 s, 46.3 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:00:45 (1788652845) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:01:14 (1788652874) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:01:45 (1788652905) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:02:10 (1788652930) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:02:37 (1788652957) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 20:02:46 (1788652966) mirror create pid 54329 mirror split pid 54330 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54329 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54330 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 20:03:54 (1788653034) Seed set to 3036 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read 000003 1788653037.332273 mapwrite 0x027d48 thru 0x0318ec (0x9ba5 bytes) 000005 1788653037.509501 mapread 0x01ec78 thru 0x02432a (0x56b3 bytes) truncating to largest ever: 0x30091 000006 1788653037.543865 trunc from 0x0318ed to 0x030090 (0x185c bytes) 000007 1788653037.581991 mapwrite 0x003c3f thru 0x00a857 (0x6c19 bytes) 000009 1788653037.687333 write 0x00d0aa thru 0x01aa36 (0xd98d bytes) 000010 1788653037.698790 read 0x02d2ed thru 0x030090 (0x2da4 bytes) 000013 1788653037.705638 write 0x03e254 thru 0x03ffff (0x1dac bytes) 000016 1788653037.713156 punch from 0x033f44 to 0x038e5e (0x4f1b bytes) 000017 1788653037.766671 write 0x0108b4 thru 0x0169be (0x610b bytes) 000018 1788653037.767867 read 0x024b61 thru 0x025138 (0x5d8 bytes) 000019 1788653037.768414 mapread 0x01de97 thru 0x024d5f (0x6ec9 bytes) 000024 1788653037.802171 MIRROR_EXTEND to 2 mirrors 000026 1788653039.488679 punch from 0x018d70 to 0x01da68 (0x4cf9 bytes) 000027 1788653039.569481 trunc from 0x040000 to 0x00c23b (0x33dc4 bytes) 000028 1788653039.633499 punch from 0x003789 to 0x00c23b (0x8ab3 bytes) 000029 1788653039.654628 MIRROR_RESYNC 2 mirrors 000030 1788653040.891898 punch from 0x008ac5 to 0x00c23b (0x3777 bytes) 000032 1788653040.990602 read 0x00642d thru 0x00b937 (0x550b bytes) 000034 1788653041.005192 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xc23c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc23c): Mirror 2: 0x92647c84 (stale) CRC-32 checksum value for chunk [0, 0xc23c): Mirror 1: 0x92647c84 Mirror 3: 0x92647c84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc23c: skipped 000035 1788653042.518477 punch from 0x004f59 to 0x00ac2a (0x5cd2 bytes) 000036 1788653042.691073 punch from 0x0077d2 to 0x00c23b (0x4a6a bytes) 000038 1788653042.713421 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc23c): Mirror 3: 0x92647c84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc23c): Mirror 3: 0x92647c84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc23c: skipped 000039 1788653043.227730 mapread 0x003c33 thru 0x00621b (0x25e9 bytes) 000041 1788653043.254720 trunc from 0x00c23c to 0x021293 (0x15058 bytes) 000042 1788653043.276289 punch from 0x008d1a to 0x01398a (0xac71 bytes) 000043 1788653043.284419 read 0x015bda thru 0x01dbd4 (0x7ffb bytes) 000044 1788653043.298862 trunc from 0x021294 to 0x028c5b (0x79c8 bytes) 000046 1788653043.341738 trunc from 0x028c5c to 0x01b60f (0xd64c bytes) 000048 1788653043.379536 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b610): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b610): Mirror 3: 0x1da959e1 (stale) CRC-32 checksum value for chunk [0, 0x1b610): Mirror 1: 0x1da959e1 Mirror 4: 0x1da959e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b610: skipped 000050 1788653045.575869 trunc from 0x01b610 to 0x01754d (0x40c2 bytes) 000053 1788653045.798358 punch from 0x0071af to 0x01712b (0xff7d bytes) 000054 1788653045.804550 trunc from 0x01754e to 0x000c9a (0x168b3 bytes) 000055 1788653045.834692 read 0x000ac7 thru 0x000c9a (0x1d4 bytes) 000056 1788653045.845571 mapread 0x000293 thru 0x000c9a (0xa08 bytes) 000057 1788653045.847183 punch from 0x000665 to 0x000c9a (0x636 bytes) 000058 1788653045.853676 read 0x00011d thru 0x000c9a (0xb7e bytes) 000059 1788653045.860254 mapwrite 0x03f6de thru 0x03ffff (0x922 bytes) 000060 1788653045.975580 mapread 0x0228f3 thru 0x02f33c (0xca4a bytes) 000061 1788653045.977113 read 0x026f92 thru 0x02ee60 (0x7ecf bytes) 000062 1788653045.984700 mapread 0x01252c thru 0x02139e (0xee73 bytes) 000063 1788653045.988649 trunc from 0x040000 to 0x01317b (0x2ce84 bytes) 000065 1788653046.026112 punch from 0x00d31a to 0x00fa28 (0x270f bytes) 000066 1788653046.065279 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1317c): Mirror 3: 0x9e8caf90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1317c): Mirror 3: 0x9e8caf90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1317c: skipped 000068 1788653046.399225 trunc from 0x01317c to 0x027331 (0x141b6 bytes) 000070 1788653046.421684 write 0x01123e thru 0x0135ce (0x2391 bytes) 000071 1788653046.432103 punch from 0x009282 to 0x014c6b (0xb9ea bytes) 000075 1788653046.443566 punch from 0x02624d to 0x027331 (0x10e5 bytes) 000076 1788653046.448668 mapwrite 0x02561f thru 0x0267c8 (0x11aa bytes) 000078 1788653046.562839 write 0x0018b7 thru 0x00e8ec (0xd036 bytes) 000080 1788653046.596263 write 0x01a3f8 thru 0x01c3f2 (0x1ffb bytes) 000081 1788653046.639956 MIRROR_RESYNC 2 mirrors 000082 1788653048.101312 mapread 0x00a072 thru 0x011a97 (0x7a26 bytes) 000083 1788653048.157281 mapwrite 0x014f43 thru 0x0237ed (0xe8ab bytes) truncating to largest ever: 0x3c337 000085 1788653048.361291 trunc from 0x027332 to 0x03c336 (0x15005 bytes) 000087 1788653048.384415 mapwrite 0x0353c7 thru 0x03f332 (0x9f6c bytes) 000088 1788653048.472345 write 0x0000f1 thru 0x0059f2 (0x5902 bytes) 000092 1788653048.486121 trunc from 0x03f333 to 0x0200ef (0x1f243 bytes) 000093 1788653048.507388 write 0x002557 thru 0x0028ae (0x358 bytes) 000094 1788653048.508069 trunc from 0x0200f0 to 0x031b62 (0x11a73 bytes) 000095 1788653048.534379 mapread 0x02243b thru 0x02d9de (0xb5a4 bytes) 000097 1788653048.568718 mapread 0x014193 thru 0x019165 (0x4fd3 bytes) 000098 1788653048.572022 write 0x00af64 thru 0x018e53 (0xdef0 bytes) 000099 1788653048.603317 mapread 0x009f11 thru 0x00d619 (0x3709 bytes) 000101 1788653048.640741 MIRROR_RESYNC 2 mirrors 000102 1788653049.758319 mapread 0x02c3f9 thru 0x02d518 (0x1120 bytes) 000103 1788653049.780175 punch from 0x02e222 to 0x031b62 (0x3941 bytes) 000104 1788653049.882851 read 0x00f651 thru 0x015c2e (0x65de bytes) 000105 1788653049.899822 mapread 0x0165bc thru 0x017960 (0x13a5 bytes) 000106 1788653049.928343 trunc from 0x031b63 to 0x03b23b (0x96d9 bytes) 000109 1788653050.006214 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b23c): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b23c): Mirror 3: 0xd33bdf3c (stale) CRC-32 checksum value for chunk [0, 0x3b23c): Mirror 1: 0xd33bdf3c Mirror 4: 0xd33bdf3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b23c: skipped 000110 1788653051.957937 trunc from 0x03b23c to 0x039bc2 (0x1679 bytes) 000111 1788653052.160572 punch from 0x0137a9 to 0x01ac85 (0x74dd bytes) 000112 1788653052.172682 mapread 0x00862f thru 0x00d4ac (0x4e7e bytes) 000113 1788653052.203605 read 0x00a824 thru 0x0163b7 (0xbb94 bytes) 000114 1788653052.229219 mapread 0x013e57 thru 0x01842c (0x45d6 bytes) 000115 1788653052.240420 trunc from 0x039bc3 to 0x026914 (0x132ae bytes) 000116 1788653052.299888 mapread 0x0042c4 thru 0x004a1b (0x758 bytes) 000121 1788653052.312734 read 0x014e18 thru 0x0150a3 (0x28c bytes) 000123 1788653052.324738 mapread 0x003140 thru 0x0036f4 (0x5b5 bytes) 000125 1788653052.330309 read 0x0265fc thru 0x026914 (0x319 bytes) 000126 1788653052.371548 MIRROR_RESYNC 3 mirrors 000127 1788653054.128398 mapread 0x001d53 thru 0x011013 (0xf2c1 bytes) 000128 1788653054.157579 punch from 0x02619f to 0x026914 (0x776 bytes) 000129 1788653054.243219 punch from 0x025bf8 to 0x026914 (0xd1d bytes) 000130 1788653054.250614 punch from 0x009f80 to 0x0133ae (0x942f bytes) 000131 1788653054.255393 trunc from 0x026915 to 0x00181f (0x250f5 bytes) 000132 1788653054.281216 mapread 0x000e9e thru 0x00181f (0x982 bytes) 000133 1788653054.289173 mapwrite 0x034873 thru 0x03ffff (0xb78d bytes) 000135 1788653054.402385 mapread 0x03adc0 thru 0x03d066 (0x22a7 bytes) 000136 1788653054.403365 write 0x01278a thru 0x01bddf (0x9656 bytes) 000138 1788653054.420460 punch from 0x001918 to 0x00b7c6 (0x9eaf bytes) 000140 1788653054.449860 mapwrite 0x01e726 thru 0x02cda9 (0xe684 bytes) 000141 1788653054.593488 mapwrite 0x032c67 thru 0x0381e3 (0x557d bytes) 000144 1788653054.672724 mapread 0x00bb6f thru 0x0146cd (0x8b5f bytes) 000145 1788653054.706574 mapread 0x01412b thru 0x022b5a (0xea30 bytes) 000148 1788653054.723529 mapread 0x029958 thru 0x0372c7 (0xd970 bytes) 000149 1788653054.742686 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x332dcdfb (stale) Mirror 4: 0x332dcdfb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5e619002 Mirror 5: 0x5e619002 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1788653056.858171 mapread 0x01c8ee thru 0x026009 (0x971c bytes) 000151 1788653056.932359 read 0x035dff thru 0x037b7a (0x1d7c bytes) 000152 1788653056.969598 mapwrite 0x01e2d1 thru 0x024f0b (0x6c3b bytes) 000153 1788653057.210731 read 0x01a904 thru 0x01f93e (0x503b bytes) 000154 1788653057.226339 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x332dcdfb (stale) Mirror 4: 0x332dcdfb (stale) Mirror 5: 0x5e619002 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x212f79cb Mirror 6: 0x212f79cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000156 1788653058.851544 punch from 0x02c6c5 to 0x0365f0 (0x9f2c bytes) 000157 1788653058.999351 trunc from 0x040000 to 0x020ee3 (0x1f11c bytes) 000158 1788653059.067505 read 0x01851e thru 0x020836 (0x8319 bytes) 000160 1788653059.093095 trunc from 0x020ee4 to 0x013038 (0xdeab bytes) 000161 1788653059.131464 mapread 0x00779a thru 0x00f643 (0x7eaa bytes) 000163 1788653059.154329 trunc from 0x013039 to 0x00cfba (0x607e bytes) 000164 1788653059.174568 mapwrite 0x039389 thru 0x03ffff (0x6c77 bytes) 000165 1788653059.266564 mapwrite 0x018402 thru 0x024412 (0xc011 bytes) 000167 1788653059.389650 punch from 0x008f78 to 0x018e3d (0xfec6 bytes) 000168 1788653059.398697 read 0x02ee1e thru 0x034c02 (0x5de5 bytes) 000169 1788653059.419194 write 0x013735 thru 0x0189e0 (0x52ac bytes) 000171 1788653059.423489 mapread 0x037ee4 thru 0x03ffff (0x811c bytes) 000172 1788653059.425329 mapwrite 0x030dfd thru 0x032e65 (0x2069 bytes) 000173 1788653059.482497 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x332dcdfb (stale) Mirror 4: 0x332dcdfb (stale) Mirror 5: 0x5e619002 (stale) Mirror 6: 0x212f79cb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2c50a700 Mirror 7: 0x2c50a700 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000174 1788653061.335549 mapwrite 0x013043 thru 0x019c47 (0x6c05 bytes) 000175 1788653061.665203 mapread 0x024e48 thru 0x030497 (0xb650 bytes) 000176 1788653061.714313 mapwrite 0x03c355 thru 0x03ffff (0x3cab bytes) 000179 1788653061.843181 trunc from 0x040000 to 0x00734d (0x38cb2 bytes) 000180 1788653061.904025 read 0x0041c2 thru 0x00734d (0x318c bytes) 000181 1788653061.922477 write 0x007ef5 thru 0x014b08 (0xcc14 bytes) 000182 1788653061.925612 mapread 0x0028b5 thru 0x005a0c (0x3158 bytes) 000183 1788653061.937300 mapread 0x0006b8 thru 0x003790 (0x30d9 bytes) 000184 1788653061.950668 read 0x00831b thru 0x00c030 (0x3d16 bytes) 000185 1788653061.951988 write 0x02f22b thru 0x0340d3 (0x4ea9 bytes) 000187 1788653061.961100 read 0x02c4ae thru 0x0323d7 (0x5f2a bytes) 000188 1788653061.994470 mapwrite 0x0338fe thru 0x03ffff (0xc702 bytes) 000189 1788653062.196524 trunc from 0x040000 to 0x02663c (0x199c3 bytes) truncating to largest ever: 0x3ff4b 000190 1788653062.280861 trunc from 0x02663d to 0x03ff4a (0x1990e bytes) 000191 1788653062.320667 write 0x00c7c5 thru 0x0141f1 (0x7a2d bytes) 000192 1788653062.328607 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ff4b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff4b): Mirror 3: 0x2a94032f (stale) Mirror 4: 0x2a94032f (stale) Mirror 5: 0x63c377f0 (stale) Mirror 6: 0x6313f325 (stale) Mirror 7: 0x8c3ccdc0 (stale) CRC-32 checksum value for chunk [0, 0x3ff4b): Mirror 1: 0xef91e462 Mirror 8: 0xef91e462 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff4b: skipped 000193 1788653064.453615 write 0x006fab thru 0x008ad4 (0x1b2a bytes) 000195 1788653064.598549 mapread 0x0346dd thru 0x0377ba (0x30de bytes) 000196 1788653064.637477 mapread 0x03e7bd thru 0x03ff4a (0x178e bytes) 000198 1788653064.650283 read 0x000117 thru 0x0036da (0x35c4 bytes) 000199 1788653064.669609 trunc from 0x03ff4b to 0x03699d (0x95ad bytes) 000200 1788653064.721451 trunc from 0x03699e to 0x02b289 (0xb714 bytes) 000201 1788653064.758249 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b28a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b28a): Mirror 3: 0x1fd63ee4 (stale) Mirror 4: 0x1fd63ee4 (stale) Mirror 5: 0xb3d3b5c4 (stale) Mirror 6: 0xc187092c (stale) Mirror 7: 0x4511921b (stale) Mirror 8: 0xe24da09a (stale) CRC-32 checksum value for chunk [0, 0x2b28a): Mirror 1: 0xa97fbe7e Mirror 9: 0xa97fbe7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b28a: skipped 000202 1788653066.909545 trunc from 0x02b28a to 0x03196b (0x66e2 bytes) 000203 1788653067.019397 mapwrite 0x021f4c thru 0x02f14c (0xd201 bytes) 000205 1788653067.132618 mapwrite 0x030db3 thru 0x039fbd (0x920b bytes) 000206 1788653067.270406 mapread 0x02f2ff thru 0x03869f (0x93a1 bytes) 000207 1788653067.273878 read 0x012be2 thru 0x01e996 (0xbdb5 bytes) 000208 1788653067.284582 write 0x0349bc thru 0x037b63 (0x31a8 bytes) 000210 1788653067.286283 read 0x0254dd thru 0x027f3b (0x2a5f bytes) 000212 1788653067.288955 mapread 0x02ef7f thru 0x0373c3 (0x8445 bytes) 000216 1788653067.291961 mapwrite 0x007e88 thru 0x00a70c (0x2885 bytes) 000218 1788653067.360211 mapwrite 0x03c48c thru 0x03ffff (0x3b74 bytes) 000220 1788653067.441504 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x332dcdfb (stale) Mirror 4: 0x332dcdfb (stale) Mirror 5: 0x5e619002 (stale) Mirror 6: 0x212f79cb (stale) Mirror 7: 0x2c50a700 (stale) Mirror 8: 0x148349ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x332dcdfb (stale) Mirror 4: 0x332dcdfb (stale) Mirror 5: 0x5e619002 (stale) Mirror 6: 0x212f79cb (stale) Mirror 7: 0x2c50a700 (stale) Mirror 8: 0x148349ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000221 1788653068.013467 mapwrite 0x0392ac thru 0x03ffff (0x6d54 bytes) 000222 1788653068.104340 read 0x037f98 thru 0x03ffff (0x8068 bytes) 000225 1788653068.109356 write 0x0349b3 thru 0x039bca (0x5218 bytes) 000229 1788653068.113912 trunc from 0x040000 to 0x036fb5 (0x904a bytes) 000230 1788653068.139447 trunc from 0x036fb6 to 0x009559 (0x2da5c bytes) 000231 1788653068.165238 mapread 0x001f8c thru 0x007d2a (0x5d9f bytes) 000234 1788653068.193701 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x955a): Mirror 4: 0x10040393 (stale) Mirror 5: 0x8f9e9d72 (stale) Mirror 6: 0x8f9e9d72 (stale) Mirror 7: 0x8f9e9d72 (stale) Mirror 8: 0xe2826fec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x955a): Mirror 4: 0x10040393 (stale) Mirror 5: 0x8f9e9d72 (stale) Mirror 6: 0x8f9e9d72 (stale) Mirror 7: 0x8f9e9d72 (stale) Mirror 8: 0xe2826fec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x955a: skipped 000237 1788653069.142384 read 0x008e0e thru 0x009559 (0x74c bytes) 000238 1788653069.163006 mapread 0x0071ff thru 0x007ec9 (0xccb bytes) 000240 1788653069.172370 mapread 0x007c06 thru 0x009559 (0x1954 bytes) 000241 1788653069.206701 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x955a): Mirror 5: 0x8f9e9d72 (stale) Mirror 6: 0x8f9e9d72 (stale) Mirror 7: 0x8f9e9d72 (stale) Mirror 8: 0xe2826fec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x955a): Mirror 5: 0x8f9e9d72 (stale) Mirror 6: 0x8f9e9d72 (stale) Mirror 7: 0x8f9e9d72 (stale) Mirror 8: 0xe2826fec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x955a: skipped 000242 1788653069.971590 read 0x000b79 thru 0x009559 (0x89e1 bytes) 000243 1788653069.995781 read 0x0002c7 thru 0x009559 (0x9293 bytes) 000245 1788653069.996876 trunc from 0x00955a to 0x0202d6 (0x16d7d bytes) 000246 1788653070.018269 punch from 0x012495 to 0x01513c (0x2ca8 bytes) 000247 1788653070.030430 write 0x0342e8 thru 0x0389a5 (0x46be bytes) 000248 1788653070.032297 mapread 0x002cad thru 0x0036e2 (0xa36 bytes) 000250 1788653070.050167 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x389a6): Mirror 5: 0x788f6e81 (stale) Mirror 6: 0x96ac2fc6 (stale) Mirror 8: 0xd70909ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389a6): Mirror 5: 0x788f6e81 (stale) Mirror 6: 0x96ac2fc6 (stale) Mirror 8: 0xd70909ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389a6: skipped 000251 1788653070.579272 MIRROR_RESYNC 4 mirrors 000252 1788653073.226711 mapread 0x03684b thru 0x0389a5 (0x215b bytes) 000253 1788653073.276264 MIRROR_RESYNC 4 mirrors 000255 1788653073.843033 punch from 0x01ff09 to 0x025497 (0x558f bytes) 000256 1788653073.909432 punch from 0x019945 to 0x0256b0 (0xbd6c bytes) 000257 1788653073.918642 mapwrite 0x0247a0 thru 0x0288a6 (0x4107 bytes) 000258 1788653073.981394 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x389a6): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389a6): Mirror 5: 0xebd4312 (stale) Mirror 6: 0xebd4312 (stale) Mirror 8: 0xebd4312 (stale) CRC-32 checksum value for chunk [0, 0x389a6): Mirror 1: 0x44cad4b3 Mirror 9: 0x44cad4b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389a6: skipped 000259 1788653075.650391 trunc from 0x0389a6 to 0x020bc2 (0x17de3 bytes) 000260 1788653075.913053 punch from 0x003657 to 0x0115cb (0xdf75 bytes) 000261 1788653075.939394 mapwrite 0x02887c thru 0x035138 (0xc8bd bytes) 000264 1788653076.163671 punch from 0x0149a8 to 0x01a6f1 (0x5d4a bytes) 000265 1788653076.173577 write 0x004ab2 thru 0x008aef (0x403e bytes) 000267 1788653076.191132 mapwrite 0x004188 thru 0x0050d6 (0xf4f bytes) 000272 1788653076.253964 read 0x03046b thru 0x0336dc (0x3272 bytes) 000273 1788653076.256407 mapread 0x030cea thru 0x035138 (0x444f bytes) 000275 1788653076.265295 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35139): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35139): Mirror 5: 0x3be1502d (stale) Mirror 6: 0x3be1502d (stale) Mirror 8: 0x3be1502d (stale) Mirror 9: 0x999c75ca (stale) CRC-32 checksum value for chunk [0, 0x35139): Mirror 1: 0x51315f66 Mirror 10: 0x51315f66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35139: skipped 000276 1788653078.060547 punch from 0x026171 to 0x027e43 (0x1cd3 bytes) 000277 1788653078.266685 MIRROR_RESYNC 6 mirrors 000279 1788653081.478503 read 0x0226cb thru 0x023289 (0xbbf bytes) 000280 1788653081.483719 read 0x01c1c1 thru 0x01f6dc (0x351c bytes) 000282 1788653081.495004 write 0x02a135 thru 0x02ad58 (0xc24 bytes) 000283 1788653081.578259 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35139): Mirror 5: 0x51315f66 (stale) Mirror 8: 0x51315f66 (stale) Mirror 9: 0x51315f66 (stale) Mirror 10: 0x51315f66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35139): Mirror 5: 0x51315f66 (stale) Mirror 8: 0x51315f66 (stale) Mirror 9: 0x51315f66 (stale) Mirror 10: 0x51315f66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35139: skipped 000285 1788653082.049805 trunc from 0x035139 to 0x03b4b9 (0x6381 bytes) 000288 1788653082.070594 punch from 0x039df5 to 0x03b4b9 (0x16c5 bytes) 000289 1788653082.076091 mapwrite 0x03da83 thru 0x03ffff (0x257d bytes) 000293 1788653082.152506 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7b6ae083 (stale) Mirror 8: 0x7b6ae083 (stale) Mirror 9: 0x7b6ae083 (stale) Mirror 10: 0x7b6ae083 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5cfd1f3c Mirror 11: 0x5cfd1f3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000294 1788653083.771156 trunc from 0x040000 to 0x0324be (0xdb41 bytes) 000295 1788653083.926166 write 0x02552b thru 0x02ecee (0x97c4 bytes) 000297 1788653083.951362 punch from 0x00c050 to 0x00c0e3 (0x94 bytes) 000298 1788653083.987433 mapread 0x023978 thru 0x0261dd (0x2866 bytes) 000299 1788653084.041370 mapread 0x004dc3 thru 0x008116 (0x3354 bytes) 000302 1788653084.069463 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x324bf): Mirror 5: 0xd14ab377 (stale) Mirror 8: 0xd14ab377 (stale) Mirror 9: 0xd14ab377 (stale) Mirror 10: 0xd14ab377 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x324bf): Mirror 5: 0xd14ab377 (stale) Mirror 8: 0xd14ab377 (stale) Mirror 9: 0xd14ab377 (stale) Mirror 10: 0xd14ab377 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x324bf: skipped 000303 1788653084.583289 mapwrite 0x034cf3 thru 0x03cee6 (0x81f4 bytes) 000305 1788653084.687474 mapread 0x0303ef thru 0x039b03 (0x9715 bytes) 000307 1788653084.689975 trunc from 0x03cee7 to 0x0121c6 (0x2ad20 bytes) 000308 1788653084.718276 write 0x0368d4 thru 0x03ffff (0x972c bytes) 000311 1788653084.725631 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7b6ae083 (stale) Mirror 8: 0x7b6ae083 (stale) Mirror 9: 0x7b6ae083 (stale) Mirror 10: 0x7b6ae083 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x233ef20e Mirror 11: 0x233ef20e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000312 1788653086.694900 read 0x0213e2 thru 0x0215d5 (0x1f4 bytes) 000313 1788653086.735194 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7b6ae083 (stale) Mirror 8: 0x7b6ae083 (stale) Mirror 9: 0x7b6ae083 (stale) Mirror 10: 0x7b6ae083 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x233ef20e Mirror 11: 0x233ef20e Mirror 12: 0x233ef20e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000314 1788653088.325797 trunc from 0x040000 to 0x00fbc4 (0x3043b bytes) 000315 1788653088.499279 mapread 0x00a37f thru 0x00b094 (0xd16 bytes) 000316 1788653088.512985 punch from 0x00d511 to 0x00d8f1 (0x3e1 bytes) 000317 1788653088.518728 mapwrite 0x01ba15 thru 0x01f66a (0x3c56 bytes) 000318 1788653088.602112 mapread 0x002e22 thru 0x00a752 (0x7931 bytes) 000319 1788653088.614158 punch from 0x0080b5 to 0x00afd0 (0x2f1c bytes) 000321 1788653088.621978 mapread 0x0026c1 thru 0x00ce14 (0xa754 bytes) 000322 1788653088.635530 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f66b): Mirror 8: 0xab6939e8 (stale) Mirror 9: 0xab6939e8 (stale) Mirror 10: 0xab6939e8 (stale) Mirror 11: 0xab6939e8 (stale) Mirror 12: 0xab6939e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f66b): Mirror 8: 0xab6939e8 (stale) Mirror 9: 0xab6939e8 (stale) Mirror 10: 0xab6939e8 (stale) Mirror 11: 0xab6939e8 (stale) Mirror 12: 0xab6939e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f66b: skipped 000323 1788653089.157698 read 0x011c0d thru 0x0133a1 (0x1795 bytes) 000324 1788653089.167964 trunc from 0x01f66b to 0x03a5af (0x1af45 bytes) 000325 1788653089.189154 trunc from 0x03a5b0 to 0x039af3 (0xabc bytes) 000326 1788653089.208083 mapwrite 0x00403a thru 0x01089a (0xc861 bytes) 000327 1788653089.334886 mapread 0x013013 thru 0x01d225 (0xa213 bytes) 000328 1788653089.386888 write 0x002c34 thru 0x00cf7e (0xa34b bytes) 000329 1788653089.390991 punch from 0x00e245 to 0x013c7b (0x5a37 bytes) 000330 1788653089.428930 read 0x016e5d thru 0x021ee8 (0xb08c bytes) 000333 1788653089.439645 mapread 0x01b8cc thru 0x01c318 (0xa4d bytes) 000334 1788653089.441628 trunc from 0x039af4 to 0x03aff4 (0x1501 bytes) 000335 1788653089.477321 mapwrite 0x022f91 thru 0x02fc02 (0xcc72 bytes) 000338 1788653089.577666 trunc from 0x03aff5 to 0x015ba7 (0x2544d bytes) 000340 1788653089.601164 write 0x03ffe1 thru 0x03ffff (0x1f bytes) 000342 1788653089.611267 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x7b6ae083 (stale) Mirror 9: 0x7b6ae083 (stale) Mirror 10: 0x7b6ae083 (stale) Mirror 11: 0x233ef20e (stale) Mirror 12: 0x233ef20e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x14ced5c0 Mirror 13: 0x14ced5c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000343 1788653091.590984 mapread 0x0336a4 thru 0x035673 (0x1fd0 bytes) 000346 1788653091.727087 trunc from 0x040000 to 0x02c791 (0x1386e bytes) 000347 1788653091.932429 mapread 0x01e4ae thru 0x02c791 (0xe2e4 bytes) 000348 1788653091.983626 mapwrite 0x028e58 thru 0x02bc2d (0x2dd6 bytes) 000349 1788653092.068307 mapwrite 0x01ae73 thru 0x024e9f (0xa02d bytes) 000350 1788653092.114645 write 0x03de8b thru 0x03eaac (0xc22 bytes) 000351 1788653092.115976 trunc from 0x03eaad to 0x0146e2 (0x2a3ca bytes) 000353 1788653092.137007 trunc from 0x0146e3 to 0x010db3 (0x392f bytes) 000354 1788653092.155977 write 0x007c08 thru 0x00cd72 (0x516b bytes) 000355 1788653092.167088 mapread 0x005190 thru 0x010db3 (0xbc24 bytes) 000356 1788653092.177701 punch from 0x00c588 to 0x010db3 (0x482c bytes) 000358 1788653092.195382 write 0x0321f9 thru 0x038ca1 (0x6aa9 bytes) 000360 1788653092.198044 mapwrite 0x00353a thru 0x00d93f (0xa406 bytes) 000361 1788653092.275414 write 0x035165 thru 0x03a755 (0x55f1 bytes) 000364 1788653092.284088 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a756): Mirror 8: 0xee2e62bf (stale) Mirror 9: 0xee2e62bf (stale) Mirror 10: 0xee2e62bf (stale) Mirror 12: 0x1aaf2b60 (stale) Mirror 13: 0x1b3b8a12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a756): Mirror 8: 0xee2e62bf (stale) Mirror 9: 0xee2e62bf (stale) Mirror 10: 0xee2e62bf (stale) Mirror 12: 0x1aaf2b60 (stale) Mirror 13: 0x1b3b8a12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a756: skipped 000367 1788653092.834914 write 0x02f6f2 thru 0x037328 (0x7c37 bytes) 000368 1788653092.863274 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a756): Mirror 8: 0xee2e62bf (stale) Mirror 9: 0xee2e62bf (stale) Mirror 12: 0x1aaf2b60 (stale) Mirror 13: 0x1b3b8a12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a756): Mirror 8: 0xee2e62bf (stale) Mirror 9: 0xee2e62bf (stale) Mirror 12: 0x1aaf2b60 (stale) Mirror 13: 0x1b3b8a12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a756: skipped 000369 1788653093.595248 trunc from 0x03a756 to 0x035b79 (0x4bdc bytes) 000371 1788653093.620413 mapwrite 0x00029f thru 0x008b23 (0x8885 bytes) 000374 1788653093.760391 mapread 0x00d492 thru 0x019e52 (0xc9c1 bytes) 000378 1788653093.796966 trunc from 0x035b7a to 0x0234fe (0x1267b bytes) 000379 1788653093.845355 read 0x00ba46 thru 0x012680 (0x6c3b bytes) 000380 1788653093.866314 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x234ff): Mirror 9: 0xb8ad69f7 (stale) Mirror 12: 0xb8ad69f7 (stale) Mirror 13: 0xc3c9b9c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x234ff): Mirror 9: 0xb8ad69f7 (stale) Mirror 12: 0xb8ad69f7 (stale) Mirror 13: 0xc3c9b9c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x234ff: skipped 000382 1788653094.376470 mapwrite 0x017078 thru 0x026dbd (0xfd46 bytes) 000383 1788653094.510741 read 0x00c435 thru 0x00e439 (0x2005 bytes) 000384 1788653094.527893 MIRROR_RESYNC 4 mirrors 000387 1788653096.054270 trunc from 0x026dbe to 0x0376f9 (0x1093c bytes) 000388 1788653096.185946 punch from 0x0015eb to 0x00e0de (0xcaf4 bytes) 000389 1788653096.196039 write 0x0263a7 thru 0x02eacc (0x8726 bytes) 000390 1788653096.212297 write 0x00132b thru 0x00e819 (0xd4ef bytes) 000392 1788653096.240636 MIRROR_RESYNC 4 mirrors 000393 1788653097.909941 write 0x0208a2 thru 0x0299de (0x913d bytes) 000394 1788653098.029362 write 0x03ee75 thru 0x03ffff (0x118b bytes) 000397 1788653098.042527 trunc from 0x040000 to 0x016d3e (0x292c1 bytes) 000399 1788653098.093578 mapread 0x004b4a thru 0x006638 (0x1aef bytes) 000400 1788653098.112948 punch from 0x0050a9 to 0x00bdd3 (0x6d2b bytes) 000401 1788653098.118512 read 0x00cdef thru 0x012e65 (0x6077 bytes) 000402 1788653098.131478 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16d3f): Mirror 12: 0xaf1eda1e (stale) Mirror 13: 0xaf1eda1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16d3f): Mirror 12: 0xaf1eda1e (stale) Mirror 13: 0xaf1eda1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16d3f: skipped 000403 1788653098.535719 trunc from 0x016d3f to 0x032d66 (0x1c028 bytes) 000407 1788653098.572069 mapread 0x019656 thru 0x0225d4 (0x8f7f bytes) 000409 1788653098.623480 write 0x018124 thru 0x01fc35 (0x7b12 bytes) 000411 1788653098.628139 punch from 0x0024b2 to 0x006874 (0x43c3 bytes) 000412 1788653098.673372 punch from 0x0257a3 to 0x02f768 (0x9fc6 bytes) 000414 1788653098.685239 mapread 0x01e2a3 thru 0x023e2e (0x5b8c bytes) 000415 1788653098.686302 mapread 0x02efeb thru 0x032d66 (0x3d7c bytes) 000416 1788653098.727441 trunc from 0x032d67 to 0x004df8 (0x2df6e bytes) 000418 1788653098.761205 punch from 0x002f62 to 0x004df8 (0x1e97 bytes) 000419 1788653098.776424 MIRROR_RESYNC 3 mirrors 000423 1788653099.969974 write 0x00b28e thru 0x0153ce (0xa141 bytes) 000424 1788653100.068868 mapwrite 0x02e2c6 thru 0x0378d4 (0x960f bytes) 000425 1788653100.245480 mapread 0x000ea9 thru 0x003368 (0x24c0 bytes) 000426 1788653100.292315 trunc from 0x0378d5 to 0x0155ad (0x22327 bytes) 000428 1788653100.320119 mapread 0x003a5b thru 0x00d3af (0x9955 bytes) 000429 1788653100.323146 mapread 0x00743c thru 0x00ef94 (0x7b59 bytes) 000431 1788653100.330562 mapread 0x015125 thru 0x0155ad (0x489 bytes) 000434 1788653100.341904 read 0x00762c thru 0x012e21 (0xb7f6 bytes) 000437 1788653100.363282 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x155ae): Mirror 13: 0xcf083e0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x155ae): Mirror 13: 0xcf083e0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x155ae: skipped 000438 1788653100.705720 punch from 0x0097e7 to 0x0155ad (0xbdc7 bytes) 000439 1788653100.713268 mapread 0x00b4f3 thru 0x0155ad (0xa0bb bytes) 000440 1788653100.734705 read 0x0150c3 thru 0x0155ad (0x4eb bytes) 000441 1788653100.735509 write 0x03c7a8 thru 0x03ffff (0x3858 bytes) 000442 1788653100.736880 mapread 0x03773a thru 0x039a55 (0x231c bytes) 000443 1788653100.746623 mapread 0x0218d6 thru 0x02e723 (0xce4e bytes) 000444 1788653100.763511 trunc from 0x040000 to 0x02cd4d (0x132b2 bytes) 000445 1788653100.782604 read 0x0142bf thru 0x022ea9 (0xebeb bytes) 000446 1788653100.792858 trunc from 0x02cd4e to 0x01c81b (0x10532 bytes) 000447 1788653100.811900 punch from 0x00c0fc to 0x01bc8a (0xfb8f bytes) 000449 1788653100.823561 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) 000450 1788653100.983370 mapwrite 0x01828b thru 0x0235ee (0xb364 bytes) 000451 1788653101.098408 write 0x021e6b thru 0x0271f4 (0x538a bytes) 000452 1788653101.099506 punch from 0x0110fe to 0x01371f (0x2622 bytes) 000453 1788653101.120620 write 0x030e55 thru 0x03bd60 (0xaf0c bytes) 000455 1788653101.126224 trunc from 0x03bd61 to 0x00a749 (0x31617 bytes) 000456 1788653101.166137 punch from 0x00a067 to 0x00a749 (0x6e3 bytes) 000457 1788653101.182939 mapwrite 0x00b209 thru 0x019ed9 (0xecd1 bytes) 000458 1788653101.322172 punch from 0x007a37 to 0x00e9df (0x6fa9 bytes) 000461 1788653101.329306 mapread 0x0105f6 thru 0x019ed9 (0x98e4 bytes) 000462 1788653101.334332 read 0x0016ae thru 0x00ee70 (0xd7c3 bytes) 000465 1788653101.354999 mapwrite 0x02d461 thru 0x03bc85 (0xe825 bytes) 000469 1788653101.556659 read 0x004147 thru 0x013742 (0xf5fc bytes) 000470 1788653101.557714 punch from 0x00ac00 to 0x015df4 (0xb1f5 bytes) 000471 1788653101.566510 mapwrite 0x0389fc thru 0x03ffff (0x7604 bytes) 000472 1788653101.737329 write 0x00bca8 thru 0x0111b7 (0x5510 bytes) 000473 1788653101.738699 read 0x03973d thru 0x03ffff (0x68c3 bytes) 000475 1788653101.741718 trunc from 0x040000 to 0x01724e (0x28db1 bytes) 000477 1788653101.761822 mapwrite 0x017178 thru 0x01b7af (0x4638 bytes) 000478 1788653101.852874 write 0x015090 thru 0x01b4c8 (0x6439 bytes) 000480 1788653101.857315 read 0x019636 thru 0x01aff0 (0x19bb bytes) 000481 1788653101.859435 mapwrite 0x002b23 thru 0x00c788 (0x9c66 bytes) 000484 1788653101.951222 trunc from 0x01b7b0 to 0x02a629 (0xee7a bytes) 000485 1788653101.973741 read 0x012cc7 thru 0x01867b (0x59b5 bytes) 000486 1788653101.975230 mapwrite 0x02dced thru 0x034e93 (0x71a7 bytes) 000487 1788653102.053011 mapread 0x02440c thru 0x033945 (0xf53a bytes) 000491 1788653102.057513 mapread 0x01694b thru 0x026592 (0xfc48 bytes) 000492 1788653102.058715 trunc from 0x034e94 to 0x02d8ec (0x75a7 bytes) 000495 1788653102.080665 mapread 0x016b6c thru 0x0219c3 (0xae58 bytes) 000496 1788653102.082339 read 0x021c8f thru 0x0255d6 (0x3948 bytes) 000497 1788653102.083527 punch from 0x0162ca to 0x01e998 (0x86cf bytes) 000501 1788653102.090695 mapread 0x004b7c thru 0x00b808 (0x6c8d bytes) 000502 1788653102.092442 mapwrite 0x01a990 thru 0x02a52e (0xfb9f bytes) 000503 1788653102.160910 read 0x022bb5 thru 0x02d8ec (0xad38 bytes) 000504 1788653102.161779 trunc from 0x02d8ed to 0x025631 (0x82bb bytes) 000505 1788653102.175501 punch from 0x010195 to 0x018f0e (0x8d7a bytes) 000506 1788653102.183660 mapread 0x0161b5 thru 0x01d354 (0x71a0 bytes) 000507 1788653102.196746 MIRROR_EXTEND to 2 mirrors 000508 1788653102.957384 punch from 0x016eeb to 0x01d867 (0x697d bytes) 000509 1788653103.060433 punch from 0x00c8e6 to 0x0180f7 (0xb812 bytes) 000511 1788653103.069484 read 0x016647 thru 0x022d77 (0xc731 bytes) 000514 1788653103.082892 read 0x007cdb thru 0x008b14 (0xe3a bytes) 000516 1788653103.095250 read 0x004998 thru 0x01273c (0xdda5 bytes) 000517 1788653103.096186 mapread 0x01a264 thru 0x0224e4 (0x8281 bytes) 000519 1788653103.098519 trunc from 0x025632 to 0x0368f2 (0x112c1 bytes) 000520 1788653103.145316 trunc from 0x0368f3 to 0x02d7c8 (0x912a bytes) 000521 1788653103.173019 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d7c9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d7c9): Mirror 2: 0xa4972bec (stale) CRC-32 checksum value for chunk [0, 0x2d7c9): Mirror 1: 0xd1e8b1c5 Mirror 3: 0xd1e8b1c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d7c9: skipped 000522 1788653104.516424 punch from 0x027db6 to 0x02d7c8 (0x5a13 bytes) 000523 1788653104.601612 read 0x00b777 thru 0x018dd6 (0xd660 bytes) 000524 1788653104.616340 read 0x007324 thru 0x014f4f (0xdc2c bytes) 000525 1788653104.631605 trunc from 0x02d7c9 to 0x0162aa (0x1751e bytes) 000526 1788653104.681124 trunc from 0x0162ab to 0x00b919 (0xa991 bytes) 000527 1788653104.698539 write 0x004ce3 thru 0x0086f1 (0x3a0f bytes) 000530 1788653104.702195 mapread 0x0029d6 thru 0x00b919 (0x8f44 bytes) 000531 1788653104.703970 punch from 0x000700 to 0x008f2a (0x882b bytes) 000532 1788653104.719378 read 0x008c1b thru 0x00b919 (0x2cff bytes) 000533 1788653104.729680 write 0x03b081 thru 0x03ffff (0x4f7f bytes) 000534 1788653104.731719 mapwrite 0x01553c thru 0x01dd08 (0x87cd bytes) 000535 1788653104.800171 write 0x02bd15 thru 0x02e899 (0x2b85 bytes) 000536 1788653104.802109 mapwrite 0x03790b thru 0x03c2b9 (0x49af bytes) 000537 1788653104.850180 mapwrite 0x026a6d thru 0x02c499 (0x5a2d bytes) 000540 1788653104.907382 punch from 0x018439 to 0x01d6f3 (0x52bb bytes) 000541 1788653104.914459 read 0x02851c thru 0x02ba67 (0x354c bytes) 000542 1788653104.915376 mapread 0x0000d6 thru 0x00de80 (0xddab bytes) 000543 1788653104.917619 read 0x037572 thru 0x03ffff (0x8a8e bytes) 000545 1788653104.918951 trunc from 0x040000 to 0x03e36d (0x1c92 bytes) 000547 1788653104.951645 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e36e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e36e): Mirror 2: 0xe16d6e2e (stale) Mirror 3: 0x36fbe243 (stale) CRC-32 checksum value for chunk [0, 0x3e36e): Mirror 1: 0x8fc71336 Mirror 4: 0x8fc71336 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e36e: skipped 000548 1788653106.587273 mapread 0x0204e8 thru 0x02766c (0x7185 bytes) 000550 1788653106.645264 read 0x0027b2 thru 0x00d0d7 (0xa926 bytes) 000551 1788653106.669521 write 0x019880 thru 0x0232ba (0x9a3b bytes) 000552 1788653106.805537 read 0x00cd8e thru 0x0145fe (0x7871 bytes) 000553 1788653106.833732 write 0x02072f thru 0x02688d (0x615f bytes) 000555 1788653106.840220 write 0x03e0e8 thru 0x03ffff (0x1f18 bytes) 000556 1788653106.852675 read 0x025603 thru 0x02f6ff (0xa0fd bytes) 000557 1788653106.858004 trunc from 0x040000 to 0x0370dd (0x8f22 bytes) 000558 1788653106.908609 mapwrite 0x00694d thru 0x0099aa (0x305e bytes) 000559 1788653107.000630 write 0x01d04c thru 0x01e35f (0x1314 bytes) 000561 1788653107.004022 mapread 0x020602 thru 0x02cc8f (0xc68e bytes) 000562 1788653107.006301 mapwrite 0x03fc0f thru 0x03ffff (0x3f1 bytes) 000564 1788653107.114559 mapread 0x00b73f thru 0x010a78 (0x533a bytes) 000567 1788653107.120157 read 0x00c35b thru 0x01c033 (0xfcd9 bytes) 000571 1788653107.125477 mapwrite 0x02e9ab thru 0x035969 (0x6fbf bytes) 000572 1788653107.198179 mapwrite 0x026935 thru 0x028caf (0x237b bytes) 000573 1788653107.264053 write 0x00095a thru 0x00f84e (0xeef5 bytes) 000575 1788653107.268323 punch from 0x00cf3e to 0x01a182 (0xd245 bytes) 000577 1788653107.303876 read 0x015523 thru 0x01db57 (0x8635 bytes) 000578 1788653107.322387 write 0x01f25f thru 0x02224e (0x2ff0 bytes) 000579 1788653107.325037 punch from 0x03b3e4 to 0x03d5d1 (0x21ee bytes) 000583 1788653107.358100 trunc from 0x040000 to 0x00c5df (0x33a20 bytes) 000587 1788653107.414594 trunc from 0x00c5e0 to 0x036dfa (0x2a81b bytes) 000589 1788653107.467422 mapwrite 0x00856c thru 0x008cb1 (0x746 bytes) 000590 1788653107.539845 punch from 0x023124 to 0x0236eb (0x5c8 bytes) 000593 1788653107.556521 write 0x001494 thru 0x00bf9e (0xab0b bytes) 000594 1788653107.564573 read 0x00dc3d thru 0x016184 (0x8548 bytes) 000595 1788653107.589389 write 0x00d159 thru 0x010e02 (0x3caa bytes) 000597 1788653107.598251 mapwrite 0x035a4f thru 0x03b084 (0x5636 bytes) 000598 1788653107.784489 read 0x00137c thru 0x00f65f (0xe2e4 bytes) 000600 1788653107.785637 punch from 0x02ca1c to 0x0313f1 (0x49d6 bytes) 000601 1788653107.800649 write 0x00d913 thru 0x015fce (0x86bc bytes) 000602 1788653107.811309 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b085): Mirror 3: 0x1e1d42e8 (stale) Mirror 4: 0xf016583f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b085): Mirror 3: 0x1e1d42e8 (stale) Mirror 4: 0xf016583f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b085: skipped 000603 1788653108.281470 read 0x006398 thru 0x007505 (0x116e bytes) 000604 1788653108.289721 write 0x01557a thru 0x01e549 (0x8fd0 bytes) 000605 1788653108.320720 read 0x0075d6 thru 0x011d39 (0xa764 bytes) 000607 1788653108.344189 trunc from 0x03b085 to 0x0316bd (0x99c7 bytes) 000608 1788653108.403133 mapwrite 0x03ccba thru 0x03ffff (0x3346 bytes) 000609 1788653108.513443 read 0x023e15 thru 0x0282e5 (0x44d1 bytes) 000611 1788653108.515357 trunc from 0x040000 to 0x0216a3 (0x1e95c bytes) 000616 1788653108.571605 write 0x03687f thru 0x03ffff (0x9781 bytes) 000618 1788653108.576129 mapwrite 0x0150da thru 0x024651 (0xf578 bytes) 000620 1788653108.666416 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x129d3d0c (stale) Mirror 4: 0x92662f40 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1f8b82a6 Mirror 5: 0x1f8b82a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000623 1788653110.363334 punch from 0x03a426 to 0x03ffff (0x5bda bytes) 000624 1788653110.498377 write 0x02ee49 thru 0x03a4ba (0xb672 bytes) 000625 1788653110.526195 MIRROR_RESYNC 4 mirrors 000626 1788653112.134701 read 0x03653b thru 0x03d51e (0x6fe4 bytes) 000628 1788653112.149897 write 0x0207a7 thru 0x02cd4d (0xc5a7 bytes) 000630 1788653112.277011 mapread 0x008ce3 thru 0x00b193 (0x24b1 bytes) 000631 1788653112.313060 trunc from 0x040000 to 0x025bf5 (0x1a40a bytes) 000632 1788653112.356290 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25bf6): Mirror 3: 0x35eac6c9 (stale) Mirror 4: 0x35eac6c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25bf6): Mirror 3: 0x35eac6c9 (stale) Mirror 4: 0x35eac6c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25bf6: skipped 000636 1788653112.875498 punch from 0x013869 to 0x014dfd (0x1595 bytes) 000637 1788653112.882648 mapwrite 0x00f3eb thru 0x01d22e (0xde44 bytes) 000638 1788653113.011447 read 0x008970 thru 0x0170d7 (0xe768 bytes) 000640 1788653113.041299 read 0x00c70c thru 0x01ab26 (0xe41b bytes) 000641 1788653113.047158 punch from 0x01d03d to 0x02396d (0x6931 bytes) 000642 1788653113.059332 mapwrite 0x017728 thru 0x022726 (0xafff bytes) 000643 1788653113.221238 read 0x021456 thru 0x025bf5 (0x47a0 bytes) 000644 1788653113.223640 punch from 0x009ad3 to 0x00ea3c (0x4f6a bytes) 000645 1788653113.268601 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25bf6): Mirror 4: 0x35eac6c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25bf6): Mirror 4: 0x35eac6c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25bf6: skipped 000646 1788653113.625653 mapwrite 0x0302ee thru 0x031da6 (0x1ab9 bytes) 000647 1788653113.729564 punch from 0x003a62 to 0x012950 (0xeeef bytes) 000648 1788653113.736901 trunc from 0x031da7 to 0x01dc44 (0x14162 bytes) 000649 1788653113.758544 trunc from 0x01dc45 to 0x0186f1 (0x5553 bytes) 000650 1788653113.791379 punch from 0x006572 to 0x013c90 (0xd71f bytes) 000653 1788653113.800671 mapread 0x012461 thru 0x014b04 (0x26a4 bytes) 000654 1788653113.814319 read 0x007776 thru 0x00bf38 (0x47c3 bytes) 000656 1788653113.821548 mapread 0x002331 thru 0x00e581 (0xc251 bytes) 000657 1788653113.830688 trunc from 0x0186f2 to 0x00ca8e (0xbc63 bytes) 000659 1788653113.852289 trunc from 0x00ca8f to 0x000248 (0xc846 bytes) 000664 1788653113.875572 read 0x000008 thru 0x000248 (0x241 bytes) 000665 1788653113.882429 punch from 0x000064 to 0x000248 (0x1e5 bytes) 000666 1788653113.887867 mapread 0x0000c2 thru 0x000248 (0x187 bytes) 000667 1788653113.898740 read 0x00008b thru 0x000248 (0x1be bytes) 000669 1788653113.901937 punch from 0x0000cb to 0x000248 (0x17e bytes) 000670 1788653113.910573 trunc from 0x000249 to 0x039c45 (0x399fd bytes) 000671 1788653113.932663 mapread 0x037b75 thru 0x039c45 (0x20d1 bytes) 000672 1788653113.959339 mapwrite 0x00023c thru 0x00aaaf (0xa874 bytes) 000673 1788653114.090186 mapwrite 0x01affb thru 0x0272e0 (0xc2e6 bytes) 000674 1788653114.187559 trunc from 0x039c46 to 0x02150f (0x18736 bytes) 000675 1788653114.237527 punch from 0x019713 to 0x02150f (0x7dfd bytes) 000676 1788653114.247895 trunc from 0x021510 to 0x00b2e9 (0x16226 bytes) 000677 1788653114.273947 mapread 0x00a9c1 thru 0x00b2e9 (0x929 bytes) 000678 1788653114.275428 punch from 0x007288 to 0x00b2e9 (0x4062 bytes) 000679 1788653114.281263 read 0x008375 thru 0x00b2e9 (0x2f75 bytes) 000680 1788653114.289840 mapread 0x002d52 thru 0x002ef4 (0x1a3 bytes) 000681 1788653114.292316 mapwrite 0x018f36 thru 0x0220f3 (0x91be bytes) 000682 1788653114.405441 trunc from 0x0220f4 to 0x01bd09 (0x63ea bytes) 000683 1788653114.441051 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1bd0a): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd0a): Mirror 4: 0xf633102b (stale) CRC-32 checksum value for chunk [0, 0x1bd0a): Mirror 1: 0x5316001 Mirror 5: 0x5316001 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd0a: skipped 000684 1788653115.705487 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1bd0a): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd0a): Mirror 4: 0xf633102b (stale) CRC-32 checksum value for chunk [0, 0x1bd0a): Mirror 1: 0x5316001 Mirror 5: 0x5316001 Mirror 6: 0x5316001 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd0a: skipped 000687 1788653117.090729 write 0x00f7cc thru 0x012847 (0x307c bytes) 000688 1788653117.215564 mapread 0x0121bf thru 0x014dba (0x2bfc bytes) 000689 1788653117.240120 trunc from 0x01bd0a to 0x00ab8b (0x1117e bytes) 000690 1788653117.275275 read 0x0082f8 thru 0x00ab8b (0x2894 bytes) 000697 1788653117.288493 mapwrite 0x02a952 thru 0x02d1df (0x288e bytes) 000698 1788653117.375515 mapwrite 0x00ed5a thru 0x00f6ed (0x994 bytes) 000699 1788653117.416298 write 0x0378ff thru 0x03a863 (0x2f65 bytes) 000700 1788653117.418340 punch from 0x002063 to 0x00d38d (0xb32b bytes) 000701 1788653117.433480 MIRROR_RESYNC 4 mirrors 000703 1788653119.310851 punch from 0x01f246 to 0x023889 (0x4644 bytes) 000704 1788653119.396520 read 0x0081b7 thru 0x01745d (0xf2a7 bytes) 000705 1788653119.417134 trunc from 0x03a864 to 0x02ebe6 (0xbc7d bytes) 000706 1788653119.463959 punch from 0x0006fc to 0x001ca9 (0x15ae bytes) 000708 1788653119.481213 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ebe7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ebe7): Mirror 4: 0xf63206a7 (stale) Mirror 5: 0xf63206a7 (stale) Mirror 6: 0xf63206a7 (stale) CRC-32 checksum value for chunk [0, 0x2ebe7): Mirror 1: 0x404c0f76 Mirror 7: 0x404c0f76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ebe7: skipped 000709 1788653120.975091 trunc from 0x02ebe7 to 0x03e62f (0xfa49 bytes) 000710 1788653121.129721 mapread 0x010636 thru 0x016981 (0x634c bytes) 000711 1788653121.176073 mapwrite 0x0328a5 thru 0x037417 (0x4b73 bytes) 000712 1788653121.295092 read 0x0313c0 thru 0x03e62f (0xd270 bytes) 000713 1788653121.312482 read 0x030411 thru 0x033750 (0x3340 bytes) 000714 1788653121.318332 MIRROR_RESYNC 5 mirrors 000715 1788653123.898469 trunc from 0x03e630 to 0x02d44b (0x111e4 bytes) 000718 1788653124.060216 mapread 0x007839 thru 0x016fdb (0xf7a3 bytes) 000720 1788653124.141961 trunc from 0x02d44c to 0x03f00d (0x11bc2 bytes) 000721 1788653124.181872 write 0x03d6c6 thru 0x03ffff (0x293a bytes) 000722 1788653124.190502 punch from 0x024084 to 0x02d226 (0x91a3 bytes) 000723 1788653124.213236 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8cb8cbb3 (stale) Mirror 5: 0x8cb8cbb3 (stale) Mirror 6: 0x8cb8cbb3 (stale) Mirror 7: 0x8cb8cbb3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa8c33756 Mirror 8: 0xa8c33756 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000724 1788653126.086166 MIRROR_RESYNC 6 mirrors 000725 1788653129.227946 mapread 0x02e3d1 thru 0x03ce00 (0xea30 bytes) 000727 1788653129.283137 MIRROR_EXTEND to 7 mirrors 000728 1788653131.133340 read 0x038eb5 thru 0x03f3a6 (0x64f2 bytes) 000730 1788653131.150333 read 0x00609b thru 0x00fd84 (0x9cea bytes) 000731 1788653131.196118 MIRROR_EXTEND to 8 mirrors 000733 1788653132.775092 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) 000734 1788653133.259562 mapread 0x017ff2 thru 0x018d60 (0xd6f bytes) 000736 1788653133.269546 write 0x0167fa thru 0x01bd8f (0x5596 bytes) 000739 1788653133.398940 punch from 0x02acc3 to 0x02e669 (0x39a7 bytes) 000740 1788653133.419420 mapread 0x01787c thru 0x0180e2 (0x867 bytes) 000741 1788653133.422148 mapread 0x02d11b thru 0x035121 (0x8007 bytes) 000742 1788653133.434072 mapwrite 0x03b5fb thru 0x03ffff (0x4a05 bytes) 000743 1788653133.500547 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa8c33756 (stale) Mirror 5: 0xa8c33756 (stale) Mirror 6: 0xa8c33756 (stale) Mirror 7: 0xa8c33756 (stale) Mirror 8: 0xa8c33756 (stale) Mirror 10: 0xa8c33756 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdb0eba7f Mirror 11: 0xdb0eba7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000744 1788653135.152493 mapread 0x005e24 thru 0x0067fe (0x9db bytes) 000745 1788653135.235598 read 0x035624 thru 0x036335 (0xd12 bytes) 000746 1788653135.262746 punch from 0x020afd to 0x0295fb (0x8aff bytes) 000748 1788653135.478826 mapwrite 0x013727 thru 0x014c9c (0x1576 bytes) 000749 1788653135.593412 read 0x028dd5 thru 0x0328e4 (0x9b10 bytes) 000751 1788653135.615339 mapwrite 0x001b27 thru 0x00d605 (0xbadf bytes) 000754 1788653135.724382 mapread 0x021a0c thru 0x0272dd (0x58d2 bytes) 000757 1788653135.727640 mapread 0x003c24 thru 0x006a42 (0x2e1f bytes) 000758 1788653135.731245 read 0x032f10 thru 0x03a906 (0x79f7 bytes) 000761 1788653135.733593 read 0x026fbc thru 0x0331bc (0xc201 bytes) 000762 1788653135.735648 punch from 0x01674c to 0x01a082 (0x3937 bytes) 000763 1788653135.741518 trunc from 0x040000 to 0x025efc (0x1a103 bytes) 000764 1788653135.790568 mapwrite 0x01e87e thru 0x0265e1 (0x7d64 bytes) 000765 1788653136.053086 write 0x0248c9 thru 0x0324e1 (0xdc19 bytes) 000767 1788653136.074642 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x324e2): Mirror 5: 0xb055f286 (stale) Mirror 6: 0xb055f286 (stale) Mirror 7: 0xb055f286 (stale) Mirror 8: 0xb055f286 (stale) Mirror 10: 0xb055f286 (stale) Mirror 11: 0x5016c2ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x324e2): Mirror 5: 0xb055f286 (stale) Mirror 6: 0xb055f286 (stale) Mirror 7: 0xb055f286 (stale) Mirror 8: 0xb055f286 (stale) Mirror 10: 0xb055f286 (stale) Mirror 11: 0x5016c2ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x324e2: skipped 000768 1788653136.778158 read 0x03181b thru 0x0324e1 (0xcc7 bytes) 000769 1788653136.794998 punch from 0x024923 to 0x02d025 (0x8703 bytes) 000770 1788653136.805830 mapwrite 0x023c08 thru 0x031e58 (0xe251 bytes) 000771 1788653137.021976 write 0x02717c thru 0x02c9e0 (0x5865 bytes) 000772 1788653137.029868 read 0x02ba6e thru 0x0324e1 (0x6a74 bytes) 000773 1788653137.047936 mapread 0x020a92 thru 0x024de3 (0x4352 bytes) 000775 1788653137.051533 mapread 0x00893c thru 0x013fe4 (0xb6a9 bytes) 000776 1788653137.082626 read 0x013c83 thru 0x015f67 (0x22e5 bytes) 000777 1788653137.083259 mapread 0x00d389 thru 0x016f8f (0x9c07 bytes) 000778 1788653137.084169 mapwrite 0x02405f thru 0x031352 (0xd2f4 bytes) 000780 1788653137.281374 punch from 0x00fb95 to 0x01a754 (0xabc0 bytes) 000781 1788653137.307120 read 0x007c4b thru 0x009f97 (0x234d bytes) 000782 1788653137.309653 read 0x00533e thru 0x011090 (0xbd53 bytes) 000784 1788653137.328774 read 0x0201d7 thru 0x02e07a (0xdea4 bytes) 000785 1788653137.349565 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x324e2): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x324e2): Mirror 5: 0xb055f286 (stale) Mirror 6: 0xb055f286 (stale) Mirror 7: 0xb055f286 (stale) Mirror 8: 0xb055f286 (stale) Mirror 10: 0xb055f286 (stale) Mirror 11: 0x5016c2ca (stale) CRC-32 checksum value for chunk [0, 0x324e2): Mirror 1: 0x2e987dd8 Mirror 12: 0x2e987dd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x324e2: skipped 000786 1788653139.545018 read 0x028b54 thru 0x02d990 (0x4e3d bytes) 000787 1788653139.614296 punch from 0x029fae to 0x03166f (0x76c2 bytes) 000788 1788653139.963356 punch from 0x018f8b to 0x025a88 (0xcafe bytes) 000790 1788653139.981746 punch from 0x01d9e0 to 0x02908a (0xb6ab bytes) 000792 1788653140.005975 punch from 0x028551 to 0x0324e1 (0x9f91 bytes) 000793 1788653140.015089 write 0x03d3be thru 0x03ffff (0x2c42 bytes) 000794 1788653140.017075 read 0x0229f5 thru 0x02eeff (0xc50b bytes) 000795 1788653140.045863 punch from 0x01672c to 0x018057 (0x192c bytes) 000797 1788653140.074285 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa8c33756 (stale) Mirror 6: 0xa8c33756 (stale) Mirror 7: 0xa8c33756 (stale) Mirror 8: 0xa8c33756 (stale) Mirror 10: 0xa8c33756 (stale) Mirror 11: 0xdb0eba7f (stale) Mirror 12: 0x6460346f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2d1926db Mirror 13: 0x2d1926db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000800 1788653142.754090 read 0x03e746 thru 0x03f28f (0xb4a bytes) 000801 1788653142.794744 write 0x01d9f6 thru 0x02b49f (0xdaaa bytes) 000802 1788653143.038240 MIRROR_RESYNC 9 mirrors 000803 1788653147.792726 MIRROR_EXTEND to 10 mirrors 000804 1788653149.222409 mapread 0x03127d thru 0x03befd (0xac81 bytes) 000805 1788653149.286224 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) 000807 1788653150.105358 mapread 0x00d11e thru 0x00f1a2 (0x2085 bytes) 000808 1788653150.134538 punch from 0x004e7b to 0x0147b9 (0xf93f bytes) 000810 1788653150.359948 trunc from 0x040000 to 0x032c82 (0xd37d bytes) 000811 1788653150.392045 mapwrite 0x01ddf8 thru 0x02b61b (0xd824 bytes) 000812 1788653150.653452 mapwrite 0x00ba9f thru 0x013ebb (0x841d bytes) 000813 1788653150.765304 punch from 0x02039a to 0x02977c (0x93e3 bytes) 000814 1788653150.782504 read 0x00dc02 thru 0x01147b (0x387a bytes) 000815 1788653150.785484 mapwrite 0x03a0cc thru 0x03f907 (0x583c bytes) 000817 1788653150.971424 trunc from 0x03f908 to 0x03ef3e (0x9c9 bytes) 000818 1788653151.026250 mapwrite 0x017674 thru 0x018d76 (0x1703 bytes) 000820 1788653151.162997 read 0x03536c thru 0x03ef3e (0x9bd3 bytes) 000822 1788653151.167052 mapread 0x03d6c2 thru 0x03ef3e (0x187d bytes) 000823 1788653151.176008 trunc from 0x03ef3f to 0x002ee9 (0x3c055 bytes) 000824 1788653151.219454 write 0x014b78 thru 0x01af27 (0x63b0 bytes) 000826 1788653151.234380 mapwrite 0x018758 thru 0x01a580 (0x1e29 bytes) 000827 1788653151.352902 mapwrite 0x01a52b thru 0x025d65 (0xb83b bytes) 000828 1788653151.455310 mapwrite 0x006174 thru 0x00d5b2 (0x743f bytes) 000829 1788653151.504421 read 0x00c9a6 thru 0x01557d (0x8bd8 bytes) 000830 1788653151.506455 trunc from 0x025d66 to 0x010684 (0x156e1 bytes) 000831 1788653151.543911 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10685): Mirror 5: 0x46a74f48 (stale) Mirror 6: 0x46a74f48 (stale) Mirror 7: 0x46a74f48 (stale) Mirror 8: 0x46a74f48 (stale) Mirror 10: 0x46a74f48 (stale) Mirror 11: 0x46a74f48 (stale) Mirror 14: 0x46a74f48 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10685): Mirror 5: 0x46a74f48 (stale) Mirror 6: 0x46a74f48 (stale) Mirror 7: 0x46a74f48 (stale) Mirror 8: 0x46a74f48 (stale) Mirror 10: 0x46a74f48 (stale) Mirror 11: 0x46a74f48 (stale) Mirror 14: 0x46a74f48 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10685: skipped 000832 1788653152.230458 trunc from 0x010685 to 0x01eac8 (0xe444 bytes) 000833 1788653152.277960 read 0x003eba thru 0x0099a4 (0x5aeb bytes) 000834 1788653152.290361 mapread 0x0108a0 thru 0x016605 (0x5d66 bytes) 000836 1788653152.337722 mapwrite 0x010e56 thru 0x011219 (0x3c4 bytes) 000837 1788653152.414819 punch from 0x003c74 to 0x006278 (0x2605 bytes) 000838 1788653152.424792 punch from 0x015165 to 0x01596b (0x807 bytes) 000839 1788653152.432628 read 0x017558 thru 0x01eac8 (0x7571 bytes) 000841 1788653152.445441 trunc from 0x01eac9 to 0x007650 (0x17478 bytes) 000842 1788653152.467341 write 0x0382d3 thru 0x03ffff (0x7d2d bytes) 000844 1788653152.474703 punch from 0x0179ec to 0x0200c4 (0x86d9 bytes) 000845 1788653152.489315 write 0x01368c thru 0x017fa6 (0x491b bytes) 000846 1788653152.517975 mapread 0x01f3e6 thru 0x023d97 (0x49b2 bytes) 000847 1788653152.521493 punch from 0x037c69 to 0x03ffff (0x8397 bytes) 000848 1788653152.551732 trunc from 0x040000 to 0x0388f0 (0x770f bytes) 000851 1788653152.592249 read 0x0116db thru 0x011b20 (0x446 bytes) 000852 1788653152.612631 punch from 0x016b14 to 0x02644b (0xf938 bytes) 000853 1788653152.626391 punch from 0x037869 to 0x0388f0 (0x1088 bytes) 000854 1788653152.636434 mapread 0x015706 thru 0x0206de (0xafd9 bytes) 000855 1788653152.790970 mapwrite 0x02f683 thru 0x033236 (0x3bb4 bytes) 000856 1788653152.892798 write 0x00ef1a thru 0x018711 (0x97f8 bytes) 000857 1788653152.893968 punch from 0x01a9f2 to 0x021419 (0x6a28 bytes) 000858 1788653152.948702 read 0x00f107 thru 0x011986 (0x2880 bytes) 000859 1788653152.950022 trunc from 0x0388f1 to 0x034a63 (0x3e8d bytes) 000860 1788653152.996147 write 0x004c04 thru 0x00a7ba (0x5bb7 bytes) 000861 1788653152.997587 trunc from 0x034a64 to 0x009d69 (0x2acfa bytes) 000862 1788653153.035239 mapread 0x000fb5 thru 0x009d69 (0x8db5 bytes) 000863 1788653153.036436 punch from 0x0037cc to 0x009d69 (0x659e bytes) 000865 1788653153.095680 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x9d6a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9d6a): Mirror 5: 0xc614f13c (stale) Mirror 6: 0xc614f13c (stale) Mirror 7: 0xc614f13c (stale) Mirror 8: 0xc614f13c (stale) Mirror 10: 0xc614f13c (stale) Mirror 11: 0xc614f13c (stale) Mirror 14: 0xc614f13c (stale) CRC-32 checksum value for chunk [0, 0x9d6a): Mirror 1: 0xdb1dbd37 Mirror 15: 0xdb1dbd37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9d6a: skipped 000866 1788653154.966734 MIRROR_RESYNC 9 mirrors 000867 1788653156.517944 MIRROR_RESYNC 9 mirrors 000868 1788653157.136524 trunc from 0x009d6a to 0x0154ba (0xb751 bytes) 000869 1788653157.293120 mapread 0x012ef6 thru 0x0154ba (0x25c5 bytes) 000871 1788653157.343131 read 0x00d184 thru 0x0154ba (0x8337 bytes) 000872 1788653157.344660 trunc from 0x0154bb to 0x02721b (0x11d61 bytes) 000873 1788653157.376562 mapwrite 0x008d08 thru 0x01856d (0xf866 bytes) 000874 1788653157.480342 read 0x00ec2b thru 0x01a95b (0xbd31 bytes) 000876 1788653157.482295 read 0x00c5ab thru 0x00eecf (0x2925 bytes) 000877 1788653157.483237 mapwrite 0x038790 thru 0x03d6ef (0x4f60 bytes) 000878 1788653157.590602 punch from 0x02f18a to 0x030859 (0x16d0 bytes) 000879 1788653157.602831 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d6f0): Mirror 5: 0xa6a00ca9 (stale) Mirror 6: 0xa6a00ca9 (stale) Mirror 8: 0xa6a00ca9 (stale) Mirror 10: 0xa6a00ca9 (stale) Mirror 11: 0xa6a00ca9 (stale) Mirror 14: 0xa6a00ca9 (stale) Mirror 15: 0xa6a00ca9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6f0): Mirror 5: 0xa6a00ca9 (stale) Mirror 6: 0xa6a00ca9 (stale) Mirror 8: 0xa6a00ca9 (stale) Mirror 10: 0xa6a00ca9 (stale) Mirror 11: 0xa6a00ca9 (stale) Mirror 14: 0xa6a00ca9 (stale) Mirror 15: 0xa6a00ca9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6f0: skipped 000880 1788653158.128599 punch from 0x028daf to 0x0338af (0xab01 bytes) 000881 1788653158.137722 punch from 0x030317 to 0x03cd9b (0xca85 bytes) 000882 1788653158.145362 mapread 0x00a385 thru 0x016551 (0xc1cd bytes) 000884 1788653158.213140 read 0x02c838 thru 0x0371a3 (0xa96c bytes) 000885 1788653158.226311 mapread 0x01d315 thru 0x01e6f9 (0x13e5 bytes) 000886 1788653158.256152 write 0x025a0a thru 0x02ed2b (0x9322 bytes) 000887 1788653158.263529 read 0x0170cf thru 0x019266 (0x2198 bytes) 000888 1788653158.267523 write 0x01bfda thru 0x0296ac (0xd6d3 bytes) 000889 1788653158.287091 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d6f0): Mirror 5: 0xa6a00ca9 (stale) Mirror 6: 0xa6a00ca9 (stale) Mirror 8: 0xa6a00ca9 (stale) Mirror 10: 0xa6a00ca9 (stale) Mirror 11: 0xa6a00ca9 (stale) Mirror 14: 0xa6a00ca9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6f0): Mirror 5: 0xa6a00ca9 (stale) Mirror 6: 0xa6a00ca9 (stale) Mirror 8: 0xa6a00ca9 (stale) Mirror 10: 0xa6a00ca9 (stale) Mirror 11: 0xa6a00ca9 (stale) Mirror 14: 0xa6a00ca9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6f0: skipped 000890 1788653159.084899 mapwrite 0x03473e thru 0x03e943 (0xa206 bytes) 000891 1788653159.316907 trunc from 0x03e944 to 0x02b023 (0x13920 bytes) 000894 1788653159.365093 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b024): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b024): Mirror 5: 0xe98475b6 (stale) Mirror 6: 0xe98475b6 (stale) Mirror 8: 0xe98475b6 (stale) Mirror 10: 0xe98475b6 (stale) Mirror 11: 0xe98475b6 (stale) Mirror 14: 0xe98475b6 (stale) CRC-32 checksum value for chunk [0, 0x2b024): Mirror 1: 0xa4b557d9 Mirror 15: 0xa4b557d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b024: skipped 000895 1788653160.828884 trunc from 0x02b024 to 0x035e57 (0xae34 bytes) 000896 1788653160.942161 trunc from 0x035e58 to 0x01eb33 (0x17324 bytes) 000898 1788653160.987732 write 0x001da2 thru 0x0119e9 (0xfc48 bytes) 000903 1788653160.999554 read 0x014bd0 thru 0x01eb33 (0x9f64 bytes) 000905 1788653161.033442 punch from 0x00b996 to 0x018fcc (0xd637 bytes) 000906 1788653161.068676 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1eb34): Mirror 6: 0x387c5d9d (stale) Mirror 8: 0x387c5d9d (stale) Mirror 10: 0x387c5d9d (stale) Mirror 11: 0x387c5d9d (stale) Mirror 14: 0x387c5d9d (stale) Mirror 15: 0xab75030b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eb34): Mirror 6: 0x387c5d9d (stale) Mirror 8: 0x387c5d9d (stale) Mirror 10: 0x387c5d9d (stale) Mirror 11: 0x387c5d9d (stale) Mirror 14: 0x387c5d9d (stale) Mirror 15: 0xab75030b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eb34: skipped 000907 1788653161.610155 mapread 0x018a51 thru 0x01a9ae (0x1f5e bytes) 000908 1788653161.631622 punch from 0x00e57c to 0x00f968 (0x13ed bytes) 000910 1788653161.637797 mapwrite 0x021b57 thru 0x02e18c (0xc636 bytes) 000911 1788653161.741623 read 0x017282 thru 0x01c69f (0x541e bytes) 000912 1788653161.748826 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e18d): Mirror 6: 0xbbacd630 (stale) Mirror 8: 0xbbacd630 (stale) Mirror 10: 0xbbacd630 (stale) Mirror 14: 0xbbacd630 (stale) Mirror 15: 0x4b676191 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e18d): Mirror 6: 0xbbacd630 (stale) Mirror 8: 0xbbacd630 (stale) Mirror 10: 0xbbacd630 (stale) Mirror 14: 0xbbacd630 (stale) Mirror 15: 0x4b676191 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e18d: skipped 000913 1788653162.275571 trunc from 0x02e18d to 0x021e00 (0xc38c bytes) 000914 1788653162.304646 write 0x006b1b thru 0x00be4a (0x5330 bytes) 000916 1788653162.323223 write 0x0108e1 thru 0x0134c2 (0x2be2 bytes) 000917 1788653162.342975 trunc from 0x021e01 to 0x02d333 (0xb533 bytes) 000918 1788653162.384106 read 0x011686 thru 0x01465e (0x2fd9 bytes) 000919 1788653162.396220 write 0x024b2a thru 0x030dfd (0xc2d4 bytes) 000920 1788653162.409038 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30dfe): Mirror 6: 0xc8fde70d (stale) Mirror 8: 0xc8fde70d (stale) Mirror 10: 0xc8fde70d (stale) Mirror 14: 0xc8fde70d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30dfe): Mirror 6: 0xc8fde70d (stale) Mirror 8: 0xc8fde70d (stale) Mirror 10: 0xc8fde70d (stale) Mirror 14: 0xc8fde70d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30dfe: skipped 000922 1788653162.881934 punch from 0x0254a8 to 0x026729 (0x1282 bytes) 000923 1788653162.890590 mapwrite 0x001b3b thru 0x010af7 (0xefbd bytes) 000924 1788653162.993436 read 0x0066c8 thru 0x00d4cf (0x6e08 bytes) 000925 1788653162.994343 read 0x021aba thru 0x0274cf (0x5a16 bytes) 000928 1788653163.009533 read 0x001863 thru 0x00bfe7 (0xa785 bytes) 000931 1788653163.012024 read 0x01c275 thru 0x025b42 (0x98ce bytes) 000932 1788653163.013350 read 0x02223e thru 0x026e5a (0x4c1d bytes) 000933 1788653163.014072 punch from 0x017ca3 to 0x01a03a (0x2398 bytes) 000935 1788653163.042654 trunc from 0x030dfe to 0x03f79a (0xe99d bytes) 000937 1788653163.082564 mapwrite 0x028cd2 thru 0x029be7 (0xf16 bytes) 000938 1788653163.180814 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 6: 0x9ccfdb4d (stale) Mirror 8: 0x9ccfdb4d (stale) Mirror 10: 0x9ccfdb4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 6: 0x9ccfdb4d (stale) Mirror 8: 0x9ccfdb4d (stale) Mirror 10: 0x9ccfdb4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f79b: skipped 000939 1788653163.680449 mapwrite 0x012f2c thru 0x016f13 (0x3fe8 bytes) 000940 1788653163.786874 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 8: 0x9ccfdb4d (stale) Mirror 10: 0x9ccfdb4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 8: 0x9ccfdb4d (stale) Mirror 10: 0x9ccfdb4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f79b: skipped 000941 1788653164.176417 mapread 0x031bbc thru 0x034139 (0x257e bytes) 000942 1788653164.202090 read 0x003f7a thru 0x010842 (0xc8c9 bytes) 000944 1788653164.215256 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 10: 0x9ccfdb4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 10: 0x9ccfdb4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f79b: skipped 000945 1788653164.629506 punch from 0x0278df to 0x02f201 (0x7923 bytes) 000947 1788653164.645898 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 10: 0x9ccfdb4d (stale) CRC-32 checksum value for chunk [0, 0x3f79b): Mirror 1: 0xbf4fb928 Mirror 11: 0xbf4fb928 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f79b: skipped 000948 1788653165.896033 write 0x0119dd thru 0x0160ff (0x4723 bytes) 000949 1788653166.009430 mapwrite 0x0033ee thru 0x005826 (0x2439 bytes) 000951 1788653166.100806 trunc from 0x03f79b to 0x029222 (0x16578 bytes) 000952 1788653166.139454 write 0x01b947 thru 0x02915a (0xd814 bytes) 000955 1788653166.157487 trunc from 0x029223 to 0x00fa1b (0x19807 bytes) 000958 1788653166.178261 trunc from 0x00fa1c to 0x019a6d (0xa052 bytes) 000959 1788653166.226603 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19a6e): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a6e): Mirror 10: 0xee6d0806 (stale) Mirror 11: 0x3c6fd0ce (stale) CRC-32 checksum value for chunk [0, 0x19a6e): Mirror 1: 0xfe45919f Mirror 12: 0xfe45919f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a6e: skipped 000960 1788653167.779906 trunc from 0x019a6e to 0x0159e1 (0x408c bytes) 000961 1788653167.874828 mapwrite 0x02fbf4 thru 0x030fe7 (0x13f4 bytes) 000962 1788653167.965118 trunc from 0x030fe8 to 0x01235f (0x1ec88 bytes) 000963 1788653167.983947 mapread 0x006f2c thru 0x00894a (0x1a1f bytes) 000964 1788653168.000921 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12360): Mirror 10: 0xd8716514 (stale) Mirror 11: 0x8fc16e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12360): Mirror 10: 0xd8716514 (stale) Mirror 11: 0x8fc16e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12360: skipped 000965 1788653168.388265 MIRROR_RESYNC 3 mirrors 000966 1788653169.435453 mapwrite 0x03a232 thru 0x03dd87 (0x3b56 bytes) 000967 1788653169.582304 read 0x013fae thru 0x021356 (0xd3a9 bytes) 000968 1788653169.593234 mapread 0x039640 thru 0x03dd87 (0x4748 bytes) 000970 1788653169.594700 read 0x039f70 thru 0x03dd87 (0x3e18 bytes) 000971 1788653169.595829 mapwrite 0x00bc01 thru 0x013db2 (0x81b2 bytes) 000972 1788653169.664519 punch from 0x009745 to 0x016cd7 (0xd593 bytes) 000973 1788653169.670580 mapwrite 0x02ecb3 thru 0x03e2a9 (0xf5f7 bytes) 000974 1788653169.805274 mapwrite 0x008e28 thru 0x00b996 (0x2b6f bytes) 000975 1788653169.919919 punch from 0x01c031 to 0x02bc2a (0xfbfa bytes) 000976 1788653169.935167 mapwrite 0x009bbb thru 0x00de39 (0x427f bytes) 000977 1788653170.077446 write 0x037a54 thru 0x03ffff (0x85ac bytes) 000978 1788653170.078687 punch from 0x03a270 to 0x03ffff (0x5d90 bytes) 000982 1788653170.137580 mapread 0x01c93e thru 0x02919d (0xc860 bytes) 000985 1788653170.277573 mapread 0x01ecb4 thru 0x0260b1 (0x73fe bytes) 000987 1788653170.283819 write 0x018e67 thru 0x02495a (0xbaf4 bytes) 000989 1788653170.287322 write 0x012054 thru 0x01b47b (0x9428 bytes) 000991 1788653170.301869 punch from 0x003ff2 to 0x010f95 (0xcfa4 bytes) 000992 1788653170.325954 punch from 0x000c8c to 0x009ac9 (0x8e3e bytes) 000993 1788653170.331543 punch from 0x027491 to 0x031a0d (0xa57d bytes) 000994 1788653170.337290 read 0x038a97 thru 0x03d478 (0x49e2 bytes) 000995 1788653170.350563 write 0x03a6c1 thru 0x03ffff (0x593f bytes) 000996 1788653170.357014 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x179:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xf0362e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xf0362e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000997 1788653170.640455 mapwrite 0x030b36 thru 0x038695 (0x7b60 bytes) 000998 1788653170.718093 trunc from 0x040000 to 0x032d5b (0xd2a4 bytes) 000999 1788653170.736371 mapread 0x012a76 thru 0x01618d (0x3718 bytes) 001000 1788653170.747009 punch from 0x00294a to 0x007fd0 (0x5687 bytes) All operations completed A-OK! PASS 70a (141s) == sanity-flr test 71: check mirror extend parent fid ==== 20:06:15 (1788653175) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:06:19 (1788653179) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267219 s, 7.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303172 s, 3.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 20:06:29 (1788653189) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3484320 .. Extending file size to 3586880 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4050336 .. Extending file size to 8280096 .. Extending file size to 7763520 .. Extending file size to 8305216 .. Extending file size to 8356992 .. Extending file size to 7953344 .. Extending file size to 8774208 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8513248 .. Extending file size to 9187552 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8827200 .. Waiting 57336 57338 57339 57342 57343 done 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) 192.168.204.152@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:07:54 (1788653274) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8640544 .. Extending file size to 1823872 .. Extending file size to 8699872 .. Extending file size to 8892640 .. Extending file size to 9110944 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8930432 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58798 58799 58801 58802 58804 done 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) 192.168.204.152@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (88s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:09:21 (1788653361) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117378 s, 17.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.509955 s, 4.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd6bd73ca (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:09:43 (1788653383) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135347 s, 15.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 20:09:51 (1788653391) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:10:01 (1788653401) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.289154 s, 7.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:10:11 (1788653411) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.292409 s, 7.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:10:21 (1788653421) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.431949 s, 4.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:10:56 (1788653456) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126366 s, 8.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } PASS 204d (66s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:12:02 (1788653522) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.258752 s, 8.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:13:06 (1788653586) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg452-server mds-ost sync done. Creating new pool test_204f oleg452-server: Pool lustre.test_204f created Adding targets to pool oleg452-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg452-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.234425 s, 8.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg452-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg452-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg452-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:14:29 (1788653669) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:14:38 (1788653678) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:14:47 (1788653687) oleg452-server: Pool lustre.test_206 created Adding targets to pool oleg452-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg452-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg452-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg452-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg452-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:15:15 (1788653715) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104324 s, 10.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982769 s, 10.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xec1ed46 Mirror 3: 0xec1ed46 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:15:29 (1788653729) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:15:33 (1788653733) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:15:37 (1788653737) Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.105655 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1590000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xdf0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xdf0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xdf0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1de0000 Read this (4 bytes): BBBB 192.168.204.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg452-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:16:16 (1788653776) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0157887 s, 519 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:16:26 (1788653786) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611523 s, 17.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f0 lmm_fid: [0x200000404:0x1f0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f0 lmm_fid: [0x200000404:0x1f0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834416 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 20:16:28.000000000 -0400 Modify: 2026-09-05 20:16:28.000000000 -0400 Change: 2026-09-05 20:16:30.000000000 -0400 Birth: 2026-09-05 20:16:28.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:16:37 (1788653797) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:16:42 (1788653802) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.403686 s, 101 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0627765 s, 65.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 20:16 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f2:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 20:16 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:16:52 (1788653812) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:17:02 (1788653822) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 1958 sec =========== 20:17:14 (1788653834) === sanity-flr: start cleanup 20:17:16 (1788653836) === === sanity-flr: finish cleanup 20:17:23 (1788653843) ===