-----============= acceptance-small: sanity-flr ============----- Wed Sep 30 12:09:30 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg309-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:09:47 (1790784587) === oleg309-client.virtnet: executing check_config_client /mnt/lustre oleg309-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg309-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b2042cd1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b2042cd1800.idle_timeout=debug disable quota as required oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:10:29 (1790784629) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8459] == sanity-flr test 0a: lfs mirror create with -N option == 12:10:31 (1790784631) PASS 0a (26s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:10:57 (1790784657) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:11:01 (1790784661) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:11:06 (1790784666) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:11:35 (1790784695) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:11:39 (1790784699) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:11:43 (1790784703) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b2042cd1800.stats=clear osc.lustre-OST0001-osc-ffff9b2042cd1800.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:11:54 (1790784714) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:12:08 (1790784728) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:12:18 (1790784738) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:12:22 (1790784742) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:12:32 (1790784752) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 12:12:42 (1790784762) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:12:51 (1790784771) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:12:59 (1790784779) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:13:08 (1790784788) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:13:18 (1790784798) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:13:25 (1790784805) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 2.48334 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115934 s, 9.0 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:13:38 (1790784818) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12016 s, 17.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:13:46 (1790784826) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:13:57 (1790784837) Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b2042cd1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b2042cd1800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b2042cd1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b2042cd1800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:15:08 (1790784908) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.273333 s, 19.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b2042cd1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b2042cd1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:15:49 (1790784949) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b2042cd1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b2042cd1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b2042cd1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b2042cd1800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:17:11 (1790785031) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.84593 s, 9.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 30 12:17 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.352s user 0m0.001s sys 0m0.200s Stopping /mnt/lustre-ost1 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b2042cd1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b2042cd1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.484s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg309-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg309-server oleg309-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b2042cd1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b2042cd1800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.407s user 0m0.001s sys 0m0.210s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg309-server: oleg309-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg309-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg309-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b2042cd1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b2042cd1800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:18:49 (1790785129) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:18:52 (1790785132) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:18:56 (1790785136) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:18:59 (1790785139) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 12:19:08 (1790785148) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.590232 s, 21.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:19:21 (1790785161) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279545 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557996 s, 18.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:19:29 (1790785169) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:19:38 (1790785178) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000743519 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101054 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000819666 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00290105 s, 353 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 12:19:50 (1790785190) create test files with size 15735523 .. 5cb9e8d4f95c3f98c8ec90ab94001ba8 - c279f7a07ee655fdbf44f1c20a0d669d - afbb43498cc0674250620348043b3474 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 12:20:17 (1790785217) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:20:37 (1790785237) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:20:45 (1790785245) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138764 s, 7.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444845 s, 23.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 12:20:56 (1790785256) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.277116 s, 7.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.594832 s, 7.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225847 s, 9.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 12:21:14 (1790785274) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 12:21:18 (1790785278) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:21:21 (1790785281) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03472 s, 30.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 12:21:29 (1790785289) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.376831 s, 8.3 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:21:40 (1790785300) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:21:51 (1790785311) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.300409 s, 34.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:22:02 (1790785322) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b2050294800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.325986 s, 32.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 12302 flags: 1 Stopping client oleg309-client.virtnet /mnt/lustre (opts:) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 12302 flags: 1 llite.lustre-ffff9b2047ac8800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:22:19 (1790785339) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:22:59 (1790785379) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:23:41 (1790785421) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:23:44 (1790785424) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.286655 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.243048 s, 12.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x26dcc30b (nosync,stale) Mirror 2: 0x26dcc30b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 78e8634d504607da0db2827b124325ad - mirror 1 checksum: ccbc7747b90796f69e6984c1b8c2d2b1 - mirror 2 checksum: 78e8634d504607da0db2827b124325ad - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-30T12:23:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-30T12:23:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 78e8634d504607da0db2827b124325ad - mirror 1 checksum: ccbc7747b90796f69e6984c1b8c2d2b1 - mirror 2 checksum: ccbc7747b90796f69e6984c1b8c2d2b1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-30T12:23:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-30T12:23:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:23:59 (1790785439) Failing mds1 on oleg309-server Stopping /mnt/lustre-mds1 (opts:) on oleg309-server 12:24:06 (1790785446) shut down facet: mds1 facet_host: oleg309-server facet_failover_host: oleg309-server Failover mds1 to oleg309-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg309-server: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg309-client: oleg309-server: ssh exited with exit code 1 Started lustre-MDT0000 12:24:32 (1790785472) targets are mounted 12:24:32 (1790785472) facet_failover done oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333927 s, 31.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:24:55 (1790785495) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:25:00 (1790785500) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:25:04 (1790785504) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0137389 s, 1.2 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:25:14 (1790785514) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:25:17 (1790785517) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00527532 s, 776 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.128435 s, 31.9 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:25:26 (1790785526) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x13:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:25:52 (1790785552) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:26:24 (1790785584) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:26:51 (1790785611) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:27:17 (1790785637) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 12:27:23 (1790785643) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 56404 mirror split pid 56405 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 56404 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 56405 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 12:28:31 (1790785711) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5714 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000003 1790785714.348423 write 0x036986 thru 0x03ffff (0x967a bytes) 000005 1790785714.350746 mapwrite 0x0385dd thru 0x03efe0 (0x6a04 bytes) 000007 1790785714.452871 read 0x03bbbd thru 0x03ffff (0x4443 bytes) 000008 1790785714.455975 mapread 0x02576b thru 0x0288a7 (0x313d bytes) truncating to largest ever: 0x34f76 000010 1790785714.517647 trunc from 0x040000 to 0x034f75 (0xb08a bytes) 000011 1790785714.554538 trunc from 0x034f76 to 0x001feb (0x32f8a bytes) 000014 1790785714.587500 read 0x00167d thru 0x001feb (0x96f bytes) 000015 1790785714.592598 mapwrite 0x024162 thru 0x02e6e5 (0xa584 bytes) 000017 1790785714.727543 write 0x015842 thru 0x024e5b (0xf61a bytes) 000018 1790785714.730720 write 0x02b635 thru 0x03a6c7 (0xf093 bytes) 000019 1790785714.748646 write 0x0127f6 thru 0x021a70 (0xf27b bytes) 000021 1790785714.755388 mapwrite 0x0177b6 thru 0x024f6b (0xd7b6 bytes) 000022 1790785714.796378 write 0x006e30 thru 0x01345b (0xc62c bytes) 000024 1790785714.798262 write 0x021212 thru 0x02e092 (0xce81 bytes) 000025 1790785714.800461 punch from 0x034148 to 0x039d45 (0x5bfe bytes) 000026 1790785714.817288 read 0x01561d thru 0x0180f1 (0x2ad5 bytes) 000029 1790785714.823026 mapwrite 0x01d1bc thru 0x02d037 (0xfe7c bytes) 000030 1790785714.903998 mapwrite 0x0050f5 thru 0x00a7a0 (0x56ac bytes) 000033 1790785714.983919 mapread 0x01459e thru 0x021950 (0xd3b3 bytes) truncating to largest ever: 0x3ce85 000034 1790785714.989387 trunc from 0x03a6c8 to 0x03ce84 (0x27bd bytes) 000035 1790785715.019194 mapwrite 0x02adec thru 0x036a81 (0xbc96 bytes) 000040 1790785715.164478 read 0x0218e8 thru 0x02e9de (0xd0f7 bytes) 000042 1790785715.166640 trunc from 0x03ce85 to 0x014068 (0x28e1c bytes) 000043 1790785715.214061 write 0x00029a thru 0x00d5c3 (0xd32a bytes) 000044 1790785715.223279 punch from 0x00175a to 0x006829 (0x50d0 bytes) 000045 1790785715.268478 mapread 0x00f27c thru 0x014068 (0x4ded bytes) 000046 1790785715.273223 trunc from 0x014069 to 0x019f39 (0x5ed1 bytes) 000047 1790785715.312409 read 0x019e75 thru 0x019f39 (0xc5 bytes) 000048 1790785715.329230 mapread 0x01330b thru 0x019f39 (0x6c2f bytes) 000049 1790785715.336929 punch from 0x0150c6 to 0x0189f6 (0x3931 bytes) 000050 1790785715.347516 punch from 0x00c0d0 to 0x019f39 (0xde6a bytes) 000051 1790785715.361084 mapread 0x00ea09 thru 0x01495b (0x5f53 bytes) 000052 1790785715.442738 mapwrite 0x02448e thru 0x02c8d8 (0x844b bytes) 000053 1790785715.594999 mapwrite 0x03f4e6 thru 0x03ffff (0xb1a bytes) 000055 1790785715.696663 write 0x0068f3 thru 0x01284f (0xbf5d bytes) 000058 1790785715.699751 mapwrite 0x033012 thru 0x03777f (0x476e bytes) 000059 1790785715.754011 trunc from 0x040000 to 0x008d90 (0x3726f bytes) 000060 1790785715.775741 trunc from 0x008d91 to 0x0215b0 (0x18820 bytes) 000061 1790785715.792182 trunc from 0x0215b1 to 0x002ee0 (0x1e6d0 bytes) 000062 1790785715.830074 mapwrite 0x022b5e thru 0x02e030 (0xb4d3 bytes) 000064 1790785715.980916 trunc from 0x02e031 to 0x0323ce (0x439e bytes) 000065 1790785715.998011 MIRROR_EXTEND to 2 mirrors 000067 1790785717.182240 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) 000069 1790785717.344561 read 0x0068a2 thru 0x0078a2 (0x1001 bytes) 000070 1790785717.352125 punch from 0x010b6e to 0x01e52f (0xd9c2 bytes) 000071 1790785717.370165 mapread 0x01916d thru 0x01ad31 (0x1bc5 bytes) 000072 1790785717.379486 mapread 0x00df47 thru 0x01598c (0x7a46 bytes) 000076 1790785717.388431 punch from 0x017ebd to 0x01978f (0x18d3 bytes) 000077 1790785717.393634 mapwrite 0x00275b thru 0x00f166 (0xca0c bytes) 000079 1790785717.451229 mapwrite 0x01eb03 thru 0x026bcb (0x80c9 bytes) 000080 1790785717.547621 trunc from 0x0323cf to 0x009c74 (0x2875a bytes) 000081 1790785717.564003 punch from 0x00379d to 0x009c74 (0x64d8 bytes) 000082 1790785717.570840 punch from 0x000a14 to 0x009c74 (0x9261 bytes) 000083 1790785717.580741 mapwrite 0x0260b9 thru 0x035b50 (0xfa98 bytes) 000084 1790785717.775861 write 0x01ccc7 thru 0x024724 (0x7a5e bytes) 000086 1790785717.794973 trunc from 0x035b51 to 0x00eba1 (0x26faf bytes) 000087 1790785717.831572 punch from 0x005aa9 to 0x00b511 (0x5a69 bytes) 000089 1790785717.846917 mapwrite 0x007b59 thru 0x014356 (0xc7fe bytes) 000090 1790785717.997853 read 0x00cd81 thru 0x014356 (0x75d6 bytes) 000091 1790785718.006863 write 0x01409a thru 0x018a48 (0x49af bytes) 000092 1790785718.020990 trunc from 0x018a49 to 0x021bdc (0x9194 bytes) 000096 1790785718.066079 punch from 0x014975 to 0x01f1f3 (0xa87f bytes) 000097 1790785718.109350 punch from 0x00145f to 0x002b27 (0x16c9 bytes) 000098 1790785718.118529 mapwrite 0x002c3e thru 0x00c321 (0x96e4 bytes) 000099 1790785718.214726 mapwrite 0x018b9f thru 0x01f0f8 (0x655a bytes) 000100 1790785718.288976 mapread 0x00e923 thru 0x01b388 (0xca66 bytes) 000101 1790785718.292567 read 0x019ad0 thru 0x021bdc (0x810d bytes) 000103 1790785718.295823 trunc from 0x021bdd to 0x02b307 (0x972b bytes) 000105 1790785718.317786 trunc from 0x02b308 to 0x038c81 (0xd97a bytes) 000106 1790785718.343150 mapread 0x02605d thru 0x030df5 (0xad99 bytes) 000107 1790785718.363852 punch from 0x027ba3 to 0x03623c (0xe69a bytes) 000108 1790785718.370772 read 0x013bc4 thru 0x01b8b6 (0x7cf3 bytes) 000112 1790785718.375117 trunc from 0x038c82 to 0x003f4c (0x34d35 bytes) 000113 1790785718.396380 read 0x000074 thru 0x003f4c (0x3ed9 bytes) 000116 1790785718.400102 mapread 0x002638 thru 0x003f4c (0x1915 bytes) 000118 1790785718.403510 trunc from 0x003f4d to 0x03c43b (0x384ef bytes) 000119 1790785718.420977 read 0x0368d8 thru 0x03c43b (0x5b64 bytes) 000121 1790785718.443984 trunc from 0x03c43c to 0x00ac0b (0x31830 bytes) 000122 1790785718.469164 read 0x001182 thru 0x00ac0b (0x9a8a bytes) 000124 1790785718.472578 read 0x002668 thru 0x00ac0b (0x85a4 bytes) 000126 1790785718.474826 mapwrite 0x013f37 thru 0x0177ef (0x38b9 bytes) 000128 1790785718.623216 punch from 0x00c374 to 0x011b4d (0x57da bytes) 000129 1790785718.632161 mapread 0x001aab thru 0x01036b (0xe8c1 bytes) 000130 1790785718.684325 punch from 0x00122c to 0x00564f (0x4424 bytes) 000131 1790785718.704409 trunc from 0x0177f0 to 0x0123c9 (0x5426 bytes) 000132 1790785718.731728 punch from 0x00aaab to 0x0123c9 (0x791f bytes) 000134 1790785718.736906 read 0x0047e2 thru 0x0123c9 (0xdbe8 bytes) 000135 1790785718.752177 write 0x03f2c7 thru 0x03ffff (0xd39 bytes) 000138 1790785718.759397 write 0x035418 thru 0x03634f (0xf38 bytes) 000139 1790785718.770126 mapwrite 0x01cd3f thru 0x01dbc9 (0xe8b bytes) 000140 1790785718.852500 trunc from 0x040000 to 0x02b34b (0x14cb4 bytes) 000141 1790785718.871399 read 0x01faf5 thru 0x028b04 (0x9010 bytes) 000142 1790785718.872704 mapread 0x01968a thru 0x020bec (0x7563 bytes) 000144 1790785718.876935 trunc from 0x02b34c to 0x03ace3 (0xf998 bytes) 000146 1790785718.899034 punch from 0x03a616 to 0x03ace3 (0x6ce bytes) 000147 1790785718.910250 write 0x021159 thru 0x021d77 (0xc1f bytes) 000148 1790785718.912463 mapwrite 0x010f8a thru 0x019dc0 (0x8e37 bytes) 000149 1790785718.975763 read 0x017e8f thru 0x01a1ed (0x235f bytes) 000154 1790785718.980929 punch from 0x02b45a to 0x02cfbe (0x1b65 bytes) 000156 1790785718.988687 punch from 0x02371f to 0x0257d8 (0x20ba bytes) 000157 1790785718.998335 read 0x02d5ed thru 0x035c03 (0x8617 bytes) 000160 1790785719.019591 MIRROR_EXTEND to 2 mirrors 000161 1790785720.156249 punch from 0x008e6a to 0x012984 (0x9b1b bytes) 000164 1790785720.342381 MIRROR_RESYNC 2 mirrors 000166 1790785721.476325 trunc from 0x03ace4 to 0x038ab6 (0x222d bytes) 000167 1790785721.589275 trunc from 0x038ab7 to 0x01a448 (0x1e66e bytes) 000168 1790785721.654929 write 0x00549b thru 0x014203 (0xed69 bytes) 000169 1790785721.674256 write 0x03a96a thru 0x03f307 (0x499e bytes) 000170 1790785721.679924 trunc from 0x03f308 to 0x02d217 (0x120f0 bytes) 000172 1790785721.701247 trunc from 0x02d218 to 0x0081b6 (0x25061 bytes) 000173 1790785721.717863 write 0x03f47e thru 0x03ffff (0xb82 bytes) 000174 1790785721.737848 trunc from 0x040000 to 0x008caa (0x37355 bytes) 000176 1790785721.758087 write 0x0192d3 thru 0x020d93 (0x7ac1 bytes) 000177 1790785721.766432 MIRROR_RESYNC 2 mirrors 000182 1790785722.622521 write 0x035126 thru 0x03ffff (0xaeda bytes) 000183 1790785722.759189 write 0x032092 thru 0x03a568 (0x84d7 bytes) 000184 1790785722.777867 punch from 0x00004a to 0x00bb20 (0xbad7 bytes) 000185 1790785722.807682 read 0x000855 thru 0x007b3a (0x72e6 bytes) 000187 1790785722.859241 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbea4972e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73c0681b Mirror 3: 0x73c0681b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000188 1790785724.503852 punch from 0x014e5b to 0x02197f (0xcb25 bytes) 000189 1790785724.700697 punch from 0x038a72 to 0x03ca4b (0x3fda bytes) 000190 1790785724.715425 write 0x013269 thru 0x01efd0 (0xbd68 bytes) 000192 1790785724.738629 MIRROR_RESYNC 3 mirrors 000195 1790785725.985027 mapread 0x017d28 thru 0x018d03 (0xfdc bytes) 000196 1790785726.015412 write 0x039c21 thru 0x03fcaa (0x608a bytes) 000198 1790785726.166669 punch from 0x01db4d to 0x025e5b (0x830f bytes) 000199 1790785726.219613 MIRROR_RESYNC 3 mirrors 000200 1790785727.589111 read 0x004e96 thru 0x00a4b2 (0x561d bytes) 000201 1790785727.604792 punch from 0x00e86f to 0x01ac65 (0xc3f7 bytes) 000202 1790785727.700604 mapread 0x0108e8 thru 0x0111a5 (0x8be bytes) 000204 1790785727.712699 write 0x03dd49 thru 0x03ffff (0x22b7 bytes) 000205 1790785727.718276 read 0x00ecb1 thru 0x015f67 (0x72b7 bytes) 000210 1790785727.747402 punch from 0x039a43 to 0x03ffff (0x65bd bytes) 000211 1790785727.761624 read 0x00a572 thru 0x016a6d (0xc4fc bytes) 000212 1790785727.762371 punch from 0x0325c1 to 0x03ffff (0xda3f bytes) 000213 1790785727.768747 write 0x02a2e3 thru 0x03627d (0xbf9b bytes) 000215 1790785727.804286 MIRROR_RESYNC 3 mirrors 000216 1790785728.878099 punch from 0x002242 to 0x00b169 (0x8f28 bytes) 000217 1790785728.951842 write 0x03ce14 thru 0x03ffff (0x31ec bytes) 000218 1790785728.965350 write 0x03110d thru 0x03aea2 (0x9d96 bytes) 000219 1790785728.979344 write 0x0055ea thru 0x014568 (0xef7f bytes) 000221 1790785729.000972 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbcebcb38 (stale) Mirror 3: 0xbcebcb38 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8ae9f722 Mirror 4: 0x8ae9f722 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000222 1790785730.277079 read 0x034dbe thru 0x039318 (0x455b bytes) 000225 1790785730.354796 mapwrite 0x02c5f9 thru 0x0323d8 (0x5de0 bytes) 000226 1790785730.628120 write 0x011d4e thru 0x014e0c (0x30bf bytes) 000228 1790785730.652749 mapwrite 0x0129e2 thru 0x018a6b (0x608a bytes) 000229 1790785730.788008 read 0x0277d8 thru 0x02896e (0x1197 bytes) 000231 1790785730.791267 punch from 0x005584 to 0x0132a1 (0xdd1e bytes) 000232 1790785730.802315 punch from 0x02d87b to 0x038b29 (0xb2af bytes) 000234 1790785730.815341 mapread 0x03f5c2 thru 0x03ffff (0xa3e bytes) 000235 1790785730.826635 read 0x03073f thru 0x0359de (0x52a0 bytes) 000236 1790785730.827328 trunc from 0x040000 to 0x0318cf (0xe730 bytes) 000237 1790785730.868189 write 0x0122ae thru 0x01e4a4 (0xc1f7 bytes) 000239 1790785730.881691 read 0x00d7df thru 0x01a580 (0xcda2 bytes) 000241 1790785730.885514 mapread 0x02edd9 thru 0x0318cf (0x2af7 bytes) 000242 1790785730.887501 punch from 0x0136cf to 0x018bd2 (0x5504 bytes) 000243 1790785730.911246 mapread 0x02f231 thru 0x0318cf (0x269f bytes) 000244 1790785730.913577 read 0x018f1f thru 0x01f256 (0x6338 bytes) 000247 1790785730.933462 mapread 0x01b622 thru 0x0236fc (0x80db bytes) 000248 1790785730.936305 read 0x01c234 thru 0x023cb2 (0x7a7f bytes) 000251 1790785730.939115 trunc from 0x0318d0 to 0x02a0b8 (0x7817 bytes) 000252 1790785730.965936 mapread 0x022db3 thru 0x02a0b8 (0x7306 bytes) 000253 1790785730.967497 trunc from 0x02a0b9 to 0x02e6f6 (0x463e bytes) 000254 1790785731.003552 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e6f7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6f7): Mirror 2: 0x432f08a0 (stale) Mirror 3: 0x432f08a0 (stale) Mirror 4: 0x8219840f (stale) CRC-32 checksum value for chunk [0, 0x2e6f7): Mirror 1: 0x8c696e9d Mirror 5: 0x8c696e9d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6f7: skipped 000255 1790785732.175773 mapwrite 0x011a12 thru 0x021626 (0xfc15 bytes) 000256 1790785732.580740 trunc from 0x02e6f7 to 0x01dee3 (0x10813 bytes) 000257 1790785732.625633 MIRROR_RESYNC 5 mirrors 000259 1790785734.013474 punch from 0x01cbc4 to 0x01dee3 (0x1320 bytes) 000260 1790785734.187970 write 0x013aa1 thru 0x01b4c5 (0x7a25 bytes) 000261 1790785734.207908 punch from 0x019c39 to 0x01d6b2 (0x3a7a bytes) 000262 1790785734.241018 trunc from 0x01dee4 to 0x035904 (0x17a21 bytes) 000264 1790785734.286918 mapread 0x004e22 thru 0x01091d (0xbafc bytes) 000265 1790785734.321817 mapwrite 0x0010bc thru 0x00f07a (0xdfbf bytes) 000266 1790785734.436759 mapwrite 0x024e09 thru 0x032d86 (0xdf7e bytes) 000268 1790785734.566937 punch from 0x00ac18 to 0x0178bc (0xcca5 bytes) 000272 1790785734.585783 write 0x021533 thru 0x02bf9a (0xaa68 bytes) 000274 1790785734.589303 read 0x022466 thru 0x02a528 (0x80c3 bytes) 000275 1790785734.597218 write 0x033ac7 thru 0x033b06 (0x40 bytes) 000276 1790785734.599519 write 0x003cb2 thru 0x006005 (0x2354 bytes) 000277 1790785734.608618 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35905): Mirror 3: 0xb94c6451 (stale) Mirror 4: 0xb94c6451 (stale) Mirror 5: 0xb94c6451 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35905): Mirror 3: 0xb94c6451 (stale) Mirror 4: 0xb94c6451 (stale) Mirror 5: 0xb94c6451 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35905: skipped 000278 1790785735.107837 write 0x036e2f thru 0x03dae7 (0x6cb9 bytes) 000279 1790785735.121879 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dae8): Mirror 3: 0x640823ee (stale) Mirror 4: 0x640823ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dae8): Mirror 3: 0x640823ee (stale) Mirror 4: 0x640823ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dae8: skipped 000282 1790785735.636609 mapread 0x03165b thru 0x03dae7 (0xc48d bytes) 000284 1790785735.683172 write 0x005b0e thru 0x00e214 (0x8707 bytes) 000286 1790785735.713905 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dae8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dae8): Mirror 3: 0x640823ee (stale) Mirror 4: 0x640823ee (stale) CRC-32 checksum value for chunk [0, 0x3dae8): Mirror 1: 0x9e2c60e1 Mirror 5: 0x9e2c60e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dae8: skipped 000288 1790785737.140086 trunc from 0x03dae8 to 0x012a99 (0x2b04e bytes) 000289 1790785737.251942 write 0x03f4d8 thru 0x03ffff (0xb28 bytes) 000292 1790785737.255926 mapread 0x02c10a thru 0x03a30c (0xe203 bytes) 000293 1790785737.287772 read 0x02eadc thru 0x03523f (0x6764 bytes) 000296 1790785737.291864 punch from 0x005a5f to 0x007909 (0x1eab bytes) 000297 1790785737.301958 write 0x015855 thru 0x019f0a (0x46b6 bytes) 000298 1790785737.315829 trunc from 0x040000 to 0x001aa7 (0x3e558 bytes) 000301 1790785737.338101 punch from 0x00191b to 0x001aa7 (0x18d bytes) 000302 1790785737.344957 read 0x00177f thru 0x001aa7 (0x329 bytes) 000303 1790785737.356105 MIRROR_RESYNC 4 mirrors 000304 1790785738.708176 trunc from 0x001aa8 to 0x015f23 (0x1447c bytes) 000305 1790785738.811184 trunc from 0x015f24 to 0x03a1de (0x242bb bytes) 000307 1790785738.836650 write 0x00709b thru 0x0079e3 (0x949 bytes) 000308 1790785738.848855 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a1df): Mirror 3: 0x3837b376 (stale) Mirror 5: 0x3837b376 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1df): Mirror 3: 0x3837b376 (stale) Mirror 5: 0x3837b376 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1df: skipped 000310 1790785739.308224 write 0x015284 thru 0x01adf1 (0x5b6e bytes) 000311 1790785739.322018 read 0x01ebd7 thru 0x02d867 (0xec91 bytes) 000312 1790785739.344828 punch from 0x008cb2 to 0x00c421 (0x3770 bytes) truncating to largest ever: 0x3dc39 000313 1790785739.360739 trunc from 0x03a1df to 0x03dc38 (0x3a5a bytes) 000314 1790785739.389813 write 0x02bc50 thru 0x037d32 (0xc0e3 bytes) 000315 1790785739.400911 punch from 0x0304d4 to 0x03c642 (0xc16f bytes) 000316 1790785739.425168 read 0x019666 thru 0x020a7a (0x7415 bytes) 000318 1790785739.447498 mapread 0x02e40f thru 0x02f8ba (0x14ac bytes) 000321 1790785739.461896 MIRROR_RESYNC 3 mirrors 000323 1790785740.657011 trunc from 0x03dc39 to 0x011ab0 (0x2c188 bytes) 000326 1790785740.773845 punch from 0x000cc7 to 0x00e712 (0xda4c bytes) 000327 1790785740.787214 punch from 0x00ebb1 to 0x011ab0 (0x2f00 bytes) 000328 1790785740.795938 punch from 0x00342b to 0x010a25 (0xd5fb bytes) 000333 1790785740.807192 read 0x00c39b thru 0x011ab0 (0x5716 bytes) 000334 1790785740.818886 trunc from 0x011ab1 to 0x014ce0 (0x3230 bytes) 000336 1790785740.843997 read 0x00415b thru 0x00ff14 (0xbdba bytes) 000337 1790785740.858651 read 0x00fd0d thru 0x014ce0 (0x4fd4 bytes) 000339 1790785740.861972 punch from 0x010d9a to 0x0132ed (0x2554 bytes) 000341 1790785740.869391 read 0x00e54d thru 0x0120bc (0x3b70 bytes) 000344 1790785740.879600 trunc from 0x014ce1 to 0x0288a3 (0x13bc3 bytes) 000345 1790785740.902805 mapread 0x003387 thru 0x00458d (0x1207 bytes) 000346 1790785740.905862 write 0x011bec thru 0x0199bb (0x7dd0 bytes) 000348 1790785740.915422 write 0x03044e thru 0x03119d (0xd50 bytes) 000349 1790785740.922109 write 0x01ba84 thru 0x01fe72 (0x43ef bytes) 000350 1790785740.957123 MIRROR_RESYNC 3 mirrors 000351 1790785742.635439 mapwrite 0x03991d thru 0x03ffff (0x66e3 bytes) 000352 1790785742.820780 punch from 0x00d548 to 0x012667 (0x5120 bytes) 000353 1790785742.829205 mapwrite 0x011406 thru 0x011745 (0x340 bytes) 000354 1790785742.895962 read 0x00bfda thru 0x0174bd (0xb4e4 bytes) 000355 1790785742.898087 punch from 0x0100dd to 0x01f732 (0xf656 bytes) 000357 1790785742.904724 trunc from 0x040000 to 0x02d75d (0x128a2 bytes) 000358 1790785742.926193 mapwrite 0x0171d4 thru 0x0233a6 (0xc1d3 bytes) 000359 1790785743.004671 punch from 0x01f63d to 0x02559e (0x5f62 bytes) 000360 1790785743.010798 trunc from 0x02d75e to 0x02e19e (0xa41 bytes) 000361 1790785743.032952 punch from 0x026f1d to 0x02e19e (0x7282 bytes) 000362 1790785743.040198 punch from 0x01496b to 0x015bf6 (0x128c bytes) 000363 1790785743.045508 mapread 0x01f24d thru 0x024a36 (0x57ea bytes) 000364 1790785743.057273 read 0x01ffda thru 0x02af02 (0xaf29 bytes) 000365 1790785743.080277 punch from 0x026fa3 to 0x029992 (0x29f0 bytes) 000368 1790785743.086986 punch from 0x023bff to 0x028bb0 (0x4fb2 bytes) 000369 1790785743.096717 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e19f): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e19f): Mirror 3: 0xe12707c7 (stale) Mirror 5: 0xe12707c7 (stale) CRC-32 checksum value for chunk [0, 0x2e19f): Mirror 1: 0xa387af Mirror 6: 0xa387af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e19f: skipped 000370 1790785744.362667 punch from 0x0083f6 to 0x012ac3 (0xa6ce bytes) 000372 1790785744.471179 mapwrite 0x025f17 thru 0x031478 (0xb562 bytes) 000373 1790785744.668375 MIRROR_RESYNC 4 mirrors 000375 1790785745.782940 mapwrite 0x00237d thru 0x003fd2 (0x1c56 bytes) 000378 1790785746.008808 write 0x01ef51 thru 0x02509a (0x614a bytes) 000379 1790785746.033761 read 0x0088c7 thru 0x00ea62 (0x619c bytes) 000382 1790785746.049746 trunc from 0x031479 to 0x002ce7 (0x2e791 bytes) 000383 1790785746.119031 write 0x037456 thru 0x0398a1 (0x244c bytes) 000384 1790785746.120303 trunc from 0x0398a2 to 0x021099 (0x18808 bytes) 000386 1790785746.146153 mapread 0x005805 thru 0x011153 (0xb94f bytes) 000388 1790785746.198893 mapwrite 0x00892b thru 0x01531e (0xc9f4 bytes) 000390 1790785746.275249 mapread 0x0110b7 thru 0x012461 (0x13ab bytes) 000391 1790785746.276805 write 0x013d6c thru 0x0144ea (0x77f bytes) 000392 1790785746.282385 mapwrite 0x031fc5 thru 0x03ee68 (0xcea4 bytes) 000394 1790785746.384892 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ee69): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee69): Mirror 3: 0xb9bef7af (stale) Mirror 5: 0xb9bef7af (stale) Mirror 6: 0xb9bef7af (stale) CRC-32 checksum value for chunk [0, 0x3ee69): Mirror 1: 0x2e55a3f3 Mirror 7: 0x2e55a3f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee69: skipped 000395 1790785747.982856 trunc from 0x03ee69 to 0x02a2a8 (0x14bc0 bytes) 000396 1790785748.154970 mapwrite 0x01327d thru 0x01544a (0x21ce bytes) 000398 1790785748.232453 write 0x006cf9 thru 0x011c26 (0xaf2e bytes) 000401 1790785748.252633 read 0x0079bc thru 0x016f7c (0xf5c1 bytes) 000404 1790785748.260177 mapread 0x016740 thru 0x024120 (0xd9e1 bytes) 000405 1790785748.290861 read 0x00c397 thru 0x01bdc5 (0xfa2f bytes) 000406 1790785748.292037 mapwrite 0x024fc1 thru 0x034675 (0xf6b5 bytes) 000407 1790785748.415366 write 0x03c794 thru 0x03ffff (0x386c bytes) 000408 1790785748.416865 read 0x020ad9 thru 0x02693f (0x5e67 bytes) 000409 1790785748.421804 mapread 0x03ff0f thru 0x03ffff (0xf1 bytes) 000410 1790785748.423484 punch from 0x00f02e to 0x01bba1 (0xcb74 bytes) 000411 1790785748.435648 trunc from 0x040000 to 0x001599 (0x3ea66 bytes) 000412 1790785748.462346 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x159a): Mirror 3: 0x931c143c (stale) Mirror 5: 0x931c143c (stale) Mirror 7: 0x931c143c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x159a): Mirror 3: 0x931c143c (stale) Mirror 5: 0x931c143c (stale) Mirror 7: 0x931c143c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x159a: skipped 000414 1790785748.925890 mapwrite 0x01f517 thru 0x01ff58 (0xa42 bytes) 000416 1790785749.029405 mapwrite 0x010f77 thru 0x01107a (0x104 bytes) 000417 1790785749.142944 punch from 0x00fe40 to 0x014917 (0x4ad8 bytes) truncating to largest ever: 0x3f362 000422 1790785749.159908 trunc from 0x01ff59 to 0x03f361 (0x1f409 bytes) 000424 1790785749.193270 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f362): Mirror 5: 0xd01cb143 (stale) Mirror 7: 0xfd55dcaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f362): Mirror 5: 0xd01cb143 (stale) Mirror 7: 0xfd55dcaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f362: skipped 000425 1790785749.541913 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f362): Mirror 7: 0xfd55dcaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f362): Mirror 7: 0xfd55dcaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f362: skipped 000426 1790785749.935765 mapwrite 0x016ae1 thru 0x01dad3 (0x6ff3 bytes) 000427 1790785750.046246 write 0x00b0ff thru 0x010974 (0x5876 bytes) 000428 1790785750.090631 trunc from 0x03f362 to 0x00d19e (0x321c3 bytes) 000429 1790785750.168017 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xd19f): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd19f): Mirror 7: 0x58fb50be (stale) CRC-32 checksum value for chunk [0, 0xd19f): Mirror 1: 0x19679491 Mirror 8: 0x19679491 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd19f: skipped 000430 1790785751.500683 punch from 0x0045f6 to 0x00d19e (0x8ba9 bytes) 000431 1790785751.736413 mapwrite 0x000198 thru 0x00eebf (0xed28 bytes) 000432 1790785751.945257 write 0x001188 thru 0x00488c (0x3705 bytes) 000433 1790785751.946326 mapwrite 0x00bc94 thru 0x018fdc (0xd349 bytes) 000434 1790785752.217883 mapwrite 0x03be32 thru 0x03f69d (0x386c bytes) 000435 1790785752.415318 punch from 0x02347c to 0x02ca69 (0x95ee bytes) 000436 1790785752.429840 trunc from 0x03f69e to 0x02e3d3 (0x112ca bytes) 000437 1790785752.475128 write 0x02d3c3 thru 0x034b42 (0x7780 bytes) 000438 1790785752.492199 trunc from 0x034b43 to 0x033cc9 (0xe79 bytes) 000439 1790785752.508065 read 0x00d9b5 thru 0x01b7de (0xde2a bytes) 000440 1790785752.518489 punch from 0x0306c3 to 0x033451 (0x2d8f bytes) 000442 1790785752.551128 read 0x0233bf thru 0x023753 (0x395 bytes) 000443 1790785752.568421 mapread 0x0253e8 thru 0x032470 (0xd089 bytes) 000444 1790785752.585562 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33cca): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cca): Mirror 7: 0xb77b7c19 (stale) Mirror 8: 0x98aa17d8 (stale) CRC-32 checksum value for chunk [0, 0x33cca): Mirror 1: 0xa8cec77c Mirror 9: 0xa8cec77c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cca: skipped 000445 1790785754.012724 mapwrite 0x034bf2 thru 0x03a89c (0x5cab bytes) 000447 1790785754.289207 write 0x02a549 thru 0x02cdcd (0x2885 bytes) 000449 1790785754.312756 MIRROR_RESYNC 4 mirrors 000450 1790785755.836984 mapwrite 0x023c27 thru 0x0282b6 (0x4690 bytes) 000452 1790785756.032252 read 0x00254a thru 0x006258 (0x3d0f bytes) 000453 1790785756.045645 read 0x012224 thru 0x016a1a (0x47f7 bytes) 000454 1790785756.057976 write 0x020bff thru 0x02b959 (0xad5b bytes) 000455 1790785756.060178 read 0x01cf3c thru 0x02a5af (0xd674 bytes) 000456 1790785756.062421 punch from 0x016127 to 0x01e05a (0x7f34 bytes) 000457 1790785756.088653 write 0x021016 thru 0x030755 (0xf740 bytes) 000458 1790785756.093026 mapread 0x00fecf thru 0x01e8b9 (0xe9eb bytes) 000459 1790785756.121461 MIRROR_RESYNC 4 mirrors 000460 1790785757.265567 trunc from 0x03a89d to 0x018d03 (0x21b99 bytes) 000461 1790785757.372418 trunc from 0x018d04 to 0x006efb (0x11e08 bytes) 000463 1790785757.394337 write 0x022878 thru 0x022d11 (0x49a bytes) 000465 1790785757.400668 MIRROR_RESYNC 4 mirrors 000468 1790785759.227520 write 0x007b4d thru 0x012c82 (0xb136 bytes) 000469 1790785759.470194 mapwrite 0x020d6e thru 0x027423 (0x66b6 bytes) 000470 1790785759.682695 mapread 0x0094c4 thru 0x012fdd (0x9b1a bytes) 000475 1790785759.707943 MIRROR_RESYNC 4 mirrors 000476 1790785761.146866 trunc from 0x027424 to 0x013eca (0x13559 bytes) 000478 1790785761.244610 punch from 0x00c45a to 0x00f4cf (0x3076 bytes) 000479 1790785761.252578 read 0x0047cb thru 0x00d335 (0x8b6b bytes) 000480 1790785761.264506 punch from 0x00d19e to 0x013eca (0x6d2d bytes) 000483 1790785761.279279 write 0x00bf8a thru 0x018759 (0xc7d0 bytes) 000485 1790785761.283603 punch from 0x01640c to 0x018759 (0x234e bytes) 000486 1790785761.314380 read 0x015313 thru 0x018759 (0x3447 bytes) 000487 1790785761.331340 write 0x00c7fc thru 0x00e26e (0x1a73 bytes) 000488 1790785761.332858 write 0x031895 thru 0x03397b (0x20e7 bytes) 000489 1790785761.333923 write 0x01fd00 thru 0x02f292 (0xf593 bytes) 000490 1790785761.352786 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3397c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3397c): Mirror 7: 0xc3523ac7 (stale) Mirror 8: 0xc3523ac7 (stale) Mirror 9: 0xc3523ac7 (stale) CRC-32 checksum value for chunk [0, 0x3397c): Mirror 1: 0xa4cf1f1d Mirror 10: 0xa4cf1f1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3397c: skipped 000491 1790785763.049289 read 0x005e7a thru 0x013b8e (0xdd15 bytes) 000492 1790785763.093274 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3397c): Mirror 7: 0xc3523ac7 (stale) Mirror 8: 0xc3523ac7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3397c): Mirror 7: 0xc3523ac7 (stale) Mirror 8: 0xc3523ac7 (stale) CRC-32 checksum value for chunk [0, 0x3397c): Mirror 1: 0xa4cf1f1d Mirror 10: 0xa4cf1f1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3397c: skipped 000493 1790785764.049172 punch from 0x00f947 to 0x01f6d9 (0xfd93 bytes) 000494 1790785764.200036 trunc from 0x03397c to 0x01add8 (0x18ba3 bytes) 000496 1790785764.252503 trunc from 0x01add9 to 0x0381dc (0x1d404 bytes) 000497 1790785764.288967 write 0x016f7d thru 0x01ddae (0x6e32 bytes) 000498 1790785764.319743 write 0x0072e5 thru 0x007b40 (0x85c bytes) 000499 1790785764.334786 mapwrite 0x0341e5 thru 0x03ffff (0xbe1b bytes) 000500 1790785764.465862 read 0x037058 thru 0x0373ea (0x393 bytes) 000502 1790785764.467228 mapwrite 0x02455d thru 0x033959 (0xf3fd bytes) 000503 1790785764.538120 mapread 0x02126f thru 0x02e919 (0xd6ab bytes) 000505 1790785764.539697 trunc from 0x040000 to 0x00cada (0x33525 bytes) 000506 1790785764.560695 trunc from 0x00cadb to 0x029a3a (0x1cf60 bytes) 000507 1790785764.578827 trunc from 0x029a3b to 0x00977f (0x202bb bytes) 000508 1790785764.599857 read 0x002e04 thru 0x00977f (0x697c bytes) 000510 1790785764.612361 write 0x015fcd thru 0x017987 (0x19bb bytes) 000511 1790785764.613613 punch from 0x0146d1 to 0x017987 (0x32b7 bytes) 000512 1790785764.629153 MIRROR_RESYNC 4 mirrors 000515 1790785765.680208 trunc from 0x017988 to 0x0283b7 (0x10a30 bytes) 000517 1790785765.833882 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x283b8): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x283b8): Mirror 7: 0xebdb16f8 (stale) Mirror 8: 0xebdb16f8 (stale) Mirror 10: 0xebdb16f8 (stale) CRC-32 checksum value for chunk [0, 0x283b8): Mirror 1: 0xebdb16f8 Mirror 11: 0xebdb16f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x283b8: skipped 000518 1790785767.415024 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x283b8): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x283b8): Mirror 7: 0xebdb16f8 (stale) Mirror 8: 0xebdb16f8 (stale) Mirror 10: 0xebdb16f8 (stale) CRC-32 checksum value for chunk [0, 0x283b8): Mirror 1: 0xebdb16f8 Mirror 11: 0xebdb16f8 Mirror 12: 0xebdb16f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x283b8: skipped 000519 1790785768.506760 read 0x01ab3e thru 0x021298 (0x675b bytes) 000520 1790785768.526555 mapwrite 0x03747c thru 0x038988 (0x150d bytes) 000521 1790785768.679927 write 0x006d0c thru 0x0162a7 (0xf59c bytes) 000524 1790785768.692026 mapread 0x00acac thru 0x0181b2 (0xd507 bytes) 000526 1790785768.704712 trunc from 0x038989 to 0x026872 (0x12116 bytes) 000527 1790785768.732031 MIRROR_RESYNC 6 mirrors 000528 1790785770.237647 trunc from 0x026873 to 0x00ad6f (0x1bb03 bytes) 000530 1790785770.330708 mapread 0x00479d thru 0x005226 (0xa8a bytes) 000531 1790785770.340814 mapread 0x00015a thru 0x000c45 (0xaec bytes) 000533 1790785770.353460 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xad70): Mirror 7: 0xf2ba68e1 (stale) Mirror 8: 0xf2ba68e1 (stale) Mirror 11: 0xf2ba68e1 (stale) Mirror 12: 0xf2ba68e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xad70): Mirror 7: 0xf2ba68e1 (stale) Mirror 8: 0xf2ba68e1 (stale) Mirror 11: 0xf2ba68e1 (stale) Mirror 12: 0xf2ba68e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xad70: skipped 000534 1790785770.676838 read 0x0099ba thru 0x00ad6f (0x13b6 bytes) 000535 1790785770.683035 write 0x001cb5 thru 0x004e7e (0x31ca bytes) 000536 1790785770.695994 mapread 0x005528 thru 0x00ad6f (0x5848 bytes) 000537 1790785770.718360 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xad70): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xad70): Mirror 7: 0xf2ba68e1 (stale) Mirror 8: 0xf2ba68e1 (stale) Mirror 11: 0xf2ba68e1 (stale) Mirror 12: 0xf2ba68e1 (stale) CRC-32 checksum value for chunk [0, 0xad70): Mirror 1: 0x3231c395 Mirror 13: 0x3231c395 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xad70: skipped 000538 1790785771.780004 mapwrite 0x026a2f thru 0x034d06 (0xe2d8 bytes) 000539 1790785771.965674 mapread 0x00e030 thru 0x01c51f (0xe4f0 bytes) 000541 1790785771.989296 mapwrite 0x029c92 thru 0x03507d (0xb3ec bytes) 000542 1790785772.114320 mapread 0x01079b thru 0x01bc9d (0xb503 bytes) 000543 1790785772.116825 write 0x022e4b thru 0x0244f9 (0x16af bytes) 000544 1790785772.122980 trunc from 0x03507e to 0x00be14 (0x29269 bytes) 000545 1790785772.154293 read 0x00030e thru 0x007354 (0x7047 bytes) 000547 1790785772.166327 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xbe15): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe15): Mirror 7: 0x3a34129e (stale) Mirror 8: 0x3a34129e (stale) Mirror 11: 0x3a34129e (stale) Mirror 12: 0x3a34129e (stale) Mirror 13: 0x6e2d8ef (stale) CRC-32 checksum value for chunk [0, 0xbe15): Mirror 1: 0x6e2d8ef Mirror 14: 0x6e2d8ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe15: skipped 000548 1790785773.313877 mapread 0x00a570 thru 0x00be14 (0x18a5 bytes) 000549 1790785773.364113 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbe15): Mirror 7: 0x3a34129e (stale) Mirror 8: 0x3a34129e (stale) Mirror 11: 0x3a34129e (stale) Mirror 13: 0x6e2d8ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe15): Mirror 7: 0x3a34129e (stale) Mirror 8: 0x3a34129e (stale) Mirror 11: 0x3a34129e (stale) Mirror 13: 0x6e2d8ef (stale) CRC-32 checksum value for chunk [0, 0xbe15): Mirror 1: 0x6e2d8ef Mirror 14: 0x6e2d8ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe15: skipped 000550 1790785774.347971 write 0x027830 thru 0x028a1d (0x11ee bytes) 000552 1790785774.432497 mapwrite 0x009ad4 thru 0x00be92 (0x23bf bytes) 000556 1790785774.499432 MIRROR_RESYNC 6 mirrors 000557 1790785776.281836 punch from 0x027aed to 0x028a1d (0xf31 bytes) 000558 1790785776.373577 mapwrite 0x008097 thru 0x00f06f (0x6fd9 bytes) 000559 1790785776.494963 trunc from 0x028a1e to 0x0208a4 (0x8179 bytes) 000560 1790785776.516485 mapread 0x004da8 thru 0x00e30b (0x9564 bytes) 000561 1790785776.517623 mapwrite 0x02a3dd thru 0x0321c5 (0x7de9 bytes) 000563 1790785776.592757 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x321c6): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x321c6): Mirror 7: 0x98b94d7f (stale) Mirror 8: 0x98b94d7f (stale) Mirror 11: 0x98b94d7f (stale) Mirror 13: 0x98b94d7f (stale) Mirror 14: 0x98b94d7f (stale) CRC-32 checksum value for chunk [0, 0x321c6): Mirror 1: 0x7b3906f9 Mirror 15: 0x7b3906f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x321c6: skipped 000564 1790785777.733183 mapread 0x01e2c3 thru 0x02966c (0xb3aa bytes) 000567 1790785777.926576 mapread 0x008f1e thru 0x014044 (0xb127 bytes) 000569 1790785778.010517 mapwrite 0x02bab7 thru 0x0309eb (0x4f35 bytes) 000570 1790785778.193628 write 0x0357d8 thru 0x03a97a (0x51a3 bytes) 000572 1790785778.196589 read 0x0083a5 thru 0x0087bb (0x417 bytes) 000574 1790785778.204842 mapread 0x034b50 thru 0x03a97a (0x5e2b bytes) 000575 1790785778.212756 trunc from 0x03a97b to 0x0253e0 (0x1559a bytes) 000576 1790785778.272624 trunc from 0x0253e1 to 0x030171 (0xad91 bytes) 000577 1790785778.317029 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30172): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30172): Mirror 7: 0x6d0d08b0 (stale) Mirror 8: 0x6d0d08b0 (stale) Mirror 11: 0x6d0d08b0 (stale) Mirror 13: 0x6d0d08b0 (stale) Mirror 14: 0x6d0d08b0 (stale) Mirror 15: 0x4e5a724f (stale) CRC-32 checksum value for chunk [0, 0x30172): Mirror 1: 0xf0af4ce0 Mirror 16: 0xf0af4ce0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30172: skipped 000578 1790785780.131885 write 0x014041 thru 0x0187bb (0x477b bytes) 000579 1790785780.265709 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x30172): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30172): Mirror 7: 0x6d0d08b0 (stale) Mirror 8: 0x6d0d08b0 (stale) Mirror 11: 0x6d0d08b0 (stale) Mirror 13: 0x6d0d08b0 (stale) Mirror 14: 0x6d0d08b0 (stale) Mirror 15: 0x4e5a724f (stale) Mirror 16: 0xf0af4ce0 (stale) CRC-32 checksum value for chunk [0, 0x30172): Mirror 1: 0x10ffcf8d Mirror 17: 0x10ffcf8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30172: skipped 000581 1790785781.720603 trunc from 0x030172 to 0x007d1e (0x28453 bytes) 000582 1790785781.816358 punch from 0x0043d0 to 0x007d1e (0x394f bytes) 000584 1790785781.822944 mapwrite 0x030fe5 thru 0x034bbb (0x3bd7 bytes) 000585 1790785781.907411 punch from 0x0275db to 0x02e7d8 (0x71fe bytes) 000587 1790785781.912798 mapread 0x027a05 thru 0x034bbb (0xd1b7 bytes) 000588 1790785781.927664 mapread 0x0134b1 thru 0x020004 (0xcb54 bytes) 000590 1790785781.942251 write 0x0077dc thru 0x017413 (0xfc38 bytes) 000591 1790785781.948705 mapread 0x025302 thru 0x025de5 (0xae4 bytes) 000592 1790785781.949679 write 0x01a36b thru 0x01a7ec (0x482 bytes) 000594 1790785781.951716 trunc from 0x034bbc to 0x0330db (0x1ae0 bytes) 000596 1790785781.973004 write 0x02e153 thru 0x0300b8 (0x1f66 bytes) 000597 1790785781.975657 trunc from 0x0330dc to 0x006b12 (0x2c5c9 bytes) 000598 1790785781.994140 punch from 0x001d90 to 0x006b12 (0x4d83 bytes) 000599 1790785781.999602 mapwrite 0x01455a thru 0x01645e (0x1f05 bytes) 000600 1790785782.063887 write 0x014d78 thru 0x015b27 (0xdb0 bytes) 000604 1790785782.066308 write 0x00aa82 thru 0x0104e1 (0x5a60 bytes) 000606 1790785782.074326 punch from 0x0032c2 to 0x003e43 (0xb82 bytes) 000607 1790785782.085301 punch from 0x00f6a6 to 0x010b33 (0x148e bytes) 000608 1790785782.090326 punch from 0x006b4a to 0x0076ff (0xbb6 bytes) 000610 1790785782.095684 punch from 0x011b88 to 0x01603e (0x44b7 bytes) 000612 1790785782.100878 punch from 0x00533b to 0x00fa49 (0xa70f bytes) 000613 1790785782.106274 punch from 0x00b855 to 0x011a2d (0x61d9 bytes) 000614 1790785782.111196 write 0x026ff9 thru 0x036ec3 (0xfecb bytes) 000615 1790785782.115268 mapread 0x024714 thru 0x027de7 (0x36d4 bytes) 000616 1790785782.122734 trunc from 0x036ec4 to 0x03a49e (0x35db bytes) 000618 1790785782.143785 mapread 0x014b65 thru 0x021981 (0xce1d bytes) 000620 1790785782.154570 write 0x02ace5 thru 0x02e843 (0x3b5f bytes) 000621 1790785782.156933 write 0x01392b thru 0x021b83 (0xe259 bytes) 000626 1790785782.160907 read 0x0345db thru 0x037283 (0x2ca9 bytes) 000627 1790785782.165995 read 0x013d65 thru 0x01c8d8 (0x8b74 bytes) 000628 1790785782.166837 trunc from 0x03a49f to 0x01222e (0x28270 bytes) 000629 1790785782.180794 write 0x003814 thru 0x0133d1 (0xfbbe bytes) 000630 1790785782.190193 punch from 0x004b5d to 0x0057b0 (0xc54 bytes) 000631 1790785782.203164 mapread 0x00f171 thru 0x013067 (0x3ef7 bytes) 000632 1790785782.204011 read 0x00929d thru 0x00fcf4 (0x6a58 bytes) 000633 1790785782.204603 trunc from 0x0133d2 to 0x020431 (0xd060 bytes) 000634 1790785782.220812 mapread 0x000e33 thru 0x00445d (0x362b bytes) 000635 1790785782.230788 read 0x01679a thru 0x01dddd (0x7644 bytes) 000636 1790785782.231526 mapread 0x010aff thru 0x01c611 (0xbb13 bytes) 000638 1790785782.234371 punch from 0x00d5bb to 0x00d823 (0x269 bytes) 000640 1790785782.238966 punch from 0x002e89 to 0x010b59 (0xdcd1 bytes) 000641 1790785782.244888 punch from 0x00e2c7 to 0x018f22 (0xac5c bytes) 000642 1790785782.249342 mapwrite 0x0375a6 thru 0x038c42 (0x169d bytes) 000643 1790785782.340518 punch from 0x0278c1 to 0x02f024 (0x7764 bytes) 000644 1790785782.346171 trunc from 0x038c43 to 0x026df8 (0x11e4a bytes) 000645 1790785782.361830 mapwrite 0x00df21 thru 0x011a7b (0x3b5b bytes) 000646 1790785782.429707 MIRROR_RESYNC 9 mirrors 000647 1790785784.785833 MIRROR_EXTEND to 10 mirrors 000648 1790785785.696465 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) 000649 1790785786.213528 punch from 0x00efd9 to 0x01c2fe (0xd326 bytes) 000650 1790785786.338909 read 0x00bd67 thru 0x013287 (0x7521 bytes) 000651 1790785786.350676 punch from 0x01a82e to 0x021715 (0x6ee8 bytes) 000652 1790785786.360798 read 0x01a49b thru 0x01d564 (0x30ca bytes) 000653 1790785786.374877 punch from 0x0132a9 to 0x0139f6 (0x74e bytes) 000654 1790785786.379749 write 0x010901 thru 0x01defc (0xd5fc bytes) 000656 1790785786.382259 write 0x00dc06 thru 0x01a9ef (0xcdea bytes) 000658 1790785786.384611 punch from 0x012e61 to 0x016bc2 (0x3d62 bytes) 000659 1790785786.399232 mapread 0x01bda3 thru 0x01d47c (0x16da bytes) 000661 1790785786.400581 read 0x00402c thru 0x00c1f5 (0x81ca bytes) 000662 1790785786.401170 read 0x00eefa thru 0x01a7d3 (0xb8da bytes) 000663 1790785786.410273 read 0x00d329 thru 0x011810 (0x44e8 bytes) 000664 1790785786.410851 trunc from 0x026df9 to 0x035035 (0xe23d bytes) 000668 1790785786.431661 punch from 0x006003 to 0x009cd2 (0x3cd0 bytes) 000671 1790785786.438889 trunc from 0x035036 to 0x01cbdd (0x18458 bytes) 000672 1790785786.453826 read 0x0060cc thru 0x00ac52 (0x4b87 bytes) 000673 1790785786.460702 read 0x006366 thru 0x00b5ce (0x5269 bytes) 000674 1790785786.466240 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1cbde): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cbde): Mirror 8: 0x7dffe5a3 (stale) Mirror 11: 0x7dffe5a3 (stale) Mirror 13: 0x7dffe5a3 (stale) Mirror 14: 0x7dffe5a3 (stale) Mirror 15: 0x7dffe5a3 (stale) Mirror 16: 0x7dffe5a3 (stale) Mirror 17: 0x7dffe5a3 (stale) Mirror 18: 0x7dffe5a3 (stale) CRC-32 checksum value for chunk [0, 0x1cbde): Mirror 1: 0x4acd13fb Mirror 19: 0x4acd13fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cbde: skipped 000675 1790785787.583903 read 0x000b02 thru 0x009c94 (0x9193 bytes) 000676 1790785787.614232 mapwrite 0x012d8b thru 0x01703d (0x42b3 bytes) 000678 1790785788.018793 mapwrite 0x01537b thru 0x02176a (0xc3f0 bytes) 000679 1790785788.187838 trunc from 0x02176b to 0x033c31 (0x124c7 bytes) 000680 1790785788.204970 write 0x02d792 thru 0x0366a9 (0x8f18 bytes) 000681 1790785788.215132 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x366aa): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366aa): Mirror 8: 0x5ebf5b55 (stale) Mirror 11: 0x5ebf5b55 (stale) Mirror 13: 0x5ebf5b55 (stale) Mirror 14: 0x5ebf5b55 (stale) Mirror 15: 0x5ebf5b55 (stale) Mirror 16: 0x5ebf5b55 (stale) Mirror 17: 0x5ebf5b55 (stale) Mirror 18: 0x5ebf5b55 (stale) Mirror 19: 0x87c76e89 (stale) CRC-32 checksum value for chunk [0, 0x366aa): Mirror 1: 0xcffe3601 Mirror 20: 0xcffe3601 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366aa: skipped 000682 1790785789.509583 write 0x03222f thru 0x03a705 (0x84d7 bytes) 000683 1790785789.668776 write 0x004cab thru 0x011489 (0xc7df bytes) 000684 1790785789.692745 mapwrite 0x02b5b3 thru 0x02e059 (0x2aa7 bytes) 000685 1790785789.833774 mapwrite 0x002030 thru 0x00cc40 (0xac11 bytes) 000686 1790785789.903133 MIRROR_RESYNC 11 mirrors 000688 1790785791.931607 MIRROR_RESYNC 11 mirrors 000689 1790785792.936121 read 0x034c6b thru 0x03711a (0x24b0 bytes) 000691 1790785792.944452 mapwrite 0x03fc72 thru 0x03ffff (0x38e bytes) 000692 1790785793.116767 write 0x003aa3 thru 0x00a811 (0x6d6f bytes) 000694 1790785793.144512 punch from 0x00ffd5 to 0x0131fb (0x3227 bytes) 000695 1790785793.164351 trunc from 0x040000 to 0x02e69f (0x11960 bytes) 000697 1790785793.191024 write 0x0009f8 thru 0x0050e1 (0x46ea bytes) 000698 1790785793.210392 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2e6a0): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6a0): Mirror 8: 0xfd6bf512 (stale) Mirror 11: 0xfd6bf512 (stale) Mirror 13: 0xfd6bf512 (stale) Mirror 14: 0xfd6bf512 (stale) Mirror 15: 0xfd6bf512 (stale) Mirror 16: 0xfd6bf512 (stale) Mirror 17: 0xfd6bf512 (stale) Mirror 18: 0xfd6bf512 (stale) Mirror 19: 0xfd6bf512 (stale) Mirror 20: 0xfd6bf512 (stale) CRC-32 checksum value for chunk [0, 0x2e6a0): Mirror 1: 0xb51aaf8d Mirror 21: 0xb51aaf8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6a0: skipped 000699 1790785794.901608 trunc from 0x02e6a0 to 0x02897f (0x5d20 bytes) 000701 1790785795.029359 MIRROR_RESYNC 12 mirrors 000704 1790785797.305690 MIRROR_EXTEND to 13 mirrors 000705 1790785798.812501 trunc from 0x028980 to 0x031ab1 (0x9132 bytes) 000706 1790785798.973823 read 0x02d686 thru 0x031ab1 (0x442c bytes) 000708 1790785798.988497 punch from 0x030b49 to 0x031ab1 (0xf69 bytes) 000711 1790785799.007315 mapwrite 0x01484a thru 0x01c10d (0x78c4 bytes) 000712 1790785799.176602 trunc from 0x031ab2 to 0x03bec5 (0xa414 bytes) 000714 1790785799.211413 read 0x0129a0 thru 0x0180ba (0x571b bytes) 000716 1790785799.213481 mapread 0x036116 thru 0x03bec5 (0x5db0 bytes) 000717 1790785799.234835 punch from 0x01c125 to 0x025bbb (0x9a97 bytes) 000718 1790785799.240454 punch from 0x013a16 to 0x014be4 (0x11cf bytes) 000719 1790785799.246147 read 0x0015f4 thru 0x0111bb (0xfbc8 bytes) 000720 1790785799.247290 write 0x01312f thru 0x017b6a (0x4a3c bytes) 000723 1790785799.264987 trunc from 0x03bec6 to 0x000306 (0x3bbbf bytes) 000724 1790785799.291813 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x307): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307): Mirror 8: 0x900fbc97 (stale) Mirror 11: 0x900fbc97 (stale) Mirror 13: 0x900fbc97 (stale) Mirror 14: 0x900fbc97 (stale) Mirror 15: 0x900fbc97 (stale) Mirror 16: 0x900fbc97 (stale) Mirror 17: 0x900fbc97 (stale) Mirror 18: 0x900fbc97 (stale) Mirror 19: 0x900fbc97 (stale) Mirror 20: 0x900fbc97 (stale) Mirror 21: 0x900fbc97 (stale) Mirror 22: 0x900fbc97 (stale) CRC-32 checksum value for chunk [0, 0x307): Mirror 1: 0x900fbc97 Mirror 23: 0x900fbc97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307: skipped 000725 1790785800.376534 punch from 0x0000c4 to 0x00024f (0x18c bytes) 000729 1790785800.485268 MIRROR_RESYNC 14 mirrors 000730 1790785802.324824 mapwrite 0x03625d thru 0x037d0c (0x1ab0 bytes) 000732 1790785802.673870 mapwrite 0x012c43 thru 0x01ba0f (0x8dcd bytes) 000733 1790785802.827597 trunc from 0x037d0d to 0x00a413 (0x2d8f9 bytes) 000734 1790785802.860459 punch from 0x006a75 to 0x009943 (0x2ecf bytes) 000735 1790785802.872157 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa414): Mirror 8: 0xecb44dbd (stale) Mirror 11: 0xecb44dbd (stale) Mirror 13: 0xecb44dbd (stale) Mirror 14: 0xecb44dbd (stale) Mirror 16: 0xecb44dbd (stale) Mirror 17: 0xecb44dbd (stale) Mirror 18: 0xecb44dbd (stale) Mirror 19: 0xecb44dbd (stale) Mirror 20: 0xecb44dbd (stale) Mirror 21: 0xecb44dbd (stale) Mirror 22: 0xecb44dbd (stale) Mirror 23: 0xecb44dbd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa414): Mirror 8: 0xecb44dbd (stale) Mirror 11: 0xecb44dbd (stale) Mirror 13: 0xecb44dbd (stale) Mirror 14: 0xecb44dbd (stale) Mirror 16: 0xecb44dbd (stale) Mirror 17: 0xecb44dbd (stale) Mirror 18: 0xecb44dbd (stale) Mirror 19: 0xecb44dbd (stale) Mirror 20: 0xecb44dbd (stale) Mirror 21: 0xecb44dbd (stale) Mirror 22: 0xecb44dbd (stale) Mirror 23: 0xecb44dbd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa414: skipped 000736 1790785803.558889 punch from 0x001b6d to 0x00a413 (0x88a7 bytes) 000738 1790785803.569654 write 0x02bffb thru 0x03a693 (0xe699 bytes) 000739 1790785803.577471 MIRROR_RESYNC 13 mirrors 000740 1790785807.087725 write 0x007647 thru 0x008e5a (0x1814 bytes) 000741 1790785807.196259 write 0x004892 thru 0x0116f3 (0xce62 bytes) 000742 1790785807.208483 mapread 0x006fc8 thru 0x007595 (0x5ce bytes) 000743 1790785807.209627 mapwrite 0x0146b5 thru 0x023536 (0xee82 bytes) 000744 1790785807.308300 punch from 0x01e091 to 0x0284f9 (0xa469 bytes) 000745 1790785807.323315 punch from 0x00c197 to 0x0162dc (0xa146 bytes) 000746 1790785807.328479 mapwrite 0x007b27 thru 0x00eeab (0x7385 bytes) 000747 1790785807.458610 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a694): Mirror 8: 0x4e7ad047 (stale) Mirror 11: 0x4e7ad047 (stale) Mirror 13: 0x4e7ad047 (stale) Mirror 14: 0x4e7ad047 (stale) Mirror 16: 0x4e7ad047 (stale) Mirror 17: 0x4e7ad047 (stale) Mirror 18: 0x4e7ad047 (stale) Mirror 19: 0x4e7ad047 (stale) Mirror 20: 0x4e7ad047 (stale) Mirror 21: 0x4e7ad047 (stale) Mirror 22: 0x4e7ad047 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a694): Mirror 8: 0x4e7ad047 (stale) Mirror 11: 0x4e7ad047 (stale) Mirror 13: 0x4e7ad047 (stale) Mirror 14: 0x4e7ad047 (stale) Mirror 16: 0x4e7ad047 (stale) Mirror 17: 0x4e7ad047 (stale) Mirror 18: 0x4e7ad047 (stale) Mirror 19: 0x4e7ad047 (stale) Mirror 20: 0x4e7ad047 (stale) Mirror 21: 0x4e7ad047 (stale) Mirror 22: 0x4e7ad047 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a694: skipped 000748 1790785808.294584 mapread 0x008ea6 thru 0x00ceb5 (0x4010 bytes) 000749 1790785808.316421 punch from 0x002861 to 0x00c5fb (0x9d9b bytes) 000750 1790785808.321768 mapwrite 0x001c8e thru 0x00d655 (0xb9c8 bytes) 000752 1790785808.421627 write 0x01fd27 thru 0x027b76 (0x7e50 bytes) 000753 1790785808.441118 mapread 0x0126ea thru 0x01b17a (0x8a91 bytes) 000754 1790785808.449617 mapread 0x00f28f thru 0x01199d (0x270f bytes) 000755 1790785808.450469 punch from 0x0279c4 to 0x02af90 (0x35cd bytes) 000758 1790785808.490557 mapread 0x038196 thru 0x03a693 (0x24fe bytes) 000759 1790785808.509925 write 0x0389d4 thru 0x03ffff (0x762c bytes) 000761 1790785808.511668 read 0x03de9b thru 0x03ffff (0x2165 bytes) 000762 1790785808.519246 punch from 0x02377e to 0x026bd8 (0x345b bytes) 000763 1790785808.570592 write 0x01f50a thru 0x0229cd (0x34c4 bytes) 000765 1790785808.572189 read 0x03dfb0 thru 0x03ffff (0x2050 bytes) 000767 1790785808.582275 mapwrite 0x01679c thru 0x01da6f (0x72d4 bytes) 000769 1790785808.722571 mapread 0x0273a8 thru 0x02d545 (0x619e bytes) 000771 1790785808.728319 trunc from 0x040000 to 0x00ad02 (0x352fd bytes) 000773 1790785808.784316 mapwrite 0x007227 thru 0x00f595 (0x836f bytes) 000775 1790785808.966685 mapread 0x006eb1 thru 0x0085ae (0x16fe bytes) 000776 1790785808.968254 mapwrite 0x01bc44 thru 0x01d4ef (0x18ac bytes) 000777 1790785809.090312 write 0x00c622 thru 0x018d42 (0xc721 bytes) 000778 1790785809.109435 read 0x01cd6b thru 0x01d4ef (0x785 bytes) 000779 1790785809.111423 mapread 0x014f82 thru 0x01b396 (0x6415 bytes) 000780 1790785809.129175 punch from 0x005a7c to 0x0117c0 (0xbd45 bytes) 000782 1790785809.161796 write 0x03608f thru 0x03cc05 (0x6b77 bytes) 000783 1790785809.162994 mapwrite 0x02190f thru 0x02738c (0x5a7e bytes) 000784 1790785809.253934 mapwrite 0x03f136 thru 0x03ffff (0xeca bytes) 000787 1790785809.343634 read 0x01eb88 thru 0x02777b (0x8bf4 bytes) 000788 1790785809.347677 write 0x00a50a thru 0x018e3b (0xe932 bytes) 000789 1790785809.370919 write 0x03e91f thru 0x03ffff (0x16e1 bytes) 000790 1790785809.371838 write 0x009179 thru 0x00b53c (0x23c4 bytes) 000791 1790785809.372780 punch from 0x025d25 to 0x02c730 (0x6a0c bytes) 000793 1790785809.407641 punch from 0x033167 to 0x034777 (0x1611 bytes) 000794 1790785809.419191 read 0x03820d thru 0x03ffff (0x7df3 bytes) 000795 1790785809.425489 read 0x036fdb thru 0x03c556 (0x557c bytes) 000796 1790785809.426175 mapread 0x02f2ab thru 0x035a6c (0x67c2 bytes) 000797 1790785809.470763 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x30c93134 (stale) Mirror 11: 0x30c93134 (stale) Mirror 13: 0x30c93134 (stale) Mirror 14: 0x30c93134 (stale) Mirror 16: 0x30c93134 (stale) Mirror 17: 0x30c93134 (stale) Mirror 18: 0x30c93134 (stale) Mirror 19: 0x30c93134 (stale) Mirror 21: 0x30c93134 (stale) Mirror 22: 0x30c93134 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x30c93134 (stale) Mirror 11: 0x30c93134 (stale) Mirror 13: 0x30c93134 (stale) Mirror 14: 0x30c93134 (stale) Mirror 16: 0x30c93134 (stale) Mirror 17: 0x30c93134 (stale) Mirror 18: 0x30c93134 (stale) Mirror 19: 0x30c93134 (stale) Mirror 21: 0x30c93134 (stale) Mirror 22: 0x30c93134 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000799 1790785810.420360 mapread 0x004d10 thru 0x0092f9 (0x45ea bytes) 000800 1790785810.483448 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x30c93134 (stale) Mirror 11: 0x30c93134 (stale) Mirror 13: 0x30c93134 (stale) Mirror 14: 0x30c93134 (stale) Mirror 16: 0x30c93134 (stale) Mirror 17: 0x30c93134 (stale) Mirror 18: 0x30c93134 (stale) Mirror 19: 0x30c93134 (stale) Mirror 22: 0x30c93134 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x30c93134 (stale) Mirror 11: 0x30c93134 (stale) Mirror 13: 0x30c93134 (stale) Mirror 14: 0x30c93134 (stale) Mirror 16: 0x30c93134 (stale) Mirror 17: 0x30c93134 (stale) Mirror 18: 0x30c93134 (stale) Mirror 19: 0x30c93134 (stale) Mirror 22: 0x30c93134 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000802 1790785811.470411 trunc from 0x040000 to 0x027c34 (0x183cb bytes) 000803 1790785811.523647 write 0x01f218 thru 0x02c3f9 (0xd1e2 bytes) 000805 1790785811.543776 write 0x0111d8 thru 0x019c96 (0x8abf bytes) 000808 1790785811.567887 mapread 0x00e224 thru 0x01cada (0xe8b7 bytes) 000809 1790785811.630640 mapwrite 0x005546 thru 0x0105cc (0xb087 bytes) 000810 1790785811.857286 trunc from 0x02c3fa to 0x01d687 (0xed72 bytes) 000811 1790785811.894837 mapwrite 0x028aab thru 0x02e906 (0x5e5c bytes) 000812 1790785812.060619 mapread 0x024d55 thru 0x029ea0 (0x514c bytes) 000813 1790785812.064919 write 0x03048c thru 0x03b826 (0xb39b bytes) 000814 1790785812.066563 mapwrite 0x00eaaa thru 0x015d1e (0x7275 bytes) 000815 1790785812.132453 punch from 0x012b8f to 0x021622 (0xea94 bytes) 000816 1790785812.144426 punch from 0x026a4c to 0x02c2d7 (0x588c bytes) 000817 1790785812.171617 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3b827): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b827): Mirror 8: 0xc9770fc3 (stale) Mirror 11: 0xc9770fc3 (stale) Mirror 13: 0xc9770fc3 (stale) Mirror 14: 0xc9770fc3 (stale) Mirror 16: 0xc9770fc3 (stale) Mirror 17: 0xc9770fc3 (stale) Mirror 18: 0xc9770fc3 (stale) Mirror 19: 0xc9770fc3 (stale) Mirror 22: 0xc9770fc3 (stale) CRC-32 checksum value for chunk [0, 0x3b827): Mirror 1: 0xb9f6a677 Mirror 23: 0xb9f6a677 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b827: skipped 000819 1790785813.608262 MIRROR_RESYNC 11 mirrors 000820 1790785816.019875 punch from 0x019ca6 to 0x026605 (0xc960 bytes) 000821 1790785816.192625 mapwrite 0x03d3e6 thru 0x03ffff (0x2c1a bytes) 000824 1790785816.441709 MIRROR_RESYNC 11 mirrors 000825 1790785818.961586 punch from 0x0336b9 to 0x03cbc5 (0x950d bytes) 000829 1790785819.060700 write 0x03139d thru 0x0338c4 (0x2528 bytes) 000830 1790785819.074807 trunc from 0x040000 to 0x001276 (0x3ed89 bytes) 000831 1790785819.109498 mapread 0x000c0d thru 0x001276 (0x66a bytes) 000832 1790785819.122914 trunc from 0x001277 to 0x03b979 (0x3a703 bytes) 000833 1790785819.193654 MIRROR_RESYNC 11 mirrors 000834 1790785822.343700 punch from 0x01a526 to 0x020702 (0x61dd bytes) 000837 1790785822.491630 MIRROR_RESYNC 11 mirrors 000839 1790785825.636018 trunc from 0x03b97a to 0x023e4e (0x17b2b bytes) 000841 1790785825.737879 write 0x031e3a thru 0x037be4 (0x5dab bytes) 000842 1790785825.739684 write 0x00986b thru 0x00d3f0 (0x3b86 bytes) 000843 1790785825.754788 read 0x0111e6 thru 0x01eaca (0xd8e5 bytes) 000845 1790785825.764102 mapread 0x02195f thru 0x02f1c5 (0xd867 bytes) 000849 1790785825.808194 mapwrite 0x023739 thru 0x032e90 (0xf758 bytes) 000852 1790785825.933194 punch from 0x02dc3d to 0x037be4 (0x9fa8 bytes) 000853 1790785825.949345 mapread 0x01db10 thru 0x025dc6 (0x82b7 bytes) 000854 1790785825.950682 trunc from 0x037be5 to 0x00c40b (0x2b7d9 bytes) 000856 1790785826.000905 mapwrite 0x0178e7 thru 0x024f6c (0xd686 bytes) 000857 1790785826.173373 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x24f6d): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24f6d): Mirror 8: 0xc4cf7146 (stale) Mirror 11: 0xc4cf7146 (stale) Mirror 13: 0xc4cf7146 (stale) Mirror 14: 0xc4cf7146 (stale) Mirror 16: 0xc4cf7146 (stale) Mirror 17: 0xc4cf7146 (stale) Mirror 18: 0xc4cf7146 (stale) Mirror 19: 0xc4cf7146 (stale) Mirror 22: 0xc4cf7146 (stale) Mirror 23: 0xc4cf7146 (stale) CRC-32 checksum value for chunk [0, 0x24f6d): Mirror 1: 0x468a78c7 Mirror 24: 0x468a78c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24f6d: skipped 000858 1790785827.669209 mapwrite 0x019261 thru 0x01ea50 (0x57f0 bytes) 000860 1790785827.979388 trunc from 0x024f6d to 0x0393c6 (0x1445a bytes) 000861 1790785828.024983 trunc from 0x0393c7 to 0x0324bf (0x6f07 bytes) 000862 1790785828.043867 mapread 0x0234fd thru 0x023604 (0x108 bytes) 000863 1790785828.063541 read 0x02d078 thru 0x02d7b6 (0x73f bytes) 000866 1790785828.074276 mapread 0x02a22b thru 0x02fd86 (0x5b5c bytes) 000867 1790785828.090002 punch from 0x011b5a to 0x01fbe2 (0xe089 bytes) 000869 1790785828.098645 trunc from 0x0324c0 to 0x026578 (0xbf47 bytes) 000870 1790785828.138213 punch from 0x00b457 to 0x0169a5 (0xb54f bytes) 000871 1790785828.145271 trunc from 0x026579 to 0x01cbb2 (0x99c6 bytes) 000872 1790785828.188290 mapread 0x0149e9 thru 0x01cbb2 (0x81ca bytes) 000873 1790785828.229625 MIRROR_RESYNC 12 mirrors 000874 1790785830.399703 punch from 0x01a7ae to 0x01cbb2 (0x2405 bytes) 000875 1790785830.517206 mapwrite 0x00ec85 thru 0x00f3c1 (0x73d bytes) 000877 1790785830.677880 trunc from 0x01cbb3 to 0x035d21 (0x1916f bytes) 000878 1790785830.756371 trunc from 0x035d22 to 0x00d054 (0x28ccd bytes) 000879 1790785830.777964 write 0x01c06f thru 0x02aff4 (0xef86 bytes) 000880 1790785830.780453 trunc from 0x02aff5 to 0x01ff0f (0xb0e5 bytes) 000881 1790785830.804641 mapread 0x00c819 thru 0x00d88c (0x1074 bytes) 000883 1790785830.829674 mapwrite 0x020928 thru 0x0219ee (0x10c7 bytes) 000884 1790785831.024011 punch from 0x013f5b to 0x01e836 (0xa8dc bytes) 000886 1790785831.041087 read 0x00c500 thru 0x0159f5 (0x94f6 bytes) 000888 1790785831.059522 mapread 0x014fbe thru 0x0204a4 (0xb4e7 bytes) 000890 1790785831.078380 read 0x00a5cf thru 0x010f37 (0x6969 bytes) 000892 1790785831.086900 MIRROR_RESYNC 12 mirrors 000893 1790785833.856615 mapread 0x00c3d9 thru 0x015e26 (0x9a4e bytes) 000897 1790785833.884218 write 0x01a250 thru 0x026341 (0xc0f2 bytes) 000898 1790785834.051901 read 0x007a90 thru 0x00bb63 (0x40d4 bytes) 000899 1790785834.062772 trunc from 0x026342 to 0x029d52 (0x3a11 bytes) 000902 1790785834.118773 trunc from 0x029d53 to 0x03ea6c (0x14d1a bytes) 000903 1790785834.182725 trunc from 0x03ea6d to 0x01fd38 (0x1ed34 bytes) 000904 1790785834.242653 mapwrite 0x011d7b thru 0x01ab90 (0x8e16 bytes) 000905 1790785834.423152 mapwrite 0x01ebdf thru 0x02962f (0xaa51 bytes) 000906 1790785834.617263 write 0x021d49 thru 0x02f233 (0xd4eb bytes) 000907 1790785834.620369 punch from 0x029a64 to 0x02d958 (0x3ef5 bytes) 000908 1790785834.689446 trunc from 0x02f234 to 0x00ba81 (0x237b2 bytes) 000910 1790785834.711285 trunc from 0x00ba82 to 0x018f6f (0xd4ee bytes) 000911 1790785834.736188 MIRROR_RESYNC 12 mirrors 000913 1790785837.169998 mapwrite 0x01ce1c thru 0x027f92 (0xb177 bytes) 000915 1790785837.503539 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27f93): Mirror 8: 0x21b1621d (stale) Mirror 11: 0x21b1621d (stale) Mirror 13: 0x21b1621d (stale) Mirror 14: 0x21b1621d (stale) Mirror 16: 0x21b1621d (stale) Mirror 17: 0x21b1621d (stale) Mirror 19: 0x21b1621d (stale) Mirror 22: 0x21b1621d (stale) Mirror 23: 0x21b1621d (stale) Mirror 24: 0x21b1621d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f93): Mirror 8: 0x21b1621d (stale) Mirror 11: 0x21b1621d (stale) Mirror 13: 0x21b1621d (stale) Mirror 14: 0x21b1621d (stale) Mirror 16: 0x21b1621d (stale) Mirror 17: 0x21b1621d (stale) Mirror 19: 0x21b1621d (stale) Mirror 22: 0x21b1621d (stale) Mirror 23: 0x21b1621d (stale) Mirror 24: 0x21b1621d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f93: skipped 000916 1790785838.198723 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x27f93): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f93): Mirror 8: 0x21b1621d (stale) Mirror 11: 0x21b1621d (stale) Mirror 13: 0x21b1621d (stale) Mirror 14: 0x21b1621d (stale) Mirror 16: 0x21b1621d (stale) Mirror 17: 0x21b1621d (stale) Mirror 19: 0x21b1621d (stale) Mirror 22: 0x21b1621d (stale) Mirror 23: 0x21b1621d (stale) Mirror 24: 0x21b1621d (stale) CRC-32 checksum value for chunk [0, 0x27f93): Mirror 1: 0x4b707e33 Mirror 25: 0x4b707e33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f93: skipped 000917 1790785839.815794 read 0x00d853 thru 0x012d71 (0x551f bytes) 000919 1790785839.848985 mapread 0x0050a3 thru 0x00651b (0x1479 bytes) 000921 1790785839.901887 trunc from 0x027f93 to 0x01f64e (0x8944 bytes) 000922 1790785840.043590 punch from 0x013603 to 0x016e4f (0x384d bytes) 000924 1790785840.051982 write 0x0309b3 thru 0x036580 (0x5bce bytes) 000925 1790785840.063504 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36581): Mirror 8: 0xb9f6d489 (stale) Mirror 11: 0xb9f6d489 (stale) Mirror 13: 0xb9f6d489 (stale) Mirror 14: 0xb9f6d489 (stale) Mirror 16: 0xb9f6d489 (stale) Mirror 17: 0xb9f6d489 (stale) Mirror 19: 0xb9f6d489 (stale) Mirror 22: 0xb9f6d489 (stale) Mirror 23: 0xb9f6d489 (stale) Mirror 24: 0xb9f6d489 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36581): Mirror 8: 0xb9f6d489 (stale) Mirror 11: 0xb9f6d489 (stale) Mirror 13: 0xb9f6d489 (stale) Mirror 14: 0xb9f6d489 (stale) Mirror 16: 0xb9f6d489 (stale) Mirror 17: 0xb9f6d489 (stale) Mirror 19: 0xb9f6d489 (stale) Mirror 22: 0xb9f6d489 (stale) Mirror 23: 0xb9f6d489 (stale) Mirror 24: 0xb9f6d489 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36581: skipped 000926 1790785840.854143 MIRROR_RESYNC 11 mirrors 000927 1790785844.297905 punch from 0x006ec3 to 0x00d59e (0x66dc bytes) 000928 1790785844.381148 write 0x017d72 thru 0x0218f4 (0x9b83 bytes) 000929 1790785844.392643 mapwrite 0x006fd6 thru 0x00eff9 (0x8024 bytes) 000930 1790785844.557706 write 0x0218d4 thru 0x028694 (0x6dc1 bytes) 000932 1790785844.571951 write 0x0349f8 thru 0x03f623 (0xac2c bytes) 000933 1790785844.579600 punch from 0x03576e to 0x0383b9 (0x2c4c bytes) 000937 1790785844.615524 trunc from 0x03f624 to 0x00ad45 (0x348de bytes) 000938 1790785844.675946 mapwrite 0x017593 thru 0x026aeb (0xf559 bytes) 000939 1790785844.774992 write 0x01984a thru 0x02287e (0x9035 bytes) 000940 1790785844.777105 mapread 0x022414 thru 0x026aeb (0x46d8 bytes) 000941 1790785844.778482 read 0x0138cf thru 0x023332 (0xfa64 bytes) 000942 1790785844.785965 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26aec): Mirror 8: 0x20385aaf (stale) Mirror 13: 0x20385aaf (stale) Mirror 14: 0x20385aaf (stale) Mirror 16: 0x20385aaf (stale) Mirror 17: 0x20385aaf (stale) Mirror 19: 0x20385aaf (stale) Mirror 22: 0x20385aaf (stale) Mirror 23: 0x20385aaf (stale) Mirror 24: 0x20385aaf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26aec): Mirror 8: 0x20385aaf (stale) Mirror 13: 0x20385aaf (stale) Mirror 14: 0x20385aaf (stale) Mirror 16: 0x20385aaf (stale) Mirror 17: 0x20385aaf (stale) Mirror 19: 0x20385aaf (stale) Mirror 22: 0x20385aaf (stale) Mirror 23: 0x20385aaf (stale) Mirror 24: 0x20385aaf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26aec: skipped 000943 1790785845.569944 mapread 0x00c76b thru 0x016fb9 (0xa84f bytes) 000944 1790785845.613168 read 0x01b1ce thru 0x026aeb (0xb91e bytes) 000945 1790785845.624184 mapwrite 0x02e6f0 thru 0x0301cf (0x1ae0 bytes) 000946 1790785845.739381 mapread 0x00e5ad thru 0x01d74d (0xf1a1 bytes) 000948 1790785845.752967 mapread 0x008e99 thru 0x00b0ba (0x2222 bytes) 000949 1790785845.754096 trunc from 0x0301d0 to 0x00945f (0x26d70 bytes) 000950 1790785845.781149 mapread 0x000ad1 thru 0x008751 (0x7c81 bytes) 000952 1790785845.803818 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x9460): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9460): Mirror 8: 0x3e1d0185 (stale) Mirror 13: 0x3e1d0185 (stale) Mirror 14: 0x3e1d0185 (stale) Mirror 16: 0x3e1d0185 (stale) Mirror 17: 0x3e1d0185 (stale) Mirror 19: 0x3e1d0185 (stale) Mirror 22: 0x3e1d0185 (stale) Mirror 23: 0x3e1d0185 (stale) Mirror 24: 0x3e1d0185 (stale) CRC-32 checksum value for chunk [0, 0x9460): Mirror 1: 0x881a5746 Mirror 25: 0x881a5746 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9460: skipped 000953 1790785846.993545 mapread 0x002460 thru 0x00945f (0x7000 bytes) 000954 1790785847.033403 punch from 0x0079f7 to 0x00945f (0x1a69 bytes) 000956 1790785847.193335 punch from 0x003cf5 to 0x005dba (0x20c6 bytes) 000957 1790785847.201807 write 0x02474f thru 0x027030 (0x28e2 bytes) 000958 1790785847.202949 read 0x003e79 thru 0x0126f5 (0xe87d bytes) 000960 1790785847.231247 write 0x00329b thru 0x01030d (0xd073 bytes) 000962 1790785847.254320 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x27031): Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27031): Mirror 8: 0xfb470c12 (stale) Mirror 13: 0xfb470c12 (stale) Mirror 14: 0xfb470c12 (stale) Mirror 16: 0xfb470c12 (stale) Mirror 17: 0xfb470c12 (stale) Mirror 19: 0xfb470c12 (stale) Mirror 22: 0xfb470c12 (stale) Mirror 23: 0xfb470c12 (stale) Mirror 24: 0xfb470c12 (stale) Mirror 25: 0x93fcc9ef (stale) CRC-32 checksum value for chunk [0, 0x27031): Mirror 1: 0x6a3f1bdd Mirror 26: 0x6a3f1bdd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27031: skipped 000965 1790785848.553764 mapread 0x00fdf5 thru 0x00ff97 (0x1a3 bytes) 000966 1790785848.564975 trunc from 0x027031 to 0x00fa54 (0x175dc bytes) 000967 1790785848.710268 mapread 0x00af95 thru 0x00b1f3 (0x25f bytes) 000969 1790785848.739952 read 0x006592 thru 0x00ca05 (0x6474 bytes) 000970 1790785848.745134 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfa55): Mirror 13: 0x169f102e (stale) Mirror 14: 0x169f102e (stale) Mirror 16: 0x169f102e (stale) Mirror 17: 0x169f102e (stale) Mirror 19: 0x169f102e (stale) Mirror 22: 0x169f102e (stale) Mirror 23: 0x169f102e (stale) Mirror 24: 0x169f102e (stale) Mirror 25: 0xe08a463e (stale) Mirror 26: 0xbfda9d7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfa55): Mirror 13: 0x169f102e (stale) Mirror 14: 0x169f102e (stale) Mirror 16: 0x169f102e (stale) Mirror 17: 0x169f102e (stale) Mirror 19: 0x169f102e (stale) Mirror 22: 0x169f102e (stale) Mirror 23: 0x169f102e (stale) Mirror 24: 0x169f102e (stale) Mirror 25: 0xe08a463e (stale) Mirror 26: 0xbfda9d7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfa55: skipped 000971 1790785849.303254 mapread 0x000e1d thru 0x00f59a (0xe77e bytes) 000972 1790785849.349383 punch from 0x006894 to 0x00fa54 (0x91c1 bytes) 000973 1790785849.355639 mapread 0x005b4a thru 0x006223 (0x6da bytes) 000974 1790785849.381249 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfa55): Mirror 14: 0x169f102e (stale) Mirror 16: 0x169f102e (stale) Mirror 17: 0x169f102e (stale) Mirror 19: 0x169f102e (stale) Mirror 22: 0x169f102e (stale) Mirror 23: 0x169f102e (stale) Mirror 24: 0x169f102e (stale) Mirror 25: 0xe08a463e (stale) Mirror 26: 0xbfda9d7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfa55): Mirror 14: 0x169f102e (stale) Mirror 16: 0x169f102e (stale) Mirror 17: 0x169f102e (stale) Mirror 19: 0x169f102e (stale) Mirror 22: 0x169f102e (stale) Mirror 23: 0x169f102e (stale) Mirror 24: 0x169f102e (stale) Mirror 25: 0xe08a463e (stale) Mirror 26: 0xbfda9d7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfa55: skipped 000976 1790785850.209036 mapread 0x00e29e thru 0x00fa54 (0x17b7 bytes) 000977 1790785850.233989 punch from 0x00513a to 0x00daed (0x89b4 bytes) 000978 1790785850.240646 trunc from 0x00fa55 to 0x020e84 (0x11430 bytes) 000979 1790785850.269837 mapread 0x020667 thru 0x020e84 (0x81e bytes) 000980 1790785850.291419 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x20e85): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e85): Mirror 14: 0x5c8249aa (stale) Mirror 16: 0x5c8249aa (stale) Mirror 17: 0x5c8249aa (stale) Mirror 19: 0x5c8249aa (stale) Mirror 22: 0x5c8249aa (stale) Mirror 23: 0x5c8249aa (stale) Mirror 24: 0x5c8249aa (stale) Mirror 25: 0x3a21c41e (stale) Mirror 26: 0x973c3ae1 (stale) CRC-32 checksum value for chunk [0, 0x20e85): Mirror 1: 0x2d91631b Mirror 27: 0x2d91631b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e85: skipped 000981 1790785851.293717 mapread 0x0158de thru 0x01d08f (0x77b2 bytes) 000982 1790785851.328811 trunc from 0x020e85 to 0x0212ff (0x47b bytes) 000985 1790785851.482885 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x21300): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21300): Mirror 14: 0x64b7c31e (stale) Mirror 16: 0x64b7c31e (stale) Mirror 17: 0x64b7c31e (stale) Mirror 19: 0x64b7c31e (stale) Mirror 22: 0x64b7c31e (stale) Mirror 23: 0x64b7c31e (stale) Mirror 24: 0x64b7c31e (stale) Mirror 25: 0x9ed084d5 (stale) Mirror 26: 0xbef70aa (stale) Mirror 27: 0xca8a24db (stale) CRC-32 checksum value for chunk [0, 0x21300): Mirror 1: 0xca8a24db Mirror 28: 0xca8a24db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21300: skipped 000986 1790785852.677886 MIRROR_RESYNC 12 mirrors 000987 1790785855.012730 read 0x01cee5 thru 0x0212ff (0x441b bytes) 000988 1790785855.023303 trunc from 0x021300 to 0x017b46 (0x97b9 bytes) 000989 1790785855.138570 trunc from 0x017b47 to 0x005c84 (0x11ec2 bytes) 000991 1790785855.166031 read 0x00318b thru 0x005c84 (0x2afa bytes) 000992 1790785855.184814 read 0x001260 thru 0x005c84 (0x4a25 bytes) 000996 1790785855.193976 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x5c85): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5c85): Mirror 14: 0xc7111290 (stale) Mirror 16: 0xc7111290 (stale) Mirror 17: 0xc7111290 (stale) Mirror 19: 0xc7111290 (stale) Mirror 22: 0xc7111290 (stale) Mirror 23: 0xc7111290 (stale) Mirror 24: 0xc7111290 (stale) Mirror 25: 0xc7111290 (stale) Mirror 26: 0xc7111290 (stale) Mirror 27: 0xc7111290 (stale) Mirror 28: 0xc7111290 (stale) CRC-32 checksum value for chunk [0, 0x5c85): Mirror 1: 0xc7111290 Mirror 29: 0xc7111290 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5c85: skipped 000997 1790785856.567180 read 0x00503c thru 0x005c84 (0xc49 bytes) 000998 1790785856.580405 trunc from 0x005c85 to 0x036c7f (0x30ffb bytes) 000999 1790785856.738172 read 0x01bc4e thru 0x027d60 (0xc113 bytes) 001000 1790785856.773250 mapread 0x02adb7 thru 0x03225e (0x74a8 bytes) All operations completed A-OK! PASS 70a (151s) == sanity-flr test 71: check mirror extend parent fid ==== 12:31:02 (1790785862) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 12:31:06 (1790785866) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.293268 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233605 s, 4.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 12:31:16 (1790785876) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6927872 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5702176 .. Extending file size to 6278688 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6956576 .. Extending file size to 7118144 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9045568 .. Extending file size to 7400576 .. Extending file size to 8018848 .. Extending file size to 8071744 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9026336 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9187520 .. done Waiting 59482 59483 59486 59488 59489 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) 192.168.203.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (87s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:32:44 (1790785964) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4689344 .. Extending file size to 702912 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7554304 .. Extending file size to 9121088 .. Extending file size to 2991648 .. Extending file size to 4956096 .. Extending file size to 7316448 .. Extending file size to 7660672 .. Extending file size to 7720416 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8404896 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 60887 60888 60889 60891 60894 failed 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) 192.168.203.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:34:01 (1790786041) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210560 2560 2205952 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144837 s, 14.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.600782 s, 3.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x40fd71d3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:34:24 (1790786064) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106929 s, 19.6 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 12:34:31 (1790786071) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x57:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:34:39 (1790786079) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.407273 s, 5.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:34:48 (1790786088) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283279 s, 7.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:34:57 (1790786097) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.443704 s, 4.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bd:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:35:32 (1790786132) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165158 s, 6.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1bd:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:36:34 (1790786194) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.293331 s, 7.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:37:33 (1790786253) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg309-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg309-server: Pool lustre.test_204f created Adding targets to pool oleg309-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg309-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.303989 s, 6.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg309-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg309-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg309-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:38:53 (1790786333) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:39:01 (1790786341) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:39:09 (1790786349) oleg309-server: Pool lustre.test_206 created Adding targets to pool oleg309-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg309-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg309-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg309-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg309-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:39:36 (1790786376) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655539 s, 16.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429805 s, 24.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3a075ba Mirror 3: 0x3a075ba PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:39:47 (1790786387) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:39:51 (1790786391) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:39:55 (1790786395) Starting client: oleg309-client.virtnet: -o user_xattr,flock 192.168.203.109@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.141203 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1010000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x23c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x23c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x23c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb30000 Read this (4 bytes): BBBB 192.168.203.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg309-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:40:30 (1790786430) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.024283 s, 337 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:40:36 (1790786436) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458852 s, 22.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x74 lmm_fid: [0x200000405:0x74:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x74 lmm_fid: [0x200000405:0x74:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611252 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 12:40:37.000000000 -0400 Modify: 2026-09-30 12:40:37.000000000 -0400 Change: 2026-09-30 12:40:39.000000000 -0400 Birth: 2026-09-30 12:40:36.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:40:44 (1790786444) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:40:47 (1790786447) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.520143 s, 78.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0495934 s, 82.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 30 12:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x76:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 30 12:40 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:40:55 (1790786455) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:41:02 (1790786462) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1900 sec =========== 12:41:11 (1790786471) === sanity-flr: start cleanup 12:41:12 (1790786472) === === sanity-flr: finish cleanup 12:41:20 (1790786480) ===