-----============= acceptance-small: sanity-flr ============----- Wed Sep 9 03:54:25 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg222-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 03:54:50 (1788940490) === oleg222-client.virtnet: executing check_config_client /mnt/lustre oleg222-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg222-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a5050ab7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a5050ab7800.idle_timeout=debug disable quota as required oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 03:55:37 (1788940537) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8232] == sanity-flr test 0a: lfs mirror create with -N option == 03:55:40 (1788940540) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:56:01 (1788940561) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:56:05 (1788940565) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:56:08 (1788940568) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:56:35 (1788940595) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:56:39 (1788940599) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 03:56:43 (1788940603) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9a5050ab7800.stats=clear osc.lustre-OST0001-osc-ffff9a5050ab7800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:56:53 (1788940613) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:57:05 (1788940625) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:57:16 (1788940636) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:57:21 (1788940641) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:57:30 (1788940650) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 03:57:41 (1788940661) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:57:50 (1788940670) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:57:59 (1788940679) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:58:06 (1788940686) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:58:17 (1788940697) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:58:25 (1788940705) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.47582 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666871 s, 15.7 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:58:36 (1788940716) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.451095 s, 32.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:58:45 (1788940725) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:58:56 (1788940736) Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a5050ab7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a5050ab7800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a5050ab7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a5050ab7800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:00:19 (1788940819) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.367549 s, 25.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a5050ab7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a5050ab7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:01:04 (1788940864) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a5050ab7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a5050ab7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a5050ab7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a5050ab7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:02:28 (1788940948) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.71765 s, 19.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 9 04:02 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.114s user 0m0.000s sys 0m0.148s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a5050ab7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a5050ab7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.242s user 0m0.001s sys 0m0.161s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a5050ab7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a5050ab7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.752s user 0m0.001s sys 0m0.173s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a5050ab7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a5050ab7800.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:04:11 (1788941051) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:04:14 (1788941054) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:04:17 (1788941057) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:04:21 (1788941061) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 04:04:29 (1788941069) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.77737 s, 16.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:04:45 (1788941085) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066435 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844446 s, 12.4 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:04:53 (1788941093) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:05:01 (1788941101) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000914415 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116599 s, 878 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110999 s, 923 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000927323 s, 1.1 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 04:05:13 (1788941113) create test files with size 15742342 .. 576ecc2534f510ab2e70bdad5afd5bbd - 8fbe77501c71e702be726a721fa5a7cb - 97d99d0992445925dcc585ccd3c5f961 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 04:05:41 (1788941141) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:05:59 (1788941159) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:06:06 (1788941166) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111845 s, 9.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701332 s, 15.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 04:06:15 (1788941175) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102121 s, 20.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.288728 s, 14.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146635 s, 14.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 04:06:31 (1788941191) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 04:06:34 (1788941194) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:06:38 (1788941198) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339002 s, 30.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 04:06:45 (1788941205) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.408372 s, 7.7 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:06:57 (1788941217) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:07:06 (1788941226) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.442922 s, 23.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:07:18 (1788941238) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9a5050d23000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.398889 s, 26.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9a5050ab1800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:07:32 (1788941252) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:08:08 (1788941288) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (53s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:09:01 (1788941341) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:09:05 (1788941345) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.259674 s, 12.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.153319 s, 20.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf55f1983 (nosync,stale) Mirror 2: 0xf55f1983 (nosync,stale) ** verify mirror 2 doesn't change original checksum: c032c8d0423120c6a620575e4ba1e211 - mirror 1 checksum: 397597493f8c5aad0527001be32bd481 - mirror 2 checksum: c032c8d0423120c6a620575e4ba1e211 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-09T04:09:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-09T04:09:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c032c8d0423120c6a620575e4ba1e211 - mirror 1 checksum: 397597493f8c5aad0527001be32bd481 - mirror 2 checksum: 397597493f8c5aad0527001be32bd481 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-09T04:09:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-09T04:09:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:09:16 (1788941356) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541153 s, 19.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:09:24 (1788941364) Failing mds1 on oleg222-server Stopping /mnt/lustre-mds1 (opts:) on oleg222-server 04:09:37 (1788941377) shut down facet: mds1 facet_host: oleg222-server facet_failover_host: oleg222-server Failover mds1 to oleg222-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 04:10:09 (1788941409) targets are mounted 04:10:09 (1788941409) facet_failover done oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023827 s, 44.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:10:26 (1788941426) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0593867 s, 345 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0338978 s, 604 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:10:39 (1788941439) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0692579 s, 296 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0134365 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:10:49 (1788941449) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00549168 s, 3.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:10:56 (1788941456) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0508607 s, 403 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:11:05 (1788941465) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00639684 s, 640 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0314098 s, 130 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:11:14 (1788941474) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0xf:0x0]: Invalid argument (22) PASS 60b (30s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:11:44 (1788941504) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:12:14 (1788941534) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:12:39 (1788941559) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:13:04 (1788941584) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 04:13:11 (1788941591) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57602 mirror split pid 57603 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57602 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57603 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 04:14:20 (1788941660) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 1662 skipping zero length zero range skipping zero length zero range fallocating to largest ever: 0x1d78d 000003 1788941662.796371 fallocate 0x018580 thru 0x01d78c (0x520d bytes) 000005 1788941662.806989 mapwrite 0x0000f9 thru 0x00e110 (0xe018 bytes) fallocating to largest ever: 0x29781 000006 1788941662.975853 fallocate 0x020425 thru 0x029780 (0x935c bytes) 000007 1788941662.985397 trunc from 0x00e111 to 0x007358 (0x6db8 bytes) 000008 1788941663.031547 trunc from 0x004ec0 to 0x007358 (0x2499 bytes) 000009 1788941663.047172 trunc from 0x007359 to 0x01c750 (0x153f8 bytes) 000010 1788941663.108873 mapread 0x006d7f thru 0x01359c (0xc81e bytes) 000011 1788941663.151262 trunc from 0x01c751 to 0x00e238 (0xe518 bytes) 000012 1788941663.175524 write 0x022b67 thru 0x02dc22 (0xb0bc bytes) 000013 1788941663.178240 write 0x000e45 thru 0x00c4bb (0xb677 bytes) 000015 1788941663.181445 write 0x03f74a thru 0x03ffff (0x8b6 bytes) fallocating to largest ever: 0x3e529 000016 1788941663.183835 fallocate 0x02ec75 thru 0x03e528 (0xf8b4 bytes) truncating to largest ever: 0x3e796 000017 1788941663.209052 trunc from 0x040000 to 0x03e795 (0x186a bytes) 000018 1788941663.272317 mapread 0x02c3e4 thru 0x030817 (0x4434 bytes) 000020 1788941663.317308 punch from 0x00acd9 to 0x01107b (0x63a3 bytes) 000022 1788941663.353318 punch from 0x027e61 to 0x02f970 (0x7b10 bytes) 000023 1788941663.364762 fallocate 0x0297e0 thru 0x0326b6 (0x8ed7 bytes) 000025 1788941663.377363 punch from 0x016847 to 0x01860f (0x1dc9 bytes) 000027 1788941663.415941 MIRROR_EXTEND to 2 mirrors 000028 1788941664.672566 fallocate 0x0007fa thru 0x009bab (0x93b2 bytes) 000029 1788941664.728540 punch from 0x00fed6 to 0x011ebd (0x1fe8 bytes) 000030 1788941664.739189 MIRROR_RESYNC 2 mirrors 000031 1788941665.822895 trunc from 0x03a32d to 0x03e795 (0x4469 bytes) 000032 1788941665.892976 trunc from 0x03e796 to 0x020360 (0x1e435 bytes) 000034 1788941665.921623 write 0x007de6 thru 0x00c302 (0x451d bytes) 000035 1788941665.934487 MIRROR_RESYNC 2 mirrors 000036 1788941666.610361 trunc from 0x020361 to 0x01e173 (0x21ed bytes) 000037 1788941666.699757 fallocate 0x00d832 thru 0x01c3f7 (0xebc6 bytes) 000038 1788941666.708614 fallocate 0x006bf0 thru 0x0112df (0xa6f0 bytes) 000039 1788941666.722766 mapwrite 0x02c832 thru 0x038aa6 (0xc275 bytes) 000040 1788941666.840129 read 0x02633b thru 0x030468 (0xa12e bytes) 000041 1788941666.849295 mapread 0x003e17 thru 0x010122 (0xc30c bytes) 000042 1788941666.873767 read 0x02e51f thru 0x02ead2 (0x5b4 bytes) 000043 1788941666.876507 trunc from 0x038aa7 to 0x03c989 (0x3ee3 bytes) 000044 1788941666.911417 write 0x011d0d thru 0x01b63e (0x9932 bytes) 000045 1788941666.937265 punch from 0x00be98 to 0x00f791 (0x38fa bytes) 000047 1788941666.954208 trunc from 0x0005a7 to 0x0052bd (0x4d17 bytes) 000048 1788941666.963747 trunc from 0x009979 to 0x0108b4 (0x6f3c bytes) 000049 1788941666.969936 trunc from 0x03c98a to 0x037299 (0x56f0 bytes) 000050 1788941667.000358 fallocate 0x00f90e thru 0x019b01 (0xa1f4 bytes) 000051 1788941667.009970 punch from 0x007b32 to 0x013796 (0xbc65 bytes) 000052 1788941667.065543 read 0x00b53a thru 0x0139d5 (0x849c bytes) fallocating to largest ever: 0x40000 000054 1788941667.094903 fallocate 0x03ce29 thru 0x03ffff (0x31d7 bytes) 000056 1788941667.112756 trunc from 0x03b666 to 0x03ba65 (0x400 bytes) 000057 1788941667.124896 mapwrite 0x014828 thru 0x018974 (0x414d bytes) 000058 1788941667.168745 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000059 1788941667.354389 mapread 0x032072 thru 0x03d79d (0xb72c bytes) 000061 1788941667.408536 write 0x030a44 thru 0x032b76 (0x2133 bytes) 000063 1788941667.420448 fallocate 0x0331df thru 0x036d28 (0x3b4a bytes) 000064 1788941667.431183 mapwrite 0x019cad thru 0x02204d (0x83a1 bytes) 000065 1788941667.506644 fallocate 0x03ee5c thru 0x03ffff (0x11a4 bytes) 000067 1788941667.516322 trunc from 0x01e190 to 0x020efc (0x2d6d bytes) 000068 1788941667.521501 mapwrite 0x003c7a thru 0x00eeba (0xb241 bytes) 000070 1788941667.556694 MIRROR_EXTEND to 2 mirrors 000071 1788941668.865133 mapwrite 0x0037ea thru 0x005f15 (0x272c bytes) 000072 1788941669.037052 punch from 0x009432 to 0x01554b (0xc11a bytes) 000073 1788941669.052734 write 0x013d81 thru 0x01df0c (0xa18c bytes) 000075 1788941669.078985 mapwrite 0x036bb0 thru 0x03e0e8 (0x7539 bytes) 000076 1788941669.171852 trunc from 0x040000 to 0x02945f (0x16ba0 bytes) 000077 1788941669.241483 fallocate 0x0222ef thru 0x0226bd (0x3cf bytes) 000078 1788941669.252855 punch from 0x01a6f0 to 0x023ded (0x96fe bytes) 000079 1788941669.266644 mapread 0x01f459 thru 0x01fd1c (0x8c4 bytes) 000080 1788941669.310480 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29460): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29460): Mirror 2: 0x51b45819 (stale) CRC-32 checksum value for chunk [0, 0x29460): Mirror 1: 0x8791b9d4 Mirror 3: 0x8791b9d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29460: skipped 000081 1788941670.447521 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29460): Mirror 2: 0x51b45819 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29460): Mirror 2: 0x51b45819 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29460: skipped 000083 1788941670.661806 trunc from 0x029460 to 0x01dc87 (0xb7d8 bytes) 000085 1788941670.711492 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1dc88): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dc88): Mirror 2: 0xba5b13e9 (stale) CRC-32 checksum value for chunk [0, 0x1dc88): Mirror 1: 0x2632462e Mirror 3: 0x2632462e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dc88: skipped 000086 1788941671.500832 read 0x017d7c thru 0x01dc87 (0x5f0c bytes) 000087 1788941671.515640 write 0x03cccd thru 0x03ffff (0x3333 bytes) 000088 1788941671.560982 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x32bdb109 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x32bdb109 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000089 1788941671.908925 read 0x0043d2 thru 0x006d8b (0x29ba bytes) 000090 1788941671.924066 mapwrite 0x036c74 thru 0x03e019 (0x73a6 bytes) 000091 1788941672.084263 punch from 0x026b55 to 0x035753 (0xebff bytes) 000092 1788941672.095463 trunc from 0x040000 to 0x03a646 (0x59b9 bytes) 000093 1788941672.145573 punch from 0x00f0c2 to 0x019940 (0xa87f bytes) 000094 1788941672.168509 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a647): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a647): Mirror 2: 0x45c0efe4 (stale) CRC-32 checksum value for chunk [0, 0x3a647): Mirror 1: 0xc39539ad Mirror 3: 0xc39539ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a647: skipped 000095 1788941673.947457 punch from 0x00f911 to 0x0123ad (0x2a9d bytes) 000096 1788941674.010179 fallocate 0x03f014 thru 0x03ffff (0xfec bytes) 000097 1788941674.016317 read 0x00f87e thru 0x01f69a (0xfe1d bytes) 000098 1788941674.034249 trunc from 0x031ad1 to 0x0336ff (0x1c2f bytes) 000099 1788941674.050214 punch from 0x008589 to 0x00fc8c (0x7704 bytes) 000100 1788941674.056211 read 0x02c171 thru 0x02c666 (0x4f6 bytes) 000101 1788941674.086044 read 0x0262da thru 0x035cbc (0xf9e3 bytes) 000102 1788941674.096964 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x32bdb109 (stale) Mirror 3: 0x2a8e7928 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbd652db3 Mirror 4: 0xbd652db3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000103 1788941675.143985 trunc from 0x040000 to 0x02f6ed (0x10912 bytes) 000104 1788941675.188387 fallocate 0x027dd5 thru 0x02d7b5 (0x59e1 bytes) 000105 1788941675.197418 fallocate 0x032bfa thru 0x03e848 (0xbc4f bytes) 000106 1788941675.205366 fallocate 0x0131db thru 0x01588f (0x26b5 bytes) 000107 1788941675.210784 trunc from 0x02f6ee to 0x017ead (0x17840 bytes) 000109 1788941675.273956 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17eae): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17eae): Mirror 2: 0x8a6e3946 (stale) Mirror 3: 0x83eb5482 (stale) Mirror 4: 0x1bf46252 (stale) CRC-32 checksum value for chunk [0, 0x17eae): Mirror 1: 0x1bf46252 Mirror 5: 0x1bf46252 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17eae: skipped 000110 1788941676.901493 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x17eae): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17eae): Mirror 2: 0x8a6e3946 (stale) Mirror 3: 0x83eb5482 (stale) Mirror 4: 0x1bf46252 (stale) CRC-32 checksum value for chunk [0, 0x17eae): Mirror 1: 0x1bf46252 Mirror 5: 0x1bf46252 Mirror 6: 0x1bf46252 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17eae: skipped 000111 1788941677.970416 MIRROR_RESYNC 6 mirrors 000113 1788941679.637176 mapread 0x00db9c thru 0x012e6c (0x52d1 bytes) 000114 1788941679.685815 read 0x004195 thru 0x0041ae (0x1a bytes) 000115 1788941679.692805 mapwrite 0x01b814 thru 0x020339 (0x4b26 bytes) 000116 1788941679.873557 mapread 0x01b105 thru 0x01db46 (0x2a42 bytes) 000117 1788941679.875812 mapwrite 0x01b5c1 thru 0x02018f (0x4bcf bytes) 000118 1788941679.900490 fallocate 0x034b06 thru 0x0382ba (0x37b5 bytes) 000119 1788941679.907670 mapwrite 0x008fce thru 0x012c28 (0x9c5b bytes) 000120 1788941680.046483 read 0x0058a0 thru 0x009ba7 (0x4308 bytes) 000121 1788941680.047939 mapwrite 0x0324e2 thru 0x03792f (0x544e bytes) 000122 1788941680.115375 MIRROR_RESYNC 6 mirrors 000123 1788941682.357928 fallocate 0x02f7fa thru 0x031960 (0x2167 bytes) 000124 1788941682.417348 mapwrite 0x03466a thru 0x03a43e (0x5dd5 bytes) 000125 1788941682.586176 mapread 0x009d40 thru 0x01959c (0xf85d bytes) 000126 1788941682.643431 punch from 0x009492 to 0x00cd22 (0x3891 bytes) 000127 1788941682.651075 punch from 0x00a8bd to 0x0121b8 (0x78fc bytes) 000128 1788941682.659647 fallocate 0x00dc75 thru 0x017a1b (0x9da7 bytes) 000130 1788941682.668259 punch from 0x028834 to 0x0375b4 (0xed81 bytes) 000131 1788941682.679242 read 0x004505 thru 0x012d97 (0xe893 bytes) 000132 1788941682.695411 trunc from 0x0050e8 to 0x00a948 (0x5861 bytes) 000133 1788941682.702384 fallocate 0x02e08d thru 0x02fd13 (0x1c87 bytes) 000135 1788941682.711384 mapread 0x022464 thru 0x02331b (0xeb8 bytes) 000136 1788941682.731898 trunc from 0x038021 to 0x03a43e (0x241e bytes) 000137 1788941682.744075 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a43f): Mirror 2: 0x214c6213 (stale) Mirror 4: 0x214c6213 (stale) Mirror 5: 0x214c6213 (stale) Mirror 6: 0x214c6213 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a43f): Mirror 2: 0x214c6213 (stale) Mirror 4: 0x214c6213 (stale) Mirror 5: 0x214c6213 (stale) Mirror 6: 0x214c6213 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a43f: skipped 000138 1788941683.222122 mapread 0x03550e thru 0x03751b (0x200e bytes) 000139 1788941683.278791 write 0x030492 thru 0x03163b (0x11aa bytes) 000140 1788941683.296387 write 0x001951 thru 0x006b46 (0x51f6 bytes) 000141 1788941683.323081 fallocate 0x01d8a9 thru 0x01ffb5 (0x270d bytes) 000142 1788941683.330223 mapread 0x016fd9 thru 0x021c72 (0xac9a bytes) 000143 1788941683.387053 mapwrite 0x00ad42 thru 0x00e88d (0x3b4c bytes) 000144 1788941683.433111 mapwrite 0x0374d8 thru 0x03ffff (0x8b28 bytes) 000145 1788941683.480799 mapread 0x022d68 thru 0x02b093 (0x832c bytes) 000147 1788941683.496122 mapwrite 0x031267 thru 0x03d76b (0xc505 bytes) 000148 1788941683.519818 punch from 0x00b307 to 0x015a78 (0xa772 bytes) 000149 1788941683.527362 mapread 0x02e158 thru 0x02e48c (0x335 bytes) 000150 1788941683.528380 read 0x00c025 thru 0x0174eb (0xb4c7 bytes) 000152 1788941683.541571 fallocate 0x00b840 thru 0x00ecfd (0x34be bytes) 000153 1788941683.548177 fallocate 0x00f4ae thru 0x0196ac (0xa1ff bytes) 000154 1788941683.553709 write 0x034ee3 thru 0x039dc2 (0x4ee0 bytes) 000155 1788941683.555597 mapwrite 0x03879f thru 0x03902b (0x88d bytes) 000156 1788941683.572597 mapwrite 0x0381ed thru 0x03b5b7 (0x33cb bytes) 000157 1788941683.592984 trunc from 0x024c6d to 0x0326b2 (0xda46 bytes) 000158 1788941683.600172 trunc from 0x00289b to 0x008f7b (0x66e1 bytes) 000160 1788941683.606473 mapread 0x035955 thru 0x03ffff (0xa6ab bytes) 000161 1788941683.607449 fallocate 0x030ce6 thru 0x03354c (0x2867 bytes) 000162 1788941683.613388 punch from 0x037359 to 0x03888f (0x1537 bytes) 000164 1788941683.619901 read 0x03c94e thru 0x03ffff (0x36b2 bytes) 000165 1788941683.620682 write 0x01ed93 thru 0x02c3d5 (0xd643 bytes) 000166 1788941683.632625 punch from 0x03e7ed to 0x03ffff (0x1813 bytes) 000167 1788941683.639090 write 0x03dc6a thru 0x03ffff (0x2396 bytes) 000168 1788941683.640116 mapwrite 0x00df3b thru 0x01565a (0x7720 bytes) 000169 1788941683.663152 trunc from 0x00d91f to 0x0149b2 (0x7094 bytes) 000170 1788941683.669976 write 0x029ad2 thru 0x02dc07 (0x4136 bytes) 000171 1788941683.681773 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x83fd2b86 (stale) Mirror 4: 0x83fd2b86 (stale) Mirror 5: 0x83fd2b86 (stale) Mirror 6: 0x83fd2b86 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x78db9ffc Mirror 7: 0x78db9ffc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000173 1788941685.078355 write 0x0090a8 thru 0x015527 (0xc480 bytes) 000174 1788941685.195570 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x83fd2b86 (stale) Mirror 4: 0x83fd2b86 (stale) Mirror 5: 0x83fd2b86 (stale) Mirror 7: 0x78db9ffc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x83fd2b86 (stale) Mirror 4: 0x83fd2b86 (stale) Mirror 5: 0x83fd2b86 (stale) Mirror 7: 0x78db9ffc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000175 1788941685.670110 mapwrite 0x02bcb2 thru 0x02ed33 (0x3082 bytes) 000176 1788941685.707834 punch from 0x0231c7 to 0x028d70 (0x5baa bytes) 000177 1788941685.713373 trunc from 0x00ca3e to 0x00f44f (0x2a12 bytes) 000178 1788941685.719091 mapread 0x005928 thru 0x011851 (0xbf2a bytes) 000179 1788941685.737227 read 0x02485a thru 0x029dbe (0x5565 bytes) 000180 1788941685.746823 write 0x012ab3 thru 0x01a237 (0x7785 bytes) 000181 1788941685.765737 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x83fd2b86 (stale) Mirror 4: 0x83fd2b86 (stale) Mirror 5: 0x83fd2b86 (stale) Mirror 7: 0x78db9ffc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9585f94 Mirror 8: 0xf9585f94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000182 1788941686.759844 read 0x007070 thru 0x00ded3 (0x6e64 bytes) 000183 1788941686.777157 punch from 0x0195ed to 0x01ac90 (0x16a4 bytes) 000185 1788941686.828070 mapread 0x003d6d thru 0x003fcb (0x25f bytes) 000188 1788941686.837082 mapread 0x00b818 thru 0x00fee6 (0x46cf bytes) 000189 1788941686.871042 write 0x0004b2 thru 0x00793b (0x748a bytes) 000190 1788941686.886976 trunc from 0x040000 to 0x03b52a (0x4ad5 bytes) 000191 1788941686.923104 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b52b): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b52b): Mirror 2: 0x7eb1e12d (stale) Mirror 4: 0x7eb1e12d (stale) Mirror 5: 0x7eb1e12d (stale) Mirror 7: 0x48cd8ff0 (stale) Mirror 8: 0xab1fa0f (stale) CRC-32 checksum value for chunk [0, 0x3b52b): Mirror 1: 0x1a68cf5d Mirror 9: 0x1a68cf5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b52b: skipped 000192 1788941688.464892 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b52b): Mirror 2: 0x7eb1e12d (stale) Mirror 5: 0x7eb1e12d (stale) Mirror 7: 0x48cd8ff0 (stale) Mirror 8: 0xab1fa0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b52b): Mirror 2: 0x7eb1e12d (stale) Mirror 5: 0x7eb1e12d (stale) Mirror 7: 0x48cd8ff0 (stale) Mirror 8: 0xab1fa0f (stale) CRC-32 checksum value for chunk [0, 0x3b52b): Mirror 1: 0x1a68cf5d Mirror 9: 0x1a68cf5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b52b: skipped 000193 1788941689.433940 mapwrite 0x02ccc7 thru 0x02ea46 (0x1d80 bytes) 000194 1788941689.624673 trunc from 0x03b52b to 0x038bd9 (0x2951 bytes) 000195 1788941689.685077 trunc from 0x038bda to 0x01ceef (0x1bcea bytes) 000196 1788941689.702884 mapwrite 0x014e47 thru 0x0218e8 (0xcaa2 bytes) 000197 1788941689.780710 trunc from 0x0218e9 to 0x03aa6e (0x19186 bytes) 000198 1788941689.812851 trunc from 0x00be43 to 0x010eb0 (0x506e bytes) 000199 1788941689.824942 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aa6f): Mirror 2: 0xa2aa931e (stale) Mirror 5: 0xa2aa931e (stale) Mirror 8: 0xb52b51a7 (stale) Mirror 9: 0xeebd1a0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa6f): Mirror 2: 0xa2aa931e (stale) Mirror 5: 0xa2aa931e (stale) Mirror 8: 0xb52b51a7 (stale) Mirror 9: 0xeebd1a0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa6f: skipped 000200 1788941690.227882 mapwrite 0x030df5 thru 0x0386b8 (0x78c4 bytes) 000201 1788941690.265364 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aa6f): Mirror 2: 0xa2aa931e (stale) Mirror 5: 0xa2aa931e (stale) Mirror 8: 0xb52b51a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa6f): Mirror 2: 0xa2aa931e (stale) Mirror 5: 0xa2aa931e (stale) Mirror 8: 0xb52b51a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa6f: skipped 000202 1788941690.604136 mapwrite 0x027cb4 thru 0x02dafd (0x5e4a bytes) 000203 1788941690.642800 fallocate 0x022e5d thru 0x030830 (0xd9d4 bytes) 000204 1788941690.649712 trunc from 0x01b26c to 0x02950a (0xe29f bytes) 000205 1788941690.657527 read 0x0194f2 thru 0x0224e5 (0x8ff4 bytes) 000206 1788941690.674816 trunc from 0x03aa6f to 0x034da4 (0x5cca bytes) 000207 1788941690.706759 mapread 0x007a74 thru 0x0094ff (0x1a8c bytes) 000208 1788941690.726150 read 0x00ee6e thru 0x01218b (0x331e bytes) 000209 1788941690.782973 mapread 0x003568 thru 0x00740e (0x3ea7 bytes) 000210 1788941690.783999 read 0x016076 thru 0x01e806 (0x8791 bytes) 000211 1788941690.784611 punch from 0x017238 to 0x01f3f5 (0x81be bytes) 000212 1788941690.803081 mapwrite 0x0117e4 thru 0x019b73 (0x8390 bytes) 000213 1788941690.841534 read 0x01d9c4 thru 0x02af0d (0xd54a bytes) 000214 1788941690.842220 mapread 0x0078d6 thru 0x00e246 (0x6971 bytes) 000215 1788941690.843108 trunc from 0x01f293 to 0x02d712 (0xe480 bytes) 000216 1788941690.854936 mapwrite 0x0038aa thru 0x00a90a (0x7061 bytes) 000217 1788941690.888497 trunc from 0x0055bf to 0x0094dc (0x3f1e bytes) 000218 1788941690.894518 fallocate 0x0142ef thru 0x019d03 (0x5a15 bytes) 000219 1788941690.900084 punch from 0x014610 to 0x01e9b3 (0xa3a4 bytes) 000220 1788941690.906217 read 0x01fa77 thru 0x02b80b (0xbd95 bytes) 000221 1788941690.929787 MIRROR_RESYNC 4 mirrors 000222 1788941692.766583 trunc from 0x012768 to 0x0169c6 (0x425f bytes) 000223 1788941692.825622 fallocate 0x032442 thru 0x03ac12 (0x87d1 bytes) 000224 1788941692.842681 trunc from 0x034da5 to 0x00a515 (0x2a88f bytes) 000225 1788941692.888241 trunc from 0x00a516 to 0x012e36 (0x8921 bytes) 000226 1788941692.938568 read 0x00f29a thru 0x012e36 (0x3b9d bytes) 000227 1788941692.958454 read 0x00a27c thru 0x012e36 (0x8bbb bytes) 000228 1788941693.019482 write 0x036638 thru 0x0391d3 (0x2b9c bytes) 000229 1788941693.020998 punch from 0x00190d to 0x00587a (0x3f6e bytes) 000230 1788941693.030897 write 0x03e9b3 thru 0x03ffff (0x164d bytes) 000231 1788941693.043036 write 0x030fb1 thru 0x035171 (0x41c1 bytes) 000232 1788941693.074631 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x53e26bab (stale) Mirror 5: 0x53e26bab (stale) Mirror 8: 0x53e26bab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6edb3281 Mirror 9: 0x6edb3281 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000233 1788941694.223962 fallocate 0x011421 thru 0x0143df (0x2fbf bytes) 000234 1788941694.275788 mapread 0x003b0e thru 0x00b0a0 (0x7593 bytes) 000235 1788941694.297956 mapread 0x00d888 thru 0x011644 (0x3dbd bytes) 000236 1788941694.309122 mapread 0x020e44 thru 0x025183 (0x4340 bytes) 000237 1788941694.320327 write 0x0187bd thru 0x0238b3 (0xb0f7 bytes) 000238 1788941694.328760 trunc from 0x040000 to 0x00d65d (0x329a2 bytes) 000239 1788941694.374081 trunc from 0x00d65e to 0x01f2a4 (0x11c47 bytes) 000240 1788941694.392358 fallocate 0x00e09a thru 0x01d7e5 (0xf74c bytes) 000241 1788941694.398251 punch from 0x004674 to 0x00f704 (0xb091 bytes) 000243 1788941694.405537 read 0x00466d thru 0x00c678 (0x800c bytes) 000244 1788941694.416123 trunc from 0x01f2a5 to 0x01deb1 (0x13f3 bytes) 000245 1788941694.432717 trunc from 0x01deb2 to 0x037317 (0x19466 bytes) 000246 1788941694.452715 fallocate 0x02d200 thru 0x03be42 (0xec43 bytes) 000247 1788941694.466163 write 0x03948a thru 0x03eeb8 (0x5a2f bytes) 000248 1788941694.473006 write 0x01444c thru 0x022d0a (0xe8bf bytes) 000249 1788941694.484988 punch from 0x0159d1 to 0x020a51 (0xb081 bytes) 000250 1788941694.502405 mapwrite 0x01bc8c thru 0x01e411 (0x2786 bytes) 000251 1788941694.556235 write 0x009149 thru 0x015d2b (0xcbe3 bytes) 000252 1788941694.559469 read 0x01f030 thru 0x01fdae (0xd7f bytes) 000253 1788941694.565406 write 0x027984 thru 0x036708 (0xed85 bytes) 000254 1788941694.611013 punch from 0x024330 to 0x029af6 (0x57c7 bytes) 000255 1788941694.635602 mapwrite 0x0138d9 thru 0x0161b0 (0x28d8 bytes) 000257 1788941694.663736 mapwrite 0x01ca1f thru 0x02267a (0x5c5c bytes) 000258 1788941694.697793 trunc from 0x03eeb9 to 0x00bafb (0x333bd bytes) 000259 1788941694.717995 mapwrite 0x010012 thru 0x019350 (0x933f bytes) 000260 1788941694.832206 trunc from 0x014ead to 0x019350 (0x44a4 bytes) 000261 1788941694.841133 mapwrite 0x00441d thru 0x0087f2 (0x43d6 bytes) 000262 1788941694.877690 fallocate 0x036ffb thru 0x03db65 (0x6b6b bytes) 000263 1788941694.891552 read 0x01abfe thru 0x020e59 (0x625c bytes) 000264 1788941694.921476 fallocate 0x024397 thru 0x029f13 (0x5b7d bytes) 000265 1788941694.953004 write 0x013dd0 thru 0x01484b (0xa7c bytes) 000267 1788941694.954572 mapread 0x0093f9 thru 0x00f265 (0x5e6d bytes) 000268 1788941694.961681 trunc from 0x03db66 to 0x024ff1 (0x18b74 bytes) 000269 1788941695.019800 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24ff2): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ff2): Mirror 2: 0x28390362 (stale) Mirror 5: 0x28390362 (stale) Mirror 8: 0x28390362 (stale) Mirror 9: 0x9b7f4d8e (stale) CRC-32 checksum value for chunk [0, 0x24ff2): Mirror 1: 0xd0cb05e4 Mirror 10: 0xd0cb05e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ff2: skipped 000271 1788941696.154930 fallocate 0x000d94 thru 0x0036fb (0x2968 bytes) 000272 1788941696.213452 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24ff2): Mirror 2: 0x28390362 (stale) Mirror 8: 0x28390362 (stale) Mirror 9: 0x9b7f4d8e (stale) Mirror 10: 0xd0cb05e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ff2): Mirror 2: 0x28390362 (stale) Mirror 8: 0x28390362 (stale) Mirror 9: 0x9b7f4d8e (stale) Mirror 10: 0xd0cb05e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ff2: skipped 000273 1788941696.728631 punch from 0x011ecb to 0x01b388 (0x94be bytes) 000274 1788941696.733905 trunc from 0x024ff2 to 0x00186b (0x23786 bytes) 000275 1788941696.759058 fallocate 0x0280ab thru 0x034f58 (0xceae bytes) 000276 1788941696.765359 mapread 0x019600 thru 0x021bf7 (0x85f8 bytes) 000277 1788941696.793482 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34f59): Mirror 2: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f59): Mirror 2: 0x5d707930 (stale) Mirror 8: 0x5d707930 (stale) Mirror 9: 0xdbe6bd25 (stale) Mirror 10: 0xe4a495f2 (stale) CRC-32 checksum value for chunk [0, 0x34f59): Mirror 1: 0xa5fdf074 Mirror 11: 0xa5fdf074 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f59: skipped 000279 1788941698.206065 trunc from 0x034f59 to 0x008109 (0x2ce4f bytes) 000280 1788941698.371137 write 0x000425 thru 0x0048d0 (0x44ac bytes) 000281 1788941698.386643 trunc from 0x00810a to 0x013b2c (0xba23 bytes) 000282 1788941698.412974 punch from 0x00a9c6 to 0x00c852 (0x1e8d bytes) 000284 1788941698.419337 mapread 0x00a5a9 thru 0x00d339 (0x2d91 bytes) 000285 1788941698.441458 trunc from 0x00bdaa to 0x013b2c (0x7d83 bytes) 000286 1788941698.446168 trunc from 0x013b2d to 0x00cba0 (0x6f8c bytes) 000287 1788941698.468671 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xcba1): Mirror 8: 0xd8a3d8c4 (stale) Mirror 9: 0x4717a52e (stale) Mirror 10: 0xe21f4051 (stale) Mirror 11: 0xf5ca8bac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcba1): Mirror 8: 0xd8a3d8c4 (stale) Mirror 9: 0x4717a52e (stale) Mirror 10: 0xe21f4051 (stale) Mirror 11: 0xf5ca8bac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcba1: skipped 000288 1788941698.825811 mapwrite 0x008961 thru 0x011625 (0x8cc5 bytes) 000289 1788941698.902078 write 0x025cc2 thru 0x029b62 (0x3ea1 bytes) 000290 1788941698.905047 mapread 0x00ff3e thru 0x0120cb (0x218e bytes) 000291 1788941698.913918 punch from 0x014425 to 0x016ca2 (0x287e bytes) 000292 1788941698.920079 read 0x00fa7a thru 0x01777d (0x7d04 bytes) 000293 1788941698.928049 mapread 0x012c4f thru 0x0139a8 (0xd5a bytes) 000294 1788941698.930034 read 0x009a07 thru 0x017347 (0xd941 bytes) 000295 1788941698.932405 trunc from 0x029b63 to 0x0103be (0x197a4 bytes) 000297 1788941698.952497 write 0x009528 thru 0x015bb4 (0xc68d bytes) 000298 1788941698.954173 trunc from 0x015bb5 to 0x0217e2 (0xbc2e bytes) 000299 1788941698.978664 trunc from 0x019acb to 0x0217e2 (0x7d18 bytes) 000300 1788941698.985025 read 0x00c36a thru 0x00ff9f (0x3c36 bytes) 000301 1788941698.986422 mapwrite 0x03aff1 thru 0x03ffff (0x500f bytes) 000302 1788941699.071091 fallocate 0x036930 thru 0x03ffff (0x96d0 bytes) 000303 1788941699.081488 trunc from 0x040000 to 0x01def3 (0x2210c bytes) 000304 1788941699.108656 trunc from 0x0010ed to 0x0088d6 (0x77ea bytes) 000305 1788941699.116130 write 0x01e802 thru 0x02bfb9 (0xd7b8 bytes) 000306 1788941699.120066 trunc from 0x0184c5 to 0x0245fc (0xc138 bytes) 000307 1788941699.144063 fallocate 0x03f4f9 thru 0x03ffff (0xb07 bytes) 000308 1788941699.150852 read 0x038e94 thru 0x039a4f (0xbbc bytes) 000309 1788941699.246568 MIRROR_RESYNC 5 mirrors 000310 1788941701.389818 MIRROR_RESYNC 5 mirrors 000311 1788941701.787028 read 0x00d5ad thru 0x01d375 (0xfdc9 bytes) 000312 1788941701.797239 mapread 0x02c471 thru 0x033a00 (0x7590 bytes) 000313 1788941701.819444 trunc from 0x040000 to 0x02ad5d (0x152a2 bytes) 000314 1788941701.874513 write 0x027f96 thru 0x02e089 (0x60f4 bytes) 000315 1788941701.881092 write 0x00832b thru 0x00fb98 (0x786e bytes) 000317 1788941701.890778 trunc from 0x02a0fc to 0x02e089 (0x3f8e bytes) 000318 1788941701.904556 fallocate 0x010b88 thru 0x016ba8 (0x6021 bytes) 000319 1788941701.913351 punch from 0x01c6ee to 0x024c59 (0x856c bytes) 000320 1788941701.918338 write 0x02dd4c thru 0x03b24c (0xd501 bytes) 000321 1788941701.925957 write 0x01883e thru 0x01cb30 (0x42f3 bytes) 000322 1788941701.935078 fallocate 0x013ec2 thru 0x01981c (0x595b bytes) 000323 1788941701.940897 mapread 0x01aa58 thru 0x01bf5c (0x1505 bytes) 000324 1788941701.942205 read 0x0342f1 thru 0x03b24c (0x6f5c bytes) 000325 1788941701.943139 read 0x032390 thru 0x03af3e (0x8baf bytes) 000327 1788941701.948861 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b24d): Mirror 8: 0xdd9a914c (stale) Mirror 10: 0xdd9a914c (stale) Mirror 11: 0xdd9a914c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b24d): Mirror 8: 0xdd9a914c (stale) Mirror 10: 0xdd9a914c (stale) Mirror 11: 0xdd9a914c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b24d: skipped 000330 1788941702.297497 mapread 0x00e987 thru 0x012372 (0x39ec bytes) 000331 1788941702.316746 trunc from 0x03b24d to 0x023232 (0x1801a bytes) 000332 1788941702.340517 trunc from 0x011525 to 0x013075 (0x1b51 bytes) 000333 1788941702.364025 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23233): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23233): Mirror 8: 0xbb0f151a (stale) Mirror 10: 0xbb0f151a (stale) Mirror 11: 0xbb0f151a (stale) CRC-32 checksum value for chunk [0, 0x23233): Mirror 1: 0x1e50b7ce Mirror 12: 0x1e50b7ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23233: skipped 000335 1788941703.343228 trunc from 0x023233 to 0x033141 (0xff0f bytes) 000336 1788941703.401174 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33142): Mirror 8: 0x7c9da2b2 (stale) Mirror 10: 0x7c9da2b2 (stale) Mirror 12: 0xd37f5573 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33142): Mirror 8: 0x7c9da2b2 (stale) Mirror 10: 0x7c9da2b2 (stale) Mirror 12: 0xd37f5573 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33142: skipped 000337 1788941703.713944 trunc from 0x033142 to 0x02d1f0 (0x5f51 bytes) 000338 1788941703.731970 fallocate 0x019af1 thru 0x026851 (0xcd61 bytes) 000339 1788941703.738061 punch from 0x021eef to 0x026e00 (0x4f12 bytes) 000340 1788941703.744023 fallocate 0x006d01 thru 0x015c9c (0xef9c bytes) 000341 1788941703.749632 mapread 0x006dcd thru 0x007f02 (0x1136 bytes) 000342 1788941703.760450 mapwrite 0x025ef7 thru 0x02fc92 (0x9d9c bytes) 000343 1788941703.805709 fallocate 0x0360b1 thru 0x03b4ab (0x53fb bytes) 000344 1788941703.812132 mapread 0x01c3f2 thru 0x020190 (0x3d9f bytes) 000345 1788941703.823630 read 0x00daa0 thru 0x01a881 (0xcde2 bytes) 000346 1788941703.833878 mapread 0x025120 thru 0x02d801 (0x86e2 bytes) 000347 1788941703.834878 trunc from 0x02fc93 to 0x02a154 (0x5b3e bytes) 000348 1788941703.861984 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a155): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a155): Mirror 8: 0x58598b77 (stale) Mirror 10: 0x58598b77 (stale) Mirror 12: 0xd39e8d6b (stale) CRC-32 checksum value for chunk [0, 0x2a155): Mirror 1: 0x87f879de Mirror 13: 0x87f879de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a155: skipped 000349 1788941704.946404 trunc from 0x02a155 to 0x02cda6 (0x2c52 bytes) 000350 1788941705.003966 trunc from 0x02cda7 to 0x0223f9 (0xa9ad bytes) 000351 1788941705.028681 read 0x002bac thru 0x00c36f (0x97c4 bytes) 000352 1788941705.039247 fallocate 0x0324d1 thru 0x03b987 (0x94b7 bytes) 000353 1788941705.045134 write 0x01ee80 thru 0x024147 (0x52c8 bytes) 000354 1788941705.056936 trunc from 0x012177 to 0x0144cc (0x2356 bytes) 000355 1788941705.067679 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b988): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b988): Mirror 8: 0x83f787b1 (stale) Mirror 10: 0x83f787b1 (stale) Mirror 12: 0x3085dc93 (stale) Mirror 13: 0xdbb722b (stale) CRC-32 checksum value for chunk [0, 0x3b988): Mirror 1: 0xf0524f3c Mirror 14: 0xf0524f3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b988: skipped 000356 1788941706.448202 mapread 0x030ee6 thru 0x03425f (0x337a bytes) 000357 1788941706.485013 write 0x026030 thru 0x02803f (0x2010 bytes) 000358 1788941706.541556 trunc from 0x022e1c to 0x025fe2 (0x31c7 bytes) 000359 1788941706.548918 read 0x02c92d thru 0x03b987 (0xf05b bytes) 000360 1788941706.561371 fallocate 0x000fc1 thru 0x01057d (0xf5bd bytes) 000361 1788941706.579042 fallocate 0x037933 thru 0x03f2e6 (0x79b4 bytes) 000362 1788941706.595013 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2e7): Mirror 10: 0xebc0e64e (stale) Mirror 12: 0xcd33c6f1 (stale) Mirror 13: 0xc23468e1 (stale) Mirror 14: 0x666a6508 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2e7): Mirror 10: 0xebc0e64e (stale) Mirror 12: 0xcd33c6f1 (stale) Mirror 13: 0xc23468e1 (stale) Mirror 14: 0x666a6508 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2e7: skipped 000364 1788941707.187366 fallocate 0x02b554 thru 0x0370a8 (0xbb55 bytes) 000365 1788941707.193774 write 0x0035a0 thru 0x00bc6d (0x86ce bytes) 000366 1788941707.229879 fallocate 0x0131ff thru 0x019c04 (0x6a06 bytes) 000367 1788941707.241401 fallocate 0x028bbb thru 0x0340c9 (0xb50f bytes) 000368 1788941707.249025 mapwrite 0x02ebbb thru 0x035404 (0x684a bytes) 000369 1788941707.315097 mapread 0x03e0c2 thru 0x03f2e6 (0x1225 bytes) 000372 1788941707.330578 mapread 0x007ed6 thru 0x008109 (0x234 bytes) 000373 1788941707.331652 write 0x01c112 thru 0x01cf1e (0xe0d bytes) 000375 1788941707.344479 mapread 0x0148fc thru 0x01f754 (0xae59 bytes) 000376 1788941707.403136 read 0x002c8a thru 0x01059b (0xd912 bytes) 000377 1788941707.417796 trunc from 0x02cd47 to 0x02ea21 (0x1cdb bytes) 000378 1788941707.426822 fallocate 0x0365a7 thru 0x0389fb (0x2455 bytes) 000379 1788941707.436493 trunc from 0x03f2e7 to 0x024cb1 (0x1a635 bytes) 000380 1788941707.507174 mapread 0x022ec5 thru 0x024cb1 (0x1ded bytes) 000381 1788941707.508416 trunc from 0x024cb2 to 0x0043bf (0x208f2 bytes) 000382 1788941707.543922 mapwrite 0x019850 thru 0x01c1d8 (0x2989 bytes) 000383 1788941707.622838 mapread 0x01362f thru 0x01c1d8 (0x8baa bytes) 000384 1788941707.628620 punch from 0x00355a to 0x00bc2b (0x86d2 bytes) 000385 1788941707.640212 trunc from 0x019602 to 0x01c1d8 (0x2bd7 bytes) 000386 1788941707.647979 fallocate 0x003b84 thru 0x01052e (0xc9ab bytes) 000387 1788941707.656113 punch from 0x00d5c9 to 0x01c1d8 (0xec10 bytes) 000388 1788941707.672334 read 0x00a906 thru 0x01713e (0xc839 bytes) 000389 1788941707.689471 mapread 0x00d465 thru 0x00dac4 (0x660 bytes) 000390 1788941707.691018 trunc from 0x01c1d9 to 0x02826d (0xc095 bytes) 000391 1788941707.731226 read 0x01bede thru 0x02826d (0xc390 bytes) 000392 1788941707.746844 read 0x022d45 thru 0x02826d (0x5529 bytes) 000393 1788941707.747583 trunc from 0x00099e to 0x0061ec (0x584f bytes) 000394 1788941707.762349 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2826e): Mirror 10: 0x455e743f (stale) Mirror 13: 0xf9f701d8 (stale) Mirror 14: 0xf3d56c8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2826e): Mirror 10: 0x455e743f (stale) Mirror 13: 0xf9f701d8 (stale) Mirror 14: 0xf3d56c8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2826e: skipped 000396 1788941708.255544 trunc from 0x027356 to 0x02826d (0xf18 bytes) 000397 1788941708.274567 punch from 0x01ac6f to 0x021cb7 (0x7049 bytes) 000398 1788941708.281898 write 0x011373 thru 0x015a34 (0x46c2 bytes) 000399 1788941708.298870 trunc from 0x01d561 to 0x022695 (0x5135 bytes) 000401 1788941708.311872 punch from 0x024100 to 0x02812f (0x4030 bytes) 000402 1788941708.317698 read 0x010924 thru 0x01b014 (0xa6f1 bytes) 000403 1788941708.326533 fallocate 0x010084 thru 0x013492 (0x340f bytes) 000404 1788941708.342150 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2826e): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2826e): Mirror 10: 0x455e743f (stale) Mirror 13: 0xf9f701d8 (stale) Mirror 14: 0xf3d56c8c (stale) CRC-32 checksum value for chunk [0, 0x2826e): Mirror 1: 0xfb007983 Mirror 15: 0xfb007983 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2826e: skipped 000405 1788941709.345573 fallocate 0x01634f thru 0x0238ee (0xd5a0 bytes) 000406 1788941709.412608 mapread 0x0050c2 thru 0x010309 (0xb248 bytes) 000407 1788941709.452017 fallocate 0x010a3e thru 0x010b52 (0x115 bytes) 000408 1788941709.458137 punch from 0x00c57c to 0x014afd (0x8582 bytes) 000410 1788941709.465385 mapwrite 0x03354c thru 0x03fa77 (0xc52c bytes) 000411 1788941709.543215 trunc from 0x03bc15 to 0x03fa77 (0x3e63 bytes) 000412 1788941709.548993 trunc from 0x03fa78 to 0x03ab4b (0x4f2c bytes) 000413 1788941709.563771 punch from 0x020e8a to 0x02f0c2 (0xe239 bytes) 000414 1788941709.569126 mapread 0x00f3c9 thru 0x013041 (0x3c79 bytes) 000415 1788941709.586143 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ab4c): Mirror 13: 0xc6929591 (stale) Mirror 14: 0x72b8efa2 (stale) Mirror 15: 0x4494be7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab4c): Mirror 13: 0xc6929591 (stale) Mirror 14: 0x72b8efa2 (stale) Mirror 15: 0x4494be7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab4c: skipped 000416 1788941709.987090 trunc from 0x03ab4c to 0x00cfb5 (0x2db96 bytes) 000417 1788941710.014167 mapread 0x003d22 thru 0x00cfb5 (0x9294 bytes) 000418 1788941710.038237 read 0x00329f thru 0x00cfb5 (0x9d17 bytes) 000419 1788941710.039183 mapwrite 0x018ee7 thru 0x0194ff (0x619 bytes) 000422 1788941710.133604 MIRROR_RESYNC 4 mirrors 000423 1788941710.959951 punch from 0x017eac to 0x0194ff (0x1654 bytes) 000424 1788941711.017196 mapread 0x00979f thru 0x014ccf (0xb531 bytes) 000425 1788941711.070583 trunc from 0x00bcfa to 0x013f31 (0x8238 bytes) 000426 1788941711.089785 trunc from 0x018f9e to 0x0194ff (0x562 bytes) 000427 1788941711.109937 punch from 0x002aea to 0x00a892 (0x7da9 bytes) 000428 1788941711.121621 mapread 0x016e27 thru 0x0194ff (0x26d9 bytes) 000429 1788941711.137960 read 0x01318b thru 0x013a06 (0x87c bytes) 000430 1788941711.142252 punch from 0x009615 to 0x00e159 (0x4b45 bytes) 000431 1788941711.152034 trunc from 0x019500 to 0x02c58a (0x1308b bytes) 000433 1788941711.220069 write 0x01902c thru 0x02480e (0xb7e3 bytes) 000434 1788941711.237057 trunc from 0x00559a to 0x00857e (0x2fe5 bytes) 000435 1788941711.244133 punch from 0x002cd8 to 0x00f676 (0xc99f bytes) 000436 1788941711.250337 mapwrite 0x03e1bf thru 0x03ffff (0x1e41 bytes) 000437 1788941711.311208 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x4c641c9c (stale) Mirror 14: 0x4c641c9c (stale) Mirror 15: 0x4c641c9c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d195b05 Mirror 16: 0x7d195b05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000438 1788941712.412019 punch from 0x026a77 to 0x02a1e5 (0x376f bytes) 000439 1788941712.512939 read 0x0327bf thru 0x03edc8 (0xc60a bytes) 000440 1788941712.542510 read 0x037102 thru 0x03be16 (0x4d15 bytes) 000441 1788941712.543068 mapread 0x011887 thru 0x012548 (0xcc2 bytes) 000442 1788941712.566182 trunc from 0x040000 to 0x00c561 (0x33a9e bytes) 000443 1788941712.730683 MIRROR_RESYNC 5 mirrors 000444 1788941713.970833 mapwrite 0x012294 thru 0x01eac7 (0xc834 bytes) 000445 1788941714.077808 mapread 0x005ddb thru 0x005f56 (0x17c bytes) 000446 1788941714.088614 trunc from 0x01130d to 0x01d575 (0xc269 bytes) 000447 1788941714.093827 trunc from 0x01eac8 to 0x0025b0 (0x1c517 bytes) 000448 1788941714.126070 write 0x021ede thru 0x025cbb (0x3dde bytes) 000449 1788941714.128631 punch from 0x01c76e to 0x025cbb (0x954e bytes) 000450 1788941714.140766 write 0x035e6a thru 0x03655c (0x6f3 bytes) 000451 1788941714.145192 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3655d): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3655d): Mirror 13: 0x115b43ee (stale) Mirror 14: 0x115b43ee (stale) Mirror 15: 0x115b43ee (stale) Mirror 16: 0x115b43ee (stale) CRC-32 checksum value for chunk [0, 0x3655d): Mirror 1: 0xec820c86 Mirror 17: 0xec820c86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3655d: skipped 000452 1788941715.303146 read 0x00e3a0 thru 0x014302 (0x5f63 bytes) 000453 1788941715.339903 read 0x022c63 thru 0x028bbc (0x5f5a bytes) 000454 1788941715.368516 read 0x014dbb thru 0x0238c2 (0xeb08 bytes) 000456 1788941715.398364 fallocate 0x0047b0 thru 0x00bd7d (0x75ce bytes) 000457 1788941715.475179 read 0x01e8f5 thru 0x021d63 (0x346f bytes) 000458 1788941715.487708 punch from 0x007e40 to 0x00a852 (0x2a13 bytes) 000459 1788941715.493170 trunc from 0x0078a6 to 0x016bb5 (0xf310 bytes) 000460 1788941715.499110 mapwrite 0x01b6f3 thru 0x02a181 (0xea8f bytes) 000461 1788941715.528641 mapwrite 0x02b25e thru 0x0317ee (0x6591 bytes) 000462 1788941715.547684 fallocate 0x02ec5d thru 0x02fe88 (0x122c bytes) 000463 1788941715.553026 trunc from 0x0145d3 to 0x022c11 (0xe63f bytes) 000464 1788941715.558205 write 0x02c6ed thru 0x0359f0 (0x9304 bytes) 000465 1788941715.564052 trunc from 0x03655d to 0x000761 (0x35dfb bytes) 000466 1788941715.590582 fallocate 0x028ced thru 0x02d668 (0x497c bytes) 000467 1788941715.597038 write 0x03ca6e thru 0x03ffff (0x3592 bytes) 000468 1788941715.598302 punch from 0x00caf2 to 0x013802 (0x6d11 bytes) 000469 1788941715.604469 mapwrite 0x024f1c thru 0x032416 (0xd4fb bytes) 000471 1788941715.711338 trunc from 0x01e721 to 0x024fb2 (0x6892 bytes) 000473 1788941715.722557 mapread 0x03e335 thru 0x03ffff (0x1ccb bytes) 000474 1788941715.724565 trunc from 0x040000 to 0x03f290 (0xd6f bytes) 000475 1788941715.758161 mapread 0x033483 thru 0x033da8 (0x926 bytes) 000476 1788941715.767501 mapread 0x0010f9 thru 0x00155c (0x464 bytes) 000478 1788941715.775462 punch from 0x00420b to 0x00d725 (0x951b bytes) 000479 1788941715.780147 fallocate 0x0103dc thru 0x01f9e5 (0xf60a bytes) 000480 1788941715.791190 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f291): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f291): Mirror 13: 0xb03d6f5d (stale) Mirror 14: 0xb03d6f5d (stale) Mirror 15: 0xb03d6f5d (stale) Mirror 16: 0xb03d6f5d (stale) Mirror 17: 0x258ec2aa (stale) CRC-32 checksum value for chunk [0, 0x3f291): Mirror 1: 0xe16e24e3 Mirror 18: 0xe16e24e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f291: skipped 000481 1788941716.695041 trunc from 0x03f291 to 0x01c977 (0x22919 bytes) 000482 1788941716.753914 mapread 0x002030 thru 0x011eae (0xfe7f bytes) 000483 1788941716.792520 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1c978): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c978): Mirror 13: 0x429796c2 (stale) Mirror 14: 0x429796c2 (stale) Mirror 15: 0x429796c2 (stale) Mirror 16: 0x429796c2 (stale) Mirror 17: 0x429796c2 (stale) Mirror 18: 0x8456cad9 (stale) CRC-32 checksum value for chunk [0, 0x1c978): Mirror 1: 0x8456cad9 Mirror 19: 0x8456cad9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c978: skipped 000484 1788941717.748602 MIRROR_RESYNC 8 mirrors 000485 1788941719.033613 mapread 0x0062b8 thru 0x013c04 (0xd94d bytes) 000486 1788941719.072393 trunc from 0x005c5f to 0x0131e0 (0xd582 bytes) 000487 1788941719.121700 read 0x012f4f thru 0x01c977 (0x9a29 bytes) 000488 1788941719.130002 read 0x004b52 thru 0x00e108 (0x95b7 bytes) 000489 1788941719.142946 trunc from 0x017d0e to 0x0196ab (0x199e bytes) 000490 1788941719.147901 trunc from 0x00a76f to 0x0115ee (0x6e80 bytes) 000491 1788941719.158456 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c978): Mirror 13: 0x8456cad9 (stale) Mirror 15: 0x8456cad9 (stale) Mirror 16: 0x8456cad9 (stale) Mirror 17: 0x8456cad9 (stale) Mirror 18: 0x8456cad9 (stale) Mirror 19: 0x8456cad9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c978): Mirror 13: 0x8456cad9 (stale) Mirror 15: 0x8456cad9 (stale) Mirror 16: 0x8456cad9 (stale) Mirror 17: 0x8456cad9 (stale) Mirror 18: 0x8456cad9 (stale) Mirror 19: 0x8456cad9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c978: skipped 000492 1788941719.604076 trunc from 0x01c978 to 0x020bff (0x4288 bytes) 000493 1788941719.642961 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x20c00): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c00): Mirror 13: 0xb14e95d9 (stale) Mirror 15: 0xb14e95d9 (stale) Mirror 16: 0xb14e95d9 (stale) Mirror 17: 0xb14e95d9 (stale) Mirror 18: 0xb14e95d9 (stale) Mirror 19: 0xb14e95d9 (stale) CRC-32 checksum value for chunk [0, 0x20c00): Mirror 1: 0xb14e95d9 Mirror 20: 0xb14e95d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c00: skipped 000494 1788941720.793255 read 0x020a8e thru 0x020bff (0x172 bytes) 000495 1788941720.816088 punch from 0x00eb2a to 0x019b43 (0xb01a bytes) 000496 1788941720.897951 trunc from 0x007ed4 to 0x00ba5b (0x3b88 bytes) 000497 1788941720.910112 write 0x000d97 thru 0x0087bb (0x7a25 bytes) 000498 1788941720.930509 punch from 0x01e172 to 0x020bff (0x2a8e bytes) 000499 1788941720.937527 trunc from 0x005c81 to 0x014a15 (0xed95 bytes) 000500 1788941720.961907 write 0x00e779 thru 0x01a040 (0xb8c8 bytes) 000502 1788941720.988815 MIRROR_RESYNC 8 mirrors 000503 1788941723.225362 trunc from 0x020c00 to 0x0302ae (0xf6af bytes) 000505 1788941723.338622 mapread 0x011527 thru 0x011725 (0x1ff bytes) 000506 1788941723.350872 punch from 0x0049ee to 0x00ba2a (0x703d bytes) 000507 1788941723.365597 write 0x017f7e thru 0x019bf5 (0x1c78 bytes) 000508 1788941723.412222 read 0x02ddd7 thru 0x0302ae (0x24d8 bytes) 000509 1788941723.428719 punch from 0x02a0e9 to 0x0302ae (0x61c6 bytes) 000510 1788941723.433694 mapread 0x012f19 thru 0x01a224 (0x730c bytes) 000511 1788941723.454945 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x302af): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302af): Mirror 13: 0x9e553b97 (stale) Mirror 15: 0x9e553b97 (stale) Mirror 16: 0x9e553b97 (stale) Mirror 17: 0x9e553b97 (stale) Mirror 18: 0x9e553b97 (stale) Mirror 19: 0x9e553b97 (stale) Mirror 20: 0x9e553b97 (stale) CRC-32 checksum value for chunk [0, 0x302af): Mirror 1: 0x6237118f Mirror 21: 0x6237118f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302af: skipped 000512 1788941724.790662 read 0x00b18d thru 0x01a06d (0xeee1 bytes) 000513 1788941724.822622 trunc from 0x0302af to 0x00a24a (0x26064 bytes) 000514 1788941724.914020 trunc from 0x0058e3 to 0x00a24a (0x4968 bytes) 000515 1788941724.926404 punch from 0x00232e to 0x0079f7 (0x56ca bytes) 000516 1788941724.935029 punch from 0x00413f to 0x00a24a (0x610c bytes) 000517 1788941724.942694 trunc from 0x00a24b to 0x020c2c (0x169e2 bytes) 000519 1788941724.978813 punch from 0x006448 to 0x007630 (0x11e9 bytes) 000520 1788941724.985125 mapwrite 0x008c77 thru 0x010b99 (0x7f23 bytes) 000522 1788941725.031632 read 0x0045ed thru 0x00812d (0x3b41 bytes) 000523 1788941725.032370 read 0x00de51 thru 0x018d63 (0xaf13 bytes) 000524 1788941725.044222 trunc from 0x020c2d to 0x01247f (0xe7ad bytes) 000526 1788941725.064023 punch from 0x006372 to 0x010da0 (0xaa2f bytes) 000527 1788941725.071943 punch from 0x00ae52 to 0x01247f (0x762e bytes) 000528 1788941725.083731 trunc from 0x012480 to 0x01d19f (0xad20 bytes) 000529 1788941725.112011 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1d1a0): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d1a0): Mirror 13: 0x51b61824 (stale) Mirror 15: 0x51b61824 (stale) Mirror 16: 0x51b61824 (stale) Mirror 17: 0x51b61824 (stale) Mirror 18: 0x51b61824 (stale) Mirror 19: 0x51b61824 (stale) Mirror 20: 0x51b61824 (stale) Mirror 21: 0x6050713c (stale) CRC-32 checksum value for chunk [0, 0x1d1a0): Mirror 1: 0x8720278b Mirror 22: 0x8720278b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d1a0: skipped 000530 1788941726.385805 fallocate 0x018fd6 thru 0x01e338 (0x5363 bytes) 000533 1788941726.480580 write 0x021191 thru 0x02bcce (0xab3e bytes) 000534 1788941726.517000 punch from 0x025c07 to 0x0260f6 (0x4f0 bytes) 000535 1788941726.564870 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bccf): Mirror 13: 0x9ada3634 (stale) Mirror 15: 0x9ada3634 (stale) Mirror 16: 0x9ada3634 (stale) Mirror 17: 0x9ada3634 (stale) Mirror 19: 0x9ada3634 (stale) Mirror 20: 0x9ada3634 (stale) Mirror 21: 0xc091659f (stale) Mirror 22: 0x14068f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bccf): Mirror 13: 0x9ada3634 (stale) Mirror 15: 0x9ada3634 (stale) Mirror 16: 0x9ada3634 (stale) Mirror 17: 0x9ada3634 (stale) Mirror 19: 0x9ada3634 (stale) Mirror 20: 0x9ada3634 (stale) Mirror 21: 0xc091659f (stale) Mirror 22: 0x14068f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bccf: skipped 000536 1788941727.349145 trunc from 0x02bccf to 0x02cca7 (0xfd9 bytes) 000537 1788941727.404906 punch from 0x002173 to 0x006a79 (0x4907 bytes) 000538 1788941727.414534 trunc from 0x000a78 to 0x009b0a (0x9093 bytes) 000539 1788941727.423390 trunc from 0x02cca8 to 0x03bfbf (0xf318 bytes) 000540 1788941727.462386 read 0x02627b thru 0x028cbb (0x2a41 bytes) 000541 1788941727.473838 trunc from 0x0236b8 to 0x02f3ec (0xbd35 bytes) 000542 1788941727.494368 trunc from 0x001169 to 0x00a4eb (0x9383 bytes) 000543 1788941727.524367 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3bfc0): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfc0): Mirror 13: 0xeef69c4b (stale) Mirror 15: 0xeef69c4b (stale) Mirror 16: 0xeef69c4b (stale) Mirror 17: 0xeef69c4b (stale) Mirror 19: 0xeef69c4b (stale) Mirror 20: 0xeef69c4b (stale) Mirror 21: 0x9e84806 (stale) Mirror 22: 0xf537cab (stale) CRC-32 checksum value for chunk [0, 0x3bfc0): Mirror 1: 0x8dc0bdac Mirror 23: 0x8dc0bdac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfc0: skipped 000544 1788941729.214057 mapwrite 0x01e61d thru 0x0228fd (0x42e1 bytes) 000545 1788941729.324322 mapread 0x02063e thru 0x029819 (0x91dc bytes) 000546 1788941729.338795 mapread 0x03a950 thru 0x03bfbf (0x1670 bytes) 000547 1788941729.360165 mapwrite 0x03d400 thru 0x03ffff (0x2c00 bytes) 000548 1788941729.404542 punch from 0x035cd9 to 0x03cf6d (0x7295 bytes) 000549 1788941729.408795 fallocate 0x0332db thru 0x03a76a (0x7490 bytes) 000551 1788941729.413011 punch from 0x027e40 to 0x02d216 (0x53d7 bytes) 000552 1788941729.417185 trunc from 0x040000 to 0x0050ba (0x3af45 bytes) 000554 1788941729.448068 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x50bb): Mirror 13: 0x443af6ec (stale) Mirror 15: 0x443af6ec (stale) Mirror 16: 0x443af6ec (stale) Mirror 17: 0x443af6ec (stale) Mirror 19: 0x443af6ec (stale) Mirror 21: 0xf117b843 (stale) Mirror 22: 0x1751ec49 (stale) Mirror 23: 0x757d7299 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x50bb): Mirror 13: 0x443af6ec (stale) Mirror 15: 0x443af6ec (stale) Mirror 16: 0x443af6ec (stale) Mirror 17: 0x443af6ec (stale) Mirror 19: 0x443af6ec (stale) Mirror 21: 0xf117b843 (stale) Mirror 22: 0x1751ec49 (stale) Mirror 23: 0x757d7299 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x50bb: skipped 000555 1788941730.055322 trunc from 0x0050bb to 0x01edfd (0x19d43 bytes) 000557 1788941730.078364 mapread 0x0145a6 thru 0x01bb1f (0x757a bytes) 000558 1788941730.113199 read 0x011875 thru 0x011df8 (0x584 bytes) 000559 1788941730.114167 fallocate 0x029542 thru 0x02f77b (0x623a bytes) 000560 1788941730.121049 trunc from 0x00c742 to 0x01c03e (0xf8fd bytes) 000561 1788941730.126733 read 0x00d68f thru 0x01cf7f (0xf8f1 bytes) 000562 1788941730.137201 mapread 0x01275c thru 0x01e83b (0xc0e0 bytes) 000563 1788941730.138409 trunc from 0x01edfe to 0x000f53 (0x1deaa bytes) 000564 1788941730.158509 read 0x00040e thru 0x000f53 (0xb46 bytes) 000565 1788941730.159169 trunc from 0x000f54 to 0x03d98a (0x3ca37 bytes) 000566 1788941730.193788 MIRROR_RESYNC 9 mirrors 000568 1788941731.458817 mapwrite 0x009376 thru 0x01260e (0x9299 bytes) 000570 1788941731.574586 punch from 0x03761d to 0x03d98a (0x636e bytes) 000571 1788941731.579807 mapread 0x02eaab thru 0x02f13a (0x690 bytes) 000572 1788941731.588890 fallocate 0x02e01b thru 0x038802 (0xa7e8 bytes) 000573 1788941731.593831 trunc from 0x03d98b to 0x035ea6 (0x7ae4 bytes) 000574 1788941731.614586 trunc from 0x035ea7 to 0x03cc46 (0x6da0 bytes) 000575 1788941731.635030 mapread 0x0175d6 thru 0x01b2d8 (0x3d03 bytes) 000576 1788941731.647935 fallocate 0x020628 thru 0x023e65 (0x383e bytes) 000577 1788941731.652848 trunc from 0x03cc47 to 0x03089d (0xc3a9 bytes) 000578 1788941731.693197 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3089e): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3089e): Mirror 13: 0x16579eee (stale) Mirror 15: 0x16579eee (stale) Mirror 16: 0x16579eee (stale) Mirror 17: 0x16579eee (stale) Mirror 19: 0x16579eee (stale) Mirror 21: 0x16579eee (stale) Mirror 22: 0x16579eee (stale) Mirror 23: 0x16579eee (stale) CRC-32 checksum value for chunk [0, 0x3089e): Mirror 1: 0xdd7d8895 Mirror 24: 0xdd7d8895 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3089e: skipped 000579 1788941733.293662 mapread 0x00643a thru 0x00a983 (0x454a bytes) 000580 1788941733.331985 write 0x03f806 thru 0x03ffff (0x7fa bytes) 000581 1788941733.408118 punch from 0x01f912 to 0x025625 (0x5d14 bytes) 000582 1788941733.424658 punch from 0x02be36 to 0x02f82c (0x39f7 bytes) 000583 1788941733.439073 fallocate 0x01258b thru 0x0210cc (0xeb42 bytes) 000584 1788941733.451927 mapwrite 0x023ca3 thru 0x024a74 (0xdd2 bytes) 000585 1788941733.515639 mapwrite 0x0175e8 thru 0x01fc29 (0x8642 bytes) 000586 1788941733.556917 mapread 0x02c646 thru 0x03aaeb (0xe4a6 bytes) 000587 1788941733.593124 read 0x019ef1 thru 0x027df3 (0xdf03 bytes) 000588 1788941733.595199 fallocate 0x02abff thru 0x02c6f5 (0x1af7 bytes) 000589 1788941733.603828 read 0x03b72a thru 0x03ffff (0x48d6 bytes) 000591 1788941733.644261 mapwrite 0x03e0dd thru 0x03ffff (0x1f23 bytes) 000592 1788941733.660137 fallocate 0x025ae4 thru 0x02bebd (0x63da bytes) 000593 1788941733.680499 fallocate 0x037ab9 thru 0x038a9e (0xfe6 bytes) 000594 1788941733.706446 MIRROR_RESYNC 10 mirrors 000595 1788941737.638753 MIRROR_EXTEND to 11 mirrors 000596 1788941738.386445 trunc from 0x040000 to 0x021943 (0x1e6bc bytes) 000597 1788941738.499490 mapread 0x00c694 thru 0x0184ab (0xbe18 bytes) 000598 1788941738.548922 mapread 0x003fd4 thru 0x00f7c6 (0xb7f3 bytes) 000599 1788941738.577699 fallocate 0x02c125 thru 0x03357f (0x745b bytes) 000600 1788941738.584199 fallocate 0x03e895 thru 0x03ffff (0x176b bytes) 000601 1788941738.600458 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc5f2d37 (stale) Mirror 15: 0xc5f2d37 (stale) Mirror 16: 0xc5f2d37 (stale) Mirror 17: 0xc5f2d37 (stale) Mirror 19: 0xc5f2d37 (stale) Mirror 22: 0xc5f2d37 (stale) Mirror 23: 0xc5f2d37 (stale) Mirror 24: 0xc5f2d37 (stale) Mirror 25: 0xc5f2d37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc5f2d37 (stale) Mirror 15: 0xc5f2d37 (stale) Mirror 16: 0xc5f2d37 (stale) Mirror 17: 0xc5f2d37 (stale) Mirror 19: 0xc5f2d37 (stale) Mirror 22: 0xc5f2d37 (stale) Mirror 23: 0xc5f2d37 (stale) Mirror 24: 0xc5f2d37 (stale) Mirror 25: 0xc5f2d37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000602 1788941739.223223 punch from 0x00ada9 to 0x011991 (0x6be9 bytes) 000603 1788941739.229787 mapread 0x00423d thru 0x00b579 (0x733d bytes) 000604 1788941739.256165 write 0x037521 thru 0x03ffff (0x8adf bytes) 000606 1788941739.269517 fallocate 0x00d6d6 thru 0x015766 (0x8091 bytes) 000608 1788941739.275530 read 0x0378f2 thru 0x0392cb (0x19da bytes) 000609 1788941739.280016 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xc5f2d37 (stale) Mirror 15: 0xc5f2d37 (stale) Mirror 16: 0xc5f2d37 (stale) Mirror 17: 0xc5f2d37 (stale) Mirror 19: 0xc5f2d37 (stale) Mirror 22: 0xc5f2d37 (stale) Mirror 23: 0xc5f2d37 (stale) Mirror 24: 0xc5f2d37 (stale) Mirror 25: 0xc5f2d37 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5f60f26 Mirror 26: 0x5f60f26 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000610 1788941740.636711 trunc from 0x040000 to 0x02647f (0x19b80 bytes) 000611 1788941740.746884 mapwrite 0x0096da thru 0x00cace (0x33f5 bytes) 000612 1788941740.782678 punch from 0x0117c2 to 0x014b9f (0x33de bytes) 000613 1788941740.794465 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26480): Mirror 15: 0x8991240d (stale) Mirror 16: 0x8991240d (stale) Mirror 17: 0x8991240d (stale) Mirror 19: 0x8991240d (stale) Mirror 22: 0x8991240d (stale) Mirror 23: 0x8991240d (stale) Mirror 24: 0x8991240d (stale) Mirror 25: 0x8991240d (stale) Mirror 26: 0x11149cb9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26480): Mirror 15: 0x8991240d (stale) Mirror 16: 0x8991240d (stale) Mirror 17: 0x8991240d (stale) Mirror 19: 0x8991240d (stale) Mirror 22: 0x8991240d (stale) Mirror 23: 0x8991240d (stale) Mirror 24: 0x8991240d (stale) Mirror 25: 0x8991240d (stale) Mirror 26: 0x11149cb9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26480: skipped 000614 1788941741.585844 trunc from 0x02554b to 0x02647f (0xf35 bytes) 000615 1788941741.593572 fallocate 0x03a86a thru 0x03ffff (0x5796 bytes) 000617 1788941741.606230 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc5f2d37 (stale) Mirror 16: 0xc5f2d37 (stale) Mirror 17: 0xc5f2d37 (stale) Mirror 19: 0xc5f2d37 (stale) Mirror 23: 0xc5f2d37 (stale) Mirror 24: 0xc5f2d37 (stale) Mirror 25: 0xc5f2d37 (stale) Mirror 26: 0x5f60f26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc5f2d37 (stale) Mirror 16: 0xc5f2d37 (stale) Mirror 17: 0xc5f2d37 (stale) Mirror 19: 0xc5f2d37 (stale) Mirror 23: 0xc5f2d37 (stale) Mirror 24: 0xc5f2d37 (stale) Mirror 25: 0xc5f2d37 (stale) Mirror 26: 0x5f60f26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000618 1788941742.361011 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc5f2d37 (stale) Mirror 16: 0xc5f2d37 (stale) Mirror 17: 0xc5f2d37 (stale) Mirror 19: 0xc5f2d37 (stale) Mirror 23: 0xc5f2d37 (stale) Mirror 25: 0xc5f2d37 (stale) Mirror 26: 0x5f60f26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc5f2d37 (stale) Mirror 16: 0xc5f2d37 (stale) Mirror 17: 0xc5f2d37 (stale) Mirror 19: 0xc5f2d37 (stale) Mirror 23: 0xc5f2d37 (stale) Mirror 25: 0xc5f2d37 (stale) Mirror 26: 0x5f60f26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000619 1788941743.075597 write 0x026997 thru 0x031ddc (0xb446 bytes) 000620 1788941743.095767 write 0x002e79 thru 0x003ba6 (0xd2e bytes) 000621 1788941743.104887 read 0x009957 thru 0x00a30e (0x9b8 bytes) 000622 1788941743.109827 trunc from 0x01b9f4 to 0x01dd2d (0x233a bytes) 000624 1788941743.120473 MIRROR_RESYNC 8 mirrors 000625 1788941747.142377 read 0x025ca6 thru 0x02f044 (0x939f bytes) 000626 1788941747.158888 MIRROR_RESYNC 8 mirrors 000628 1788941747.801777 trunc from 0x03a0b5 to 0x03b31f (0x126b bytes) 000632 1788941747.846839 trunc from 0x040000 to 0x003fcb (0x3c034 bytes) 000634 1788941747.877804 mapwrite 0x00b723 thru 0x00e14d (0x2a2b bytes) 000635 1788941747.979993 fallocate 0x032bff thru 0x03ffff (0xd401 bytes) 000636 1788941747.990130 trunc from 0x00167a to 0x00525e (0x3be5 bytes) 000637 1788941747.995620 fallocate 0x02dd9f thru 0x039cc1 (0xbf23 bytes) 000638 1788941747.999693 trunc from 0x040000 to 0x00e186 (0x31e79 bytes) 000639 1788941748.018698 trunc from 0x005aff to 0x00afa3 (0x54a5 bytes) 000640 1788941748.024140 punch from 0x000afe to 0x001e06 (0x1309 bytes) 000641 1788941748.029882 punch from 0x00ba33 to 0x00e186 (0x2754 bytes) 000642 1788941748.039230 write 0x01d288 thru 0x022198 (0x4f11 bytes) 000643 1788941748.041717 trunc from 0x022199 to 0x00fec9 (0x122cf bytes) 000644 1788941748.056140 write 0x024381 thru 0x029bec (0x586c bytes) 000646 1788941748.058005 write 0x006891 thru 0x00fa0b (0x917b bytes) 000647 1788941748.073987 fallocate 0x03cc69 thru 0x03ffff (0x3397 bytes) 000648 1788941748.078756 punch from 0x013cda to 0x01a4da (0x6801 bytes) 000649 1788941748.083155 mapwrite 0x029f41 thru 0x02e226 (0x42e6 bytes) 000650 1788941748.126912 trunc from 0x02e227 to 0x035fd4 (0x7dae bytes) 000651 1788941748.149826 mapwrite 0x0387d8 thru 0x03ffff (0x7828 bytes) 000652 1788941748.192899 punch from 0x03e70c to 0x03ffff (0x18f4 bytes) 000654 1788941748.199703 trunc from 0x040000 to 0x03e356 (0x1ca9 bytes) 000655 1788941748.216583 write 0x0256c8 thru 0x02a7d9 (0x5112 bytes) 000656 1788941748.218039 trunc from 0x00a9fd to 0x019b7a (0xf17e bytes) 000657 1788941748.223792 trunc from 0x03e357 to 0x015657 (0x28cff bytes) 000658 1788941748.243910 mapwrite 0x029e56 thru 0x039d2f (0xfeda bytes) 000659 1788941748.305010 mapwrite 0x00cabf thru 0x01bfaa (0xf4ec bytes) 000660 1788941748.350983 trunc from 0x039d30 to 0x02e6f2 (0xb63d bytes) 000662 1788941748.371770 punch from 0x0039cc to 0x01051d (0xcb52 bytes) 000664 1788941748.379892 trunc from 0x01494d to 0x018a2f (0x40e3 bytes) 000665 1788941748.390031 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e6f3): Mirror 15: 0x8706e7f3 (stale) Mirror 17: 0x8706e7f3 (stale) Mirror 19: 0x8706e7f3 (stale) Mirror 23: 0x8706e7f3 (stale) Mirror 25: 0x8706e7f3 (stale) Mirror 26: 0x8706e7f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6f3): Mirror 15: 0x8706e7f3 (stale) Mirror 17: 0x8706e7f3 (stale) Mirror 19: 0x8706e7f3 (stale) Mirror 23: 0x8706e7f3 (stale) Mirror 25: 0x8706e7f3 (stale) Mirror 26: 0x8706e7f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6f3: skipped 000666 1788941748.930023 trunc from 0x02e6f3 to 0x026a90 (0x7c62 bytes) 000667 1788941748.956004 punch from 0x010bc8 to 0x0141a5 (0x35de bytes) 000668 1788941748.960812 fallocate 0x0362c0 thru 0x03a748 (0x4489 bytes) 000669 1788941748.967544 read 0x021714 thru 0x026a90 (0x537d bytes) 000670 1788941748.978750 trunc from 0x026a91 to 0x018c7c (0xde14 bytes) 000671 1788941748.994616 fallocate 0x038980 thru 0x03ae4c (0x24cd bytes) 000672 1788941748.998574 write 0x015c80 thru 0x01e6cf (0x8a50 bytes) 000673 1788941749.022147 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ae4d): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae4d): Mirror 15: 0x9ecc28d7 (stale) Mirror 17: 0x9ecc28d7 (stale) Mirror 19: 0x9ecc28d7 (stale) Mirror 23: 0x9ecc28d7 (stale) Mirror 25: 0x9ecc28d7 (stale) Mirror 26: 0x9ecc28d7 (stale) CRC-32 checksum value for chunk [0, 0x3ae4d): Mirror 1: 0x6119d204 Mirror 27: 0x6119d204 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae4d: skipped 000674 1788941750.275808 trunc from 0x03ae4d to 0x028ca7 (0x121a5 bytes) 000675 1788941750.335730 fallocate 0x0335ee thru 0x035667 (0x207a bytes) 000676 1788941750.341810 write 0x00fde8 thru 0x01574e (0x5967 bytes) 000677 1788941750.352827 mapwrite 0x03d817 thru 0x03ffff (0x27e9 bytes) 000678 1788941750.410672 fallocate 0x03445c thru 0x03db1c (0x96c1 bytes) 000679 1788941750.415726 trunc from 0x040000 to 0x03ea31 (0x15ce bytes) 000680 1788941750.435893 punch from 0x028fcc to 0x02b60a (0x263f bytes) 000681 1788941750.440805 trunc from 0x0224d1 to 0x0292c1 (0x6df1 bytes) 000682 1788941750.445774 mapwrite 0x03f23a thru 0x03ffff (0xdc6 bytes) 000684 1788941750.488780 MIRROR_RESYNC 8 mirrors 000685 1788941751.891773 punch from 0x006d71 to 0x016602 (0xf892 bytes) 000686 1788941751.938779 MIRROR_RESYNC 8 mirrors 000687 1788941753.304688 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000688 1788941753.658853 write 0x0137ac thru 0x0187cb (0x5020 bytes) 000689 1788941753.709526 mapread 0x039ac6 thru 0x03ffff (0x653a bytes) 000690 1788941753.730459 fallocate 0x004125 thru 0x00a696 (0x6572 bytes) 000691 1788941753.735690 trunc from 0x01ca21 to 0x028774 (0xbd54 bytes) 000693 1788941753.741914 punch from 0x0327af to 0x03aa00 (0x8252 bytes) 000694 1788941753.746910 trunc from 0x0112a4 to 0x01bea1 (0xabfe bytes) 000695 1788941753.761024 write 0x0202b8 thru 0x0255ca (0x5313 bytes) 000696 1788941753.771391 mapwrite 0x01a256 thru 0x0288ba (0xe665 bytes) 000697 1788941753.817473 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xe1549773 (stale) Mirror 17: 0xe1549773 (stale) Mirror 23: 0xe1549773 (stale) Mirror 25: 0xe1549773 (stale) Mirror 26: 0xe1549773 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xe1549773 (stale) Mirror 17: 0xe1549773 (stale) Mirror 23: 0xe1549773 (stale) Mirror 25: 0xe1549773 (stale) Mirror 26: 0xe1549773 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000698 1788941754.206164 trunc from 0x02b3b1 to 0x02f5a1 (0x41f1 bytes) 000699 1788941754.209968 punch from 0x03c00c to 0x03d919 (0x190e bytes) 000700 1788941754.215709 write 0x03d790 thru 0x03ffff (0x2870 bytes) 000701 1788941754.226643 punch from 0x03a787 to 0x03ffff (0x5879 bytes) 000702 1788941754.236549 trunc from 0x040000 to 0x01dbd3 (0x2242c bytes) 000703 1788941754.255159 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1dbd4): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbd4): Mirror 15: 0x73bd9452 (stale) Mirror 17: 0x73bd9452 (stale) Mirror 23: 0x73bd9452 (stale) Mirror 25: 0x73bd9452 (stale) Mirror 26: 0x73bd9452 (stale) CRC-32 checksum value for chunk [0, 0x1dbd4): Mirror 1: 0xaad7c194 Mirror 27: 0xaad7c194 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbd4: skipped 000704 1788941755.090605 trunc from 0x01dbd4 to 0x00eeb5 (0xed1e bytes) 000705 1788941755.160016 trunc from 0x002d61 to 0x0096fe (0x699e bytes) 000707 1788941755.167491 punch from 0x0089f5 to 0x00eeb5 (0x64c1 bytes) 000708 1788941755.174360 fallocate 0x03c90a thru 0x03ffff (0x36f6 bytes) 000709 1788941755.183636 trunc from 0x00eeb6 to 0x02227e (0x133c9 bytes) 000710 1788941755.225418 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2227f): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2227f): Mirror 15: 0xc00b2cc (stale) Mirror 17: 0xc00b2cc (stale) Mirror 23: 0xc00b2cc (stale) Mirror 25: 0xc00b2cc (stale) Mirror 26: 0xc00b2cc (stale) Mirror 27: 0xebaf8cab (stale) CRC-32 checksum value for chunk [0, 0x2227f): Mirror 1: 0x35d83284 Mirror 28: 0x35d83284 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2227f: skipped 000711 1788941756.520908 read 0x021451 thru 0x02227e (0xe2e bytes) 000712 1788941756.540049 punch from 0x00e4b4 to 0x01ba7f (0xd5cc bytes) 000713 1788941756.617471 punch from 0x001810 to 0x00e6f8 (0xcee9 bytes) 000714 1788941756.623930 write 0x02e1ab thru 0x0393bd (0xb213 bytes) 000715 1788941756.649929 punch from 0x022ae8 to 0x02fa3d (0xcf56 bytes) 000716 1788941756.677019 trunc from 0x0393be to 0x01ad87 (0x1e636 bytes) 000718 1788941756.747649 write 0x033a58 thru 0x03404c (0x5f5 bytes) 000720 1788941756.755446 read 0x0182d6 thru 0x02500b (0xcd36 bytes) 000721 1788941756.779127 punch from 0x024038 to 0x030a61 (0xca2a bytes) 000722 1788941756.786454 punch from 0x00b147 to 0x01238d (0x7247 bytes) 000723 1788941756.793370 read 0x0085cd thru 0x01227f (0x9cb3 bytes) 000724 1788941756.810397 mapread 0x01e468 thru 0x028a50 (0xa5e9 bytes) 000725 1788941756.811363 trunc from 0x031e93 to 0x03404c (0x21ba bytes) 000726 1788941756.833561 mapwrite 0x00b67d thru 0x01abc8 (0xf54c bytes) 000727 1788941756.865006 mapwrite 0x03e4fa thru 0x03ffff (0x1b06 bytes) 000728 1788941756.941692 mapread 0x0104b9 thru 0x012b03 (0x264b bytes) 000729 1788941756.946078 read 0x03adc7 thru 0x03ffff (0x5239 bytes) 000730 1788941756.948692 write 0x00b041 thru 0x01252a (0x74ea bytes) 000731 1788941756.951310 trunc from 0x0167ab to 0x019d2e (0x3584 bytes) 000732 1788941756.962114 write 0x00faad thru 0x01c91c (0xce70 bytes) 000733 1788941756.966236 trunc from 0x040000 to 0x0091e4 (0x36e1b bytes) 000734 1788941756.986841 mapwrite 0x019255 thru 0x01ff56 (0x6d02 bytes) 000735 1788941757.065129 mapread 0x0104de thru 0x01ca5a (0xc57d bytes) 000736 1788941757.067554 punch from 0x01519c to 0x01ff56 (0xadbb bytes) 000737 1788941757.073345 read 0x01d763 thru 0x01ff56 (0x27f4 bytes) 000738 1788941757.107439 trunc from 0x002e83 to 0x00bfa3 (0x9121 bytes) 000739 1788941757.113331 read 0x01cfdf thru 0x01ff56 (0x2f78 bytes) 000741 1788941757.115036 write 0x003e3e thru 0x0083c9 (0x458c bytes) 000742 1788941757.125726 mapread 0x01f968 thru 0x01ff56 (0x5ef bytes) 000744 1788941757.127023 read 0x009801 thru 0x00b470 (0x1c70 bytes) 000745 1788941757.127583 punch from 0x015dc7 to 0x01fa50 (0x9c8a bytes) 000746 1788941757.135653 punch from 0x008bb4 to 0x015055 (0xc4a2 bytes) 000747 1788941757.154243 mapwrite 0x01a068 thru 0x01d75e (0x36f7 bytes) 000749 1788941757.200563 write 0x03bc42 thru 0x03f6fc (0x3abb bytes) 000750 1788941757.202664 fallocate 0x0118b0 thru 0x015269 (0x39ba bytes) 000751 1788941757.211931 fallocate 0x03e7af thru 0x03ff16 (0x1768 bytes) 000752 1788941757.225446 MIRROR_RESYNC 8 mirrors 000753 1788941760.122595 read 0x026b53 thru 0x02a1b6 (0x3664 bytes) 000754 1788941760.130700 trunc from 0x03ff17 to 0x0224c0 (0x1da56 bytes) 000755 1788941760.184300 trunc from 0x0224c1 to 0x01702a (0xb496 bytes) 000756 1788941760.202764 fallocate 0x0382c2 thru 0x03ffff (0x7d3e bytes) 000757 1788941760.207843 trunc from 0x040000 to 0x02b8fb (0x14704 bytes) 000758 1788941760.228433 punch from 0x01195e to 0x01c204 (0xa8a7 bytes) 000759 1788941760.233649 mapwrite 0x03c046 thru 0x03cc18 (0xbd3 bytes) 000761 1788941760.272929 read 0x02ce5c thru 0x02d00b (0x1b0 bytes) 000762 1788941760.283302 punch from 0x02b0a8 to 0x02e433 (0x338c bytes) 000763 1788941760.293741 MIRROR_RESYNC 8 mirrors 000764 1788941761.929913 read 0x00d2ba thru 0x00f0b6 (0x1dfd bytes) 000765 1788941761.935442 trunc from 0x03b5c1 to 0x03cc18 (0x1658 bytes) 000766 1788941761.974549 punch from 0x011d97 to 0x021c08 (0xfe72 bytes) 000767 1788941761.978790 write 0x00e997 thru 0x012acb (0x4135 bytes) 000768 1788941761.990184 mapread 0x018589 thru 0x01e265 (0x5cdd bytes) 000769 1788941762.008656 read 0x0057b9 thru 0x011be4 (0xc42c bytes) 000770 1788941762.023090 MIRROR_RESYNC 8 mirrors 000771 1788941764.354043 MIRROR_EXTEND to 9 mirrors 000772 1788941765.170781 trunc from 0x03cc19 to 0x000030 (0x3cbe8 bytes) 000773 1788941765.237153 read 0x00000b thru 0x000030 (0x26 bytes) 000775 1788941765.244747 mapread 0x000019 thru 0x000030 (0x18 bytes) 000777 1788941765.246443 mapwrite 0x013ac3 thru 0x01f11b (0xb659 bytes) 000778 1788941765.321485 write 0x01bdc9 thru 0x021116 (0x534e bytes) 000779 1788941765.322735 trunc from 0x00a263 to 0x0127e3 (0x8581 bytes) 000780 1788941765.329399 punch from 0x01ec59 to 0x020951 (0x1cf9 bytes) 000781 1788941765.347252 mapwrite 0x028261 thru 0x031f03 (0x9ca3 bytes) 000782 1788941765.464688 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31f04): Mirror 15: 0x3fdd4399 (stale) Mirror 17: 0x3fdd4399 (stale) Mirror 23: 0x3fdd4399 (stale) Mirror 25: 0x3fdd4399 (stale) Mirror 26: 0x3fdd4399 (stale) Mirror 27: 0x3fdd4399 (stale) Mirror 29: 0x3fdd4399 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f04): Mirror 15: 0x3fdd4399 (stale) Mirror 17: 0x3fdd4399 (stale) Mirror 23: 0x3fdd4399 (stale) Mirror 25: 0x3fdd4399 (stale) Mirror 26: 0x3fdd4399 (stale) Mirror 27: 0x3fdd4399 (stale) Mirror 29: 0x3fdd4399 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f04: skipped 000783 1788941766.062476 read 0x02fdb5 thru 0x031f03 (0x214f bytes) 000784 1788941766.069102 write 0x006280 thru 0x00d083 (0x6e04 bytes) 000785 1788941766.084418 trunc from 0x031f04 to 0x035087 (0x3184 bytes) 000786 1788941766.107313 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35088): Mirror 15: 0x33f7340f (stale) Mirror 17: 0x33f7340f (stale) Mirror 23: 0x33f7340f (stale) Mirror 26: 0x33f7340f (stale) Mirror 27: 0x33f7340f (stale) Mirror 29: 0x33f7340f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35088): Mirror 15: 0x33f7340f (stale) Mirror 17: 0x33f7340f (stale) Mirror 23: 0x33f7340f (stale) Mirror 26: 0x33f7340f (stale) Mirror 27: 0x33f7340f (stale) Mirror 29: 0x33f7340f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35088: skipped 000787 1788941766.557970 trunc from 0x035088 to 0x03806a (0x2fe3 bytes) 000788 1788941766.581070 trunc from 0x03806b to 0x003d48 (0x34322 bytes) 000789 1788941766.598708 mapread 0x003149 thru 0x003d48 (0xc00 bytes) 000790 1788941766.605021 mapread 0x000548 thru 0x0039b3 (0x346c bytes) 000791 1788941766.615439 write 0x007a05 thru 0x00e478 (0x6a74 bytes) 000792 1788941766.616723 trunc from 0x0067ca to 0x008627 (0x1e5e bytes) 000793 1788941766.630991 write 0x03a27c thru 0x03ffff (0x5d84 bytes) 000794 1788941766.632226 punch from 0x01a1eb to 0x027f88 (0xdd9e bytes) 000795 1788941766.637074 trunc from 0x004064 to 0x011764 (0xd701 bytes) 000796 1788941766.649525 MIRROR_RESYNC 7 mirrors 000797 1788941767.659229 mapread 0x012236 thru 0x01bf89 (0x9d54 bytes) 000798 1788941767.681384 punch from 0x03aa09 to 0x03ffff (0x55f7 bytes) 000799 1788941767.726400 MIRROR_RESYNC 7 mirrors 000800 1788941768.882823 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000801 1788941769.255995 mapread 0x00ecec thru 0x00f386 (0x69b bytes) 000803 1788941769.271655 fallocate 0x02ab92 thru 0x0360b0 (0xb51f bytes) 000804 1788941769.321081 write 0x00aac9 thru 0x014d14 (0xa24c bytes) 000805 1788941769.336820 mapwrite 0x007f38 thru 0x00b71f (0x37e8 bytes) 000806 1788941769.360695 fallocate 0x01c30f thru 0x01cf74 (0xc66 bytes) 000807 1788941769.366315 fallocate 0x03d82a thru 0x03ffff (0x27d6 bytes) 000808 1788941769.371438 trunc from 0x026274 to 0x02ffa1 (0x9d2e bytes) 000809 1788941769.376944 trunc from 0x040000 to 0x0295c8 (0x16a37 bytes) 000810 1788941769.398993 write 0x037d60 thru 0x03ffff (0x82a0 bytes) 000812 1788941769.400914 trunc from 0x01c335 to 0x0260b2 (0x9d7e bytes) 000813 1788941769.410696 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x5431b931 (stale) Mirror 23: 0x5431b931 (stale) Mirror 26: 0x5431b931 (stale) Mirror 27: 0x5431b931 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x5431b931 (stale) Mirror 23: 0x5431b931 (stale) Mirror 26: 0x5431b931 (stale) Mirror 27: 0x5431b931 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000814 1788941769.808009 mapwrite 0x022fc2 thru 0x02641e (0x345d bytes) 000815 1788941769.842389 fallocate 0x039210 thru 0x03ffff (0x6df0 bytes) 000816 1788941769.850042 punch from 0x00d74a to 0x01d154 (0xfa0b bytes) 000817 1788941769.856422 trunc from 0x040000 to 0x0071e2 (0x38e1d bytes) 000819 1788941769.886106 trunc from 0x0071e3 to 0x00c8dd (0x56fb bytes) 000821 1788941769.910757 trunc from 0x004215 to 0x008427 (0x4213 bytes) 000822 1788941769.916494 punch from 0x004407 to 0x00c8dd (0x84d7 bytes) 000824 1788941769.923419 trunc from 0x00c8de to 0x01c6eb (0xfe0e bytes) 000825 1788941769.943689 punch from 0x019975 to 0x01c6eb (0x2d77 bytes) 000826 1788941769.949682 write 0x0021c2 thru 0x00dd8e (0xbbcd bytes) 000827 1788941769.960658 read 0x018619 thru 0x01bafc (0x34e4 bytes) 000828 1788941769.974068 MIRROR_RESYNC 5 mirrors 000829 1788941771.155877 mapwrite 0x0244a4 thru 0x028482 (0x3fdf bytes) 000830 1788941771.299251 punch from 0x024b28 to 0x028482 (0x395b bytes) 000832 1788941771.316016 fallocate 0x00b6ae thru 0x00cddd (0x1730 bytes) 000833 1788941771.319820 trunc from 0x028483 to 0x032cdb (0xa859 bytes) 000834 1788941771.355085 mapwrite 0x023e04 thru 0x0248f4 (0xaf1 bytes) 000835 1788941771.406991 trunc from 0x032cdc to 0x003296 (0x2fa45 bytes) 000836 1788941771.454840 mapwrite 0x0259d8 thru 0x02e34f (0x8978 bytes) 000837 1788941771.519609 read 0x011404 thru 0x01469c (0x3299 bytes) 000839 1788941771.529242 write 0x03d7b3 thru 0x03ffff (0x284d bytes) 000840 1788941771.532519 fallocate 0x034ef4 thru 0x03ffcf (0xb0dc bytes) 000841 1788941771.547344 trunc from 0x010062 to 0x011ce0 (0x1c7f bytes) 000842 1788941771.554460 fallocate 0x02f872 thru 0x0386c2 (0x8e51 bytes) 000843 1788941771.563459 MIRROR_RESYNC 5 mirrors 000845 1788941773.088488 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) 000846 1788941773.533775 trunc from 0x002a60 to 0x005f5b (0x34fc bytes) 000847 1788941773.583496 mapread 0x001015 thru 0x00f040 (0xe02c bytes) 000848 1788941773.619594 trunc from 0x040000 to 0x00dace (0x32531 bytes) 000849 1788941773.649960 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xdacf): Mirror 15: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdacf): Mirror 15: 0xc058d091 (stale) Mirror 23: 0xc058d091 (stale) Mirror 26: 0xc058d091 (stale) CRC-32 checksum value for chunk [0, 0xdacf): Mirror 1: 0xfbd808ed Mirror 27: 0xfbd808ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdacf: skipped 000850 1788941774.903793 trunc from 0x00a535 to 0x00dace (0x359a bytes) 000851 1788941774.956698 fallocate 0x0116d8 thru 0x0165c0 (0x4ee9 bytes) 000852 1788941774.961667 fallocate 0x00e1c5 thru 0x0148c3 (0x66ff bytes) 000854 1788941774.967414 trunc from 0x0165c1 to 0x019563 (0x2fa3 bytes) 000855 1788941774.989707 mapwrite 0x01c309 thru 0x0289bf (0xc6b7 bytes) 000856 1788941775.082195 trunc from 0x003074 to 0x00b0b8 (0x8045 bytes) 000857 1788941775.093205 trunc from 0x0289c0 to 0x02b05f (0x26a0 bytes) 000858 1788941775.139153 punch from 0x002320 to 0x00efc1 (0xcca2 bytes) 000859 1788941775.145139 punch from 0x020733 to 0x0281ef (0x7abd bytes) 000860 1788941775.153663 punch from 0x001f29 to 0x002079 (0x151 bytes) 000862 1788941775.163893 fallocate 0x024bdc thru 0x03148f (0xc8b4 bytes) 000863 1788941775.172766 read 0x00cd6d thru 0x01bf5c (0xf1f0 bytes) 000864 1788941775.181874 trunc from 0x02b060 to 0x01fc15 (0xb44a bytes) 000865 1788941775.211787 fallocate 0x0056d4 thru 0x00bb0f (0x643c bytes) 000866 1788941775.221773 read 0x01a709 thru 0x01fc15 (0x550d bytes) 000867 1788941775.222683 mapread 0x00c083 thru 0x01bc47 (0xfbc5 bytes) 000868 1788941775.224448 mapwrite 0x00991c thru 0x00a242 (0x927 bytes) 000869 1788941775.262069 trunc from 0x0058b0 to 0x00f23b (0x998c bytes) 000871 1788941775.277250 fallocate 0x024b7a thru 0x03202f (0xd4b6 bytes) 000872 1788941775.287599 read 0x0100fa thru 0x019da8 (0x9caf bytes) 000873 1788941775.288315 mapwrite 0x02d6b3 thru 0x038a7a (0xb3c8 bytes) 000875 1788941775.338755 punch from 0x003bf9 to 0x011251 (0xd659 bytes) 000876 1788941775.345624 trunc from 0x038a7b to 0x00b75e (0x2d31c bytes) 000877 1788941775.366842 fallocate 0x0023c7 thru 0x010a2c (0xe666 bytes) 000878 1788941775.373562 mapread 0x00b515 thru 0x00b75e (0x24a bytes) 000879 1788941775.381610 read 0x0087f4 thru 0x00b75e (0x2f6b bytes) 000880 1788941775.393390 trunc from 0x005c9e to 0x00b75e (0x5ac1 bytes) 000882 1788941775.398835 mapwrite 0x02ac1d thru 0x03643e (0xb822 bytes) 000883 1788941775.450720 punch from 0x014605 to 0x01ae6a (0x6866 bytes) 000884 1788941775.457431 mapwrite 0x02ea05 thru 0x033bca (0x51c6 bytes) 000885 1788941775.474601 read 0x0280d7 thru 0x028b8c (0xab6 bytes) 000886 1788941775.475244 trunc from 0x02b37f to 0x03623f (0xaec1 bytes) 000887 1788941775.480982 punch from 0x00b90c to 0x010655 (0x4d4a bytes) 000888 1788941775.485818 read 0x009e06 thru 0x01700a (0xd205 bytes) 000889 1788941775.499423 trunc from 0x02de19 to 0x03643e (0x8626 bytes) 000890 1788941775.504596 punch from 0x02bb95 to 0x031f59 (0x63c5 bytes) 000891 1788941775.511633 mapwrite 0x036b11 thru 0x03ffff (0x94ef bytes) 000892 1788941775.567040 trunc from 0x0074be to 0x007df0 (0x933 bytes) 000893 1788941775.574019 trunc from 0x036bf3 to 0x03cf58 (0x6366 bytes) 000894 1788941775.578656 fallocate 0x03bdfa thru 0x03ffff (0x4206 bytes) 000895 1788941775.583564 mapread 0x02d9f5 thru 0x03bfd7 (0xe5e3 bytes) 000896 1788941775.655498 read 0x03c569 thru 0x03ffff (0x3a97 bytes) 000897 1788941775.666633 punch from 0x034eb9 to 0x035cfe (0xe46 bytes) 000898 1788941775.672629 mapwrite 0x037454 thru 0x037ea6 (0xa53 bytes) 000899 1788941775.692468 punch from 0x00bfdb to 0x018833 (0xc859 bytes) 000900 1788941775.706904 mapread 0x035ffa thru 0x038ecd (0x2ed4 bytes) 000901 1788941775.712346 mapwrite 0x02ad84 thru 0x03ac95 (0xff12 bytes) 000902 1788941775.745384 write 0x00bc56 thru 0x00e650 (0x29fb bytes) 000903 1788941775.762910 punch from 0x02617d to 0x035c4a (0xface bytes) 000904 1788941775.768753 mapwrite 0x020c72 thru 0x02983e (0x8bcd bytes) 000905 1788941775.806621 read 0x0280b5 thru 0x037731 (0xf67d bytes) 000906 1788941775.817491 read 0x01b555 thru 0x01d83b (0x22e7 bytes) 000907 1788941775.817985 mapwrite 0x00dfad thru 0x012421 (0x4475 bytes) 000908 1788941775.838595 mapread 0x02c48c thru 0x034df6 (0x896b bytes) 000909 1788941775.840624 write 0x019080 thru 0x01a8bb (0x183c bytes) 000910 1788941775.844435 fallocate 0x00ca7b thru 0x0105be (0x3b44 bytes) 000911 1788941775.872709 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xb18644ec (stale) Mirror 23: 0xb18644ec (stale) Mirror 26: 0xb18644ec (stale) Mirror 27: 0xcc91cadb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6e7799a4 Mirror 28: 0x6e7799a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000912 1788941777.283634 mapread 0x005c1d thru 0x010025 (0xa409 bytes) 000913 1788941777.315340 fallocate 0x02b6a9 thru 0x032d76 (0x76ce bytes) 000914 1788941777.358560 read 0x02c7f6 thru 0x033678 (0x6e83 bytes) 000915 1788941777.368504 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xb18644ec (stale) Mirror 23: 0xb18644ec (stale) Mirror 26: 0xb18644ec (stale) Mirror 27: 0xcc91cadb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xb18644ec (stale) Mirror 23: 0xb18644ec (stale) Mirror 26: 0xb18644ec (stale) Mirror 27: 0xcc91cadb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000916 1788941777.683615 read 0x0208c8 thru 0x02436b (0x3aa4 bytes) 000917 1788941777.698030 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xb18644ec (stale) Mirror 26: 0xb18644ec (stale) Mirror 27: 0xcc91cadb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xb18644ec (stale) Mirror 26: 0xb18644ec (stale) Mirror 27: 0xcc91cadb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000918 1788941778.045074 mapread 0x024ef0 thru 0x025a51 (0xb62 bytes) 000919 1788941778.058685 write 0x0202e7 thru 0x02f59a (0xf2b4 bytes) 000920 1788941778.075515 write 0x016763 thru 0x01e0b6 (0x7954 bytes) 000921 1788941778.086572 mapread 0x00f70c thru 0x015a77 (0x636c bytes) 000922 1788941778.102553 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xb18644ec (stale) Mirror 26: 0xb18644ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xb18644ec (stale) Mirror 26: 0xb18644ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000923 1788941778.390129 trunc from 0x00a405 to 0x013831 (0x942d bytes) 000924 1788941778.395747 punch from 0x03da0c to 0x03ed25 (0x131a bytes) 000926 1788941778.401884 fallocate 0x01e10f thru 0x02486c (0x675e bytes) 000927 1788941778.406957 fallocate 0x0199e8 thru 0x028896 (0xeeaf bytes) 000928 1788941778.415894 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0xb18644ec (stale) Mirror 26: 0xb18644ec (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcbf256cd Mirror 27: 0xcbf256cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000929 1788941779.405234 trunc from 0x040000 to 0x0372ae (0x8d51 bytes) 000930 1788941779.480520 punch from 0x00c680 to 0x015984 (0x9305 bytes) 000931 1788941779.487982 trunc from 0x0193d5 to 0x022ed4 (0x9b00 bytes) 000932 1788941779.494913 trunc from 0x0372af to 0x00a869 (0x2ca45 bytes) 000933 1788941779.521999 fallocate 0x00dc8f thru 0x01ba24 (0xdd96 bytes) 000934 1788941779.529200 mapread 0x003e43 thru 0x00a869 (0x6a27 bytes) 000936 1788941779.556166 mapread 0x0093dc thru 0x00a869 (0x148e bytes) 000937 1788941779.557587 mapwrite 0x025de9 thru 0x034c15 (0xee2d bytes) 000938 1788941779.642744 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34c16): Mirror 26: 0x3eff0440 (stale) Mirror 27: 0xfab4197b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c16): Mirror 26: 0x3eff0440 (stale) Mirror 27: 0xfab4197b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c16: skipped 000939 1788941779.973745 trunc from 0x032938 to 0x034c15 (0x22de bytes) 000940 1788941779.979513 read 0x003507 thru 0x012b8d (0xf687 bytes) 000941 1788941779.992192 write 0x01ffb6 thru 0x024cfa (0x4d45 bytes) 000943 1788941780.006702 punch from 0x01565d to 0x0196b2 (0x4056 bytes) 000944 1788941780.010997 mapwrite 0x02b9bb thru 0x037920 (0xbf66 bytes) 000945 1788941780.069867 trunc from 0x01c8b5 to 0x01dcb6 (0x1402 bytes) 000946 1788941780.074948 mapread 0x003212 thru 0x00730c (0x40fb bytes) 000947 1788941780.078949 punch from 0x02b68d to 0x02b988 (0x2fc bytes) 000948 1788941780.089968 MIRROR_RESYNC 3 mirrors 000949 1788941780.811344 trunc from 0x037921 to 0x0320fa (0x5826 bytes) 000951 1788941780.898292 write 0x00bdb1 thru 0x01a89b (0xeaeb bytes) 000952 1788941780.910933 punch from 0x02040f to 0x02eb76 (0xe768 bytes) 000953 1788941780.921114 mapread 0x00696b thru 0x01365c (0xccf2 bytes) 000954 1788941780.945571 trunc from 0x0086a4 to 0x0114a2 (0x8dff bytes) 000955 1788941780.959307 trunc from 0x02bed4 to 0x0320fa (0x6227 bytes) 000956 1788941780.967431 fallocate 0x015367 thru 0x022d02 (0xd99c bytes) 000957 1788941780.974779 read 0x0290d7 thru 0x02d7e2 (0x470c bytes) 000958 1788941780.992434 mapwrite 0x026875 thru 0x031eb1 (0xb63d bytes) 000959 1788941781.056928 fallocate 0x000963 thru 0x00ea64 (0xe102 bytes) 000961 1788941781.082487 MIRROR_RESYNC 3 mirrors 000962 1788941782.285820 MIRROR_EXTEND to 4 mirrors 000963 1788941782.966036 trunc from 0x019b03 to 0x01aba6 (0x10a4 bytes) 000964 1788941782.995065 trunc from 0x0320fb to 0x0248ca (0xd830 bytes) 000965 1788941783.022265 read 0x011e3b thru 0x01e045 (0xc20b bytes) 000966 1788941783.032122 read 0x024512 thru 0x0248ca (0x3b9 bytes) 000967 1788941783.044352 read 0x011d7f thru 0x01278f (0xa11 bytes) 000968 1788941783.045570 mapwrite 0x011728 thru 0x016ae7 (0x53c0 bytes) 000969 1788941783.062216 fallocate 0x02f842 thru 0x03643d (0x6bfc bytes) 000970 1788941783.067544 punch from 0x03302e to 0x03643d (0x3410 bytes) 000971 1788941783.072333 mapwrite 0x025784 thru 0x0342bf (0xeb3c bytes) 000972 1788941783.107494 mapwrite 0x004bde thru 0x012442 (0xd865 bytes) 000973 1788941783.125924 trunc from 0x0195a6 to 0x021c5a (0x86b5 bytes) 000974 1788941783.135584 MIRROR_RESYNC 4 mirrors 000975 1788941784.686668 mapread 0x00ad86 thru 0x017ceb (0xcf66 bytes) 000976 1788941784.736977 write 0x03f7f7 thru 0x03f884 (0x8e bytes) 000977 1788941784.767098 write 0x018a01 thru 0x01bceb (0x32eb bytes) 000978 1788941784.775880 trunc from 0x0235c3 to 0x02e532 (0xaf70 bytes) 000979 1788941784.781500 mapwrite 0x029c1a thru 0x039899 (0xfc80 bytes) 000980 1788941784.827934 write 0x01ce84 thru 0x01ffe7 (0x3164 bytes) 000981 1788941784.839578 mapread 0x001e23 thru 0x002357 (0x535 bytes) 000982 1788941784.851532 read 0x00af20 thru 0x0198eb (0xe9cc bytes) 000983 1788941784.860966 write 0x01c222 thru 0x02407a (0x7e59 bytes) 000984 1788941784.862604 mapwrite 0x03dd69 thru 0x03ffff (0x2297 bytes) 000985 1788941784.906169 mapwrite 0x02e753 thru 0x034d8c (0x663a bytes) 000986 1788941784.924489 fallocate 0x01d3f7 thru 0x0218a6 (0x44b0 bytes) 000987 1788941784.929758 trunc from 0x040000 to 0x0255c5 (0x1aa3a bytes) 000989 1788941784.959132 punch from 0x000126 to 0x00f4f2 (0xf3cd bytes) 000990 1788941784.965614 trunc from 0x012e65 to 0x018730 (0x58cc bytes) 000992 1788941784.972163 mapwrite 0x01512f thru 0x023c66 (0xeb38 bytes) 000993 1788941785.009062 trunc from 0x0255c6 to 0x03e846 (0x19281 bytes) 000994 1788941785.029248 mapwrite 0x012e95 thru 0x01ea37 (0xbba3 bytes) 000995 1788941785.052556 trunc from 0x03e847 to 0x022b06 (0x1bd40 bytes) 000996 1788941785.070843 mapread 0x012835 thru 0x013b2d (0x12f9 bytes) 000997 1788941785.071713 trunc from 0x014cd7 to 0x01a2ca (0x55f4 bytes) 000998 1788941785.077946 mapread 0x012801 thru 0x01efaf (0xc7af bytes) 000999 1788941785.088925 mapread 0x00fa2e thru 0x017839 (0x7e0c bytes) 001000 1788941785.090514 mapwrite 0x00a4db thru 0x01166f (0x7195 bytes) All operations completed A-OK! PASS 70a (131s) == sanity-flr test 71: check mirror extend parent fid ==== 04:16:30 (1788941790) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x170:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 04:16:41 (1788941801) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198439 s, 10.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172089 s, 6.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 04:16:50 (1788941810) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7868832 .. Extending file size to 8387808 .. Extending file size to 8560640 .. Extending file size to 8788288 .. Extending file size to 8824448 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9130368 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9173920 .. Extending file size to 9292704 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61405 61406 61407 61410 61412 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:18:18 (1788941898) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1164288 .. Extending file size to 8770016 .. done Extending file size to 1668320 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8719648 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8913664 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62792 62793 62794 62797 62799 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (88s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:19:46 (1788941986) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1268864 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2048 1269256 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24212 3582808 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18548 3588472 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42760 7171280 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1268864 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2048 1269256 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24212 3582808 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18548 3588472 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42760 7171280 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.273293 s, 7.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.280947 s, 7.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcb85eba2 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:20:12 (1788942012) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154981 s, 13.5 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 04:20:21 (1788942021) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x17f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2be:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2be:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2bf:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x2dc:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:20:30 (1788942030) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138517 s, 15.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:20:40 (1788942040) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149959 s, 14.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2dd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2dd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2de:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:20:49 (1788942049) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180664 s, 11.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2df:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e0:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:21:26 (1788942086) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663025 s, 15.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x60:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:22:20 (1788942140) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145516 s, 14.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:23:20 (1788942200) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg222-server: Pool lustre.test_204f created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120413 s, 17.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg222-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:24:35 (1788942275) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:24:46 (1788942286) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:24:57 (1788942297) oleg222-server: Pool lustre.test_206 created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg222-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:25:29 (1788942329) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746377 s, 14.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494332 s, 21.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3bc41982 Mirror 3: 0x3bc41982 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:25:42 (1788942342) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:25:45 (1788942345) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:25:50 (1788942350) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0665847 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2020000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2020000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2020000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x820000 Read this (4 bytes): BBBB 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:26:32 (1788942392) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0263501 s, 311 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:26:42 (1788942402) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983839 s, 10.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1a0 lmm_fid: [0x200000405:0x1a0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1a0 lmm_fid: [0x200000405:0x1a0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2cf:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611552 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 04:26:43.000000000 -0400 Modify: 2026-09-09 04:26:43.000000000 -0400 Change: 2026-09-09 04:26:45.000000000 -0400 Birth: 2026-09-09 04:26:43.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:26:53 (1788942413) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614972 s, 17.1 MB/s oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg222-server: O/280000401/d13/2ed: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:27:16 (1788942436) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.279245 s, 147 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0258363 s, 159 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 9 04:27 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 9 04:27 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:27:26 (1788942446) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:27:36 (1788942456) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1997 sec =========== 04:27:46 (1788942466) === sanity-flr: start cleanup 04:27:48 (1788942468) === === sanity-flr: finish cleanup 04:27:57 (1788942477) ===