-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 10:49:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg623-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:50:05 (1788619805) === oleg623-client.virtnet: executing check_config_client /mnt/lustre oleg623-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg623-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c6104c0d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c6104c0d800.idle_timeout=debug disable quota as required oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:50:38 (1788619838) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7624] == sanity-flr test 0a: lfs mirror create with -N option == 10:50:40 (1788619840) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:50:58 (1788619858) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:51:01 (1788619861) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:51:04 (1788619864) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:51:27 (1788619887) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:51:30 (1788619890) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:51:34 (1788619894) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c6104c0d800.stats=clear osc.lustre-OST0001-osc-ffff9c6104c0d800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:51:41 (1788619901) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:51:52 (1788619912) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:52:00 (1788619920) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:52:04 (1788619924) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:52:11 (1788619931) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 10:52:20 (1788619940) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:52:27 (1788619947) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:52:30 (1788619950) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:52:37 (1788619957) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:52:47 (1788619967) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:52:54 (1788619974) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.61295 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144013 s, 7.3 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:53:04 (1788619984) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.193096 s, 48.9 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:53:15 (1788619995) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:53:28 (1788620008) Stopping /mnt/lustre-ost1 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c6104c0d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c6104c0d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg623-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c6104c0d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c6104c0d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg623-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (68s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:54:36 (1788620076) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.320089 s, 19.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c6104c0d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c6104c0d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg623-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:55:19 (1788620119) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c6104c0d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c6104c0d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg623-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c6104c0d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c6104c0d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg623-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:56:37 (1788620197) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.54869 s, 8.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 10:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.634s user 0m0.002s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c6104c0d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c6104c0d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.288s user 0m0.000s sys 0m0.181s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg623-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c6104c0d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c6104c0d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.076s user 0m0.001s sys 0m0.193s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg623-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg623-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c6104c0d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c6104c0d800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:58:13 (1788620293) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:58:16 (1788620296) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:58:20 (1788620300) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 10:58:23 (1788620303) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 10:58:30 (1788620310) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.621016 s, 20.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:58:42 (1788620322) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424008 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396953 s, 26.4 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:58:49 (1788620329) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:58:56 (1788620336) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000972511 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00262095 s, 391 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00330648 s, 310 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000977528 s, 1.0 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 10:59:06 (1788620346) create test files with size 15747788 .. 6844b2bb6ba819c3fc891fdc705ad453 - ae948c8fa5a0c998486d511f77b9b7b3 - 64c99d4a02cbea2f2956fe434a31f796 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg623-client.virtnet /mnt/lustre (opts:) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 10:59:28 (1788620368) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:59:45 (1788620385) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:59:51 (1788620391) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183772 s, 5.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498445 s, 21.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 11:00:02 (1788620402) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.264004 s, 7.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.503205 s, 8.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158454 s, 13.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (22s) == sanity-flr test 42: lfs mirror verify ================= 11:00:24 (1788620424) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 11:00:28 (1788620428) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:00:31 (1788620431) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461598 s, 22.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 11:00:38 (1788620438) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:00:42 (1788620442) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:00:51 (1788620451) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.264245 s, 39.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg623-client.virtnet /mnt/lustre (opts:) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:01:03 (1788620463) llite.lustre-ffff9c6108d68000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.324102 s, 32.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg623-client.virtnet /mnt/lustre (opts:) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c61088ff000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:01:17 (1788620477) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:01:54 (1788620514) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:02:29 (1788620549) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 11:02:32 (1788620552) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.229844 s, 13.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.189977 s, 16.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd1fdb0d8 (nosync,stale) Mirror 2: 0xd1fdb0d8 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 11301ccb8b832ae12be73f01740a88e1 - mirror 1 checksum: 57e04093c254386838e3272d3b243f62 - mirror 2 checksum: 11301ccb8b832ae12be73f01740a88e1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T11:02:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T11:02:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 11301ccb8b832ae12be73f01740a88e1 - mirror 1 checksum: 57e04093c254386838e3272d3b243f62 - mirror 2 checksum: 57e04093c254386838e3272d3b243f62 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T11:02:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T11:02:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:02:46 (1788620566) Failing mds1 on oleg623-server Stopping /mnt/lustre-mds1 (opts:) on oleg623-server 11:02:54 (1788620574) shut down facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server Failover mds1 to oleg623-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-MDT0000 11:03:16 (1788620596) targets are mounted 11:03:16 (1788620596) facet_failover done oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371778 s, 28.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:03:34 (1788620614) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:03:38 (1788620618) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:03:41 (1788620621) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00955861 s, 1.7 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:03:50 (1788620630) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:03:54 (1788620634) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110689 s, 370 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.124619 s, 32.9 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:04:05 (1788620645) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:04:32 (1788620672) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:05:00 (1788620700) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:05:25 (1788620725) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:05:50 (1788620750) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 11:05:59 (1788620759) mirror create pid 54337 mirror split pid 54338 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54337 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54338 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 11:07:06 (1788620826) Seed set to 828 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0xcc4a 000003 1788620829.254863 trunc from 00000000 to 0x00cc49 (0xcc4a bytes) truncating to largest ever: 0x1587c 000004 1788620829.272413 trunc from 0x00cc4a to 0x01587b (0x8c32 bytes) 000006 1788620829.289571 mapread 0x001325 thru 0x0055e2 (0x42be bytes) 000009 1788620829.311456 read 0x00f318 thru 0x012f9c (0x3c85 bytes) 000010 1788620829.316221 punch from 0x0120d2 to 0x01587b (0x37aa bytes) 000011 1788620829.322213 write 0x005977 thru 0x014591 (0xec1b bytes) 000012 1788620829.328673 mapread 0x0148f4 thru 0x01587b (0xf88 bytes) 000013 1788620829.334967 mapwrite 0x03d8b6 thru 0x03ffff (0x274a bytes) 000014 1788620829.461968 read 0x03c26b thru 0x03ffff (0x3d95 bytes) 000015 1788620829.466969 mapread 0x02a093 thru 0x02c88c (0x27fa bytes) 000016 1788620829.495336 mapread 0x02a3b3 thru 0x03768d (0xd2db bytes) 000017 1788620829.496816 read 0x0155d8 thru 0x0225e3 (0xd00c bytes) 000018 1788620829.498037 mapwrite 0x027108 thru 0x03117f (0xa078 bytes) 000019 1788620829.620879 mapread 0x011587 thru 0x017353 (0x5dcd bytes) 000022 1788620829.622524 punch from 0x019a18 to 0x01aba1 (0x118a bytes) 000024 1788620829.639644 mapread 0x0382f4 thru 0x03cbbe (0x48cb bytes) 000025 1788620829.641565 punch from 0x0325f6 to 0x03ecc0 (0xc6cb bytes) truncating to largest ever: 0x218cc 000030 1788620829.661890 trunc from 0x040000 to 0x0218cb (0x1e734 bytes) 000031 1788620829.688182 write 0x014baa thru 0x018259 (0x36b0 bytes) 000032 1788620829.689403 mapwrite 0x0206ed thru 0x024c12 (0x4526 bytes) truncating to largest ever: 0x3d9d1 000033 1788620829.795703 trunc from 0x024c13 to 0x03d9d0 (0x18dbe bytes) 000034 1788620829.818893 mapwrite 0x017d7e thru 0x0277b6 (0xfa39 bytes) 000035 1788620829.917881 write 0x01cd42 thru 0x01e5af (0x186e bytes) 000040 1788620829.932428 read 0x010116 thru 0x0170c0 (0x6fab bytes) 000041 1788620829.935235 write 0x01be7e thru 0x027b34 (0xbcb7 bytes) 000042 1788620829.936520 trunc from 0x03d9d1 to 0x024d31 (0x18c9f bytes) 000044 1788620829.980295 mapread 0x000373 thru 0x003777 (0x3405 bytes) 000045 1788620829.982529 mapread 0x00e6b7 thru 0x01a83f (0xc189 bytes) 000046 1788620829.986260 write 0x021a81 thru 0x028d74 (0x72f4 bytes) 000047 1788620829.993810 mapwrite 0x0077fd thru 0x00a042 (0x2846 bytes) 000049 1788620830.119828 punch from 0x021bad to 0x028d74 (0x71c8 bytes) 000050 1788620830.125987 trunc from 0x028d75 to 0x016bc2 (0x121b2 bytes) 000051 1788620830.154473 read 0x00afd5 thru 0x015e9e (0xaeca bytes) 000052 1788620830.155740 mapread 0x010fc2 thru 0x016bc2 (0x5c01 bytes) 000054 1788620830.158122 mapwrite 0x029160 thru 0x036d1e (0xdbbf bytes) 000055 1788620830.350462 write 0x024fc0 thru 0x03411b (0xf15c bytes) 000056 1788620830.357108 punch from 0x0063f9 to 0x014618 (0xe220 bytes) 000057 1788620830.414285 mapread 0x010c20 thru 0x016d22 (0x6103 bytes) 000059 1788620830.470788 mapread 0x0224c8 thru 0x02fed2 (0xda0b bytes) 000060 1788620830.480915 read 0x0190fa thru 0x01c1af (0x30b6 bytes) 000061 1788620830.481950 write 0x00e2d4 thru 0x015003 (0x6d30 bytes) 000062 1788620830.501181 mapread 0x021ac5 thru 0x0262f9 (0x4835 bytes) 000065 1788620830.503005 trunc from 0x036d1f to 0x010e34 (0x25eea bytes) 000066 1788620830.545006 mapwrite 0x002f00 thru 0x01154c (0xe64d bytes) 000067 1788620830.650605 mapwrite 0x02ef29 thru 0x03191b (0x29f3 bytes) 000069 1788620830.764748 mapwrite 0x012aed thru 0x012b37 (0x4b bytes) 000070 1788620830.826983 read 0x02ef26 thru 0x03191b (0x29f6 bytes) 000071 1788620830.829267 read 0x021dc5 thru 0x02cb75 (0xadb1 bytes) 000073 1788620830.848943 punch from 0x01430a to 0x01cf4d (0x8c44 bytes) 000074 1788620830.854536 punch from 0x009f00 to 0x00d2dc (0x33dd bytes) 000076 1788620830.860500 punch from 0x0023ea to 0x008b86 (0x679d bytes) 000077 1788620830.868782 trunc from 0x03191c to 0x01161a (0x20301 bytes) 000078 1788620830.908169 read 0x0076bb thru 0x01161a (0x9f60 bytes) 000081 1788620830.923253 punch from 0x00a0d9 to 0x010aa2 (0x69ca bytes) 000082 1788620830.935830 MIRROR_EXTEND to 2 mirrors 000083 1788620831.865140 write 0x02ef1b thru 0x03158f (0x2675 bytes) 000084 1788620831.953960 write 0x01141a thru 0x01d46b (0xc052 bytes) 000086 1788620831.969253 mapwrite 0x03a97c thru 0x03ffff (0x5684 bytes) 000087 1788620832.103929 mapwrite 0x00a5b6 thru 0x01996b (0xf3b6 bytes) 000088 1788620832.235709 write 0x02d069 thru 0x03c2cd (0xf265 bytes) 000089 1788620832.241247 mapread 0x013e86 thru 0x0167e1 (0x295c bytes) 000090 1788620832.243213 punch from 0x00ba2e to 0x01574e (0x9d21 bytes) 000091 1788620832.272262 read 0x0042d6 thru 0x00e5e4 (0xa30f bytes) 000093 1788620832.285891 write 0x018c27 thru 0x01ae5c (0x2236 bytes) 000099 1788620832.297291 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000100 1788620832.479269 mapwrite 0x01171c thru 0x01fe65 (0xe74a bytes) 000101 1788620832.650152 mapwrite 0x017259 thru 0x022552 (0xb2fa bytes) 000102 1788620832.806591 mapwrite 0x02da20 thru 0x03c00e (0xe5ef bytes) 000104 1788620832.887525 trunc from 0x040000 to 0x00ceb6 (0x33149 bytes) 000105 1788620832.908107 read 0x0038f5 thru 0x00ab7c (0x7288 bytes) 000106 1788620832.916147 trunc from 0x00ceb7 to 0x025fb1 (0x190fb bytes) 000107 1788620832.931066 trunc from 0x025fb2 to 0x03a1ed (0x1423c bytes) 000108 1788620832.960990 mapwrite 0x00fdac thru 0x00ff3b (0x190 bytes) 000109 1788620833.017307 trunc from 0x03a1ee to 0x01a274 (0x1ff79 bytes) 000110 1788620833.041564 punch from 0x00ad75 to 0x018c23 (0xdeaf bytes) 000111 1788620833.049870 mapread 0x00251c thru 0x00ada4 (0x8889 bytes) 000113 1788620833.065051 write 0x01a268 thru 0x01c476 (0x220f bytes) 000114 1788620833.070248 mapwrite 0x03cf5f thru 0x03ee74 (0x1f16 bytes) 000115 1788620833.229580 punch from 0x015602 to 0x0156ee (0xed bytes) 000116 1788620833.235805 mapwrite 0x01fda7 thru 0x02b09a (0xb2f4 bytes) 000118 1788620833.370420 write 0x00f8c3 thru 0x01e642 (0xed80 bytes) 000120 1788620833.376159 read 0x01214d thru 0x01c6c5 (0xa579 bytes) 000121 1788620833.377521 mapwrite 0x020025 thru 0x02d171 (0xd14d bytes) 000123 1788620833.506014 punch from 0x030f78 to 0x032fde (0x2067 bytes) 000125 1788620833.520420 trunc from 0x03ee75 to 0x035ceb (0x9189 bytes) 000126 1788620833.575082 trunc from 0x035cec to 0x007d6f (0x2df7c bytes) 000128 1788620833.624018 write 0x0114d5 thru 0x0163cc (0x4ef8 bytes) 000129 1788620833.625102 read 0x00eb10 thru 0x0163cc (0x78bd bytes) 000131 1788620833.642463 MIRROR_EXTEND to 2 mirrors 000135 1788620834.532215 mapwrite 0x00433c thru 0x00f8be (0xb583 bytes) 000136 1788620834.812462 mapread 0x008220 thru 0x00bdfa (0x3bdb bytes) 000137 1788620834.831901 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000139 1788620835.054841 mapwrite 0x03872a thru 0x03889a (0x171 bytes) 000140 1788620835.116754 write 0x0037c8 thru 0x00e0db (0xa914 bytes) 000141 1788620835.128090 mapread 0x0291e9 thru 0x0323bb (0x91d3 bytes) 000142 1788620835.151478 trunc from 0x03889b to 0x032544 (0x6356 bytes) 000143 1788620835.170411 punch from 0x016282 to 0x01c111 (0x5e90 bytes) 000145 1788620835.192835 write 0x03896e thru 0x03ffff (0x7692 bytes) 000146 1788620835.194568 punch from 0x0246b8 to 0x031d2b (0xd674 bytes) 000147 1788620835.205605 punch from 0x000f27 to 0x007b19 (0x6bf3 bytes) truncating to largest ever: 0x3e95e 000150 1788620835.211148 trunc from 0x040000 to 0x03e95d (0x16a2 bytes) 000151 1788620835.226824 punch from 0x000f41 to 0x0073b0 (0x6470 bytes) 000152 1788620835.231478 punch from 0x014b92 to 0x023d8b (0xf1fa bytes) 000155 1788620835.237182 read 0x00539d thru 0x007066 (0x1cca bytes) 000156 1788620835.243575 trunc from 0x03e95e to 0x0266db (0x18282 bytes) 000159 1788620835.262076 read 0x013761 thru 0x01e62b (0xaecb bytes) 000161 1788620835.275100 read 0x01f435 thru 0x0266db (0x72a7 bytes) 000163 1788620835.276552 punch from 0x0055f4 to 0x00b2cb (0x5cd8 bytes) 000165 1788620835.282862 mapread 0x0184cb thru 0x01bc33 (0x3769 bytes) 000168 1788620835.300546 mapread 0x014e73 thru 0x017402 (0x2590 bytes) 000170 1788620835.304835 write 0x02587d thru 0x02abc0 (0x5344 bytes) 000171 1788620835.306543 read 0x00aab2 thru 0x016b02 (0xc051 bytes) 000172 1788620835.317557 trunc from 0x02abc1 to 0x0079b4 (0x2320c bytes) 000173 1788620835.341896 mapwrite 0x01e0a8 thru 0x01ed73 (0xccc bytes) 000174 1788620835.472844 trunc from 0x01ed74 to 0x00b717 (0x1365c bytes) 000176 1788620835.509967 mapread 0x00a873 thru 0x00b717 (0xea5 bytes) 000179 1788620835.527493 read 0x00611f thru 0x008575 (0x2457 bytes) 000180 1788620835.535455 mapwrite 0x03be71 thru 0x03ffff (0x418f bytes) 000186 1788620835.652413 punch from 0x012781 to 0x021495 (0xed15 bytes) 000187 1788620835.659042 punch from 0x020e58 to 0x029a2b (0x8bd4 bytes) 000188 1788620835.667812 write 0x02242c thru 0x02f7ab (0xd380 bytes) 000190 1788620835.682871 MIRROR_EXTEND to 2 mirrors 000191 1788620837.087195 mapread 0x009291 thru 0x00bd8c (0x2afc bytes) 000192 1788620837.123646 mapread 0x017e6c thru 0x0263db (0xe570 bytes) 000193 1788620837.150033 trunc from 0x040000 to 0x0292b1 (0x16d4e bytes) 000195 1788620837.323086 MIRROR_RESYNC 2 mirrors 000198 1788620838.419888 MIRROR_EXTEND to 3 mirrors 000200 1788620840.136816 mapread 0x00ec29 thru 0x01012b (0x1503 bytes) 000201 1788620840.181697 read 0x010707 thru 0x0193ff (0x8cf9 bytes) 000202 1788620840.195729 trunc from 0x0292b2 to 0x0232f2 (0x5fbf bytes) 000206 1788620840.352886 mapread 0x021b01 thru 0x0232f2 (0x17f2 bytes) 000207 1788620840.368484 punch from 0x0141ad to 0x017d19 (0x3b6d bytes) 000208 1788620840.373282 mapread 0x013616 thru 0x0151c4 (0x1baf bytes) 000209 1788620840.384525 trunc from 0x0232f3 to 0x031199 (0xdea7 bytes) 000211 1788620840.405842 read 0x02f1fa thru 0x031199 (0x1fa0 bytes) 000212 1788620840.417884 mapread 0x01bcc3 thru 0x02b3c2 (0xf700 bytes) 000215 1788620840.421088 read 0x02cf86 thru 0x031199 (0x4214 bytes) 000216 1788620840.425828 MIRROR_RESYNC 3 mirrors 000217 1788620842.004726 trunc from 0x03119a to 0x007792 (0x29a07 bytes) 000218 1788620842.099602 mapread 0x0011eb thru 0x007792 (0x65a8 bytes) 000220 1788620842.124934 read 0x001464 thru 0x007792 (0x632f bytes) 000222 1788620842.127791 mapread 0x0011a1 thru 0x007792 (0x65f2 bytes) 000223 1788620842.129349 punch from 0x0071b3 to 0x007792 (0x5e0 bytes) 000224 1788620842.136831 trunc from 0x007793 to 0x022ead (0x1b71b bytes) 000225 1788620842.183098 MIRROR_RESYNC 3 mirrors 000227 1788620843.463336 punch from 0x015588 to 0x01af7c (0x59f5 bytes) 000228 1788620843.610976 punch from 0x021860 to 0x022ead (0x164e bytes) 000231 1788620843.626493 mapread 0x0109a7 thru 0x015419 (0x4a73 bytes) 000233 1788620843.687456 read 0x016d74 thru 0x022ead (0xc13a bytes) 000234 1788620843.704756 read 0x005719 thru 0x013153 (0xda3b bytes) 000235 1788620843.719080 mapread 0x00ef75 thru 0x0159b8 (0x6a44 bytes) 000237 1788620843.728315 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22eae): Mirror 3: 0x9dea8948 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22eae): Mirror 3: 0x9dea8948 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22eae: skipped 000238 1788620844.147548 read 0x00a3ec thru 0x00bb26 (0x173b bytes) 000240 1788620844.158368 punch from 0x02237b to 0x022ead (0xb33 bytes) 000241 1788620844.165334 write 0x007412 thru 0x016c57 (0xf846 bytes) 000242 1788620844.188667 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22eae): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22eae): Mirror 3: 0x9dea8948 (stale) CRC-32 checksum value for chunk [0, 0x22eae): Mirror 1: 0xef8b7dbb Mirror 4: 0xef8b7dbb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22eae: skipped 000243 1788620845.366114 punch from 0x002f49 to 0x010bec (0xdca4 bytes) 000244 1788620845.521406 trunc from 0x022eae to 0x035c19 (0x12d6c bytes) 000245 1788620845.615004 punch from 0x00e565 to 0x0186a2 (0xa13e bytes) 000246 1788620845.627364 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35c1a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c1a): Mirror 3: 0x6aeceab9 (stale) Mirror 4: 0x33ea69ec (stale) CRC-32 checksum value for chunk [0, 0x35c1a): Mirror 1: 0x6aeceab9 Mirror 5: 0x6aeceab9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c1a: skipped 000247 1788620847.120835 mapread 0x035760 thru 0x035c19 (0x4ba bytes) 000248 1788620847.180289 mapread 0x01588d thru 0x016ab1 (0x1225 bytes) 000249 1788620847.245851 punch from 0x02242d to 0x02f3d7 (0xcfab bytes) 000250 1788620847.432657 trunc from 0x035c1a to 0x003469 (0x327b0 bytes) 000252 1788620847.484373 punch from 0x0012be to 0x003469 (0x21ac bytes) 000253 1788620847.504831 trunc from 0x00346a to 0x018e84 (0x15a1b bytes) 000254 1788620847.554437 punch from 0x00783d to 0x00b1df (0x39a3 bytes) 000255 1788620847.565717 read 0x003541 thru 0x0119a8 (0xe468 bytes) 000256 1788620847.592982 punch from 0x014705 to 0x018e84 (0x4780 bytes) 000257 1788620847.605492 write 0x003911 thru 0x007c08 (0x42f8 bytes) 000258 1788620847.607325 mapread 0x00bf7c thru 0x0159c7 (0x9a4c bytes) 000259 1788620847.621877 MIRROR_RESYNC 4 mirrors 000261 1788620849.350064 trunc from 0x018e85 to 0x03e076 (0x251f2 bytes) 000262 1788620849.446371 MIRROR_RESYNC 4 mirrors 000264 1788620851.267473 mapread 0x00b853 thru 0x0103b3 (0x4b61 bytes) 000265 1788620851.329423 trunc from 0x03e077 to 0x001315 (0x3cd61 bytes) 000266 1788620851.488377 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1316): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1316): Mirror 3: 0xfd9c8c38 (stale) Mirror 4: 0xfd9c8c38 (stale) Mirror 5: 0xfd9c8c38 (stale) CRC-32 checksum value for chunk [0, 0x1316): Mirror 1: 0xfd9c8c38 Mirror 6: 0xfd9c8c38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1316: skipped 000268 1788620852.724501 trunc from 0x001316 to 0x02f007 (0x2dcf2 bytes) 000270 1788620853.006460 mapwrite 0x0248ee thru 0x0297a4 (0x4eb7 bytes) 000272 1788620853.270928 read 0x02060d thru 0x024dbd (0x47b1 bytes) 000273 1788620853.276648 mapread 0x0031c8 thru 0x0077d6 (0x460f bytes) 000274 1788620853.313467 mapwrite 0x012915 thru 0x01360f (0xcfb bytes) 000275 1788620853.453466 trunc from 0x02f008 to 0x00ca76 (0x22591 bytes) 000276 1788620853.484402 read 0x005abb thru 0x00ca76 (0x6fbc bytes) 000278 1788620853.496983 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xca77): Mirror 4: 0x19407daa (stale) Mirror 5: 0x19407daa (stale) Mirror 6: 0x4f40b1e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xca77): Mirror 4: 0x19407daa (stale) Mirror 5: 0x19407daa (stale) Mirror 6: 0x4f40b1e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xca77: skipped 000280 1788620853.902538 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xca77): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xca77): Mirror 4: 0x19407daa (stale) Mirror 5: 0x19407daa (stale) Mirror 6: 0x4f40b1e4 (stale) CRC-32 checksum value for chunk [0, 0xca77): Mirror 1: 0x4f40b1e4 Mirror 7: 0x4f40b1e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xca77: skipped 000281 1788620854.935046 trunc from 0x00ca77 to 0x02157a (0x14b04 bytes) 000283 1788620855.109720 punch from 0x00b630 to 0x014146 (0x8b17 bytes) 000285 1788620855.119499 trunc from 0x02157b to 0x039e78 (0x188fe bytes) 000286 1788620855.152091 mapread 0x0325c6 thru 0x039e78 (0x78b3 bytes) 000289 1788620855.182872 trunc from 0x039e79 to 0x016ebd (0x22fbb bytes) 000291 1788620855.204440 punch from 0x016636 to 0x016ebd (0x888 bytes) 000292 1788620855.210629 write 0x037e57 thru 0x03ffff (0x81a9 bytes) 000293 1788620855.211950 write 0x0133a9 thru 0x016225 (0x2e7d bytes) 000294 1788620855.222901 read 0x03d722 thru 0x03ffff (0x28de bytes) 000298 1788620855.226815 trunc from 0x040000 to 0x00c81d (0x337e2 bytes) 000299 1788620855.250297 mapread 0x006e01 thru 0x00c81d (0x5a1d bytes) 000300 1788620855.260700 read 0x008956 thru 0x00c81d (0x3ec8 bytes) 000301 1788620855.261823 write 0x0168a6 thru 0x01c6df (0x5e3a bytes) 000304 1788620855.265753 read 0x007d90 thru 0x00d21b (0x548c bytes) 000305 1788620855.277353 trunc from 0x01c6e0 to 0x01ab7d (0x1b62 bytes) 000306 1788620855.293671 trunc from 0x01ab7e to 0x00c65d (0xe520 bytes) 000308 1788620855.317957 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xc65e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc65e): Mirror 4: 0x3668c21f (stale) Mirror 5: 0x3668c21f (stale) Mirror 6: 0xce1e4b9c (stale) Mirror 7: 0xce1e4b9c (stale) CRC-32 checksum value for chunk [0, 0xc65e): Mirror 1: 0xce1e4b9c Mirror 8: 0xce1e4b9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc65e: skipped 000309 1788620856.314341 mapread 0x0058de thru 0x005e6c (0x58f bytes) 000310 1788620856.330041 write 0x003ae1 thru 0x012e68 (0xf388 bytes) 000311 1788620856.511878 trunc from 0x012e69 to 0x036bf4 (0x23d8c bytes) 000314 1788620856.574801 read 0x028f39 thru 0x02a444 (0x150c bytes) 000315 1788620856.582304 mapread 0x028572 thru 0x02fa35 (0x74c4 bytes) 000317 1788620856.624482 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36bf5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bf5): Mirror 4: 0x3055423c (stale) Mirror 5: 0x3055423c (stale) Mirror 6: 0x68e2efd0 (stale) Mirror 7: 0x68e2efd0 (stale) Mirror 8: 0x68e2efd0 (stale) CRC-32 checksum value for chunk [0, 0x36bf5): Mirror 1: 0x7a14907 Mirror 9: 0x7a14907 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bf5: skipped 000319 1788620858.232357 punch from 0x028443 to 0x02ab87 (0x2745 bytes) 000320 1788620858.465766 read 0x004c52 thru 0x00cae0 (0x7e8f bytes) 000322 1788620858.481823 mapwrite 0x02ecac thru 0x03de10 (0xf165 bytes) 000323 1788620858.635211 punch from 0x00d14d to 0x010f4d (0x3e01 bytes) 000326 1788620858.642928 mapread 0x00086c thru 0x006670 (0x5e05 bytes) 000327 1788620858.653718 write 0x01aeda thru 0x01d496 (0x25bd bytes) 000329 1788620858.667746 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3de11): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de11): Mirror 4: 0x5d4c7356 (stale) Mirror 5: 0x5d4c7356 (stale) Mirror 6: 0x63a076fa (stale) Mirror 7: 0x63a076fa (stale) Mirror 8: 0x63a076fa (stale) Mirror 9: 0x57a6e6fb (stale) CRC-32 checksum value for chunk [0, 0x3de11): Mirror 1: 0x9cb2a80 Mirror 10: 0x9cb2a80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de11: skipped 000330 1788620860.124089 mapwrite 0x007888 thru 0x00d4f3 (0x5c6c bytes) 000332 1788620860.539686 MIRROR_RESYNC 8 mirrors 000333 1788620862.588573 mapread 0x039a69 thru 0x03de10 (0x43a8 bytes) 000334 1788620862.624124 read 0x0136a3 thru 0x01f376 (0xbcd4 bytes) 000336 1788620862.637058 mapread 0x01955e thru 0x02281f (0x92c2 bytes) 000337 1788620862.654858 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000339 1788620863.227485 trunc from 0x03de11 to 0x03c884 (0x158c bytes) 000340 1788620863.404069 write 0x025846 thru 0x027197 (0x1952 bytes) 000341 1788620863.430417 punch from 0x03a64c to 0x03c884 (0x2239 bytes) 000343 1788620863.457958 MIRROR_RESYNC 7 mirrors 000344 1788620865.409993 mapread 0x001d46 thru 0x01008d (0xe348 bytes) 000347 1788620865.428652 write 0x01d70a thru 0x02bcf0 (0xe5e7 bytes) 000348 1788620865.600569 MIRROR_RESYNC 7 mirrors 000349 1788620867.783891 trunc from 0x03c885 to 0x0230b0 (0x197d4 bytes) 000351 1788620867.902768 MIRROR_RESYNC 7 mirrors 000352 1788620869.677775 mapwrite 0x01ed0a thru 0x0202ac (0x15a3 bytes) 000353 1788620869.927853 read 0x0010d8 thru 0x0037f1 (0x271a bytes) 000356 1788620869.935778 punch from 0x00d3e4 to 0x0138d1 (0x64ee bytes) 000357 1788620869.947189 MIRROR_RESYNC 7 mirrors 000359 1788620871.590685 mapread 0x00143a thru 0x002e87 (0x1a4e bytes) 000360 1788620871.599921 trunc from 0x0230b1 to 0x037537 (0x14487 bytes) 000361 1788620871.670915 write 0x014cf7 thru 0x016b6e (0x1e78 bytes) 000362 1788620871.689374 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37538): Mirror 6: 0x1e14f068 (stale) Mirror 7: 0x1e14f068 (stale) Mirror 8: 0x1e14f068 (stale) Mirror 9: 0x1e14f068 (stale) Mirror 10: 0x1e14f068 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37538): Mirror 6: 0x1e14f068 (stale) Mirror 7: 0x1e14f068 (stale) Mirror 8: 0x1e14f068 (stale) Mirror 9: 0x1e14f068 (stale) Mirror 10: 0x1e14f068 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37538: skipped 000363 1788620872.290595 read 0x0081a1 thru 0x0099e6 (0x1846 bytes) 000364 1788620872.297758 read 0x010a10 thru 0x01862c (0x7c1d bytes) 000365 1788620872.328736 write 0x0240d9 thru 0x032d34 (0xec5c bytes) 000366 1788620872.330577 read 0x01b3e5 thru 0x01d6ba (0x22d6 bytes) 000368 1788620872.335160 trunc from 0x037538 to 0x01c53e (0x1aff9 bytes) 000369 1788620872.360887 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c53f): Mirror 7: 0xedf87138 (stale) Mirror 8: 0xedf87138 (stale) Mirror 9: 0xedf87138 (stale) Mirror 10: 0xedf87138 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c53f): Mirror 7: 0xedf87138 (stale) Mirror 8: 0xedf87138 (stale) Mirror 9: 0xedf87138 (stale) Mirror 10: 0xedf87138 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c53f: skipped 000370 1788620872.859575 trunc from 0x01c53f to 0x013b9d (0x89a1 bytes) 000375 1788620872.881302 mapwrite 0x01aece thru 0x01eae5 (0x3c18 bytes) 000377 1788620873.048201 mapwrite 0x01a9ed thru 0x029612 (0xec26 bytes) 000379 1788620873.210004 punch from 0x0178b9 to 0x0242d0 (0xca18 bytes) 000380 1788620873.234582 MIRROR_RESYNC 5 mirrors 000381 1788620875.411199 mapread 0x019b27 thru 0x01a479 (0x953 bytes) 000382 1788620875.428170 MIRROR_EXTEND to 6 mirrors 000384 1788620876.433599 mapwrite 0x00029e thru 0x00aeeb (0xac4e bytes) 000386 1788620876.731955 write 0x00a2d4 thru 0x0103b0 (0x60dd bytes) 000387 1788620876.733173 mapread 0x025382 thru 0x029612 (0x4291 bytes) 000388 1788620876.753041 read 0x02945d thru 0x029612 (0x1b6 bytes) 000389 1788620876.755359 punch from 0x0152c4 to 0x0200c7 (0xae04 bytes) 000390 1788620876.778951 mapread 0x018ced thru 0x01b987 (0x2c9b bytes) 000392 1788620876.804713 MIRROR_RESYNC 6 mirrors 000393 1788620878.433109 punch from 0x021168 to 0x029612 (0x84ab bytes) 000395 1788620878.497914 punch from 0x0014bd to 0x00c283 (0xadc7 bytes) 000396 1788620878.507240 MIRROR_RESYNC 6 mirrors 000397 1788620880.504572 mapread 0x017a34 thru 0x0267d1 (0xed9e bytes) 000398 1788620880.649510 MIRROR_RESYNC 6 mirrors 000399 1788620881.537172 read 0x020486 thru 0x026d87 (0x6902 bytes) 000400 1788620881.539062 read 0x00554a thru 0x005a47 (0x4fe bytes) 000401 1788620881.560747 write 0x00d515 thru 0x00f9eb (0x24d7 bytes) 000402 1788620881.727458 write 0x0305e6 thru 0x0329c1 (0x23dc bytes) 000404 1788620881.730130 read 0x0254ce thru 0x0329c1 (0xd4f4 bytes) 000405 1788620881.763804 punch from 0x00619e to 0x00f9e7 (0x984a bytes) 000406 1788620881.788480 trunc from 0x0329c2 to 0x03791d (0x4f5c bytes) 000407 1788620881.822296 mapread 0x01cc51 thru 0x024e37 (0x81e7 bytes) 000408 1788620881.823383 punch from 0x005a25 to 0x014c41 (0xf21d bytes) 000409 1788620881.829792 punch from 0x0230de to 0x031d29 (0xec4c bytes) 000412 1788620881.847784 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3791e): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3791e): Mirror 7: 0x4c8aacae (stale) Mirror 8: 0x4c8aacae (stale) Mirror 9: 0x4c8aacae (stale) Mirror 10: 0x4c8aacae (stale) Mirror 11: 0x4c8aacae (stale) CRC-32 checksum value for chunk [0, 0x3791e): Mirror 1: 0x8d7cf847 Mirror 12: 0x8d7cf847 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3791e: skipped 000413 1788620883.119106 mapwrite 0x0070bd thru 0x015932 (0xe876 bytes) 000414 1788620883.355806 read 0x02f6cc thru 0x03791d (0x8252 bytes) 000415 1788620883.381580 write 0x013ffb thru 0x014596 (0x59c bytes) 000417 1788620883.384528 write 0x02c108 thru 0x038ecf (0xcdc8 bytes) 000419 1788620883.399898 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 8: 0x9a8ac2be (stale) Mirror 9: 0x9a8ac2be (stale) Mirror 10: 0x9a8ac2be (stale) Mirror 11: 0x9a8ac2be (stale) Mirror 12: 0x90385290 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 8: 0x9a8ac2be (stale) Mirror 9: 0x9a8ac2be (stale) Mirror 10: 0x9a8ac2be (stale) Mirror 11: 0x9a8ac2be (stale) Mirror 12: 0x90385290 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ed0: skipped 000422 1788620884.087094 punch from 0x027e58 to 0x0317f0 (0x9999 bytes) 000423 1788620884.098715 read 0x02f46b thru 0x038ecf (0x9a65 bytes) 000424 1788620884.118800 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 8: 0x9a8ac2be (stale) Mirror 9: 0x9a8ac2be (stale) Mirror 10: 0x9a8ac2be (stale) Mirror 11: 0x9a8ac2be (stale) Mirror 12: 0x90385290 (stale) CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 1: 0x43b8bd4f Mirror 13: 0x43b8bd4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ed0: skipped 000426 1788620885.586190 mapwrite 0x02488b thru 0x02d550 (0x8cc6 bytes) 000427 1788620885.946907 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 8: 0x9a8ac2be (stale) Mirror 9: 0x9a8ac2be (stale) Mirror 10: 0x9a8ac2be (stale) Mirror 11: 0x9a8ac2be (stale) Mirror 12: 0x90385290 (stale) Mirror 13: 0x43b8bd4f (stale) CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 1: 0x8d573cf Mirror 14: 0x8d573cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ed0: skipped 000428 1788620887.250962 punch from 0x010c6c to 0x010e6e (0x203 bytes) 000430 1788620887.451942 punch from 0x025d47 to 0x025ecb (0x185 bytes) 000431 1788620887.461657 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 8: 0x9a8ac2be (stale) Mirror 9: 0x9a8ac2be (stale) Mirror 10: 0x9a8ac2be (stale) Mirror 11: 0x9a8ac2be (stale) Mirror 12: 0x90385290 (stale) Mirror 13: 0x43b8bd4f (stale) Mirror 14: 0x8d573cf (stale) CRC-32 checksum value for chunk [0, 0x38ed0): Mirror 1: 0x2c4a4447 Mirror 15: 0x2c4a4447 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ed0: skipped 000436 1788620889.350348 trunc from 0x038ed0 to 0x00ffa3 (0x28f2c bytes) 000437 1788620889.541236 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xffa4): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xffa4): Mirror 8: 0x258f9f9d (stale) Mirror 9: 0x258f9f9d (stale) Mirror 10: 0x258f9f9d (stale) Mirror 11: 0x258f9f9d (stale) Mirror 12: 0xe629662b (stale) Mirror 13: 0x52ac519d (stale) Mirror 14: 0x52ac519d (stale) Mirror 15: 0x52ac519d (stale) CRC-32 checksum value for chunk [0, 0xffa4): Mirror 1: 0x52ac519d Mirror 16: 0x52ac519d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xffa4: skipped 000438 1788620891.208232 trunc from 0x00ffa4 to 0x00e04b (0x1f58 bytes) 000440 1788620891.399995 mapread 0x0055b8 thru 0x0062d5 (0xd1e bytes) 000442 1788620891.426464 read 0x0004af thru 0x0009af (0x501 bytes) 000443 1788620891.438082 read 0x00567c thru 0x00e04b (0x89d0 bytes) 000444 1788620891.452015 mapwrite 0x008aa2 thru 0x00a45b (0x19ba bytes) 000445 1788620891.539330 punch from 0x00ba17 to 0x00e04b (0x2635 bytes) 000447 1788620891.563915 mapread 0x000720 thru 0x003ce0 (0x35c1 bytes) 000449 1788620891.594987 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xe04c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe04c): Mirror 8: 0x1b96e3f5 (stale) Mirror 9: 0x1b96e3f5 (stale) Mirror 10: 0x1b96e3f5 (stale) Mirror 11: 0x1b96e3f5 (stale) Mirror 12: 0xd16e403c (stale) Mirror 13: 0x9a9863e2 (stale) Mirror 14: 0x9a9863e2 (stale) Mirror 15: 0x9a9863e2 (stale) Mirror 16: 0x9a9863e2 (stale) CRC-32 checksum value for chunk [0, 0xe04c): Mirror 1: 0x3419e6d8 Mirror 17: 0x3419e6d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe04c: skipped 000451 1788620893.434675 mapread 0x0065c3 thru 0x00b9b6 (0x53f4 bytes) 000452 1788620893.551086 punch from 0x0070a1 to 0x00e04b (0x6fab bytes) 000453 1788620893.768968 mapread 0x0098c2 thru 0x00e04b (0x478a bytes) 000454 1788620893.792227 mapread 0x002536 thru 0x00ced3 (0xa99e bytes) 000455 1788620893.793274 mapwrite 0x01d638 thru 0x023161 (0x5b2a bytes) 000456 1788620893.968932 mapwrite 0x01d55b thru 0x026143 (0x8be9 bytes) 000457 1788620894.075845 punch from 0x00c243 to 0x01b666 (0xf424 bytes) 000459 1788620894.088234 write 0x03190c thru 0x034ab1 (0x31a6 bytes) 000461 1788620894.091231 write 0x015bde thru 0x019dd5 (0x41f8 bytes) 000462 1788620894.099940 punch from 0x00e8fe to 0x017027 (0x872a bytes) 000463 1788620894.112851 punch from 0x0061f3 to 0x0145f2 (0xe400 bytes) 000464 1788620894.125769 MIRROR_RESYNC 11 mirrors 000465 1788620896.769762 mapread 0x02ec0a thru 0x034ab1 (0x5ea8 bytes) 000466 1788620896.809475 mapwrite 0x00e12e thru 0x011eb6 (0x3d89 bytes) 000467 1788620897.125794 mapread 0x006dfc thru 0x00751d (0x722 bytes) 000468 1788620897.140112 mapwrite 0x034d9a thru 0x03ffff (0xb266 bytes) 000469 1788620897.313836 read 0x024079 thru 0x027be6 (0x3b6e bytes) 000472 1788620897.330741 read 0x01ab6c thru 0x024c27 (0xa0bc bytes) 000474 1788620897.369353 MIRROR_RESYNC 11 mirrors 000476 1788620899.768897 read 0x00ea7f thru 0x01915a (0xa6dc bytes) 000477 1788620899.797958 read 0x02d568 thru 0x03ad5f (0xd7f8 bytes) 000478 1788620899.816838 punch from 0x02abb9 to 0x03a4ab (0xf8f3 bytes) 000480 1788620899.905559 write 0x01b25f thru 0x024d26 (0x9ac8 bytes) 000481 1788620899.919153 write 0x023076 thru 0x02a9ca (0x7955 bytes) 000484 1788620899.933110 read 0x02fd95 thru 0x03fb86 (0xfdf2 bytes) 000485 1788620899.952024 mapread 0x02d854 thru 0x03bfa1 (0xe74e bytes) 000486 1788620899.953042 write 0x025653 thru 0x03223b (0xcbe9 bytes) 000487 1788620899.954426 read 0x0047c2 thru 0x00fd93 (0xb5d2 bytes) 000488 1788620899.955226 mapread 0x010e87 thru 0x01a2d4 (0x944e bytes) 000489 1788620899.956203 punch from 0x00499d to 0x0122b0 (0xd914 bytes) 000490 1788620899.970594 punch from 0x037677 to 0x03a92c (0x32b6 bytes) 000491 1788620899.977832 mapread 0x0282a3 thru 0x037cad (0xfa0b bytes) 000492 1788620899.987696 punch from 0x026df8 to 0x02b8e0 (0x4ae9 bytes) 000493 1788620899.996150 read 0x004cfe thru 0x012962 (0xdc65 bytes) 000494 1788620899.996887 write 0x02d6f0 thru 0x034d22 (0x7633 bytes) 000495 1788620899.997955 punch from 0x0179b1 to 0x01e37a (0x69ca bytes) 000496 1788620900.012204 mapread 0x024d35 thru 0x02c67b (0x7947 bytes) 000498 1788620900.022726 punch from 0x001235 to 0x002b24 (0x18f0 bytes) 000499 1788620900.027991 punch from 0x022d8c to 0x02a777 (0x79ec bytes) 000500 1788620900.035489 punch from 0x03fa7c to 0x03ffff (0x584 bytes) 000501 1788620900.053051 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x31af1b8a (stale) Mirror 9: 0x31af1b8a (stale) Mirror 10: 0x31af1b8a (stale) Mirror 11: 0x31af1b8a (stale) Mirror 12: 0x31af1b8a (stale) Mirror 13: 0x31af1b8a (stale) Mirror 14: 0x31af1b8a (stale) Mirror 15: 0x31af1b8a (stale) Mirror 17: 0x31af1b8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x31af1b8a (stale) Mirror 9: 0x31af1b8a (stale) Mirror 10: 0x31af1b8a (stale) Mirror 11: 0x31af1b8a (stale) Mirror 12: 0x31af1b8a (stale) Mirror 13: 0x31af1b8a (stale) Mirror 14: 0x31af1b8a (stale) Mirror 15: 0x31af1b8a (stale) Mirror 17: 0x31af1b8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000502 1788620900.955675 read 0x024bb8 thru 0x02f0ba (0xa503 bytes) 000503 1788620900.974060 punch from 0x0386a4 to 0x03ffff (0x795c bytes) 000506 1788620900.988978 write 0x003334 thru 0x00a598 (0x7265 bytes) 000509 1788620901.032369 write 0x010645 thru 0x014d69 (0x4725 bytes) 000512 1788620901.048210 punch from 0x01221e to 0x01a765 (0x8548 bytes) 000514 1788620901.068802 mapread 0x01ed67 thru 0x0258d1 (0x6b6b bytes) 000515 1788620901.089017 mapwrite 0x00c45b thru 0x00cb7d (0x723 bytes) 000517 1788620901.165572 mapwrite 0x018819 thru 0x019a7b (0x1263 bytes) 000518 1788620901.249901 mapread 0x0354f0 thru 0x0374fd (0x200e bytes) 000519 1788620901.263429 read 0x0303d0 thru 0x0322d2 (0x1f03 bytes) 000520 1788620901.264062 punch from 0x01604a to 0x02548f (0xf446 bytes) truncating to largest ever: 0x3f916 000521 1788620901.268927 trunc from 0x040000 to 0x03f915 (0x6ea bytes) 000523 1788620901.304833 read 0x0176e4 thru 0x0183bd (0xcda bytes) 000524 1788620901.321468 write 0x01cc5d thru 0x02a876 (0xdc1a bytes) 000526 1788620901.324637 punch from 0x02c767 to 0x03b5b5 (0xee4f bytes) 000528 1788620901.354834 mapread 0x020339 thru 0x023735 (0x33fd bytes) 000529 1788620901.358165 mapwrite 0x01bdaa thru 0x01efb2 (0x3209 bytes) 000530 1788620901.450526 punch from 0x0158e6 to 0x021c78 (0xc393 bytes) 000531 1788620901.462177 mapwrite 0x024b67 thru 0x0297ec (0x4c86 bytes) 000532 1788620901.617229 MIRROR_RESYNC 10 mirrors 000533 1788620903.656197 trunc from 0x03f916 to 0x01cf2b (0x229ea bytes) 000534 1788620903.854845 punch from 0x009bc0 to 0x00d623 (0x3a64 bytes) 000535 1788620903.866150 trunc from 0x01cf2c to 0x006eef (0x1603c bytes) 000537 1788620903.925136 write 0x02f6e0 thru 0x03ddd7 (0xe6f8 bytes) 000540 1788620903.931422 mapwrite 0x02f811 thru 0x033b8b (0x437b bytes) 000541 1788620904.029256 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ddd8): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddd8): Mirror 8: 0x79cab93f (stale) Mirror 9: 0x79cab93f (stale) Mirror 10: 0x79cab93f (stale) Mirror 11: 0x79cab93f (stale) Mirror 12: 0x79cab93f (stale) Mirror 13: 0x79cab93f (stale) Mirror 14: 0x79cab93f (stale) Mirror 15: 0x79cab93f (stale) Mirror 17: 0x79cab93f (stale) CRC-32 checksum value for chunk [0, 0x3ddd8): Mirror 1: 0x133bb520 Mirror 18: 0x133bb520 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddd8: skipped 000545 1788620905.812317 trunc from 0x03ddd8 to 0x02a36d (0x13a6a bytes) 000546 1788620906.039343 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a36e): Mirror 8: 0x8b1c86f6 (stale) Mirror 9: 0x8b1c86f6 (stale) Mirror 10: 0x8b1c86f6 (stale) Mirror 11: 0x8b1c86f6 (stale) Mirror 12: 0x8b1c86f6 (stale) Mirror 13: 0x8b1c86f6 (stale) Mirror 14: 0x8b1c86f6 (stale) Mirror 15: 0x8b1c86f6 (stale) Mirror 17: 0x8b1c86f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a36e): Mirror 8: 0x8b1c86f6 (stale) Mirror 9: 0x8b1c86f6 (stale) Mirror 10: 0x8b1c86f6 (stale) Mirror 11: 0x8b1c86f6 (stale) Mirror 12: 0x8b1c86f6 (stale) Mirror 13: 0x8b1c86f6 (stale) Mirror 14: 0x8b1c86f6 (stale) Mirror 15: 0x8b1c86f6 (stale) Mirror 17: 0x8b1c86f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a36e: skipped 000548 1788620906.869117 mapread 0x027b3b thru 0x02a36d (0x2833 bytes) 000550 1788620906.926033 read 0x0191fc thru 0x01ddbf (0x4bc4 bytes) 000551 1788620906.943899 mapwrite 0x01c998 thru 0x027f8f (0xb5f8 bytes) 000553 1788620907.035525 trunc from 0x02a36e to 0x037fab (0xdc3e bytes) 000554 1788620907.061462 read 0x0128d4 thru 0x0211da (0xe907 bytes) 000555 1788620907.100747 trunc from 0x037fac to 0x03d28d (0x52e2 bytes) 000556 1788620907.152945 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3d28e): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d28e): Mirror 8: 0x70832693 (stale) Mirror 9: 0x70832693 (stale) Mirror 10: 0x70832693 (stale) Mirror 11: 0x70832693 (stale) Mirror 12: 0x70832693 (stale) Mirror 13: 0x70832693 (stale) Mirror 14: 0x70832693 (stale) Mirror 15: 0x70832693 (stale) Mirror 17: 0x70832693 (stale) CRC-32 checksum value for chunk [0, 0x3d28e): Mirror 1: 0xa6cbcf44 Mirror 18: 0xa6cbcf44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d28e: skipped 000557 1788620909.067184 mapwrite 0x03db4a thru 0x03ffff (0x24b6 bytes) 000558 1788620909.248835 read 0x01eb7a thru 0x022347 (0x37ce bytes) 000560 1788620909.264015 read 0x000f28 thru 0x007f0d (0x6fe6 bytes) 000561 1788620909.302703 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x4dad681b (stale) Mirror 9: 0x4dad681b (stale) Mirror 10: 0x4dad681b (stale) Mirror 11: 0x4dad681b (stale) Mirror 12: 0x4dad681b (stale) Mirror 13: 0x4dad681b (stale) Mirror 14: 0x4dad681b (stale) Mirror 15: 0x4dad681b (stale) Mirror 17: 0x4dad681b (stale) Mirror 18: 0xe23543e0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2656e647 Mirror 19: 0x2656e647 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000562 1788620910.779189 punch from 0x039529 to 0x03ffff (0x6ad7 bytes) 000565 1788620910.901370 punch from 0x024c77 to 0x02cfe3 (0x836d bytes) 000566 1788620910.907607 punch from 0x033fb6 to 0x03e0b2 (0xa0fd bytes) 000567 1788620910.923321 MIRROR_RESYNC 12 mirrors 000568 1788620913.216889 mapwrite 0x03fa4f thru 0x03ffff (0x5b1 bytes) 000571 1788620913.412694 mapread 0x030e29 thru 0x03ffff (0xf1d7 bytes) 000572 1788620913.446808 mapwrite 0x021367 thru 0x02e0a3 (0xcd3d bytes) 000573 1788620913.543861 read 0x03a3d8 thru 0x03ffff (0x5c28 bytes) 000575 1788620913.546336 write 0x00360b thru 0x012e20 (0xf816 bytes) 000577 1788620913.558583 write 0x01625e thru 0x02574d (0xf4f0 bytes) 000578 1788620913.565141 mapwrite 0x011854 thru 0x01be47 (0xa5f4 bytes) 000579 1788620913.641822 mapwrite 0x010d6a thru 0x01fc2f (0xeec6 bytes) 000580 1788620913.711923 punch from 0x020547 to 0x0217cb (0x1285 bytes) 000581 1788620913.717131 read 0x0347e3 thru 0x03534b (0xb69 bytes) 000582 1788620913.717731 read 0x03fa54 thru 0x03ffff (0x5ac bytes) 000583 1788620913.718248 punch from 0x00862e to 0x008cd6 (0x6a9 bytes) 000584 1788620913.726120 trunc from 0x040000 to 0x002a5e (0x3d5a1 bytes) 000586 1788620913.759261 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a5f): Mirror 8: 0xbc692bd9 (stale) Mirror 9: 0xbc692bd9 (stale) Mirror 10: 0xbc692bd9 (stale) Mirror 12: 0xbc692bd9 (stale) Mirror 13: 0xbc692bd9 (stale) Mirror 14: 0xbc692bd9 (stale) Mirror 15: 0xbc692bd9 (stale) Mirror 17: 0xbc692bd9 (stale) Mirror 18: 0xbc692bd9 (stale) Mirror 19: 0xbc692bd9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a5f): Mirror 8: 0xbc692bd9 (stale) Mirror 9: 0xbc692bd9 (stale) Mirror 10: 0xbc692bd9 (stale) Mirror 12: 0xbc692bd9 (stale) Mirror 13: 0xbc692bd9 (stale) Mirror 14: 0xbc692bd9 (stale) Mirror 15: 0xbc692bd9 (stale) Mirror 17: 0xbc692bd9 (stale) Mirror 18: 0xbc692bd9 (stale) Mirror 19: 0xbc692bd9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a5f: skipped 000587 1788620914.279769 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a5f): Mirror 8: 0xbc692bd9 (stale) Mirror 9: 0xbc692bd9 (stale) Mirror 10: 0xbc692bd9 (stale) Mirror 12: 0xbc692bd9 (stale) Mirror 13: 0xbc692bd9 (stale) Mirror 14: 0xbc692bd9 (stale) Mirror 15: 0xbc692bd9 (stale) Mirror 18: 0xbc692bd9 (stale) Mirror 19: 0xbc692bd9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a5f): Mirror 8: 0xbc692bd9 (stale) Mirror 9: 0xbc692bd9 (stale) Mirror 10: 0xbc692bd9 (stale) Mirror 12: 0xbc692bd9 (stale) Mirror 13: 0xbc692bd9 (stale) Mirror 14: 0xbc692bd9 (stale) Mirror 15: 0xbc692bd9 (stale) Mirror 18: 0xbc692bd9 (stale) Mirror 19: 0xbc692bd9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a5f: skipped 000588 1788620914.874948 mapwrite 0x006871 thru 0x009c28 (0x33b8 bytes) 000590 1788620915.042226 punch from 0x000bd0 to 0x009c28 (0x9059 bytes) 000592 1788620915.050797 punch from 0x007e44 to 0x009c28 (0x1de5 bytes) 000593 1788620915.055412 trunc from 0x009c29 to 0x035fc0 (0x2c398 bytes) 000595 1788620915.085387 read 0x02dd5e thru 0x035fc0 (0x8263 bytes) 000597 1788620915.100697 mapwrite 0x031bbd thru 0x03e6f2 (0xcb36 bytes) 000598 1788620915.237945 write 0x01b989 thru 0x0295c9 (0xdc41 bytes) 000601 1788620915.256882 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e6f3): Mirror 8: 0x55c50455 (stale) Mirror 9: 0x55c50455 (stale) Mirror 10: 0x55c50455 (stale) Mirror 12: 0x55c50455 (stale) Mirror 13: 0x55c50455 (stale) Mirror 14: 0x55c50455 (stale) Mirror 18: 0x55c50455 (stale) Mirror 19: 0x55c50455 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6f3): Mirror 8: 0x55c50455 (stale) Mirror 9: 0x55c50455 (stale) Mirror 10: 0x55c50455 (stale) Mirror 12: 0x55c50455 (stale) Mirror 13: 0x55c50455 (stale) Mirror 14: 0x55c50455 (stale) Mirror 18: 0x55c50455 (stale) Mirror 19: 0x55c50455 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6f3: skipped 000602 1788620915.933082 mapwrite 0x017fa1 thru 0x01c2a1 (0x4301 bytes) 000603 1788620916.032797 read 0x01d4d4 thru 0x01f494 (0x1fc1 bytes) 000607 1788620916.042385 write 0x03e43c thru 0x03ffff (0x1bc4 bytes) 000608 1788620916.055436 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xa8071962 (stale) Mirror 9: 0xa8071962 (stale) Mirror 10: 0xa8071962 (stale) Mirror 13: 0xa8071962 (stale) Mirror 14: 0xa8071962 (stale) Mirror 18: 0xa8071962 (stale) Mirror 19: 0xa8071962 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xa8071962 (stale) Mirror 9: 0xa8071962 (stale) Mirror 10: 0xa8071962 (stale) Mirror 13: 0xa8071962 (stale) Mirror 14: 0xa8071962 (stale) Mirror 18: 0xa8071962 (stale) Mirror 19: 0xa8071962 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000609 1788620916.585693 mapread 0x01da61 thru 0x027857 (0x9df7 bytes) 000611 1788620916.615152 trunc from 0x040000 to 0x02cd21 (0x132de bytes) 000615 1788620916.645260 write 0x0073b0 thru 0x010398 (0x8fe9 bytes) 000616 1788620916.669964 punch from 0x004681 to 0x0053b2 (0xd32 bytes) 000617 1788620916.696029 punch from 0x001d7d to 0x010c56 (0xeeda bytes) 000619 1788620916.701364 mapwrite 0x029e2b thru 0x0362a9 (0xc47f bytes) 000620 1788620916.785929 write 0x000eff thru 0x003e80 (0x2f82 bytes) 000621 1788620916.795458 mapread 0x006c0b thru 0x00f43d (0x8833 bytes) 000622 1788620916.806824 trunc from 0x0362aa to 0x0368e6 (0x63d bytes) 000625 1788620916.828467 trunc from 0x0368e7 to 0x021543 (0x153a3 bytes) 000626 1788620916.847856 punch from 0x00c119 to 0x011648 (0x5530 bytes) 000627 1788620916.859530 read 0x00bd90 thru 0x018330 (0xc5a1 bytes) 000628 1788620916.875416 trunc from 0x021544 to 0x01e8d1 (0x2c72 bytes) 000632 1788620916.901254 trunc from 0x01e8d2 to 0x016228 (0x86a9 bytes) 000633 1788620916.935032 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x16229): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16229): Mirror 8: 0x409b6a62 (stale) Mirror 9: 0x409b6a62 (stale) Mirror 10: 0x409b6a62 (stale) Mirror 13: 0x409b6a62 (stale) Mirror 14: 0x409b6a62 (stale) Mirror 18: 0x409b6a62 (stale) Mirror 19: 0x409b6a62 (stale) CRC-32 checksum value for chunk [0, 0x16229): Mirror 1: 0x33bc66ff Mirror 20: 0x33bc66ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16229: skipped 000636 1788620918.188053 punch from 0x00e82f to 0x016228 (0x79fa bytes) 000639 1788620918.272845 mapread 0x010e94 thru 0x016228 (0x5395 bytes) 000641 1788620918.297389 trunc from 0x016229 to 0x0133d7 (0x2e51 bytes) 000643 1788620918.319436 mapwrite 0x001e9a thru 0x0118b1 (0xfa18 bytes) 000644 1788620918.375420 mapwrite 0x010cc9 thru 0x01c8bb (0xbbf3 bytes) 000646 1788620918.451599 write 0x032685 thru 0x03770b (0x5087 bytes) 000648 1788620918.454084 read 0x01dd4e thru 0x02cdc2 (0xf075 bytes) 000649 1788620918.465724 MIRROR_RESYNC 9 mirrors 000650 1788620920.834904 mapread 0x0153bb thru 0x01aa36 (0x567c bytes) 000653 1788620920.858114 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000656 1788620921.268164 MIRROR_RESYNC 8 mirrors 000657 1788620921.915137 write 0x02af2d thru 0x02b647 (0x71b bytes) 000658 1788620922.003580 read 0x029b48 thru 0x03770b (0xdbc4 bytes) 000659 1788620922.014122 trunc from 0x03770c to 0x0397b3 (0x20a8 bytes) 000660 1788620922.036531 read 0x00f976 thru 0x0169cc (0x7057 bytes) 000661 1788620922.058309 mapwrite 0x025dc6 thru 0x02ce54 (0x708f bytes) 000662 1788620922.110528 read 0x01579f thru 0x01ae16 (0x5678 bytes) 000663 1788620922.111219 mapwrite 0x02041a thru 0x0214a1 (0x1088 bytes) 000665 1788620922.143853 trunc from 0x0397b4 to 0x01166f (0x28144 bytes) 000666 1788620922.164882 write 0x01bd3f thru 0x02a410 (0xe6d2 bytes) 000667 1788620922.167529 read 0x0188c5 thru 0x02272b (0x9e67 bytes) 000669 1788620922.178815 trunc from 0x02a411 to 0x019186 (0x1128a bytes) 000670 1788620922.212931 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x19187): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19187): Mirror 8: 0xa81fcfc6 (stale) Mirror 9: 0xa81fcfc6 (stale) Mirror 10: 0xa81fcfc6 (stale) Mirror 13: 0xa81fcfc6 (stale) Mirror 18: 0xa81fcfc6 (stale) Mirror 19: 0xa81fcfc6 (stale) Mirror 20: 0xa81fcfc6 (stale) CRC-32 checksum value for chunk [0, 0x19187): Mirror 1: 0xcf3348e7 Mirror 21: 0xcf3348e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19187: skipped 000671 1788620923.137835 read 0x00e59f thru 0x01799b (0x93fd bytes) 000673 1788620923.166776 read 0x0023a5 thru 0x00d3c9 (0xb025 bytes) 000674 1788620923.203131 MIRROR_RESYNC 9 mirrors 000675 1788620924.915487 MIRROR_EXTEND to 10 mirrors 000676 1788620925.917976 mapread 0x000d12 thru 0x00ee12 (0xe101 bytes) 000677 1788620925.954784 write 0x03da61 thru 0x03ffff (0x259f bytes) 000678 1788620926.109733 punch from 0x03a402 to 0x03ffff (0x5bfe bytes) 000680 1788620926.132372 punch from 0x01c813 to 0x0286c0 (0xbeae bytes) 000682 1788620926.156124 mapwrite 0x02e44b thru 0x0388d3 (0xa489 bytes) 000684 1788620926.399396 mapwrite 0x016bc6 thru 0x01a1e4 (0x361f bytes) 000685 1788620926.462522 trunc from 0x040000 to 0x033766 (0xc899 bytes) 000686 1788620926.507546 mapread 0x02d943 thru 0x033766 (0x5e24 bytes) 000687 1788620926.508580 punch from 0x013769 to 0x02254c (0xede4 bytes) 000688 1788620926.514645 mapwrite 0x006a99 thru 0x0072d4 (0x83c bytes) 000690 1788620926.560103 mapread 0x01a62f thru 0x0223d6 (0x7da8 bytes) 000691 1788620926.569466 trunc from 0x033767 to 0x0164a9 (0x1d2bd bytes) 000692 1788620926.586752 mapread 0x000406 thru 0x008109 (0x7d04 bytes) 000693 1788620926.597755 MIRROR_RESYNC 10 mirrors 000694 1788620928.456509 read 0x00363a thru 0x010c08 (0xd5cf bytes) 000696 1788620928.469451 trunc from 0x0164aa to 0x02cc6c (0x167c3 bytes) 000698 1788620928.558987 MIRROR_RESYNC 10 mirrors 000701 1788620930.663348 trunc from 0x02cc6d to 0x03973d (0xcad1 bytes) 000703 1788620930.780021 read 0x016ee8 thru 0x018790 (0x18a9 bytes) 000704 1788620930.787865 read 0x011a8c thru 0x01f30f (0xd884 bytes) 000705 1788620930.804106 read 0x0092a7 thru 0x00c84b (0x35a5 bytes) 000707 1788620930.805491 mapread 0x02dfc5 thru 0x0381a2 (0xa1de bytes) 000708 1788620930.814595 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3973e): Mirror 8: 0x4639231b (stale) Mirror 9: 0x4639231b (stale) Mirror 10: 0x4639231b (stale) Mirror 13: 0x4639231b (stale) Mirror 18: 0x4639231b (stale) Mirror 20: 0x4639231b (stale) Mirror 21: 0x4639231b (stale) Mirror 22: 0x4639231b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3973e): Mirror 8: 0x4639231b (stale) Mirror 9: 0x4639231b (stale) Mirror 10: 0x4639231b (stale) Mirror 13: 0x4639231b (stale) Mirror 18: 0x4639231b (stale) Mirror 20: 0x4639231b (stale) Mirror 21: 0x4639231b (stale) Mirror 22: 0x4639231b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3973e: skipped 000712 1788620931.653691 mapread 0x033a10 thru 0x035346 (0x1937 bytes) 000713 1788620931.678666 write 0x011bda thru 0x01df48 (0xc36f bytes) 000715 1788620931.689977 mapwrite 0x01e3df thru 0x02836f (0x9f91 bytes) 000716 1788620931.757761 punch from 0x029c71 to 0x02be11 (0x21a1 bytes) 000717 1788620931.767050 MIRROR_RESYNC 9 mirrors 000718 1788620934.305120 read 0x00bb62 thru 0x010bb3 (0x5052 bytes) 000721 1788620934.318055 write 0x03b7ce thru 0x03ffff (0x4832 bytes) 000722 1788620934.415392 read 0x017d16 thru 0x024d53 (0xd03e bytes) 000723 1788620934.434637 mapread 0x022716 thru 0x023046 (0x931 bytes) 000726 1788620934.436424 write 0x0169af thru 0x024b26 (0xe178 bytes) 000727 1788620934.441326 write 0x030103 thru 0x03e728 (0xe626 bytes) 000728 1788620934.443285 write 0x023b4b thru 0x0259c3 (0x1e79 bytes) 000729 1788620934.449962 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xda961dfb (stale) Mirror 9: 0xda961dfb (stale) Mirror 10: 0xda961dfb (stale) Mirror 13: 0xda961dfb (stale) Mirror 18: 0xda961dfb (stale) Mirror 20: 0xda961dfb (stale) Mirror 21: 0xda961dfb (stale) Mirror 22: 0xda961dfb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xde3a0abe Mirror 23: 0xde3a0abe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000730 1788620936.069668 write 0x01ed68 thru 0x026e3b (0x80d4 bytes) 000731 1788620936.243069 mapwrite 0x039512 thru 0x03b21e (0x1d0d bytes) 000732 1788620936.374888 punch from 0x025c24 to 0x02e654 (0x8a31 bytes) 000733 1788620936.380422 mapwrite 0x03b6ef thru 0x03ffff (0x4911 bytes) 000735 1788620936.427487 punch from 0x0179ed to 0x01b331 (0x3945 bytes) 000736 1788620936.433207 trunc from 0x040000 to 0x0203af (0x1fc50 bytes) 000738 1788620936.461347 mapwrite 0x03b673 thru 0x03ffff (0x498d bytes) 000739 1788620936.545921 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xda961dfb (stale) Mirror 9: 0xda961dfb (stale) Mirror 10: 0xda961dfb (stale) Mirror 13: 0xda961dfb (stale) Mirror 18: 0xda961dfb (stale) Mirror 20: 0xda961dfb (stale) Mirror 21: 0xda961dfb (stale) Mirror 22: 0xda961dfb (stale) Mirror 23: 0xde3a0abe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4ebb9777 Mirror 24: 0x4ebb9777 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1788620938.016479 mapread 0x005055 thru 0x00d13b (0x80e7 bytes) 000742 1788620938.135808 MIRROR_RESYNC 11 mirrors 000743 1788620940.197842 punch from 0x031f30 to 0x03e01e (0xc0ef bytes) 000745 1788620940.323405 mapwrite 0x00fdd4 thru 0x019535 (0x9762 bytes) 000746 1788620940.568818 MIRROR_RESYNC 11 mirrors 000748 1788620942.764156 trunc from 0x040000 to 0x033726 (0xc8d9 bytes) 000751 1788620942.888322 punch from 0x00646b to 0x006de7 (0x97d bytes) 000752 1788620942.894962 trunc from 0x033727 to 0x03d843 (0xa11d bytes) 000755 1788620942.931162 write 0x01af6d thru 0x02af32 (0xffc6 bytes) 000756 1788620942.947423 trunc from 0x03d844 to 0x009d90 (0x33ab3 bytes) 000758 1788620942.997343 write 0x0002c3 thru 0x000bd8 (0x916 bytes) 000759 1788620943.005396 punch from 0x0000f6 to 0x009aca (0x99d5 bytes) 000760 1788620943.032033 MIRROR_RESYNC 11 mirrors 000761 1788620944.722564 punch from 0x00841c to 0x009740 (0x1325 bytes) 000763 1788620944.823497 write 0x02066a thru 0x021d9c (0x1733 bytes) 000764 1788620944.825487 read 0x01fc4e thru 0x021d9c (0x214f bytes) 000768 1788620944.835003 punch from 0x00f238 to 0x019f11 (0xacda bytes) 000769 1788620944.845595 write 0x01927c thru 0x024c48 (0xb9cd bytes) 000770 1788620944.852187 mapwrite 0x017970 thru 0x02733a (0xf9cb bytes) 000772 1788620944.977646 mapread 0x020cb8 thru 0x02733a (0x6683 bytes) 000773 1788620944.985309 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2733b): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2733b): Mirror 8: 0x8c3045b5 (stale) Mirror 9: 0x8c3045b5 (stale) Mirror 10: 0x8c3045b5 (stale) Mirror 13: 0x8c3045b5 (stale) Mirror 18: 0x8c3045b5 (stale) Mirror 20: 0x8c3045b5 (stale) Mirror 21: 0x8c3045b5 (stale) Mirror 22: 0x8c3045b5 (stale) Mirror 23: 0x8c3045b5 (stale) Mirror 24: 0x8c3045b5 (stale) CRC-32 checksum value for chunk [0, 0x2733b): Mirror 1: 0xb93c0dec Mirror 25: 0xb93c0dec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2733b: skipped 000774 1788620946.108988 trunc from 0x02733b to 0x00d6c2 (0x19c78 bytes) 000775 1788620946.252301 punch from 0x005d91 to 0x0079a5 (0x1c15 bytes) 000776 1788620946.263035 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0xd6c3): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd6c3): Mirror 8: 0x8b55f5a5 (stale) Mirror 9: 0x8b55f5a5 (stale) Mirror 10: 0x8b55f5a5 (stale) Mirror 13: 0x8b55f5a5 (stale) Mirror 18: 0x8b55f5a5 (stale) Mirror 20: 0x8b55f5a5 (stale) Mirror 21: 0x8b55f5a5 (stale) Mirror 22: 0x8b55f5a5 (stale) Mirror 23: 0x8b55f5a5 (stale) Mirror 24: 0x8b55f5a5 (stale) Mirror 25: 0x8b55f5a5 (stale) CRC-32 checksum value for chunk [0, 0xd6c3): Mirror 1: 0x8b55f5a5 Mirror 26: 0x8b55f5a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd6c3: skipped 000778 1788620947.808259 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0xd6c3): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd6c3): Mirror 8: 0x8b55f5a5 (stale) Mirror 9: 0x8b55f5a5 (stale) Mirror 10: 0x8b55f5a5 (stale) Mirror 13: 0x8b55f5a5 (stale) Mirror 18: 0x8b55f5a5 (stale) Mirror 20: 0x8b55f5a5 (stale) Mirror 21: 0x8b55f5a5 (stale) Mirror 22: 0x8b55f5a5 (stale) Mirror 23: 0x8b55f5a5 (stale) Mirror 24: 0x8b55f5a5 (stale) Mirror 25: 0x8b55f5a5 (stale) CRC-32 checksum value for chunk [0, 0xd6c3): Mirror 1: 0x8b55f5a5 Mirror 26: 0x8b55f5a5 Mirror 27: 0x8b55f5a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd6c3: skipped 000780 1788620949.114114 punch from 0x0063e7 to 0x00667e (0x298 bytes) 000781 1788620949.204468 mapread 0x00082f thru 0x005232 (0x4a04 bytes) 000783 1788620949.237347 mapread 0x00bd97 thru 0x00d6c2 (0x192c bytes) 000784 1788620949.238245 read 0x00483c thru 0x009433 (0x4bf8 bytes) 000786 1788620949.240579 trunc from 0x00d6c3 to 0x022c9f (0x155dd bytes) 000787 1788620949.268136 MIRROR_RESYNC 14 mirrors 000788 1788620952.217379 mapread 0x00e642 thru 0x016ffa (0x89b9 bytes) 000789 1788620952.240403 trunc from 0x022ca0 to 0x00a804 (0x1849b bytes) 000790 1788620952.326979 mapwrite 0x00ad67 thru 0x01557d (0xa817 bytes) 000792 1788620952.437500 punch from 0x005bf8 to 0x00fd55 (0xa15e bytes) 000793 1788620952.444893 read 0x0037f5 thru 0x005e8b (0x2697 bytes) 000794 1788620952.462115 punch from 0x00293c to 0x006748 (0x3e0d bytes) 000795 1788620952.469775 mapread 0x0146e0 thru 0x01557d (0xe9e bytes) 000798 1788620952.471583 write 0x01fa09 thru 0x0288e2 (0x8eda bytes) 000800 1788620952.474046 write 0x021318 thru 0x029f4e (0x8c37 bytes) 000801 1788620952.475491 punch from 0x020d59 to 0x021365 (0x60d bytes) 000802 1788620952.493965 punch from 0x01b395 to 0x021cb1 (0x691d bytes) 000803 1788620952.500280 mapread 0x01ee14 thru 0x024713 (0x5900 bytes) 000804 1788620952.519429 mapwrite 0x00add7 thru 0x00d320 (0x254a bytes) 000805 1788620952.604924 read 0x028c71 thru 0x029f4e (0x12de bytes) 000806 1788620952.605924 write 0x0021a0 thru 0x004a23 (0x2884 bytes) 000807 1788620952.620265 mapwrite 0x02ee5a thru 0x03e282 (0xf429 bytes) 000808 1788620952.736041 write 0x03f01c thru 0x03ffff (0xfe4 bytes) 000811 1788620952.741976 write 0x03994b thru 0x03e454 (0x4b0a bytes) 000813 1788620952.743678 mapwrite 0x03dacc thru 0x03ffff (0x2534 bytes) 000814 1788620952.849254 mapwrite 0x023e48 thru 0x0294a5 (0x565e bytes) 000816 1788620952.984672 trunc from 0x040000 to 0x01d79d (0x22862 bytes) 000818 1788620953.037098 mapread 0x014dd3 thru 0x01d79d (0x89cb bytes) 000820 1788620953.040483 read 0x009c85 thru 0x01378e (0x9b0a bytes) 000821 1788620953.041077 mapwrite 0x016ad7 thru 0x023642 (0xcb6c bytes) 000822 1788620953.200004 MIRROR_RESYNC 14 mirrors 000823 1788620955.708233 punch from 0x0095d8 to 0x00c33a (0x2d63 bytes) 000824 1788620955.839746 punch from 0x0075df to 0x00c7ed (0x520f bytes) 000825 1788620955.846367 mapread 0x00efa4 thru 0x01894b (0x99a8 bytes) 000826 1788620955.870557 read 0x017fb9 thru 0x01abed (0x2c35 bytes) 000827 1788620955.879198 trunc from 0x023643 to 0x01f97b (0x3cc7 bytes) 000828 1788620955.918193 read 0x01c168 thru 0x01f97b (0x3814 bytes) 000830 1788620955.934510 read 0x011fcf thru 0x012b46 (0xb78 bytes) 000831 1788620955.935183 read 0x00e725 thru 0x013ce1 (0x55bd bytes) 000832 1788620955.935726 punch from 0x00d85b to 0x01d36d (0xfb13 bytes) 000834 1788620955.953036 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1f97c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f97c): Mirror 8: 0x5d4885d7 (stale) Mirror 9: 0x5d4885d7 (stale) Mirror 10: 0x5d4885d7 (stale) Mirror 13: 0x5d4885d7 (stale) Mirror 18: 0x5d4885d7 (stale) Mirror 20: 0x5d4885d7 (stale) Mirror 21: 0x5d4885d7 (stale) Mirror 22: 0x5d4885d7 (stale) Mirror 23: 0x5d4885d7 (stale) Mirror 24: 0x5d4885d7 (stale) Mirror 25: 0x5d4885d7 (stale) Mirror 26: 0x5d4885d7 (stale) Mirror 27: 0x5d4885d7 (stale) CRC-32 checksum value for chunk [0, 0x1f97c): Mirror 1: 0xd3ddd125 Mirror 28: 0xd3ddd125 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f97c: skipped 000836 1788620957.299161 MIRROR_RESYNC 15 mirrors 000839 1788620960.000028 trunc from 0x01f97c to 0x0355fc (0x15c81 bytes) 000840 1788620960.223123 punch from 0x0355cb to 0x0355fc (0x32 bytes) 000841 1788620960.227925 trunc from 0x0355fd to 0x02d933 (0x7cc9 bytes) 000842 1788620960.248426 mapwrite 0x03ce98 thru 0x03dad2 (0xc3b bytes) 000843 1788620960.325260 mapwrite 0x002b63 thru 0x00e41e (0xb8bc bytes) 000845 1788620960.400919 read 0x022fc8 thru 0x032674 (0xf6ad bytes) 000846 1788620960.434653 read 0x015229 thru 0x01fad5 (0xa8ad bytes) 000847 1788620960.459474 read 0x030dd3 thru 0x032263 (0x1491 bytes) 000850 1788620960.461040 trunc from 0x03dad3 to 0x029a1e (0x140b4 bytes) 000851 1788620960.506995 trunc from 0x029a1f to 0x03bca3 (0x12285 bytes) 000854 1788620960.538443 trunc from 0x03bca4 to 0x00284d (0x39456 bytes) 000855 1788620960.577479 mapwrite 0x039983 thru 0x03ffff (0x667d bytes) 000856 1788620960.715456 punch from 0x00276e to 0x0065b9 (0x3e4c bytes) 000857 1788620960.725103 punch from 0x015fd3 to 0x01bb5a (0x5b88 bytes) 000858 1788620960.734874 trunc from 0x040000 to 0x0368bd (0x9742 bytes) 000859 1788620960.774244 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x368be): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x368be): Mirror 8: 0xf34fbf49 (stale) Mirror 9: 0xf34fbf49 (stale) Mirror 10: 0xf34fbf49 (stale) Mirror 13: 0xf34fbf49 (stale) Mirror 18: 0xf34fbf49 (stale) Mirror 20: 0xf34fbf49 (stale) Mirror 21: 0xf34fbf49 (stale) Mirror 22: 0xf34fbf49 (stale) Mirror 23: 0xf34fbf49 (stale) Mirror 24: 0xf34fbf49 (stale) Mirror 25: 0xf34fbf49 (stale) Mirror 26: 0xf34fbf49 (stale) Mirror 27: 0xf34fbf49 (stale) Mirror 28: 0xf34fbf49 (stale) CRC-32 checksum value for chunk [0, 0x368be): Mirror 1: 0x7a923173 Mirror 29: 0x7a923173 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x368be: skipped 000860 1788620962.808715 trunc from 0x0368be to 0x0095d6 (0x2d2e7 bytes) 000861 1788620963.054448 write 0x03712d thru 0x03ffff (0x8ed3 bytes) 000863 1788620963.062135 MIRROR_RESYNC 16 mirrors 000864 1788620967.268117 write 0x0054c5 thru 0x00ef46 (0x9a82 bytes) 000865 1788620967.454667 trunc from 0x040000 to 0x02e05b (0x11fa4 bytes) 000866 1788620967.516733 mapwrite 0x01088e thru 0x01cd9f (0xc512 bytes) 000867 1788620967.774277 trunc from 0x02e05c to 0x02af8b (0x30d0 bytes) 000868 1788620967.850358 read 0x0028cc thru 0x00547d (0x2bb2 bytes) 000869 1788620967.865319 mapwrite 0x031aaf thru 0x034d94 (0x32e6 bytes) 000871 1788620967.986009 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34d95): Mirror 9: 0xab8722e0 (stale) Mirror 10: 0xab8722e0 (stale) Mirror 13: 0xab8722e0 (stale) Mirror 18: 0xab8722e0 (stale) Mirror 20: 0xab8722e0 (stale) Mirror 21: 0xab8722e0 (stale) Mirror 22: 0xab8722e0 (stale) Mirror 23: 0xab8722e0 (stale) Mirror 24: 0xab8722e0 (stale) Mirror 25: 0xab8722e0 (stale) Mirror 26: 0xab8722e0 (stale) Mirror 27: 0xab8722e0 (stale) Mirror 28: 0xab8722e0 (stale) Mirror 29: 0xab8722e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d95): Mirror 9: 0xab8722e0 (stale) Mirror 10: 0xab8722e0 (stale) Mirror 13: 0xab8722e0 (stale) Mirror 18: 0xab8722e0 (stale) Mirror 20: 0xab8722e0 (stale) Mirror 21: 0xab8722e0 (stale) Mirror 22: 0xab8722e0 (stale) Mirror 23: 0xab8722e0 (stale) Mirror 24: 0xab8722e0 (stale) Mirror 25: 0xab8722e0 (stale) Mirror 26: 0xab8722e0 (stale) Mirror 27: 0xab8722e0 (stale) Mirror 28: 0xab8722e0 (stale) Mirror 29: 0xab8722e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d95: skipped 000872 1788620969.496001 trunc from 0x034d95 to 0x03aaa8 (0x5d14 bytes) 000873 1788620969.524411 punch from 0x00fd47 to 0x01e423 (0xe6dd bytes) 000874 1788620969.530856 punch from 0x037dff to 0x03aaa8 (0x2caa bytes) 000875 1788620969.537202 read 0x001527 thru 0x00ea4b (0xd525 bytes) 000877 1788620969.549198 mapwrite 0x026ade thru 0x02aa70 (0x3f93 bytes) 000878 1788620969.677098 read 0x008cff thru 0x011d58 (0x905a bytes) 000881 1788620969.694994 mapread 0x03393c thru 0x03aaa8 (0x716d bytes) 000882 1788620969.696842 mapwrite 0x039bfe thru 0x03ffff (0x6402 bytes) 000883 1788620969.907604 MIRROR_RESYNC 15 mirrors 000884 1788620972.667836 trunc from 0x040000 to 0x025839 (0x1a7c6 bytes) 000885 1788620972.823453 mapwrite 0x018d79 thru 0x01f4a2 (0x672a bytes) 000886 1788620972.931254 read 0x023491 thru 0x025839 (0x23a9 bytes) 000887 1788620972.937118 mapwrite 0x02202c thru 0x02664e (0x4623 bytes) 000888 1788620973.013846 trunc from 0x02664f to 0x036c8b (0x1063d bytes) 000890 1788620973.038694 read 0x000596 thru 0x00c00c (0xba77 bytes) 000893 1788620973.050223 punch from 0x01f1c7 to 0x02b799 (0xc5d3 bytes) 000896 1788620973.056787 trunc from 0x036c8c to 0x01f449 (0x17842 bytes) 000897 1788620973.076291 punch from 0x003570 to 0x00d490 (0x9f21 bytes) 000898 1788620973.085168 MIRROR_SPLIT mirror 12 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f44a): Mirror 9: 0x18d92271 (stale) Mirror 10: 0x18d92271 (stale) Mirror 13: 0x18d92271 (stale) Mirror 18: 0x18d92271 (stale) Mirror 20: 0x18d92271 (stale) Mirror 21: 0x18d92271 (stale) Mirror 22: 0x18d92271 (stale) Mirror 23: 0x18d92271 (stale) Mirror 24: 0x18d92271 (stale) Mirror 25: 0x18d92271 (stale) Mirror 26: 0x18d92271 (stale) Mirror 28: 0x18d92271 (stale) Mirror 29: 0x18d92271 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f44a): Mirror 9: 0x18d92271 (stale) Mirror 10: 0x18d92271 (stale) Mirror 13: 0x18d92271 (stale) Mirror 18: 0x18d92271 (stale) Mirror 20: 0x18d92271 (stale) Mirror 21: 0x18d92271 (stale) Mirror 22: 0x18d92271 (stale) Mirror 23: 0x18d92271 (stale) Mirror 24: 0x18d92271 (stale) Mirror 25: 0x18d92271 (stale) Mirror 26: 0x18d92271 (stale) Mirror 28: 0x18d92271 (stale) Mirror 29: 0x18d92271 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f44a: skipped 000899 1788620973.923547 punch from 0x002c76 to 0x006744 (0x3acf bytes) 000900 1788620973.930272 trunc from 0x01f44a to 0x035d99 (0x16950 bytes) 000901 1788620973.950890 trunc from 0x035d9a to 0x01d75a (0x1863f bytes) 000902 1788620973.969068 read 0x00c51d thru 0x00cdc9 (0x8ad bytes) 000903 1788620973.975140 read 0x00170a thru 0x007136 (0x5a2d bytes) 000904 1788620973.988203 trunc from 0x01d75b to 0x00a5bc (0x1319e bytes) 000906 1788620974.015116 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa5bd): Mirror 9: 0x6bb7fa8b (stale) Mirror 10: 0x6bb7fa8b (stale) Mirror 13: 0x6bb7fa8b (stale) Mirror 20: 0x6bb7fa8b (stale) Mirror 21: 0x6bb7fa8b (stale) Mirror 22: 0x6bb7fa8b (stale) Mirror 23: 0x6bb7fa8b (stale) Mirror 24: 0x6bb7fa8b (stale) Mirror 25: 0x6bb7fa8b (stale) Mirror 26: 0x6bb7fa8b (stale) Mirror 28: 0x6bb7fa8b (stale) Mirror 29: 0x6bb7fa8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa5bd): Mirror 9: 0x6bb7fa8b (stale) Mirror 10: 0x6bb7fa8b (stale) Mirror 13: 0x6bb7fa8b (stale) Mirror 20: 0x6bb7fa8b (stale) Mirror 21: 0x6bb7fa8b (stale) Mirror 22: 0x6bb7fa8b (stale) Mirror 23: 0x6bb7fa8b (stale) Mirror 24: 0x6bb7fa8b (stale) Mirror 25: 0x6bb7fa8b (stale) Mirror 26: 0x6bb7fa8b (stale) Mirror 28: 0x6bb7fa8b (stale) Mirror 29: 0x6bb7fa8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa5bd: skipped 000907 1788620974.775531 punch from 0x003371 to 0x005500 (0x2190 bytes) 000908 1788620974.790649 read 0x0082e9 thru 0x00a5bc (0x22d4 bytes) 000909 1788620974.798163 read 0x001707 thru 0x00a5bc (0x8eb6 bytes) 000910 1788620974.810448 trunc from 0x00a5bd to 0x03654b (0x2bf8f bytes) 000911 1788620974.848767 mapwrite 0x02c7b4 thru 0x03414c (0x7999 bytes) 000913 1788620974.952007 mapwrite 0x03c64b thru 0x03ffff (0x39b5 bytes) 000917 1788620975.093774 trunc from 0x040000 to 0x0395a5 (0x6a5a bytes) 000918 1788620975.160529 punch from 0x0233a2 to 0x02fbe0 (0xc83f bytes) 000920 1788620975.170926 write 0x021e52 thru 0x02ee85 (0xd034 bytes) 000922 1788620975.205900 trunc from 0x0395a6 to 0x026b19 (0x12a8c bytes) 000923 1788620975.279712 trunc from 0x026b1a to 0x00adf6 (0x1bd23 bytes) 000924 1788620975.322130 mapwrite 0x02208b thru 0x026233 (0x41a9 bytes) 000925 1788620975.583271 punch from 0x023e80 to 0x026233 (0x23b4 bytes) 000926 1788620975.592129 mapread 0x016bcc thru 0x026233 (0xf668 bytes) 000927 1788620975.800372 write 0x015695 thru 0x01bcf3 (0x665f bytes) 000928 1788620975.809049 mapread 0x011572 thru 0x012e62 (0x18f1 bytes) 000929 1788620975.822914 mapwrite 0x01c3fb thru 0x022292 (0x5e98 bytes) 000931 1788620975.943776 write 0x01de5d thru 0x023670 (0x5814 bytes) 000933 1788620975.961473 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26234): Mirror 9: 0xf5e1ac29 (stale) Mirror 10: 0xf5e1ac29 (stale) Mirror 13: 0xf5e1ac29 (stale) Mirror 20: 0xf5e1ac29 (stale) Mirror 21: 0xf5e1ac29 (stale) Mirror 22: 0xf5e1ac29 (stale) Mirror 23: 0xf5e1ac29 (stale) Mirror 24: 0xf5e1ac29 (stale) Mirror 25: 0xf5e1ac29 (stale) Mirror 26: 0xf5e1ac29 (stale) Mirror 29: 0xf5e1ac29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26234): Mirror 9: 0xf5e1ac29 (stale) Mirror 10: 0xf5e1ac29 (stale) Mirror 13: 0xf5e1ac29 (stale) Mirror 20: 0xf5e1ac29 (stale) Mirror 21: 0xf5e1ac29 (stale) Mirror 22: 0xf5e1ac29 (stale) Mirror 23: 0xf5e1ac29 (stale) Mirror 24: 0xf5e1ac29 (stale) Mirror 25: 0xf5e1ac29 (stale) Mirror 26: 0xf5e1ac29 (stale) Mirror 29: 0xf5e1ac29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26234: skipped 000934 1788620976.828902 write 0x019560 thru 0x01a664 (0x1105 bytes) 000936 1788620976.847399 trunc from 0x026234 to 0x0306e7 (0xa4b4 bytes) 000937 1788620976.907340 mapwrite 0x017f1b thru 0x021321 (0x9407 bytes) 000939 1788620977.051036 mapwrite 0x0150f0 thru 0x0169c6 (0x18d7 bytes) 000940 1788620977.193993 read 0x00e60a thru 0x019eef (0xb8e6 bytes) 000941 1788620977.206146 read 0x00627e thru 0x00bdd8 (0x5b5b bytes) 000942 1788620977.215265 mapread 0x02a627 thru 0x0306e7 (0x60c1 bytes) 000943 1788620977.221674 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x306e8): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x306e8): Mirror 9: 0xbcefae0d (stale) Mirror 10: 0xbcefae0d (stale) Mirror 13: 0xbcefae0d (stale) Mirror 20: 0xbcefae0d (stale) Mirror 21: 0xbcefae0d (stale) Mirror 22: 0xbcefae0d (stale) Mirror 23: 0xbcefae0d (stale) Mirror 24: 0xbcefae0d (stale) Mirror 25: 0xbcefae0d (stale) Mirror 26: 0xbcefae0d (stale) Mirror 29: 0xbcefae0d (stale) CRC-32 checksum value for chunk [0, 0x306e8): Mirror 1: 0xd7f73c1a Mirror 30: 0xd7f73c1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x306e8: skipped 000945 1788620978.537216 mapread 0x00156e thru 0x00f73a (0xe1cd bytes) 000946 1788620978.615229 mapwrite 0x004d80 thru 0x0145dc (0xf85d bytes) 000947 1788620978.985440 mapread 0x02fad2 thru 0x0306e7 (0xc16 bytes) 000948 1788620978.999791 mapread 0x004b67 thru 0x013b6c (0xf006 bytes) 000949 1788620979.000714 write 0x00633e thru 0x011cb9 (0xb97c bytes) 000951 1788620979.006122 read 0x000065 thru 0x0032e4 (0x3280 bytes) 000953 1788620979.015374 read 0x027240 thru 0x02c7cf (0x5590 bytes) 000954 1788620979.033826 write 0x03e864 thru 0x03ffff (0x179c bytes) 000955 1788620979.035433 mapwrite 0x002566 thru 0x00944f (0x6eea bytes) 000957 1788620979.117259 punch from 0x02e94c to 0x030195 (0x184a bytes) 000958 1788620979.124544 mapwrite 0x03d632 thru 0x03ffff (0x29ce bytes) 000959 1788620979.183118 read 0x00bc0b thru 0x00fc2b (0x4021 bytes) 000960 1788620979.183943 punch from 0x003c7a to 0x0075b1 (0x3938 bytes) 000961 1788620979.189067 write 0x007365 thru 0x0170b7 (0xfd53 bytes) 000962 1788620979.199960 read 0x031131 thru 0x033004 (0x1ed4 bytes) 000964 1788620979.201994 mapread 0x039188 thru 0x03ffff (0x6e78 bytes) 000966 1788620979.203832 punch from 0x00623a to 0x00ea0f (0x87d6 bytes) 000967 1788620979.222968 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6458c812 (stale) Mirror 10: 0x6458c812 (stale) Mirror 13: 0x6458c812 (stale) Mirror 20: 0x6458c812 (stale) Mirror 21: 0x6458c812 (stale) Mirror 22: 0x6458c812 (stale) Mirror 24: 0x6458c812 (stale) Mirror 25: 0x6458c812 (stale) Mirror 26: 0x6458c812 (stale) Mirror 29: 0x6458c812 (stale) Mirror 30: 0xcfda1fd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6458c812 (stale) Mirror 10: 0x6458c812 (stale) Mirror 13: 0x6458c812 (stale) Mirror 20: 0x6458c812 (stale) Mirror 21: 0x6458c812 (stale) Mirror 22: 0x6458c812 (stale) Mirror 24: 0x6458c812 (stale) Mirror 25: 0x6458c812 (stale) Mirror 26: 0x6458c812 (stale) Mirror 29: 0x6458c812 (stale) Mirror 30: 0xcfda1fd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000968 1788620979.970542 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6458c812 (stale) Mirror 10: 0x6458c812 (stale) Mirror 13: 0x6458c812 (stale) Mirror 20: 0x6458c812 (stale) Mirror 21: 0x6458c812 (stale) Mirror 24: 0x6458c812 (stale) Mirror 25: 0x6458c812 (stale) Mirror 26: 0x6458c812 (stale) Mirror 29: 0x6458c812 (stale) Mirror 30: 0xcfda1fd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6458c812 (stale) Mirror 10: 0x6458c812 (stale) Mirror 13: 0x6458c812 (stale) Mirror 20: 0x6458c812 (stale) Mirror 21: 0x6458c812 (stale) Mirror 24: 0x6458c812 (stale) Mirror 25: 0x6458c812 (stale) Mirror 26: 0x6458c812 (stale) Mirror 29: 0x6458c812 (stale) Mirror 30: 0xcfda1fd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000969 1788620980.842790 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6458c812 (stale) Mirror 10: 0x6458c812 (stale) Mirror 13: 0x6458c812 (stale) Mirror 20: 0x6458c812 (stale) Mirror 21: 0x6458c812 (stale) Mirror 24: 0x6458c812 (stale) Mirror 25: 0x6458c812 (stale) Mirror 26: 0x6458c812 (stale) Mirror 29: 0x6458c812 (stale) Mirror 30: 0xcfda1fd6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x24ff84b1 Mirror 31: 0x24ff84b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000970 1788620981.982766 mapread 0x014050 thru 0x014184 (0x135 bytes) 000971 1788620982.002997 read 0x012f85 thru 0x015064 (0x20e0 bytes) 000972 1788620982.017591 mapread 0x005bde thru 0x00fa91 (0x9eb4 bytes) 000973 1788620982.047297 write 0x00e478 thru 0x01619f (0x7d28 bytes) 000975 1788620982.171558 trunc from 0x040000 to 0x00226c (0x3dd93 bytes) 000976 1788620982.206207 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x226d): Mirror 9: 0x7b8f763c (stale) Mirror 10: 0x7b8f763c (stale) Mirror 13: 0x7b8f763c (stale) Mirror 20: 0x7b8f763c (stale) Mirror 21: 0x7b8f763c (stale) Mirror 24: 0x7b8f763c (stale) Mirror 25: 0x7b8f763c (stale) Mirror 26: 0x7b8f763c (stale) Mirror 29: 0x7b8f763c (stale) Mirror 30: 0x7b8f763c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x226d): Mirror 9: 0x7b8f763c (stale) Mirror 10: 0x7b8f763c (stale) Mirror 13: 0x7b8f763c (stale) Mirror 20: 0x7b8f763c (stale) Mirror 21: 0x7b8f763c (stale) Mirror 24: 0x7b8f763c (stale) Mirror 25: 0x7b8f763c (stale) Mirror 26: 0x7b8f763c (stale) Mirror 29: 0x7b8f763c (stale) Mirror 30: 0x7b8f763c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x226d: skipped 000977 1788620982.866541 write 0x038551 thru 0x03bab8 (0x3568 bytes) 000978 1788620982.868997 mapread 0x024701 thru 0x02c40e (0x7d0e bytes) 000981 1788620982.889737 read 0x00c6f8 thru 0x014960 (0x8269 bytes) 000982 1788620982.897084 mapread 0x00e619 thru 0x00ffef (0x19d7 bytes) 000983 1788620982.898975 write 0x00933c thru 0x013999 (0xa65e bytes) 000984 1788620982.904186 mapwrite 0x0044d9 thru 0x00952f (0x5057 bytes) 000985 1788620983.006810 write 0x03e5c5 thru 0x03ffff (0x1a3b bytes) 000987 1788620983.013212 punch from 0x00105d to 0x003adb (0x2a7f bytes) 000988 1788620983.029405 mapwrite 0x02cd79 thru 0x039fc3 (0xd24b bytes) 000989 1788620983.114711 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x6458c812 (stale) Mirror 13: 0x6458c812 (stale) Mirror 20: 0x6458c812 (stale) Mirror 21: 0x6458c812 (stale) Mirror 24: 0x6458c812 (stale) Mirror 25: 0x6458c812 (stale) Mirror 26: 0x6458c812 (stale) Mirror 29: 0x6458c812 (stale) Mirror 30: 0xcfda1fd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x6458c812 (stale) Mirror 13: 0x6458c812 (stale) Mirror 20: 0x6458c812 (stale) Mirror 21: 0x6458c812 (stale) Mirror 24: 0x6458c812 (stale) Mirror 25: 0x6458c812 (stale) Mirror 26: 0x6458c812 (stale) Mirror 29: 0x6458c812 (stale) Mirror 30: 0xcfda1fd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000991 1788620983.720370 trunc from 0x040000 to 0x013767 (0x2c898 bytes) 000992 1788620983.760445 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x13768): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13768): Mirror 10: 0x2b2af4a (stale) Mirror 13: 0x2b2af4a (stale) Mirror 20: 0x2b2af4a (stale) Mirror 21: 0x2b2af4a (stale) Mirror 24: 0x2b2af4a (stale) Mirror 25: 0x2b2af4a (stale) Mirror 26: 0x2b2af4a (stale) Mirror 29: 0x2b2af4a (stale) Mirror 30: 0x58260cd0 (stale) CRC-32 checksum value for chunk [0, 0x13768): Mirror 1: 0x652b8f1a Mirror 31: 0x652b8f1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13768: skipped 000993 1788620985.157516 read 0x008873 thru 0x00972c (0xeba bytes) 000995 1788620985.193345 mapread 0x012b8a thru 0x013767 (0xbde bytes) 000996 1788620985.244483 trunc from 0x013768 to 0x01eeb0 (0xb749 bytes) 000997 1788620985.401016 read 0x006118 thru 0x00bc6c (0x5b55 bytes) 000998 1788620985.429932 write 0x028cd2 thru 0x02bdce (0x30fd bytes) 000999 1788620985.434656 read 0x007294 thru 0x00f1e6 (0x7f53 bytes) 001000 1788620985.436088 trunc from 0x02bdcf to 0x0084e0 (0x238ee bytes) All operations completed A-OK! PASS 70a (165s) == sanity-flr test 71: check mirror extend parent fid ==== 11:09:51 (1788620991) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 11:09:55 (1788620995) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.349271 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308473 s, 3.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 11:10:05 (1788621005) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre2 Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4721280 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3113568 .. Extending file size to 5525376 .. Extending file size to 8597344 .. Extending file size to 8552352 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9058688 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8672800 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57420 57421 57423 57425 57426 done 192.168.206.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre2 (opts:) 192.168.206.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:11:29 (1788621089) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre2 Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5722848 .. Extending file size to 1059712 .. Extending file size to 6095616 .. Extending file size to 7118336 .. Extending file size to 8288992 .. Extending file size to 8199680 .. Extending file size to 8614656 .. Extending file size to 9069632 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9017088 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9099072 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58743 58744 58746 58747 58750 done 192.168.206.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre2 (opts:) 192.168.206.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:12:48 (1788621168) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172882 s, 12.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.448694 s, 4.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9ba4dd63 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:13:13 (1788621193) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0834848 s, 25.1 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 11:13:22 (1788621202) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:13:30 (1788621210) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269811 s, 7.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:13:38 (1788621218) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126404 s, 16.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:13:45 (1788621225) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237043 s, 8.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:14:20 (1788621260) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg623-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132436 s, 7.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:15:23 (1788621323) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg623-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169662 s, 12.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:16:15 (1788621375) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg623-server mds-ost sync done. Creating new pool test_204f oleg623-server: Pool lustre.test_204f created Adding targets to pool oleg623-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg623-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142448 s, 14.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg623-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg623-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg623-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:17:29 (1788621449) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:17:37 (1788621457) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:17:45 (1788621465) oleg623-server: Pool lustre.test_206 created Adding targets to pool oleg623-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg623-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg623-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg623-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg623-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:18:10 (1788621490) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882753 s, 11.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314851 s, 33.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3326d4f2 Mirror 3: 0x3326d4f2 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:18:19 (1788621499) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:18:22 (1788621502) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:18:26 (1788621506) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.105022 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2330000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1390000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1390000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1390000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x11e0000 Read this (4 bytes): BBBB 192.168.206.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:19:00 (1788621540) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.037603 s, 218 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:19:07 (1788621547) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513762 s, 20.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x210 lmm_fid: [0x200000404:0x210:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x210 lmm_fid: [0x200000404:0x210:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834448 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 11:19:08.000000000 -0400 Modify: 2026-09-05 11:19:09.000000000 -0400 Change: 2026-09-05 11:19:10.000000000 -0400 Birth: 2026-09-05 11:19:09.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:19:15 (1788621555) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:19:18 (1788621558) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.548756 s, 74.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0520357 s, 78.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 11:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x212:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 11:19 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:19:27 (1788621567) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:19:35 (1788621575) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1796 sec =========== 11:19:44 (1788621584) === sanity-flr: start cleanup 11:19:46 (1788621586) === === sanity-flr: finish cleanup 11:19:52 (1788621592) ===