-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 12:52:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg601-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:52:36 (1788540756) === oleg601-client.virtnet: executing check_config_client /mnt/lustre oleg601-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg601-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8fa28b7d8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8fa28b7d8800.idle_timeout=debug disable quota as required oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:53:12 (1788540792) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8212] == sanity-flr test 0a: lfs mirror create with -N option == 12:53:14 (1788540794) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:53:28 (1788540808) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:53:31 (1788540811) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:53:34 (1788540814) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:53:54 (1788540834) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:53:57 (1788540837) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:54:01 (1788540841) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8fa28b7d8800.stats=clear osc.lustre-OST0001-osc-ffff8fa28b7d8800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:54:08 (1788540848) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:54:17 (1788540857) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:54:25 (1788540865) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:54:29 (1788540869) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:54:36 (1788540876) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 12:54:45 (1788540885) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:54:51 (1788540891) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:54:58 (1788540898) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:55:05 (1788540905) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:55:14 (1788540914) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:55:21 (1788540921) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.554291 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562306 s, 18.6 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:55:29 (1788540929) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.559945 s, 31.8 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:55:37 (1788540937) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:55:46 (1788540946) Stopping /mnt/lustre-ost1 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fa28b7d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fa28b7d8800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg601-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg601-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fa28b7d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fa28b7d8800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg601-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (127s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:57:53 (1788541073) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.135717 s, 38.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fa28b7d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fa28b7d8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg601-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:58:32 (1788541112) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fa28b7d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fa28b7d8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg601-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fa28b7d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fa28b7d8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg601-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (63s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:59:35 (1788541175) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.90667 s, 27.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 12:59 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.852s user 0m0.000s sys 0m0.201s Stopping /mnt/lustre-ost1 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fa28b7d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fa28b7d8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.949s user 0m0.002s sys 0m0.171s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg601-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg601-server oleg601-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fa28b7d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fa28b7d8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.972s user 0m0.001s sys 0m0.169s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg601-server: oleg601-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg601-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg601-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8fa28b7d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fa28b7d8800.ost_server_uuid in FULL state after 0 sec PASS 33b (79s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:00:55 (1788541255) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:00:58 (1788541258) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:01:02 (1788541262) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 13:01:05 (1788541265) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 13:01:12 (1788541272) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.421996 s, 29.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:01:23 (1788541283) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295541 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253987 s, 41.3 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:01:29 (1788541289) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:01:37 (1788541297) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125648 s, 815 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107851 s, 949 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000963751 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00339722 s, 301 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 13:01:46 (1788541306) create test files with size 15732828 .. 172df60dac6395fb93b5ede60c6cd5c9 - 9a515c2650fcd116e14f21ac6ad87150 - f16b2f5088d6578474aa8a87a52a7943 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg601-client.virtnet /mnt/lustre (opts:) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 13:02:06 (1788541326) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:02:21 (1788541341) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:02:28 (1788541348) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651649 s, 16.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355487 s, 29.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 13:02:36 (1788541356) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0995369 s, 21.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.293436 s, 14.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178265 s, 11.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 13:02:51 (1788541371) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 13:02:55 (1788541375) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:02:58 (1788541378) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245997 s, 42.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 13:03:06 (1788541386) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.271212 s, 11.6 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:03:15 (1788541395) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:03:25 (1788541405) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.244171 s, 42.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg601-client.virtnet /mnt/lustre (opts:) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:03:36 (1788541416) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8fa288853800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.351754 s, 29.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg601-client.virtnet /mnt/lustre (opts:) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8fa2853a7000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:03:52 (1788541432) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:04:26 (1788541466) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:04:59 (1788541499) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 13:05:03 (1788541503) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.295663 s, 10.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.19956 s, 15.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xcc34f1d (nosync,stale) Mirror 2: 0xcc34f1d (nosync,stale) ** verify mirror 2 doesn't change original checksum: 82ec14e1ccca9e696941d410b88cadf9 - mirror 1 checksum: 996880d6ce9b425d0f2e7d8a42d26e23 - mirror 2 checksum: 82ec14e1ccca9e696941d410b88cadf9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T13:05:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T13:05:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 82ec14e1ccca9e696941d410b88cadf9 - mirror 1 checksum: 996880d6ce9b425d0f2e7d8a42d26e23 - mirror 2 checksum: 996880d6ce9b425d0f2e7d8a42d26e23 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T13:05:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T13:05:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:05:12 (1788541512) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266412 s, 39.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:05:20 (1788541520) Failing mds1 on oleg601-server Stopping /mnt/lustre-mds1 (opts:) on oleg601-server 13:05:31 (1788541531) shut down facet: mds1 facet_host: oleg601-server facet_failover_host: oleg601-server Failover mds1 to oleg601-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg601-server: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg601-client: oleg601-server: ssh exited with exit code 1 Started lustre-MDT0000 13:05:54 (1788541554) targets are mounted 13:05:54 (1788541554) facet_failover done oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239332 s, 43.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:06:09 (1788541569) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0445804 s, 459 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0229983 s, 891 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:06:18 (1788541578) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0599366 s, 342 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00904848 s, 2.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:06:26 (1788541586) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00671666 s, 2.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:06:33 (1788541593) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0472035 s, 434 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:06:41 (1788541601) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0241941 s, 169 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0309946 s, 132 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:06:49 (1788541609) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:07:09 (1788541629) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:07:36 (1788541656) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:08:00 (1788541680) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:08:24 (1788541704) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 13:08:31 (1788541711) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57551 mirror split pid 57552 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57551 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57552 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 13:09:38 (1788541778) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 1780 000001 1788541780.232497 mapwrite 0x000639 thru 0x00ec79 (0xe641 bytes) 000002 1788541780.329392 mapread 0x00c49e thru 0x00ec79 (0x27dc bytes) 000003 1788541780.336601 MIRROR_EXTEND to 2 mirrors 000004 1788541780.844074 write 0x03d534 thru 0x03ffff (0x2acc bytes) fallocating to largest ever: 0x1cbee 000005 1788541780.873571 fallocate 0x016f51 thru 0x01cbed (0x5c9d bytes) 000006 1788541780.876913 write 0x017b30 thru 0x01ae47 (0x3318 bytes) 000007 1788541780.887938 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5d717ba7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x370915f8 Mirror 3: 0x370915f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000008 1788541781.672442 read 0x01608e thru 0x01734b (0x12be bytes) truncating to largest ever: 0x24e84 000009 1788541781.683007 trunc from 0x040000 to 0x024e83 (0x1b17c bytes) 000011 1788541781.721899 mapread 0x019111 thru 0x020f83 (0x7e73 bytes) 000012 1788541781.737459 punch from 0x01cfa9 to 0x020e63 (0x3ebb bytes) 000014 1788541781.742695 write 0x024ee6 thru 0x028207 (0x3322 bytes) fallocating to largest ever: 0x2c74c 000015 1788541781.747721 fallocate 0x01f64f thru 0x02c74b (0xd0fd bytes) 000016 1788541781.754364 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c74c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c74c): Mirror 2: 0x45748e0f (stale) Mirror 3: 0x4a46415f (stale) CRC-32 checksum value for chunk [0, 0x2c74c): Mirror 1: 0xdf307d0b Mirror 4: 0xdf307d0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c74c: skipped 000017 1788541783.026013 MIRROR_RESYNC 4 mirrors 000018 1788541784.415365 punch from 0x010b84 to 0x012117 (0x1594 bytes) 000019 1788541784.468517 write 0x030958 thru 0x0331d1 (0x287a bytes) 000020 1788541784.469806 punch from 0x01fd8c to 0x02194b (0x1bc0 bytes) 000021 1788541784.485261 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x331d2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x331d2): Mirror 2: 0x85a15da1 (stale) Mirror 3: 0x85a15da1 (stale) Mirror 4: 0x85a15da1 (stale) CRC-32 checksum value for chunk [0, 0x331d2): Mirror 1: 0x367f9418 Mirror 5: 0x367f9418 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x331d2: skipped 000022 1788541785.388107 mapwrite 0x00b09d thru 0x011ecf (0x6e33 bytes) fallocating to largest ever: 0x3ef2d 000023 1788541785.539987 fallocate 0x030f34 thru 0x03ef2c (0xdff9 bytes) 000024 1788541785.552974 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x331d2): Mirror 2: 0x85a15da1 (stale) Mirror 4: 0x85a15da1 (stale) Mirror 5: 0x367f9418 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x331d2): Mirror 2: 0x85a15da1 (stale) Mirror 4: 0x85a15da1 (stale) Mirror 5: 0x367f9418 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x331d2: skipped 000025 1788541785.935360 trunc from 0x0331d2 to 0x035c5e (0x2a8d bytes) 000026 1788541785.960585 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35c5f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c5f): Mirror 2: 0x66851342 (stale) Mirror 4: 0x66851342 (stale) Mirror 5: 0xa1556d9c (stale) CRC-32 checksum value for chunk [0, 0x35c5f): Mirror 1: 0xc37f56f6 Mirror 6: 0xc37f56f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c5f: skipped 000027 1788541786.852011 mapread 0x0234c2 thru 0x023727 (0x266 bytes) 000029 1788541786.866821 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35c5f): Mirror 2: 0x66851342 (stale) Mirror 5: 0xa1556d9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c5f): Mirror 2: 0x66851342 (stale) Mirror 5: 0xa1556d9c (stale) CRC-32 checksum value for chunk [0, 0x35c5f): Mirror 1: 0xc37f56f6 Mirror 6: 0xc37f56f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c5f: skipped 000030 1788541787.563783 trunc from 0x035c5f to 0x02a90f (0xb34f bytes) 000031 1788541787.637375 trunc from 0x01ad5e to 0x028ddb (0xe07e bytes) 000032 1788541787.646329 mapread 0x017c99 thru 0x01825b (0x5c3 bytes) 000033 1788541787.664052 fallocate 0x012309 thru 0x01aa56 (0x874e bytes) 000035 1788541787.673317 write 0x02e2e7 thru 0x03d9c7 (0xf6e1 bytes) 000036 1788541787.676433 mapwrite 0x006238 thru 0x009394 (0x315d bytes) 000037 1788541787.707800 write 0x02a0c8 thru 0x030073 (0x5fac bytes) 000038 1788541787.723816 mapwrite 0x014d59 thru 0x01c1d3 (0x747b bytes) 000039 1788541787.769324 mapwrite 0x00c75c thru 0x010682 (0x3f27 bytes) 000040 1788541787.796032 mapwrite 0x0089aa thru 0x010ec2 (0x8519 bytes) 000041 1788541787.818319 fallocate 0x00cca3 thru 0x01819e (0xb4fc bytes) truncating to largest ever: 0x3ef3f 000042 1788541787.838256 trunc from 0x03d9c8 to 0x03ef3e (0x1577 bytes) 000043 1788541787.853761 trunc from 0x006da5 to 0x00ed2d (0x7f89 bytes) 000044 1788541787.859461 punch from 0x03a797 to 0x03ef3e (0x47a8 bytes) 000045 1788541787.866026 mapread 0x00902d thru 0x0182af (0xf283 bytes) 000046 1788541787.884871 trunc from 0x03ef3f to 0x03496f (0xa5cf bytes) 000047 1788541787.905981 trunc from 0x02c82b to 0x032ee6 (0x66bc bytes) 000048 1788541787.919504 mapwrite 0x0309f3 thru 0x03f6b3 (0xecc1 bytes) 000049 1788541788.072099 fallocate 0x0054a1 thru 0x00eb55 (0x96b5 bytes) 000050 1788541788.090877 trunc from 0x03f6b4 to 0x02fa01 (0xfcb2 bytes) 000052 1788541788.114161 write 0x028afc thru 0x02e4f6 (0x59fb bytes) 000053 1788541788.128659 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2fa02): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa02): Mirror 2: 0xf3892cac (stale) Mirror 5: 0xf3892cac (stale) Mirror 6: 0x57f337ce (stale) CRC-32 checksum value for chunk [0, 0x2fa02): Mirror 1: 0xb0fab5dd Mirror 7: 0xb0fab5dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa02: skipped 000054 1788541789.232715 read 0x02bbe9 thru 0x02fa01 (0x3e19 bytes) 000055 1788541789.245704 mapread 0x00f1ec thru 0x01994d (0xa762 bytes) 000056 1788541789.270782 punch from 0x006a12 to 0x007bf2 (0x11e1 bytes) 000057 1788541789.329531 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2fa02): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa02): Mirror 2: 0xf3892cac (stale) Mirror 5: 0xf3892cac (stale) Mirror 6: 0x57f337ce (stale) Mirror 7: 0xb0fab5dd (stale) CRC-32 checksum value for chunk [0, 0x2fa02): Mirror 1: 0x348a8e5 Mirror 8: 0x348a8e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa02: skipped 000059 1788541790.489124 mapread 0x0081ff thru 0x010d4a (0x8b4c bytes) 000060 1788541790.521410 mapread 0x01c4ba thru 0x02a9c9 (0xe510 bytes) 000061 1788541790.561413 punch from 0x0166a7 to 0x01cb72 (0x64cc bytes) 000062 1788541790.598993 read 0x0076ab thru 0x011d29 (0xa67f bytes) 000063 1788541790.612747 trunc from 0x002e85 to 0x007224 (0x43a0 bytes) 000064 1788541790.623681 mapread 0x00ff70 thru 0x01ae04 (0xae95 bytes) 000066 1788541790.640129 trunc from 0x02fa02 to 0x03ad2b (0xb32a bytes) 000067 1788541790.678368 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ad2c): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad2c): Mirror 2: 0x9cbd64e0 (stale) Mirror 5: 0x94d2a178 (stale) Mirror 6: 0x46712e59 (stale) Mirror 7: 0x39735a36 (stale) Mirror 8: 0x6545ea4f (stale) CRC-32 checksum value for chunk [0, 0x3ad2c): Mirror 1: 0x8c9bb202 Mirror 9: 0x8c9bb202 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad2c: skipped 000068 1788541791.776863 trunc from 0x009e46 to 0x0190bd (0xf278 bytes) 000070 1788541791.812041 read 0x02d132 thru 0x02d8fc (0x7cb bytes) 000071 1788541791.820489 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ad2c): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad2c): Mirror 2: 0x9cbd64e0 (stale) Mirror 5: 0x94d2a178 (stale) Mirror 6: 0x46712e59 (stale) Mirror 7: 0x39735a36 (stale) Mirror 8: 0x6545ea4f (stale) Mirror 9: 0x8c9bb202 (stale) CRC-32 checksum value for chunk [0, 0x3ad2c): Mirror 1: 0xb45b8c45 Mirror 10: 0xb45b8c45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad2c: skipped 000072 1788541792.891136 write 0x03a598 thru 0x03e2f8 (0x3d61 bytes) 000073 1788541792.947951 trunc from 0x03e2f9 to 0x01750f (0x26de9 bytes) 000075 1788541792.985371 mapread 0x000a2d thru 0x010812 (0xfde6 bytes) 000076 1788541793.018632 mapwrite 0x01a9f2 thru 0x027ed9 (0xd4e8 bytes) 000077 1788541793.162898 read 0x002a9f thru 0x00ecd0 (0xc232 bytes) 000078 1788541793.163881 trunc from 0x00d97a to 0x01bdf9 (0xe480 bytes) 000079 1788541793.173652 fallocate 0x02abc9 thru 0x039c75 (0xf0ad bytes) 000080 1788541793.183304 trunc from 0x027eda to 0x005eb5 (0x22024 bytes) 000081 1788541793.233620 trunc from 0x005eb6 to 0x01ea2b (0x18b76 bytes) 000082 1788541793.247929 fallocate 0x027b49 thru 0x037265 (0xf71d bytes) 000083 1788541793.256421 read 0x0184e1 thru 0x01d45a (0x4f7a bytes) 000084 1788541793.290524 trunc from 0x008707 to 0x00f59c (0x6e96 bytes) 000085 1788541793.303021 fallocate 0x02d973 thru 0x03c6c6 (0xed54 bytes) 000086 1788541793.310486 read 0x01c719 thru 0x028401 (0xbce9 bytes) 000087 1788541793.311426 fallocate 0x008de2 thru 0x012815 (0x9a34 bytes) 000088 1788541793.324934 trunc from 0x00c433 to 0x00d030 (0xbfe bytes) 000089 1788541793.333897 trunc from 0x03c6c7 to 0x027851 (0x14e75 bytes) 000090 1788541793.365085 fallocate 0x031d89 thru 0x03289c (0xb14 bytes) 000091 1788541793.377348 trunc from 0x03289d to 0x0001a4 (0x326f8 bytes) 000092 1788541793.397135 write 0x03941f thru 0x03f87b (0x645d bytes) 000093 1788541793.398587 fallocate 0x01eb28 thru 0x02aae9 (0xbfc2 bytes) 000094 1788541793.416553 mapread 0x002dc6 thru 0x011af3 (0xed2e bytes) 000095 1788541793.436203 read 0x0380c7 thru 0x03f87b (0x77b5 bytes) 000096 1788541793.438050 mapread 0x01c2e4 thru 0x01f7ad (0x34ca bytes) 000097 1788541793.453077 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f87c): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f87c): Mirror 2: 0x13d693a1 (stale) Mirror 5: 0xc3ced90d (stale) Mirror 6: 0x8178b02a (stale) Mirror 7: 0x735b438f (stale) Mirror 8: 0x4246d128 (stale) Mirror 9: 0x270819e4 (stale) Mirror 10: 0x1730f522 (stale) CRC-32 checksum value for chunk [0, 0x3f87c): Mirror 1: 0x8e0899ea Mirror 11: 0x8e0899ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f87c: skipped 000098 1788541794.540244 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f87c): Mirror 2: 0x13d693a1 (stale) Mirror 5: 0xc3ced90d (stale) Mirror 6: 0x8178b02a (stale) Mirror 8: 0x4246d128 (stale) Mirror 9: 0x270819e4 (stale) Mirror 10: 0x1730f522 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f87c): Mirror 2: 0x13d693a1 (stale) Mirror 5: 0xc3ced90d (stale) Mirror 6: 0x8178b02a (stale) Mirror 8: 0x4246d128 (stale) Mirror 9: 0x270819e4 (stale) Mirror 10: 0x1730f522 (stale) CRC-32 checksum value for chunk [0, 0x3f87c): Mirror 1: 0x8e0899ea Mirror 11: 0x8e0899ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f87c: skipped 000099 1788541795.457928 read 0x02a17f thru 0x038bff (0xea81 bytes) 000101 1788541795.519763 trunc from 0x02fff3 to 0x033115 (0x3123 bytes) 000103 1788541795.617467 punch from 0x01d482 to 0x01d704 (0x283 bytes) 000104 1788541795.621390 fallocate 0x03466c thru 0x036594 (0x1f29 bytes) 000105 1788541795.635586 mapread 0x00a483 thru 0x00dd29 (0x38a7 bytes) 000106 1788541795.668827 mapread 0x0148ac thru 0x014b76 (0x2cb bytes) 000107 1788541795.669618 trunc from 0x008f29 to 0x00cb26 (0x3bfe bytes) 000108 1788541795.681533 fallocate 0x020ce2 thru 0x0304aa (0xf7c9 bytes) 000109 1788541795.688778 punch from 0x01a519 to 0x024045 (0x9b2d bytes) 000111 1788541795.696829 mapwrite 0x03fc08 thru 0x03ffff (0x3f8 bytes) 000112 1788541795.797889 trunc from 0x040000 to 0x005adf (0x3a520 bytes) 000113 1788541795.862622 trunc from 0x003191 to 0x005adf (0x294f bytes) 000114 1788541795.872079 trunc from 0x005ae0 to 0x00c655 (0x6b76 bytes) 000115 1788541795.916806 fallocate 0x0348c1 thru 0x03ab10 (0x6250 bytes) 000116 1788541795.926083 trunc from 0x02e665 to 0x0385b3 (0x9f4f bytes) 000117 1788541795.930909 trunc from 0x00a393 to 0x0167a8 (0xc416 bytes) 000118 1788541795.938964 mapread 0x014961 thru 0x01ad36 (0x63d6 bytes) 000119 1788541795.953440 read 0x004f58 thru 0x00f85f (0xa908 bytes) 000120 1788541795.964832 read 0x01c4b5 thru 0x020e37 (0x4983 bytes) 000121 1788541795.966504 punch from 0x0196a6 to 0x027bcf (0xe52a bytes) 000122 1788541795.976150 mapwrite 0x011263 thru 0x01a46d (0x920b bytes) 000123 1788541796.027350 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ab11): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab11): Mirror 2: 0x6b9e0a9d (stale) Mirror 5: 0xe3e16f8e (stale) Mirror 6: 0x11669d69 (stale) Mirror 8: 0x619f823d (stale) Mirror 9: 0x5a7e7305 (stale) Mirror 10: 0xc605320b (stale) Mirror 11: 0x5cba4bc9 (stale) CRC-32 checksum value for chunk [0, 0x3ab11): Mirror 1: 0x6a5568f9 Mirror 12: 0x6a5568f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab11: skipped 000124 1788541796.981853 trunc from 0x0290a9 to 0x02c3d0 (0x3328 bytes) 000126 1788541797.027147 punch from 0x01f3b5 to 0x02af30 (0xbb7c bytes) 000127 1788541797.033161 mapread 0x0357db thru 0x03ab10 (0x5336 bytes) 000128 1788541797.051103 write 0x0126a6 thru 0x016b52 (0x44ad bytes) 000129 1788541797.070098 fallocate 0x014b08 thru 0x017cb6 (0x31af bytes) 000130 1788541797.076713 trunc from 0x0186ea to 0x020124 (0x7a3b bytes) 000131 1788541797.081314 trunc from 0x03ab11 to 0x01f059 (0x1bab7 bytes) 000132 1788541797.109660 fallocate 0x00fb9f thru 0x01f181 (0xf5e3 bytes) 000133 1788541797.115805 mapwrite 0x00c200 thru 0x01a200 (0xe001 bytes) 000134 1788541797.161752 read 0x0032e4 thru 0x008653 (0x5370 bytes) 000136 1788541797.172039 read 0x0131af thru 0x018dba (0x5c0c bytes) 000137 1788541797.173606 read 0x01d226 thru 0x01f181 (0x1f5c bytes) 000138 1788541797.174310 mapread 0x002856 thru 0x00a52d (0x7cd8 bytes) 000139 1788541797.181651 fallocate 0x00fc7f thru 0x01427b (0x45fd bytes) 000140 1788541797.187610 trunc from 0x015b28 to 0x018a16 (0x2eef bytes) 000141 1788541797.194193 mapread 0x01ee4c thru 0x01f181 (0x336 bytes) 000143 1788541797.201913 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1f182): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f182): Mirror 2: 0x998e5307 (stale) Mirror 5: 0x998e5307 (stale) Mirror 6: 0xe4318677 (stale) Mirror 8: 0x66047821 (stale) Mirror 9: 0x8e2efcb5 (stale) Mirror 10: 0x6578cb6a (stale) Mirror 11: 0xdd9b8074 (stale) Mirror 12: 0xf193057c (stale) CRC-32 checksum value for chunk [0, 0x1f182): Mirror 1: 0x73f094a7 Mirror 13: 0x73f094a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f182: skipped 000144 1788541798.405642 write 0x015984 thru 0x0222a2 (0xc91f bytes) 000145 1788541798.502007 mapread 0x01baeb thru 0x0222a2 (0x67b8 bytes) 000146 1788541798.503046 punch from 0x01faf7 to 0x0222a2 (0x27ac bytes) 000147 1788541798.521743 mapwrite 0x027cb5 thru 0x03460f (0xc95b bytes) 000148 1788541798.612458 mapread 0x027193 thru 0x033ad5 (0xc943 bytes) 000149 1788541798.613832 fallocate 0x011279 thru 0x01e8a8 (0xd630 bytes) 000150 1788541798.618845 mapread 0x00b570 thru 0x011ea8 (0x6939 bytes) 000151 1788541798.631031 trunc from 0x0232a2 to 0x027cc6 (0x4a25 bytes) 000152 1788541798.636454 write 0x02e9bf thru 0x036d06 (0x8348 bytes) 000153 1788541798.638712 trunc from 0x00456a to 0x008e2f (0x48c6 bytes) 000154 1788541798.644459 mapread 0x01550a thru 0x01a64e (0x5145 bytes) 000155 1788541798.645827 trunc from 0x036d07 to 0x001c8c (0x3507a bytes) 000156 1788541798.661880 trunc from 0x000aef to 0x001c8c (0x119e bytes) 000157 1788541798.667449 mapwrite 0x0145e8 thru 0x018765 (0x417e bytes) 000158 1788541798.794552 trunc from 0x007e5f to 0x011b9c (0x9d3e bytes) 000159 1788541798.799820 read 0x0077fb thru 0x013b31 (0xc337 bytes) 000160 1788541798.816381 mapwrite 0x00006d thru 0x00f17a (0xf10e bytes) 000161 1788541798.883306 punch from 0x006698 to 0x00bf6a (0x58d3 bytes) 000162 1788541798.905592 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x18766): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18766): Mirror 2: 0xdeee53bf (stale) Mirror 5: 0xdeee53bf (stale) Mirror 6: 0xda3864d0 (stale) Mirror 8: 0x54c02585 (stale) Mirror 9: 0x5ff9d2d7 (stale) Mirror 10: 0x6b082b5b (stale) Mirror 11: 0x1dfa717 (stale) Mirror 12: 0x53ff4fae (stale) Mirror 13: 0xca37f723 (stale) CRC-32 checksum value for chunk [0, 0x18766): Mirror 1: 0xf812406d Mirror 14: 0xf812406d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18766: skipped 000163 1788541799.850387 read 0x017985 thru 0x018765 (0xde1 bytes) 000164 1788541799.860346 fallocate 0x0137be thru 0x020027 (0xc86a bytes) 000165 1788541799.914115 mapread 0x01e475 thru 0x020027 (0x1bb3 bytes) 000166 1788541799.932648 trunc from 0x00b3f0 to 0x013167 (0x7d78 bytes) 000167 1788541799.938599 mapread 0x00ca77 thru 0x00fdab (0x3335 bytes) 000168 1788541799.946651 mapread 0x01bfc8 thru 0x01eb0c (0x2b45 bytes) 000169 1788541799.951831 MIRROR_RESYNC 11 mirrors 000170 1788541802.876562 trunc from 0x01a894 to 0x020027 (0x5794 bytes) 000172 1788541802.907516 trunc from 0x020028 to 0x015114 (0xaf13 bytes) 000173 1788541802.923791 read 0x014822 thru 0x015114 (0x8f3 bytes) 000174 1788541802.929076 write 0x03faa4 thru 0x03ffff (0x55c bytes) 000175 1788541802.930481 fallocate 0x022d72 thru 0x031976 (0xec05 bytes) 000176 1788541802.933955 mapread 0x004184 thru 0x009c8c (0x5b09 bytes) 000178 1788541802.953887 read 0x027fa0 thru 0x02d483 (0x54e4 bytes) 000179 1788541802.962493 read 0x0231bb thru 0x02c48c (0x92d2 bytes) 000180 1788541802.966746 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x18545038 (stale) Mirror 5: 0x18545038 (stale) Mirror 8: 0x18545038 (stale) Mirror 9: 0x18545038 (stale) Mirror 10: 0x18545038 (stale) Mirror 11: 0x18545038 (stale) Mirror 12: 0x18545038 (stale) Mirror 13: 0x18545038 (stale) Mirror 14: 0x18545038 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x18545038 (stale) Mirror 5: 0x18545038 (stale) Mirror 8: 0x18545038 (stale) Mirror 9: 0x18545038 (stale) Mirror 10: 0x18545038 (stale) Mirror 11: 0x18545038 (stale) Mirror 12: 0x18545038 (stale) Mirror 13: 0x18545038 (stale) Mirror 14: 0x18545038 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000181 1788541803.467679 mapwrite 0x033143 thru 0x03f386 (0xc244 bytes) 000182 1788541803.544075 trunc from 0x01ec0d to 0x02c8c9 (0xdcbd bytes) 000183 1788541803.548624 mapread 0x012018 thru 0x012667 (0x650 bytes) 000184 1788541803.564307 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x18545038 (stale) Mirror 5: 0x18545038 (stale) Mirror 8: 0x18545038 (stale) Mirror 9: 0x18545038 (stale) Mirror 10: 0x18545038 (stale) Mirror 11: 0x18545038 (stale) Mirror 12: 0x18545038 (stale) Mirror 13: 0x18545038 (stale) Mirror 14: 0x18545038 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7569b71e Mirror 15: 0x7569b71e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000185 1788541804.589068 trunc from 0x040000 to 0x02d69e (0x12961 bytes) 000186 1788541804.644418 trunc from 0x00a22f to 0x01527e (0xb050 bytes) 000187 1788541804.650082 trunc from 0x01517a to 0x01ce67 (0x7cee bytes) 000188 1788541804.659616 read 0x00ea8d thru 0x01b6aa (0xcc1e bytes) 000189 1788541804.670376 write 0x00adf0 thru 0x00bf4f (0x1160 bytes) 000190 1788541804.682081 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2d69f): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d69f): Mirror 2: 0x5e45079a (stale) Mirror 5: 0x5e45079a (stale) Mirror 8: 0x5e45079a (stale) Mirror 9: 0x5e45079a (stale) Mirror 10: 0x5e45079a (stale) Mirror 11: 0x5e45079a (stale) Mirror 12: 0x5e45079a (stale) Mirror 13: 0x5e45079a (stale) Mirror 14: 0x5e45079a (stale) Mirror 15: 0x2421ca55 (stale) CRC-32 checksum value for chunk [0, 0x2d69f): Mirror 1: 0x2b538c74 Mirror 16: 0x2b538c74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d69f: skipped 000191 1788541805.536362 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2d69f): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d69f): Mirror 2: 0x5e45079a (stale) Mirror 5: 0x5e45079a (stale) Mirror 8: 0x5e45079a (stale) Mirror 9: 0x5e45079a (stale) Mirror 10: 0x5e45079a (stale) Mirror 11: 0x5e45079a (stale) Mirror 12: 0x5e45079a (stale) Mirror 13: 0x5e45079a (stale) Mirror 14: 0x5e45079a (stale) Mirror 15: 0x2421ca55 (stale) CRC-32 checksum value for chunk [0, 0x2d69f): Mirror 1: 0x2b538c74 Mirror 16: 0x2b538c74 Mirror 17: 0x2b538c74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d69f: skipped 000192 1788541806.482645 mapwrite 0x0104a8 thru 0x01bc8f (0xb7e8 bytes) 000193 1788541806.625651 trunc from 0x02d69f to 0x033d36 (0x6698 bytes) 000194 1788541806.672526 trunc from 0x01d86d to 0x0223e7 (0x4b7b bytes) 000195 1788541806.679707 trunc from 0x02c96e to 0x033d36 (0x73c9 bytes) 000196 1788541806.701713 MIRROR_RESYNC 13 mirrors 000197 1788541809.031608 fallocate 0x02c015 thru 0x037910 (0xb8fc bytes) 000198 1788541809.066688 trunc from 0x037911 to 0x0115fa (0x26316 bytes) 000199 1788541809.088277 read 0x001b8a thru 0x005681 (0x3af8 bytes) 000201 1788541809.100070 read 0x004371 thru 0x006ff8 (0x2c88 bytes) 000202 1788541809.115068 mapread 0x00ca18 thru 0x00e36f (0x1958 bytes) 000203 1788541809.116000 trunc from 0x006c7b to 0x00ff82 (0x9308 bytes) 000204 1788541809.121016 write 0x019718 thru 0x0258c4 (0xc1ad bytes) 000205 1788541809.122953 write 0x00a9c8 thru 0x012242 (0x787b bytes) 000206 1788541809.128987 trunc from 0x024afb to 0x0258c4 (0xdca bytes) 000207 1788541809.144525 mapwrite 0x00eb30 thru 0x019455 (0xa926 bytes) 000208 1788541809.163340 mapwrite 0x0353f0 thru 0x03566a (0x27b bytes) 000210 1788541809.194641 trunc from 0x03566b to 0x02350a (0x12160 bytes) 000211 1788541809.214162 MIRROR_RESYNC 13 mirrors 000212 1788541810.858690 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) 000215 1788541811.355198 trunc from 0x02350b to 0x019aa1 (0x9a69 bytes) 000216 1788541811.392791 fallocate 0x01c51b thru 0x01dd27 (0x180d bytes) 000217 1788541811.396647 mapread 0x01bf36 thru 0x01dd27 (0x1df2 bytes) 000218 1788541811.417506 read 0x00fa71 thru 0x01dd27 (0xe2b7 bytes) 000219 1788541811.424704 trunc from 0x01428d to 0x01dd27 (0x9a9b bytes) 000220 1788541811.429512 fallocate 0x02e1ee thru 0x02f8dc (0x16ef bytes) 000221 1788541811.432732 punch from 0x006c18 to 0x00e0c5 (0x74ae bytes) 000222 1788541811.435949 punch from 0x00de7a to 0x00ea62 (0xbe9 bytes) 000223 1788541811.440667 mapread 0x017b16 thru 0x01dd27 (0x6212 bytes) 000224 1788541811.447087 mapread 0x00a9f9 thru 0x017722 (0xcd2a bytes) 000225 1788541811.456064 read 0x012014 thru 0x019c62 (0x7c4f bytes) 000226 1788541811.457332 write 0x019918 thru 0x023927 (0xa010 bytes) 000227 1788541811.459581 trunc from 0x023928 to 0x01d4e4 (0x6443 bytes) 000228 1788541811.474041 mapread 0x00db54 thru 0x015a68 (0x7f15 bytes) 000229 1788541811.475106 read 0x00e0fa thru 0x00fd87 (0x1c8e bytes) 000230 1788541811.476443 mapwrite 0x0262ce thru 0x02d827 (0x755a bytes) 000231 1788541811.519955 fallocate 0x01a180 thru 0x027119 (0xcf9a bytes) 000232 1788541811.531019 fallocate 0x03061b thru 0x0382ad (0x7c93 bytes) 000233 1788541811.538162 read 0x009176 thru 0x016643 (0xd4ce bytes) 000234 1788541811.541660 mapwrite 0x03c0c9 thru 0x03ffff (0x3f37 bytes) 000235 1788541811.594025 write 0x0296a4 thru 0x02f4b6 (0x5e13 bytes) 000236 1788541811.596647 write 0x02dde1 thru 0x034e6b (0x708b bytes) 000238 1788541811.600096 trunc from 0x0093de to 0x00aa56 (0x1679 bytes) 000239 1788541811.604834 mapwrite 0x02bb5e thru 0x02ed64 (0x3207 bytes) 000241 1788541811.632098 MIRROR_RESYNC 12 mirrors 000242 1788541815.133770 write 0x01ebcf thru 0x024c08 (0x603a bytes) 000243 1788541815.180649 trunc from 0x028f01 to 0x02975c (0x85c bytes) 000244 1788541815.196158 write 0x01062d thru 0x01866b (0x803f bytes) 000245 1788541815.205713 trunc from 0x040000 to 0x03b5fe (0x4a01 bytes) 000246 1788541815.222614 fallocate 0x02570d thru 0x026e1a (0x170e bytes) 000247 1788541815.227214 trunc from 0x03b5ff to 0x0030ef (0x3850f bytes) 000248 1788541815.242636 mapread 0x000d11 thru 0x0030ef (0x23df bytes) 000249 1788541815.258828 write 0x03f568 thru 0x03ffff (0xa98 bytes) 000250 1788541815.259889 trunc from 0x040000 to 0x00c02b (0x33fd4 bytes) 000251 1788541815.273619 trunc from 0x004ebc to 0x00c02b (0x7170 bytes) 000252 1788541815.283932 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0xc02c): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc02c): Mirror 2: 0xc901b544 (stale) Mirror 5: 0xc901b544 (stale) Mirror 8: 0xc901b544 (stale) Mirror 9: 0xc901b544 (stale) Mirror 11: 0xc901b544 (stale) Mirror 12: 0xc901b544 (stale) Mirror 13: 0xc901b544 (stale) Mirror 14: 0xc901b544 (stale) Mirror 15: 0xc901b544 (stale) Mirror 16: 0xc901b544 (stale) Mirror 17: 0xc901b544 (stale) CRC-32 checksum value for chunk [0, 0xc02c): Mirror 1: 0x5d0e36c3 Mirror 18: 0x5d0e36c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc02c: skipped 000253 1788541816.146638 write 0x03306f thru 0x03ffff (0xcf91 bytes) 000254 1788541816.225338 mapwrite 0x03bc44 thru 0x03ffff (0x43bc bytes) 000255 1788541816.261480 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7db9b1ad (stale) Mirror 5: 0x7db9b1ad (stale) Mirror 8: 0x7db9b1ad (stale) Mirror 9: 0x7db9b1ad (stale) Mirror 11: 0x7db9b1ad (stale) Mirror 12: 0x7db9b1ad (stale) Mirror 13: 0x7db9b1ad (stale) Mirror 14: 0x7db9b1ad (stale) Mirror 15: 0x7db9b1ad (stale) Mirror 16: 0x7db9b1ad (stale) Mirror 17: 0x7db9b1ad (stale) Mirror 18: 0x54db4060 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2eb91405 Mirror 19: 0x2eb91405 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000256 1788541817.272921 trunc from 0x019643 to 0x0273d1 (0xdd8f bytes) 000257 1788541817.312062 read 0x00eaec thru 0x0160dd (0x75f2 bytes) 000258 1788541817.321269 fallocate 0x01770c thru 0x021536 (0x9e2b bytes) 000259 1788541817.327864 mapwrite 0x00277e thru 0x00fe99 (0xd71c bytes) 000260 1788541817.383762 mapwrite 0x0294b5 thru 0x0384f7 (0xf043 bytes) 000261 1788541817.426543 mapwrite 0x00d909 thru 0x00f071 (0x1769 bytes) 000262 1788541817.442368 MIRROR_RESYNC 14 mirrors 000263 1788541819.367525 trunc from 0x01fc7e to 0x023141 (0x34c4 bytes) 000264 1788541819.401086 mapread 0x013740 thru 0x021ee2 (0xe7a3 bytes) 000265 1788541819.419975 trunc from 0x01231a to 0x01dcb9 (0xb9a0 bytes) 000266 1788541819.423416 trunc from 0x01bcc4 to 0x0204ac (0x47e9 bytes) 000267 1788541819.427441 fallocate 0x021cde thru 0x0297dd (0x7b00 bytes) 000268 1788541819.431494 trunc from 0x031ad1 to 0x03a531 (0x8a61 bytes) 000269 1788541819.434686 mapwrite 0x028842 thru 0x03541f (0xcbde bytes) 000270 1788541819.473416 mapread 0x015ad9 thru 0x025a9f (0xffc7 bytes) 000271 1788541819.510089 MIRROR_RESYNC 14 mirrors 000272 1788541821.399989 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) 000273 1788541821.912888 fallocate 0x03bc1e thru 0x03c881 (0xc64 bytes) 000274 1788541821.980344 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcad3c4c (stale) Mirror 5: 0xcad3c4c (stale) Mirror 8: 0xcad3c4c (stale) Mirror 9: 0xcad3c4c (stale) Mirror 11: 0xcad3c4c (stale) Mirror 12: 0xcad3c4c (stale) Mirror 13: 0xcad3c4c (stale) Mirror 15: 0xcad3c4c (stale) Mirror 16: 0xcad3c4c (stale) Mirror 17: 0xcad3c4c (stale) Mirror 19: 0xcad3c4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcad3c4c (stale) Mirror 5: 0xcad3c4c (stale) Mirror 8: 0xcad3c4c (stale) Mirror 9: 0xcad3c4c (stale) Mirror 11: 0xcad3c4c (stale) Mirror 12: 0xcad3c4c (stale) Mirror 13: 0xcad3c4c (stale) Mirror 15: 0xcad3c4c (stale) Mirror 16: 0xcad3c4c (stale) Mirror 17: 0xcad3c4c (stale) Mirror 19: 0xcad3c4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000275 1788541822.443916 trunc from 0x040000 to 0x02826a (0x17d95 bytes) 000276 1788541822.457979 fallocate 0x029891 thru 0x02fbf8 (0x6368 bytes) 000277 1788541822.467627 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2826b): Mirror 2: 0x4452e76b (stale) Mirror 5: 0x4452e76b (stale) Mirror 8: 0x4452e76b (stale) Mirror 11: 0x4452e76b (stale) Mirror 12: 0x4452e76b (stale) Mirror 13: 0x4452e76b (stale) Mirror 15: 0x4452e76b (stale) Mirror 16: 0x4452e76b (stale) Mirror 17: 0x4452e76b (stale) Mirror 19: 0x4452e76b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2826b): Mirror 2: 0x4452e76b (stale) Mirror 5: 0x4452e76b (stale) Mirror 8: 0x4452e76b (stale) Mirror 11: 0x4452e76b (stale) Mirror 12: 0x4452e76b (stale) Mirror 13: 0x4452e76b (stale) Mirror 15: 0x4452e76b (stale) Mirror 16: 0x4452e76b (stale) Mirror 17: 0x4452e76b (stale) Mirror 19: 0x4452e76b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2826b: skipped 000278 1788541822.954016 mapwrite 0x015b2c thru 0x01e35e (0x8833 bytes) 000279 1788541822.989920 mapread 0x019cb6 thru 0x02089c (0x6be7 bytes) 000280 1788541823.004035 mapwrite 0x0377f9 thru 0x03b69e (0x3ea6 bytes) 000281 1788541823.089288 mapwrite 0x015b44 thru 0x0182c3 (0x2780 bytes) 000282 1788541823.113738 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b69f): Mirror 2: 0x406c0070 (stale) Mirror 5: 0x406c0070 (stale) Mirror 8: 0x406c0070 (stale) Mirror 11: 0x406c0070 (stale) Mirror 12: 0x406c0070 (stale) Mirror 13: 0x406c0070 (stale) Mirror 15: 0x406c0070 (stale) Mirror 16: 0x406c0070 (stale) Mirror 19: 0x406c0070 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b69f): Mirror 2: 0x406c0070 (stale) Mirror 5: 0x406c0070 (stale) Mirror 8: 0x406c0070 (stale) Mirror 11: 0x406c0070 (stale) Mirror 12: 0x406c0070 (stale) Mirror 13: 0x406c0070 (stale) Mirror 15: 0x406c0070 (stale) Mirror 16: 0x406c0070 (stale) Mirror 19: 0x406c0070 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b69f: skipped 000283 1788541823.719768 punch from 0x006e94 to 0x0115a7 (0xa714 bytes) 000285 1788541823.734144 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b69f): Mirror 2: 0x406c0070 (stale) Mirror 5: 0x406c0070 (stale) Mirror 8: 0x406c0070 (stale) Mirror 11: 0x406c0070 (stale) Mirror 13: 0x406c0070 (stale) Mirror 15: 0x406c0070 (stale) Mirror 16: 0x406c0070 (stale) Mirror 19: 0x406c0070 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b69f): Mirror 2: 0x406c0070 (stale) Mirror 5: 0x406c0070 (stale) Mirror 8: 0x406c0070 (stale) Mirror 11: 0x406c0070 (stale) Mirror 13: 0x406c0070 (stale) Mirror 15: 0x406c0070 (stale) Mirror 16: 0x406c0070 (stale) Mirror 19: 0x406c0070 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b69f: skipped 000286 1788541824.264082 trunc from 0x03b69f to 0x014f20 (0x2677e bytes) 000287 1788541824.283728 punch from 0x011417 to 0x014f20 (0x3b0a bytes) 000288 1788541824.287097 write 0x035fbc thru 0x03ca4a (0x6a8f bytes) 000289 1788541824.289892 mapread 0x01536c thru 0x01f7ec (0xa481 bytes) 000290 1788541824.302339 write 0x03dca2 thru 0x03e07b (0x3da bytes) 000291 1788541824.306641 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e07c): Mirror 2: 0x48f0e981 (stale) Mirror 5: 0x48f0e981 (stale) Mirror 8: 0x48f0e981 (stale) Mirror 11: 0x48f0e981 (stale) Mirror 15: 0x48f0e981 (stale) Mirror 16: 0x48f0e981 (stale) Mirror 19: 0x48f0e981 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e07c): Mirror 2: 0x48f0e981 (stale) Mirror 5: 0x48f0e981 (stale) Mirror 8: 0x48f0e981 (stale) Mirror 11: 0x48f0e981 (stale) Mirror 15: 0x48f0e981 (stale) Mirror 16: 0x48f0e981 (stale) Mirror 19: 0x48f0e981 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e07c: skipped 000292 1788541824.722828 punch from 0x0011e8 to 0x001ad6 (0x8ef bytes) 000293 1788541824.730788 punch from 0x033dda to 0x03e07b (0xa2a2 bytes) 000294 1788541824.734791 read 0x036dd4 thru 0x03e07b (0x72a8 bytes) 000295 1788541824.743259 trunc from 0x03e07c to 0x00ccdf (0x3139c bytes) 000296 1788541824.765854 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xcce0): Mirror 5: 0x20e09745 (stale) Mirror 8: 0x20e09745 (stale) Mirror 11: 0x20e09745 (stale) Mirror 15: 0x20e09745 (stale) Mirror 16: 0x20e09745 (stale) Mirror 19: 0x20e09745 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcce0): Mirror 5: 0x20e09745 (stale) Mirror 8: 0x20e09745 (stale) Mirror 11: 0x20e09745 (stale) Mirror 15: 0x20e09745 (stale) Mirror 16: 0x20e09745 (stale) Mirror 19: 0x20e09745 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcce0: skipped 000297 1788541825.089977 write 0x0253f7 thru 0x02ec75 (0x987f bytes) 000299 1788541825.096822 mapwrite 0x007aa3 thru 0x00f268 (0x77c6 bytes) 000301 1788541825.137021 mapread 0x006e99 thru 0x0136d7 (0xc83f bytes) 000302 1788541825.144902 trunc from 0x00f4b7 to 0x00fe50 (0x99a bytes) 000303 1788541825.149110 mapread 0x01828a thru 0x01cca5 (0x4a1c bytes) 000304 1788541825.158100 trunc from 0x02c6dc to 0x02ec75 (0x259a bytes) 000305 1788541825.166776 trunc from 0x02b9ad to 0x02dd43 (0x2397 bytes) 000306 1788541825.178868 trunc from 0x02ec76 to 0x010920 (0x1e355 bytes) 000307 1788541825.196404 mapwrite 0x02d19b thru 0x02ebbc (0x1a22 bytes) 000308 1788541825.254476 fallocate 0x028efd thru 0x03039a (0x749e bytes) 000309 1788541825.267708 mapread 0x01f269 thru 0x02dedb (0xec73 bytes) 000310 1788541825.294576 read 0x011fc6 thru 0x019d5c (0x7d97 bytes) 000312 1788541825.306803 trunc from 0x03039b to 0x01cc8b (0x1370f bytes) 000313 1788541825.323130 trunc from 0x00479f to 0x00e234 (0x9a96 bytes) 000314 1788541825.329016 read 0x0034d7 thru 0x011dcd (0xe8f7 bytes) 000315 1788541825.344059 write 0x023455 thru 0x0298f2 (0x649e bytes) 000316 1788541825.346006 fallocate 0x03e47c thru 0x03e74c (0x2d1 bytes) 000317 1788541825.355388 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x298f3): Mirror 5: 0x442ebfcd (stale) Mirror 8: 0x442ebfcd (stale) Mirror 15: 0x442ebfcd (stale) Mirror 16: 0x442ebfcd (stale) Mirror 19: 0x442ebfcd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x298f3): Mirror 5: 0x442ebfcd (stale) Mirror 8: 0x442ebfcd (stale) Mirror 15: 0x442ebfcd (stale) Mirror 16: 0x442ebfcd (stale) Mirror 19: 0x442ebfcd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x298f3: skipped 000318 1788541825.889035 trunc from 0x0298f3 to 0x036272 (0xc980 bytes) 000319 1788541825.924721 punch from 0x032142 to 0x035f97 (0x3e56 bytes) 000320 1788541825.931766 mapread 0x004a6f thru 0x0107ad (0xbd3f bytes) 000321 1788541825.984841 punch from 0x030c51 to 0x036272 (0x5622 bytes) 000322 1788541826.010596 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36273): Mirror 5: 0xda133093 (stale) Mirror 15: 0xda133093 (stale) Mirror 16: 0xda133093 (stale) Mirror 19: 0xda133093 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36273): Mirror 5: 0xda133093 (stale) Mirror 15: 0xda133093 (stale) Mirror 16: 0xda133093 (stale) Mirror 19: 0xda133093 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36273: skipped 000323 1788541826.428092 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36273): Mirror 15: 0xda133093 (stale) Mirror 16: 0xda133093 (stale) Mirror 19: 0xda133093 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36273): Mirror 15: 0xda133093 (stale) Mirror 16: 0xda133093 (stale) Mirror 19: 0xda133093 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36273: skipped 000324 1788541826.726721 trunc from 0x004ac1 to 0x00afaa (0x64ea bytes) 000325 1788541826.734935 trunc from 0x01167b to 0x01f433 (0xddb9 bytes) truncating to largest ever: 0x3f592 000326 1788541826.749739 trunc from 0x036273 to 0x03f591 (0x931f bytes) 000327 1788541826.808755 punch from 0x025646 to 0x02fdf8 (0xa7b3 bytes) 000328 1788541826.826045 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f592): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f592): Mirror 15: 0x5e9c24aa (stale) Mirror 16: 0x5e9c24aa (stale) Mirror 19: 0x5e9c24aa (stale) CRC-32 checksum value for chunk [0, 0x3f592): Mirror 1: 0x31bf404b Mirror 20: 0x31bf404b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f592: skipped 000329 1788541827.845938 mapwrite 0x02e757 thru 0x030bc3 (0x246d bytes) 000330 1788541827.939028 mapread 0x03bb77 thru 0x03f591 (0x3a1b bytes) 000331 1788541827.948955 write 0x02f39c thru 0x038f97 (0x9bfc bytes) 000332 1788541827.955051 mapread 0x01e7a7 thru 0x025d45 (0x759f bytes) fallocating to largest ever: 0x40000 000333 1788541827.965615 fallocate 0x03b161 thru 0x03ffff (0x4e9f bytes) 000334 1788541827.969829 trunc from 0x021b91 to 0x024385 (0x27f5 bytes) 000335 1788541827.973411 trunc from 0x02374f to 0x0295fc (0x5eae bytes) 000336 1788541827.979136 read 0x00a664 thru 0x0194de (0xee7b bytes) 000337 1788541827.992006 write 0x00a6d8 thru 0x00aaa4 (0x3cd bytes) 000338 1788541827.993448 trunc from 0x040000 to 0x03883e (0x77c1 bytes) 000339 1788541828.027508 punch from 0x0053a3 to 0x00d178 (0x7dd6 bytes) 000340 1788541828.058366 fallocate 0x02e357 thru 0x03df1a (0xfbc4 bytes) 000341 1788541828.072629 trunc from 0x03df1b to 0x0153e8 (0x28b32 bytes) 000342 1788541828.098118 read 0x00ac3f thru 0x0153e8 (0xa7aa bytes) 000343 1788541828.115484 write 0x03a80a thru 0x03ffff (0x57f6 bytes) 000344 1788541828.122255 trunc from 0x040000 to 0x005d15 (0x3a2ea bytes) 000345 1788541828.150847 punch from 0x002ee5 to 0x005689 (0x27a5 bytes) 000346 1788541828.158659 trunc from 0x005d16 to 0x00a609 (0x48f4 bytes) 000347 1788541828.195633 read 0x002515 thru 0x003fed (0x1ad9 bytes) 000348 1788541828.212919 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xa60a): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa60a): Mirror 15: 0xe67df71 (stale) Mirror 16: 0xe67df71 (stale) Mirror 19: 0xe67df71 (stale) Mirror 20: 0x381da86f (stale) CRC-32 checksum value for chunk [0, 0xa60a): Mirror 1: 0x9c33846b Mirror 21: 0x9c33846b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa60a: skipped 000349 1788541829.085030 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa60a): Mirror 15: 0xe67df71 (stale) Mirror 16: 0xe67df71 (stale) Mirror 20: 0x381da86f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa60a): Mirror 15: 0xe67df71 (stale) Mirror 16: 0xe67df71 (stale) Mirror 20: 0x381da86f (stale) CRC-32 checksum value for chunk [0, 0xa60a): Mirror 1: 0x9c33846b Mirror 21: 0x9c33846b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa60a: skipped 000350 1788541829.637022 trunc from 0x00a60a to 0x03f9ce (0x353c5 bytes) 000351 1788541829.758002 fallocate 0x034694 thru 0x03ffff (0xb96c bytes) 000352 1788541829.767656 MIRROR_RESYNC 5 mirrors 000353 1788541830.695454 read 0x01260b thru 0x01c0c8 (0x9abe bytes) 000354 1788541830.701850 write 0x01ced0 thru 0x01f995 (0x2ac6 bytes) 000355 1788541830.740293 trunc from 0x03f9cf to 0x0104a4 (0x2f52a bytes) 000356 1788541830.755142 mapread 0x00b703 thru 0x0104a4 (0x4da2 bytes) 000357 1788541830.772031 trunc from 0x00c0f7 to 0x00f42b (0x3335 bytes) 000358 1788541830.778761 trunc from 0x0104a5 to 0x031b37 (0x21693 bytes) 000359 1788541830.792274 mapread 0x021408 thru 0x02918c (0x7d85 bytes) 000360 1788541830.802296 punch from 0x00a66d to 0x00e03d (0x39d1 bytes) 000361 1788541830.806681 read 0x023d7f thru 0x02585f (0x1ae1 bytes) 000362 1788541830.820857 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31b38): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b38): Mirror 15: 0x2f750fad (stale) Mirror 16: 0x2f750fad (stale) Mirror 20: 0x2f750fad (stale) Mirror 21: 0x2f750fad (stale) CRC-32 checksum value for chunk [0, 0x31b38): Mirror 1: 0x2f750fad Mirror 22: 0x2f750fad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b38: skipped 000363 1788541831.836097 mapread 0x0281b2 thru 0x031b37 (0x9986 bytes) 000364 1788541831.865973 fallocate 0x0290e5 thru 0x034276 (0xb192 bytes) 000365 1788541831.899471 mapwrite 0x011906 thru 0x0135a0 (0x1c9b bytes) 000366 1788541831.925016 fallocate 0x034e79 thru 0x03ffff (0xb187 bytes) 000367 1788541831.937028 trunc from 0x000b87 to 0x00d36d (0xc7e7 bytes) 000368 1788541831.941740 punch from 0x0161e1 to 0x017b1b (0x193b bytes) 000369 1788541831.945641 punch from 0x016f7e to 0x01fa47 (0x8aca bytes) 000370 1788541831.949908 read 0x03170d thru 0x0365b6 (0x4eaa bytes) 000371 1788541831.960355 trunc from 0x040000 to 0x022552 (0x1daad bytes) 000372 1788541831.988927 fallocate 0x034e98 thru 0x0354ae (0x617 bytes) 000373 1788541831.996799 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22553): Mirror 15: 0x32646b5e (stale) Mirror 20: 0x32646b5e (stale) Mirror 21: 0x32646b5e (stale) Mirror 22: 0x32646b5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22553): Mirror 15: 0x32646b5e (stale) Mirror 20: 0x32646b5e (stale) Mirror 21: 0x32646b5e (stale) Mirror 22: 0x32646b5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22553: skipped 000374 1788541832.341845 trunc from 0x00048b to 0x004a03 (0x4579 bytes) 000375 1788541832.347078 trunc from 0x022553 to 0x0092a3 (0x192af bytes) 000376 1788541832.365274 trunc from 0x0092a4 to 0x02bf74 (0x22cd1 bytes) 000377 1788541832.384692 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2bf75): Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf75): Mirror 15: 0x16e812ad (stale) Mirror 20: 0x16e812ad (stale) Mirror 21: 0x16e812ad (stale) Mirror 22: 0x16e812ad (stale) CRC-32 checksum value for chunk [0, 0x2bf75): Mirror 1: 0xbf751335 Mirror 23: 0xbf751335 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf75: skipped 000378 1788541833.264466 fallocate 0x03c268 thru 0x03ffff (0x3d98 bytes) 000379 1788541833.291940 trunc from 0x01837d to 0x01dc2b (0x58af bytes) 000380 1788541833.297624 write 0x0351ea thru 0x038242 (0x3059 bytes) 000381 1788541833.325375 fallocate 0x01e17b thru 0x024831 (0x66b7 bytes) 000382 1788541833.340022 trunc from 0x040000 to 0x02cbdb (0x13424 bytes) 000383 1788541833.375809 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2cbdc): Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cbdc): Mirror 15: 0xb43a773e (stale) Mirror 20: 0xb43a773e (stale) Mirror 21: 0xb43a773e (stale) Mirror 22: 0xb43a773e (stale) Mirror 23: 0xffb908f1 (stale) CRC-32 checksum value for chunk [0, 0x2cbdc): Mirror 1: 0xffb908f1 Mirror 24: 0xffb908f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cbdc: skipped 000385 1788541834.275745 mapwrite 0x03adbc thru 0x03ffff (0x5244 bytes) 000386 1788541834.360754 trunc from 0x040000 to 0x039b0b (0x64f4 bytes) 000387 1788541834.397386 punch from 0x0060cc to 0x012283 (0xc1b8 bytes) 000389 1788541834.408314 read 0x016001 thru 0x016e9b (0xe9b bytes) 000390 1788541834.418880 trunc from 0x00ef60 to 0x010916 (0x19b7 bytes) 000391 1788541834.423267 mapwrite 0x008b6a thru 0x00970f (0xba6 bytes) 000392 1788541834.450782 read 0x024621 thru 0x030c69 (0xc649 bytes) 000394 1788541834.460413 trunc from 0x0077b6 to 0x00ecc6 (0x7511 bytes) 000395 1788541834.470981 mapwrite 0x00f29e thru 0x017a9d (0x8800 bytes) 000396 1788541834.504715 mapwrite 0x01f2b0 thru 0x02972e (0xa47f bytes) 000397 1788541834.534586 trunc from 0x039b0c to 0x03d8b2 (0x3da7 bytes) 000399 1788541834.561006 read 0x010de0 thru 0x01deeb (0xd10c bytes) 000401 1788541834.564303 fallocate 0x017eba thru 0x02101e (0x9165 bytes) 000402 1788541834.568975 trunc from 0x03d8b3 to 0x023f4f (0x19963 bytes) 000403 1788541834.589775 mapread 0x001f15 thru 0x008c3a (0x6d26 bytes) 000404 1788541834.599783 write 0x03e5e7 thru 0x03ffff (0x1a19 bytes) 000405 1788541834.601364 fallocate 0x031a7a thru 0x037403 (0x598a bytes) 000406 1788541834.605866 trunc from 0x040000 to 0x021080 (0x1ef7f bytes) 000408 1788541834.621803 punch from 0x005703 to 0x00915f (0x3a5d bytes) 000409 1788541834.626327 punch from 0x001753 to 0x00f151 (0xd9ff bytes) 000410 1788541834.631599 trunc from 0x021081 to 0x0335b1 (0x12531 bytes) 000411 1788541834.649437 MIRROR_RESYNC 7 mirrors 000412 1788541836.174301 trunc from 0x0335b2 to 0x0381e3 (0x4c32 bytes) 000413 1788541836.252027 trunc from 0x0381e4 to 0x03d950 (0x576d bytes) 000414 1788541836.272054 fallocate 0x03f591 thru 0x03ffff (0xa6f bytes) 000415 1788541836.280744 mapread 0x01f772 thru 0x028b79 (0x9408 bytes) 000416 1788541836.309064 read 0x015ad8 thru 0x016c4d (0x1176 bytes) 000417 1788541836.317879 MIRROR_RESYNC 7 mirrors 000418 1788541837.950698 mapwrite 0x03a45c thru 0x03ffff (0x5ba4 bytes) 000419 1788541838.034882 fallocate 0x0385d4 thru 0x039dc1 (0x17ee bytes) 000420 1788541838.046460 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xb80dbd63 (stale) Mirror 20: 0xb80dbd63 (stale) Mirror 21: 0xb80dbd63 (stale) Mirror 22: 0xb80dbd63 (stale) Mirror 23: 0xb80dbd63 (stale) Mirror 24: 0xb80dbd63 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x63943c88 Mirror 25: 0x63943c88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000421 1788541839.016283 mapwrite 0x02d5d5 thru 0x03778a (0xa1b6 bytes) 000422 1788541839.164819 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xb80dbd63 (stale) Mirror 21: 0xb80dbd63 (stale) Mirror 22: 0xb80dbd63 (stale) Mirror 23: 0xb80dbd63 (stale) Mirror 24: 0xb80dbd63 (stale) Mirror 25: 0x63943c88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xb80dbd63 (stale) Mirror 21: 0xb80dbd63 (stale) Mirror 22: 0xb80dbd63 (stale) Mirror 23: 0xb80dbd63 (stale) Mirror 24: 0xb80dbd63 (stale) Mirror 25: 0x63943c88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000423 1788541839.680361 fallocate 0x031284 thru 0x03ffff (0xed7c bytes) 000424 1788541839.686403 mapread 0x035a83 thru 0x03c5c2 (0x6b40 bytes) 000425 1788541839.701739 trunc from 0x006704 to 0x013c45 (0xd542 bytes) 000426 1788541839.706305 punch from 0x005e47 to 0x015088 (0xf242 bytes) 000427 1788541839.711057 punch from 0x00a33e to 0x01730b (0xcfce bytes) 000428 1788541839.715242 mapread 0x0318ef thru 0x03c5e0 (0xacf2 bytes) 000429 1788541839.716203 punch from 0x0012de to 0x009984 (0x86a7 bytes) 000430 1788541839.720014 trunc from 0x040000 to 0x01d85a (0x227a5 bytes) 000431 1788541839.742349 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1d85b): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d85b): Mirror 20: 0xf6097e39 (stale) Mirror 21: 0xf6097e39 (stale) Mirror 22: 0xf6097e39 (stale) Mirror 23: 0xf6097e39 (stale) Mirror 24: 0xf6097e39 (stale) Mirror 25: 0xf6097e39 (stale) CRC-32 checksum value for chunk [0, 0x1d85b): Mirror 1: 0xfa079ece Mirror 26: 0xfa079ece lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d85b: skipped 000432 1788541840.616566 punch from 0x018c54 to 0x01d85a (0x4c07 bytes) 000433 1788541840.660071 mapread 0x0082ff thru 0x008603 (0x305 bytes) 000434 1788541840.665669 mapread 0x00fd76 thru 0x01c3ae (0xc639 bytes) 000435 1788541840.684459 mapwrite 0x0399dd thru 0x03f23a (0x585e bytes) 000436 1788541840.743604 mapwrite 0x025e63 thru 0x02e2ec (0x848a bytes) 000437 1788541840.775368 trunc from 0x03f23b to 0x006b14 (0x38726 bytes) 000438 1788541840.794948 mapwrite 0x0253c0 thru 0x02b6c6 (0x6307 bytes) 000439 1788541840.842601 fallocate 0x018904 thru 0x01ed9a (0x6497 bytes) 000440 1788541840.850751 trunc from 0x02b6c7 to 0x01b40e (0x102b8 bytes) 000441 1788541840.872802 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1b40f): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b40f): Mirror 20: 0x7a3ec775 (stale) Mirror 21: 0x7a3ec775 (stale) Mirror 22: 0x7a3ec775 (stale) Mirror 23: 0x7a3ec775 (stale) Mirror 24: 0x7a3ec775 (stale) Mirror 25: 0x7a3ec775 (stale) Mirror 26: 0xcbb18a0d (stale) CRC-32 checksum value for chunk [0, 0x1b40f): Mirror 1: 0x708cc15c Mirror 27: 0x708cc15c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b40f: skipped 000442 1788541842.010459 fallocate 0x03c84e thru 0x03ffff (0x37b2 bytes) 000445 1788541842.062359 punch from 0x037112 to 0x0381cc (0x10bb bytes) 000446 1788541842.067203 mapread 0x00d1e7 thru 0x01a329 (0xd143 bytes) 000447 1788541842.104997 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xb80dbd63 (stale) Mirror 21: 0xb80dbd63 (stale) Mirror 22: 0xb80dbd63 (stale) Mirror 23: 0xb80dbd63 (stale) Mirror 24: 0xb80dbd63 (stale) Mirror 25: 0x63943c88 (stale) Mirror 26: 0x1965135c (stale) Mirror 27: 0xab8c4586 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xab8c4586 Mirror 28: 0xab8c4586 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000448 1788541842.996230 punch from 0x034e98 to 0x0391b2 (0x431b bytes) 000449 1788541843.058823 punch from 0x030556 to 0x03f4d3 (0xef7e bytes) 000450 1788541843.072129 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xb80dbd63 (stale) Mirror 22: 0xb80dbd63 (stale) Mirror 23: 0xb80dbd63 (stale) Mirror 24: 0xb80dbd63 (stale) Mirror 25: 0x63943c88 (stale) Mirror 26: 0x1965135c (stale) Mirror 27: 0xab8c4586 (stale) Mirror 28: 0xab8c4586 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xb80dbd63 (stale) Mirror 22: 0xb80dbd63 (stale) Mirror 23: 0xb80dbd63 (stale) Mirror 24: 0xb80dbd63 (stale) Mirror 25: 0x63943c88 (stale) Mirror 26: 0x1965135c (stale) Mirror 27: 0xab8c4586 (stale) Mirror 28: 0xab8c4586 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000451 1788541843.704904 write 0x030ffa thru 0x032e4e (0x1e55 bytes) 000452 1788541843.725944 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xb80dbd63 (stale) Mirror 22: 0xb80dbd63 (stale) Mirror 23: 0xb80dbd63 (stale) Mirror 24: 0xb80dbd63 (stale) Mirror 25: 0x63943c88 (stale) Mirror 26: 0x1965135c (stale) Mirror 27: 0xab8c4586 (stale) Mirror 28: 0xab8c4586 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc8323ab3 Mirror 29: 0xc8323ab3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000453 1788541845.129393 mapread 0x004f21 thru 0x014273 (0xf353 bytes) 000454 1788541845.184312 punch from 0x03c3cf to 0x03ffff (0x3c31 bytes) 000455 1788541845.223506 punch from 0x026bbe to 0x027ffc (0x143f bytes) 000456 1788541845.227353 mapwrite 0x011b96 thru 0x016c94 (0x50ff bytes) 000458 1788541845.250977 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xb80dbd63 (stale) Mirror 22: 0xb80dbd63 (stale) Mirror 23: 0xb80dbd63 (stale) Mirror 24: 0xb80dbd63 (stale) Mirror 25: 0x63943c88 (stale) Mirror 26: 0x1965135c (stale) Mirror 27: 0xab8c4586 (stale) Mirror 28: 0xab8c4586 (stale) Mirror 29: 0xc8323ab3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3eb03ff5 Mirror 30: 0x3eb03ff5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000459 1788541846.076997 write 0x0282bc thru 0x035ece (0xdc13 bytes) 000460 1788541846.117803 write 0x02859b thru 0x0342a2 (0xbd08 bytes) 000463 1788541846.120797 write 0x00a37d thru 0x00e1b7 (0x3e3b bytes) 000465 1788541846.128208 read 0x0303c5 thru 0x0385b9 (0x81f5 bytes) 000466 1788541846.133781 mapwrite 0x011632 thru 0x020037 (0xea06 bytes) 000467 1788541846.186580 fallocate 0x01e7e6 thru 0x027391 (0x8bac bytes) 000468 1788541846.198670 read 0x019baf thru 0x01c7e3 (0x2c35 bytes) 000469 1788541846.199580 trunc from 0x040000 to 0x03fc7b (0x384 bytes) 000471 1788541846.257721 trunc from 0x0262f0 to 0x035893 (0xf5a4 bytes) 000472 1788541846.264410 trunc from 0x03fc7c to 0x003904 (0x3c377 bytes) 000473 1788541846.288275 read 0x001bc0 thru 0x002c71 (0x10b2 bytes) 000474 1788541846.299655 fallocate 0x012475 thru 0x01a010 (0x7b9c bytes) 000475 1788541846.305683 fallocate 0x0028eb thru 0x006711 (0x3e27 bytes) 000476 1788541846.317529 mapwrite 0x03035d thru 0x038cc4 (0x8968 bytes) 000477 1788541846.381832 fallocate 0x01739b thru 0x018edf (0x1b45 bytes) 000478 1788541846.388929 punch from 0x00d0b4 to 0x00e787 (0x16d4 bytes) 000479 1788541846.398322 mapread 0x011189 thru 0x01d639 (0xc4b1 bytes) 000480 1788541846.405422 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38cc5): Mirror 22: 0x2c4fa90 (stale) Mirror 23: 0x2c4fa90 (stale) Mirror 24: 0x2c4fa90 (stale) Mirror 25: 0x2c4fa90 (stale) Mirror 26: 0xf14bc4f4 (stale) Mirror 27: 0x709d1a0b (stale) Mirror 28: 0x709d1a0b (stale) Mirror 29: 0xbb1552ee (stale) Mirror 30: 0xe369619e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cc5): Mirror 22: 0x2c4fa90 (stale) Mirror 23: 0x2c4fa90 (stale) Mirror 24: 0x2c4fa90 (stale) Mirror 25: 0x2c4fa90 (stale) Mirror 26: 0xf14bc4f4 (stale) Mirror 27: 0x709d1a0b (stale) Mirror 28: 0x709d1a0b (stale) Mirror 29: 0xbb1552ee (stale) Mirror 30: 0xe369619e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cc5: skipped 000481 1788541846.866267 MIRROR_RESYNC 10 mirrors 000482 1788541848.477070 trunc from 0x008ebd to 0x01578a (0xc8ce bytes) 000483 1788541848.530219 fallocate 0x02c24c thru 0x0379cb (0xb780 bytes) 000484 1788541848.536458 mapwrite 0x02bba8 thru 0x02fcac (0x4105 bytes) 000485 1788541848.619592 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38cc5): Mirror 22: 0x4f418dae (stale) Mirror 24: 0x4f418dae (stale) Mirror 25: 0x4f418dae (stale) Mirror 26: 0x4f418dae (stale) Mirror 27: 0x4f418dae (stale) Mirror 28: 0x4f418dae (stale) Mirror 29: 0x4f418dae (stale) Mirror 30: 0x4f418dae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cc5): Mirror 22: 0x4f418dae (stale) Mirror 24: 0x4f418dae (stale) Mirror 25: 0x4f418dae (stale) Mirror 26: 0x4f418dae (stale) Mirror 27: 0x4f418dae (stale) Mirror 28: 0x4f418dae (stale) Mirror 29: 0x4f418dae (stale) Mirror 30: 0x4f418dae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cc5: skipped 000486 1788541849.264377 read 0x01a81b thru 0x01ac1f (0x405 bytes) 000488 1788541849.280022 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38cc5): Mirror 22: 0x4f418dae (stale) Mirror 24: 0x4f418dae (stale) Mirror 26: 0x4f418dae (stale) Mirror 27: 0x4f418dae (stale) Mirror 28: 0x4f418dae (stale) Mirror 29: 0x4f418dae (stale) Mirror 30: 0x4f418dae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cc5): Mirror 22: 0x4f418dae (stale) Mirror 24: 0x4f418dae (stale) Mirror 26: 0x4f418dae (stale) Mirror 27: 0x4f418dae (stale) Mirror 28: 0x4f418dae (stale) Mirror 29: 0x4f418dae (stale) Mirror 30: 0x4f418dae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cc5: skipped 000489 1788541849.787634 fallocate 0x017a25 thru 0x02295e (0xaf3a bytes) 000491 1788541849.800446 write 0x0167c9 thru 0x01dc72 (0x74aa bytes) 000492 1788541849.823460 write 0x03b743 thru 0x03f1c4 (0x3a82 bytes) 000493 1788541849.827782 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f1c5): Mirror 22: 0x4c0bb9c3 (stale) Mirror 24: 0x4c0bb9c3 (stale) Mirror 26: 0x4c0bb9c3 (stale) Mirror 27: 0x4c0bb9c3 (stale) Mirror 29: 0x4c0bb9c3 (stale) Mirror 30: 0x4c0bb9c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1c5): Mirror 22: 0x4c0bb9c3 (stale) Mirror 24: 0x4c0bb9c3 (stale) Mirror 26: 0x4c0bb9c3 (stale) Mirror 27: 0x4c0bb9c3 (stale) Mirror 29: 0x4c0bb9c3 (stale) Mirror 30: 0x4c0bb9c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1c5: skipped 000495 1788541850.291736 fallocate 0x039116 thru 0x03ffff (0x6eea bytes) 000496 1788541850.296909 punch from 0x00ed9d to 0x01d59b (0xe7ff bytes) 000497 1788541850.302615 punch from 0x0116de to 0x015edf (0x4802 bytes) 000498 1788541850.306893 mapwrite 0x03ca56 thru 0x03ffff (0x35aa bytes) 000499 1788541850.368418 trunc from 0x02f67f to 0x031eb0 (0x2832 bytes) 000500 1788541850.374012 trunc from 0x040000 to 0x03fb39 (0x4c6 bytes) 000501 1788541850.386372 fallocate 0x03b762 thru 0x03ec50 (0x34ef bytes) 000502 1788541850.391868 mapread 0x005b59 thru 0x00b816 (0x5cbe bytes) 000503 1788541850.405660 punch from 0x0244c2 to 0x0299a5 (0x54e4 bytes) 000504 1788541850.409478 fallocate 0x033a35 thru 0x039ef7 (0x64c3 bytes) 000505 1788541850.413733 punch from 0x0079f5 to 0x0106b8 (0x8cc4 bytes) 000507 1788541850.419443 punch from 0x018c8d to 0x01ae33 (0x21a7 bytes) 000508 1788541850.422983 fallocate 0x03fc15 thru 0x03ffff (0x3eb bytes) 000509 1788541850.427495 read 0x00c318 thru 0x0185fb (0xc2e4 bytes) 000510 1788541850.435254 fallocate 0x023a3b thru 0x02fd57 (0xc31d bytes) 000511 1788541850.438887 mapread 0x013cd6 thru 0x01d9ff (0x9d2a bytes) 000512 1788541850.447384 fallocate 0x0060ec thru 0x008696 (0x25ab bytes) 000513 1788541850.451479 punch from 0x032eab to 0x03e9ea (0xbb40 bytes) 000514 1788541850.459793 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe8cab30e (stale) Mirror 24: 0xe8cab30e (stale) Mirror 26: 0xe8cab30e (stale) Mirror 27: 0xe8cab30e (stale) Mirror 30: 0xe8cab30e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe8cab30e (stale) Mirror 24: 0xe8cab30e (stale) Mirror 26: 0xe8cab30e (stale) Mirror 27: 0xe8cab30e (stale) Mirror 30: 0xe8cab30e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1788541850.983847 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe8cab30e (stale) Mirror 24: 0xe8cab30e (stale) Mirror 26: 0xe8cab30e (stale) Mirror 30: 0xe8cab30e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe8cab30e (stale) Mirror 24: 0xe8cab30e (stale) Mirror 26: 0xe8cab30e (stale) Mirror 30: 0xe8cab30e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000516 1788541851.496482 mapwrite 0x019cc8 thru 0x0220ff (0x8438 bytes) 000517 1788541851.539999 punch from 0x018d0c to 0x020305 (0x75fa bytes) 000518 1788541851.544632 mapwrite 0x038303 thru 0x03cab5 (0x47b3 bytes) 000519 1788541851.575912 mapwrite 0x030959 thru 0x037432 (0x6ada bytes) 000520 1788541851.634160 read 0x018100 thru 0x0191da (0x10db bytes) 000521 1788541851.642821 fallocate 0x003eae thru 0x011601 (0xd754 bytes) 000522 1788541851.651592 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe8cab30e (stale) Mirror 24: 0xe8cab30e (stale) Mirror 26: 0xe8cab30e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe8cab30e (stale) Mirror 24: 0xe8cab30e (stale) Mirror 26: 0xe8cab30e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000523 1788541851.945594 fallocate 0x01c282 thru 0x01e7a5 (0x2524 bytes) 000524 1788541851.949895 read 0x00d9c0 thru 0x0114f2 (0x3b33 bytes) 000525 1788541851.958671 trunc from 0x040000 to 0x008463 (0x37b9c bytes) 000528 1788541851.984849 punch from 0x002209 to 0x008463 (0x625b bytes) 000529 1788541851.989682 read 0x007c7e thru 0x008463 (0x7e6 bytes) 000530 1788541851.998875 punch from 0x002087 to 0x008463 (0x63dd bytes) 000531 1788541852.003613 fallocate 0x023043 thru 0x0315f7 (0xe5b5 bytes) 000532 1788541852.007704 punch from 0x02d708 to 0x0315f7 (0x3ef0 bytes) 000533 1788541852.012594 punch from 0x01b1eb to 0x02a6e9 (0xf4ff bytes) 000534 1788541852.022548 read 0x01e674 thru 0x023b7b (0x5508 bytes) 000536 1788541852.037306 trunc from 0x02a12c to 0x030f93 (0x6e68 bytes) 000537 1788541852.045678 mapwrite 0x02298f thru 0x023fe4 (0x1656 bytes) 000538 1788541852.059385 mapread 0x00881d thru 0x011458 (0x8c3c bytes) 000539 1788541852.060383 trunc from 0x0315f8 to 0x015957 (0x1bca0 bytes) 000540 1788541852.078080 read 0x00ba08 thru 0x00c6e4 (0xcdd bytes) 000541 1788541852.078887 mapwrite 0x02c526 thru 0x039823 (0xd2fe bytes) 000542 1788541852.165370 punch from 0x02951e to 0x0365be (0xd0a1 bytes) 000543 1788541852.170401 trunc from 0x039824 to 0x02991f (0xff04 bytes) 000544 1788541852.194879 mapwrite 0x02f871 thru 0x03a2ea (0xaa7a bytes) 000545 1788541852.274629 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a2eb): Mirror 22: 0x273f530e (stale) Mirror 26: 0x273f530e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2eb): Mirror 22: 0x273f530e (stale) Mirror 26: 0x273f530e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2eb: skipped 000546 1788541852.549788 trunc from 0x03a2eb to 0x0071c3 (0x33127 bytes) 000547 1788541852.569475 mapread 0x004289 thru 0x006bd7 (0x294f bytes) 000548 1788541852.587102 fallocate 0x031f8f thru 0x035ef5 (0x3f67 bytes) 000549 1788541852.592389 mapwrite 0x00759f thru 0x00ee02 (0x7864 bytes) 000550 1788541852.619485 trunc from 0x00e663 to 0x01a921 (0xc2bf bytes) 000551 1788541852.628125 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35ef6): Mirror 22: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ef6): Mirror 22: 0xa294e97a (stale) Mirror 26: 0xa294e97a (stale) CRC-32 checksum value for chunk [0, 0x35ef6): Mirror 1: 0x1bd1c2 Mirror 27: 0x1bd1c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ef6: skipped 000552 1788541853.436375 trunc from 0x035ef6 to 0x03f493 (0x959e bytes) 000553 1788541853.505205 trunc from 0x0321b6 to 0x03f493 (0xd2de bytes) 000554 1788541853.511345 trunc from 0x03f494 to 0x01077c (0x2ed17 bytes) 000555 1788541853.571825 mapread 0x002043 thru 0x01077c (0xe73a bytes) 000557 1788541853.602878 write 0x00ecb9 thru 0x01039a (0x16e2 bytes) 000558 1788541853.603837 mapwrite 0x0367b2 thru 0x0385e7 (0x1e36 bytes) 000559 1788541853.680652 mapwrite 0x014b37 thru 0x017cc8 (0x3192 bytes) 000560 1788541853.709103 read 0x031130 thru 0x0329d9 (0x18aa bytes) 000561 1788541853.722494 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x385e8): Mirror 22: 0xde91df5 (stale) Mirror 26: 0xde91df5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385e8): Mirror 22: 0xde91df5 (stale) Mirror 26: 0xde91df5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385e8: skipped 000562 1788541854.078101 read 0x00d9cc thru 0x01c8a8 (0xeedd bytes) 000563 1788541854.091420 fallocate 0x01d02a thru 0x022520 (0x54f7 bytes) 000566 1788541854.100311 fallocate 0x03c2b8 thru 0x03ca16 (0x75f bytes) 000567 1788541854.105764 mapread 0x00e17b thru 0x01579c (0x7622 bytes) 000568 1788541854.106682 trunc from 0x0385e8 to 0x0129c1 (0x25c26 bytes) 000569 1788541854.127554 fallocate 0x0321b6 thru 0x03ffff (0xde4a bytes) 000570 1788541854.139325 mapwrite 0x001c84 thru 0x00fb7c (0xdef9 bytes) 000571 1788541854.205342 trunc from 0x00e69c to 0x0129c1 (0x4326 bytes) 000572 1788541854.209337 write 0x02d5ef thru 0x03ac23 (0xd635 bytes) 000573 1788541854.212607 trunc from 0x005e0e to 0x008692 (0x2885 bytes) 000575 1788541854.218529 read 0x02df27 thru 0x030e5b (0x2f35 bytes) 000576 1788541854.219633 write 0x01eb9b thru 0x01f652 (0xab8 bytes) 000577 1788541854.228404 mapwrite 0x0340c6 thru 0x037ec3 (0x3dfe bytes) 000578 1788541854.244237 mapread 0x01ed94 thru 0x02524e (0x64bb bytes) 000581 1788541854.246891 read 0x039854 thru 0x03ac23 (0x13d0 bytes) 000582 1788541854.251216 MIRROR_RESYNC 3 mirrors 000583 1788541855.141577 trunc from 0x03ac24 to 0x027a92 (0x13191 bytes) 000584 1788541855.196671 write 0x0341c7 thru 0x03ffff (0xbe39 bytes) 000585 1788541855.198266 punch from 0x021275 to 0x021561 (0x2ed bytes) 000586 1788541855.202669 trunc from 0x040000 to 0x017f33 (0x280cc bytes) 000587 1788541855.217706 fallocate 0x01dca0 thru 0x02d003 (0xf364 bytes) 000588 1788541855.225481 MIRROR_RESYNC 3 mirrors 000589 1788541855.978318 read 0x00837f thru 0x011cb8 (0x993a bytes) 000590 1788541856.003620 trunc from 0x00465c to 0x00ccb5 (0x865a bytes) 000591 1788541856.044999 write 0x023260 thru 0x0308c9 (0xd66a bytes) 000592 1788541856.051648 mapwrite 0x0077c9 thru 0x011400 (0x9c38 bytes) 000593 1788541856.129466 trunc from 0x0308ca to 0x012ab5 (0x1de14 bytes) 000594 1788541856.168860 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12ab6): Mirror 26: 0xc61e2277 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12ab6): Mirror 26: 0xc61e2277 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12ab6: skipped 000595 1788541856.538102 trunc from 0x012ab6 to 0x022e4d (0x10398 bytes) 000597 1788541856.577948 write 0x033ebf thru 0x03f8da (0xba1c bytes) 000598 1788541856.582059 fallocate 0x01ab9e thru 0x02784d (0xccb0 bytes) 000599 1788541856.589817 mapread 0x02035d thru 0x0273c5 (0x7069 bytes) 000600 1788541856.613833 punch from 0x014258 to 0x01c949 (0x86f2 bytes) 000601 1788541856.618429 read 0x02f8ac thru 0x037f19 (0x866e bytes) 000602 1788541856.624114 read 0x00f887 thru 0x01413d (0x48b7 bytes) 000603 1788541856.634238 trunc from 0x01dc3b to 0x02d78e (0xfb54 bytes) 000604 1788541856.640584 read 0x004817 thru 0x014816 (0x10000 bytes) 000605 1788541856.641698 mapwrite 0x0164f7 thru 0x0191f7 (0x2d01 bytes) 000607 1788541856.659472 fallocate 0x02a61d thru 0x031767 (0x714b bytes) 000608 1788541856.665491 trunc from 0x00a4c6 to 0x011cb9 (0x77f4 bytes) 000610 1788541856.671954 fallocate 0x012045 thru 0x0196fd (0x76b9 bytes) 000611 1788541856.676202 mapwrite 0x014e50 thru 0x01cdac (0x7f5d bytes) 000612 1788541856.697756 punch from 0x004382 to 0x00d30f (0x8f8e bytes) 000613 1788541856.703447 mapread 0x03c1ec thru 0x03d21e (0x1033 bytes) 000614 1788541856.709244 MIRROR_RESYNC 2 mirrors 000615 1788541857.511735 mapwrite 0x02b4f7 thru 0x030d77 (0x5881 bytes) 000616 1788541857.624437 mapwrite 0x0338f2 thru 0x0348cb (0xfda bytes) 000617 1788541857.634692 mapread 0x035a24 thru 0x03f8da (0x9eb7 bytes) 000619 1788541857.647153 fallocate 0x00bbe0 thru 0x01bba2 (0xffc3 bytes) 000620 1788541857.652003 read 0x0271bd thru 0x02c391 (0x51d5 bytes) 000621 1788541857.652584 punch from 0x02256e to 0x02f05e (0xcaf1 bytes) 000622 1788541857.664774 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 26: 0xfe912863 (stale) CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 1: 0x5f9ff30 Mirror 27: 0x5f9ff30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8db: skipped 000623 1788541858.607877 punch from 0x02acef to 0x035a37 (0xad49 bytes) 000624 1788541858.645812 MIRROR_RESYNC 3 mirrors 000625 1788541859.517266 read 0x015923 thru 0x01b637 (0x5d15 bytes) 000626 1788541859.525751 write 0x00b76a thru 0x00c755 (0xfec bytes) 000627 1788541859.554078 mapread 0x029fb2 thru 0x02c44d (0x249c bytes) 000628 1788541859.568579 mapwrite 0x032715 thru 0x03dddf (0xb6cb bytes) 000629 1788541859.599035 mapwrite 0x030f62 thru 0x033680 (0x271f bytes) 000631 1788541859.628080 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 27: 0x3ce13e0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 27: 0x3ce13e0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8db: skipped 000632 1788541859.935138 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 27: 0x3ce13e0b (stale) CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 1: 0x92aa15a1 Mirror 28: 0x92aa15a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8db: skipped 000633 1788541860.904557 mapread 0x0255bc thru 0x028c95 (0x36da bytes) 000635 1788541860.945054 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 27: 0x3ce13e0b (stale) CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 1: 0x92aa15a1 Mirror 28: 0x92aa15a1 Mirror 29: 0x92aa15a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8db: skipped 000636 1788541861.745744 punch from 0x006724 to 0x0160fa (0xf9d7 bytes) 000637 1788541861.781435 punch from 0x0070e7 to 0x016a23 (0xf93d bytes) 000638 1788541861.785748 mapread 0x002257 thru 0x00eac4 (0xc86e bytes) 000639 1788541861.801661 mapread 0x01497b thru 0x01afa5 (0x662b bytes) 000640 1788541861.811061 punch from 0x033f51 to 0x03f858 (0xb908 bytes) 000641 1788541861.815542 read 0x00dc6a thru 0x00f8fb (0x1c92 bytes) 000642 1788541861.816164 write 0x031bf9 thru 0x03578c (0x3b94 bytes) 000643 1788541861.828094 mapwrite 0x00594d thru 0x014cd0 (0xf384 bytes) 000644 1788541861.849058 punch from 0x00fc41 to 0x01c24e (0xc60e bytes) 000645 1788541861.853788 fallocate 0x01590d thru 0x016bf7 (0x12eb bytes) 000646 1788541861.858720 punch from 0x00426d to 0x004afd (0x891 bytes) 000647 1788541861.865961 MIRROR_RESYNC 4 mirrors 000648 1788541862.879551 mapread 0x0015a6 thru 0x001862 (0x2bd bytes) 000649 1788541862.883825 punch from 0x03ac31 to 0x03f8da (0x4caa bytes) 000650 1788541862.925102 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 28: 0xa9c3d8e1 (stale) Mirror 29: 0xa9c3d8e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8db): Mirror 28: 0xa9c3d8e1 (stale) Mirror 29: 0xa9c3d8e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8db: skipped 000651 1788541863.233743 trunc from 0x03f8db to 0x039737 (0x61a3 bytes) 000652 1788541863.254071 trunc from 0x008a8d to 0x0181f3 (0xf767 bytes) 000654 1788541863.261249 mapread 0x003213 thru 0x004698 (0x1486 bytes) 000655 1788541863.273788 fallocate 0x015331 thru 0x01d2fe (0x7fce bytes) 000657 1788541863.278736 trunc from 0x039738 to 0x03a16c (0xa35 bytes) 000658 1788541863.293151 write 0x00207d thru 0x00f201 (0xd185 bytes) 000659 1788541863.303720 trunc from 0x019901 to 0x02856c (0xec6c bytes) 000660 1788541863.308279 mapwrite 0x030f6f thru 0x034872 (0x3904 bytes) 000661 1788541863.333055 trunc from 0x03a16d to 0x00f086 (0x2b0e6 bytes) 000662 1788541863.356463 MIRROR_RESYNC 3 mirrors 000663 1788541863.878835 fallocate 0x0337ff thru 0x03d0f2 (0x98f4 bytes) 000664 1788541863.904116 MIRROR_RESYNC 3 mirrors 000665 1788541864.467417 punch from 0x014dce to 0x019f6a (0x519d bytes) 000666 1788541864.495787 MIRROR_RESYNC 3 mirrors 000667 1788541865.137478 write 0x02fa62 thru 0x0332fb (0x389a bytes) 000668 1788541865.168376 write 0x03aff1 thru 0x03ffff (0x500f bytes) 000669 1788541865.173468 write 0x035fc2 thru 0x03ffff (0xa03e bytes) 000670 1788541865.178665 read 0x01f1a5 thru 0x028ba7 (0x9a03 bytes) 000672 1788541865.188144 trunc from 0x040000 to 0x024297 (0x1bd68 bytes) 000673 1788541865.203058 mapwrite 0x038cc8 thru 0x03ffff (0x7338 bytes) 000674 1788541865.244249 punch from 0x00ec76 to 0x0193e1 (0xa76c bytes) 000675 1788541865.248665 trunc from 0x02e2f0 to 0x035429 (0x713a bytes) 000676 1788541865.253023 mapread 0x033eb1 thru 0x03995b (0x5aab bytes) 000677 1788541865.260817 mapread 0x020907 thru 0x022b20 (0x221a bytes) 000678 1788541865.262455 trunc from 0x040000 to 0x03d48a (0x2b75 bytes) 000679 1788541865.275628 mapwrite 0x01d0c9 thru 0x026305 (0x923d bytes) 000680 1788541865.307885 write 0x0387bb thru 0x03c90c (0x4152 bytes) 000681 1788541865.309635 fallocate 0x030691 thru 0x032fc4 (0x2934 bytes) 000682 1788541865.315706 trunc from 0x03d48b to 0x01dc9e (0x1f7ec bytes) 000683 1788541865.333055 read 0x00be25 thru 0x0136f4 (0x78d0 bytes) 000684 1788541865.343479 write 0x021cbe thru 0x0257a2 (0x3ae5 bytes) 000685 1788541865.345611 punch from 0x005a10 to 0x00a486 (0x4a77 bytes) 000686 1788541865.351488 punch from 0x01556e to 0x024c4c (0xf6df bytes) 000687 1788541865.360885 trunc from 0x01dd35 to 0x0257a2 (0x7a6e bytes) 000688 1788541865.367512 mapread 0x01fbb8 thru 0x0257a2 (0x5beb bytes) 000689 1788541865.375466 trunc from 0x0257a3 to 0x03cb99 (0x173f7 bytes) 000690 1788541865.396854 punch from 0x024f00 to 0x0258d2 (0x9d3 bytes) 000691 1788541865.402787 write 0x01b50f thru 0x02ab0e (0xf600 bytes) 000692 1788541865.417226 read 0x03876a thru 0x03cb99 (0x4430 bytes) 000693 1788541865.417824 fallocate 0x02f461 thru 0x03b9a6 (0xc546 bytes) 000694 1788541865.432114 trunc from 0x02d273 to 0x0305ba (0x3348 bytes) 000695 1788541865.438243 trunc from 0x0055eb to 0x0086f9 (0x310f bytes) 000696 1788541865.443350 trunc from 0x03cb9a to 0x018e64 (0x23d35 bytes) 000697 1788541865.471838 punch from 0x00e374 to 0x013a31 (0x56be bytes) 000698 1788541865.477939 trunc from 0x007939 to 0x007d9f (0x467 bytes) 000699 1788541865.487021 mapwrite 0x03ea4e thru 0x03ffff (0x15b2 bytes) 000700 1788541865.562558 write 0x0079c9 thru 0x011f6c (0xa5a4 bytes) 000702 1788541865.574327 trunc from 0x040000 to 0x03d9a6 (0x2659 bytes) 000703 1788541865.615024 trunc from 0x03d9a7 to 0x01c670 (0x21336 bytes) 000704 1788541865.634713 MIRROR_RESYNC 3 mirrors 000705 1788541866.375705 mapread 0x013c03 thru 0x014848 (0xc46 bytes) 000706 1788541866.387955 read 0x010956 thru 0x0126a0 (0x1d4b bytes) 000707 1788541866.392059 write 0x01c2d6 thru 0x01d9f9 (0x1724 bytes) 000710 1788541866.445278 fallocate 0x002e99 thru 0x00c0d2 (0x923a bytes) 000711 1788541866.454820 fallocate 0x03bd2a thru 0x03ffff (0x42d6 bytes) 000712 1788541866.459731 read 0x00962d thru 0x015a7a (0xc44e bytes) 000714 1788541866.479766 mapread 0x00317a thru 0x007a89 (0x4910 bytes) 000715 1788541866.481328 read 0x00ec83 thru 0x01b470 (0xc7ee bytes) 000716 1788541866.481901 punch from 0x000b7c to 0x005c8f (0x5114 bytes) 000717 1788541866.500436 punch from 0x004f99 to 0x00b008 (0x6070 bytes) 000718 1788541866.506442 write 0x00b85c thru 0x012059 (0x67fe bytes) 000720 1788541866.518554 write 0x02be23 thru 0x039658 (0xd836 bytes) 000721 1788541866.521083 mapwrite 0x0122ed thru 0x01f6a0 (0xd3b4 bytes) 000722 1788541866.561668 punch from 0x035584 to 0x039658 (0x40d5 bytes) 000723 1788541866.569433 MIRROR_RESYNC 3 mirrors 000724 1788541867.515152 mapwrite 0x004bb3 thru 0x00d641 (0x8a8f bytes) 000725 1788541867.586686 mapread 0x02c6d9 thru 0x035565 (0x8e8d bytes) 000726 1788541867.603603 write 0x0081d6 thru 0x011966 (0x9791 bytes) 000727 1788541867.612459 mapread 0x006a4e thru 0x0090d2 (0x2685 bytes) 000728 1788541867.616753 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39659): Mirror 28: 0xb5463126 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39659): Mirror 28: 0xb5463126 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39659: skipped 000730 1788541867.929920 fallocate 0x00652b thru 0x013887 (0xd35d bytes) 000731 1788541867.950662 MIRROR_RESYNC 2 mirrors 000733 1788541868.584472 mapread 0x00d9c3 thru 0x019a4f (0xc08d bytes) 000734 1788541868.614849 trunc from 0x039659 to 0x021b06 (0x17b52 bytes) 000735 1788541868.657644 mapwrite 0x00ba5f thru 0x00ccb8 (0x125a bytes) 000736 1788541868.685700 trunc from 0x00f659 to 0x017f39 (0x88e1 bytes) 000738 1788541868.691292 mapread 0x015684 thru 0x01f834 (0xa1b1 bytes) 000739 1788541868.713258 mapwrite 0x020753 thru 0x021090 (0x93e bytes) 000740 1788541868.744804 fallocate 0x034fe4 thru 0x03ffff (0xb01c bytes) 000741 1788541868.749589 punch from 0x01adb5 to 0x021194 (0x63e0 bytes) 000742 1788541868.780582 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) 000744 1788541868.908582 trunc from 0x00da87 to 0x017aa0 (0xa01a bytes) 000746 1788541868.914347 fallocate 0x0010db thru 0x0047a6 (0x36cc bytes) 000747 1788541868.919055 read 0x00b650 thru 0x0183e8 (0xcd99 bytes) 000749 1788541868.929928 punch from 0x00ff91 to 0x01957b (0x95eb bytes) 000750 1788541868.934502 fallocate 0x02150b thru 0x029e9e (0x8994 bytes) 000751 1788541868.938524 mapwrite 0x027a5c thru 0x037376 (0xf91b bytes) 000753 1788541869.012113 write 0x010900 thru 0x015225 (0x4926 bytes) 000754 1788541869.022951 trunc from 0x037377 to 0x033db9 (0x35bd bytes) 000755 1788541869.034307 mapread 0x01201a thru 0x02108b (0xf072 bytes) 000756 1788541869.056904 read 0x017e95 thru 0x01b95f (0x3acb bytes) 000757 1788541869.057631 read 0x0313c7 thru 0x033db9 (0x29f3 bytes) 000758 1788541869.058582 trunc from 0x033dba to 0x01c3fe (0x179bb bytes) 000759 1788541869.077613 fallocate 0x014d85 thru 0x01ed84 (0xa000 bytes) 000762 1788541869.086296 read 0x005e0d thru 0x0130d9 (0xd2cd bytes) 000764 1788541869.102890 punch from 0x008fa3 to 0x00e9f8 (0x5a56 bytes) 000765 1788541869.108712 mapwrite 0x034ecc thru 0x036bb7 (0x1cec bytes) 000766 1788541869.205422 trunc from 0x036bb8 to 0x01f4e1 (0x176d6 bytes) 000767 1788541869.229131 write 0x021590 thru 0x02c393 (0xae04 bytes) 000768 1788541869.237672 punch from 0x01d1bd to 0x025284 (0x80c8 bytes) 000769 1788541869.270551 mapread 0x0091d8 thru 0x00d0d1 (0x3efa bytes) 000770 1788541869.318769 mapwrite 0x012647 thru 0x014e16 (0x27d0 bytes) 000772 1788541869.342567 mapread 0x00eed9 thru 0x01cc41 (0xdd69 bytes) 000774 1788541869.351223 trunc from 0x02c394 to 0x00a6e3 (0x21cb0 bytes) 000775 1788541869.368590 mapread 0x00506f thru 0x008526 (0x34b8 bytes) 000776 1788541869.374286 mapread 0x00401b thru 0x006bee (0x2bd4 bytes) 000777 1788541869.375799 read 0x000981 thru 0x00a4a5 (0x9b25 bytes) 000778 1788541869.376580 read 0x007997 thru 0x00a6e3 (0x2d4d bytes) 000779 1788541869.377552 fallocate 0x0207a8 thru 0x0247f9 (0x4052 bytes) 000780 1788541869.392784 punch from 0x003f20 to 0x0053fd (0x14de bytes) 000781 1788541869.403705 mapread 0x0038bf thru 0x007a71 (0x41b3 bytes) 000782 1788541869.410014 mapread 0x008d76 thru 0x00a6e3 (0x196e bytes) 000783 1788541869.413827 MIRROR_EXTEND to 2 mirrors 000784 1788541869.886356 punch from 0x00601f to 0x00a6e3 (0x46c5 bytes) 000785 1788541869.921083 trunc from 0x00a6e4 to 0x03ea7e (0x3439b bytes) 000786 1788541869.960149 trunc from 0x001744 to 0x008535 (0x6df2 bytes) 000787 1788541869.964899 trunc from 0x017626 to 0x020581 (0x8f5c bytes) 000788 1788541869.970111 fallocate 0x03b28b thru 0x03d0fe (0x1e74 bytes) 000789 1788541869.974141 mapread 0x027621 thru 0x032ded (0xb7cd bytes) 000790 1788541870.003394 mapread 0x029276 thru 0x02a47a (0x1205 bytes) 000791 1788541870.004375 punch from 0x036795 to 0x03b9aa (0x5216 bytes) 000792 1788541870.011747 read 0x03086e thru 0x03cc74 (0xc407 bytes) 000793 1788541870.027942 punch from 0x01a135 to 0x021ff9 (0x7ec5 bytes) 000794 1788541870.032631 punch from 0x018200 to 0x01903b (0xe3c bytes) 000795 1788541870.038395 trunc from 0x017b54 to 0x024b78 (0xd025 bytes) 000796 1788541870.043311 trunc from 0x03ea7f to 0x00b7aa (0x332d4 bytes) 000797 1788541870.076086 fallocate 0x01bf78 thru 0x02246d (0x64f6 bytes) 000798 1788541870.084128 mapread 0x000c1e thru 0x001f0c (0x12ef bytes) 000799 1788541870.099442 read 0x00aa05 thru 0x00b7aa (0xda6 bytes) 000800 1788541870.100433 trunc from 0x00b7ab to 0x034a6f (0x292c5 bytes) 000801 1788541870.117710 mapwrite 0x03e36f thru 0x03ffff (0x1c91 bytes) 000802 1788541870.168495 punch from 0x00b03a to 0x010440 (0x5407 bytes) 000803 1788541870.174013 punch from 0x01cd80 to 0x024ac0 (0x7d41 bytes) 000804 1788541870.183356 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe713bb8a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x68bc5107 Mirror 3: 0x68bc5107 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000805 1788541870.867555 trunc from 0x040000 to 0x027ad2 (0x1852d bytes) 000806 1788541870.948628 mapwrite 0x01241e thru 0x012930 (0x513 bytes) 000807 1788541870.967513 mapwrite 0x013ebb thru 0x01d7b6 (0x98fc bytes) 000809 1788541871.008120 write 0x034ae8 thru 0x03e7e6 (0x9cff bytes) 000810 1788541871.013397 MIRROR_RESYNC 3 mirrors 000811 1788541871.911324 MIRROR_EXTEND to 4 mirrors 000813 1788541872.524297 punch from 0x0273a1 to 0x02c8c6 (0x5526 bytes) 000815 1788541872.586145 write 0x022ee1 thru 0x029c4e (0x6d6e bytes) 000816 1788541872.600113 mapwrite 0x0164a4 thru 0x023898 (0xd3f5 bytes) 000818 1788541872.643306 mapwrite 0x034231 thru 0x035d4d (0x1b1d bytes) 000820 1788541872.660970 trunc from 0x03e7e7 to 0x02a3ee (0x143f8 bytes) 000821 1788541872.676163 fallocate 0x00c4f3 thru 0x00d232 (0xd40 bytes) 000822 1788541872.680392 mapread 0x00fcfa thru 0x017cd3 (0x7fda bytes) 000823 1788541872.692137 fallocate 0x00be22 thru 0x011cb0 (0x5e8f bytes) 000824 1788541872.696360 fallocate 0x02280e thru 0x031aeb (0xf2de bytes) 000825 1788541872.700597 trunc from 0x023b17 to 0x0287d5 (0x4cbf bytes) 000826 1788541872.704672 mapwrite 0x036989 thru 0x03ffff (0x9677 bytes) 000827 1788541872.744141 write 0x0390b2 thru 0x03afd5 (0x1f24 bytes) 000828 1788541872.745093 read 0x026201 thru 0x02b587 (0x5387 bytes) 000829 1788541872.753455 trunc from 0x040000 to 0x02aff5 (0x1500a bytes) 000830 1788541872.767467 mapread 0x01206d thru 0x0132eb (0x127f bytes) 000831 1788541872.768333 mapread 0x010b69 thru 0x013ec6 (0x335e bytes) 000832 1788541872.769058 trunc from 0x007249 to 0x00f5b2 (0x836a bytes) 000833 1788541872.773323 write 0x0313e1 thru 0x03e9ac (0xd5cc bytes) 000834 1788541872.776272 trunc from 0x0387b5 to 0x03e9ac (0x61f8 bytes) 000835 1788541872.788483 read 0x01d551 thru 0x024437 (0x6ee7 bytes) 000836 1788541872.796801 read 0x008c4c thru 0x00ffb6 (0x736b bytes) 000837 1788541872.798150 mapwrite 0x004ca9 thru 0x008406 (0x375e bytes) 000838 1788541872.810442 read 0x02504d thru 0x03490c (0xf8c0 bytes) 000839 1788541872.811139 trunc from 0x03e9ad to 0x03de93 (0xb19 bytes) 000840 1788541872.832512 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3de94): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de94): Mirror 2: 0x92646152 (stale) Mirror 3: 0x92646152 (stale) Mirror 4: 0x92646152 (stale) CRC-32 checksum value for chunk [0, 0x3de94): Mirror 1: 0xfe2356a4 Mirror 5: 0xfe2356a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de94: skipped 000842 1788541873.761776 read 0x00d971 thru 0x00eac6 (0x1156 bytes) 000843 1788541873.767704 punch from 0x03db68 to 0x03de93 (0x32c bytes) 000844 1788541873.802418 fallocate 0x02cc87 thru 0x03055d (0x38d7 bytes) 000845 1788541873.808249 read 0x03b292 thru 0x03de93 (0x2c02 bytes) 000846 1788541873.818573 trunc from 0x0112ce to 0x019083 (0x7db6 bytes) 000848 1788541873.825887 mapread 0x03bc51 thru 0x03de93 (0x2243 bytes) 000849 1788541873.827127 punch from 0x012337 to 0x0215ba (0xf284 bytes) 000850 1788541873.831515 write 0x03ed5c thru 0x03ffff (0x12a4 bytes) 000851 1788541873.835416 fallocate 0x006c16 thru 0x014dd5 (0xe1c0 bytes) 000852 1788541873.841832 trunc from 0x022bea to 0x028d51 (0x6168 bytes) 000853 1788541873.846040 mapread 0x033ceb thru 0x03c988 (0x8c9e bytes) 000854 1788541873.847578 read 0x011798 thru 0x017a49 (0x62b2 bytes) 000855 1788541873.855529 mapread 0x00f75c thru 0x01750e (0x7db3 bytes) 000856 1788541873.857039 trunc from 0x017130 to 0x021965 (0xa836 bytes) 000857 1788541873.861111 fallocate 0x03a8b5 thru 0x03ffff (0x574b bytes) 000858 1788541873.865421 trunc from 0x016d1f to 0x01cf60 (0x6242 bytes) 000859 1788541873.869456 mapread 0x01a045 thru 0x0259d6 (0xb992 bytes) 000860 1788541873.935462 mapwrite 0x010383 thru 0x014cfe (0x497c bytes) 000861 1788541873.979637 read 0x01f090 thru 0x02ea04 (0xf975 bytes) 000862 1788541873.980299 write 0x00e96d thru 0x013099 (0x472d bytes) 000863 1788541873.981271 mapread 0x007cbc thru 0x009bcb (0x1f10 bytes) 000865 1788541873.996460 mapwrite 0x02d823 thru 0x0347d4 (0x6fb2 bytes) 000866 1788541874.026397 punch from 0x007a0a to 0x0142ef (0xc8e6 bytes) 000867 1788541874.034967 read 0x02d4da thru 0x03c6bf (0xf1e6 bytes) 000868 1788541874.035580 punch from 0x020640 to 0x02f05d (0xea1e bytes) 000869 1788541874.041506 punch from 0x006897 to 0x014c36 (0xe3a0 bytes) 000870 1788541874.059836 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x24ce1786 (stale) Mirror 4: 0x24ce1786 (stale) Mirror 5: 0xa2ab7ef1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x24ce1786 (stale) Mirror 4: 0x24ce1786 (stale) Mirror 5: 0xa2ab7ef1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000871 1788541874.562995 fallocate 0x01a6bd thru 0x0281f8 (0xdb3c bytes) 000872 1788541874.567595 fallocate 0x0075be thru 0x015185 (0xdbc8 bytes) 000873 1788541874.576629 read 0x023e5c thru 0x0282c0 (0x4465 bytes) 000874 1788541874.584544 fallocate 0x0066a3 thru 0x00bf3b (0x5899 bytes) 000875 1788541874.588729 fallocate 0x01f6aa thru 0x022658 (0x2faf bytes) 000876 1788541874.592922 trunc from 0x022635 to 0x02d5d9 (0xafa5 bytes) 000877 1788541874.597151 write 0x00218a thru 0x010fe6 (0xee5d bytes) 000878 1788541874.609611 trunc from 0x03b215 to 0x03ffff (0x4deb bytes) 000879 1788541874.613665 fallocate 0x0233a6 thru 0x030440 (0xd09b bytes) 000880 1788541874.617534 mapread 0x02e539 thru 0x03c374 (0xde3c bytes) 000881 1788541874.644760 mapread 0x02e7b7 thru 0x030462 (0x1cac bytes) 000882 1788541874.645836 punch from 0x001b44 to 0x007410 (0x58cd bytes) 000884 1788541874.659194 mapwrite 0x029fe3 thru 0x02d4a4 (0x34c2 bytes) 000885 1788541874.672432 mapread 0x02aad4 thru 0x0376ca (0xcbf7 bytes) 000886 1788541874.674016 punch from 0x027810 to 0x02f0df (0x78d0 bytes) 000887 1788541874.678042 punch from 0x01a364 to 0x0240cd (0x9d6a bytes) 000889 1788541874.686441 MIRROR_RESYNC 4 mirrors 000890 1788541875.807417 punch from 0x026aa1 to 0x02f961 (0x8ec1 bytes) 000891 1788541875.846448 punch from 0x012f9e to 0x0148ac (0x190f bytes) 000892 1788541875.860041 trunc from 0x040000 to 0x012582 (0x2da7d bytes) 000895 1788541875.878378 trunc from 0x00f28f to 0x0123ea (0x315c bytes) 000896 1788541875.885672 read 0x0118f1 thru 0x012582 (0xc92 bytes) 000897 1788541875.890892 read 0x00fd92 thru 0x012582 (0x27f1 bytes) 000898 1788541875.908771 write 0x022ef7 thru 0x024e92 (0x1f9c bytes) 000899 1788541875.910031 trunc from 0x024e93 to 0x032989 (0xdaf7 bytes) 000900 1788541875.929444 fallocate 0x01509a thru 0x019b0a (0x4a71 bytes) 000901 1788541875.933858 read 0x0034c4 thru 0x00727d (0x3dba bytes) 000902 1788541875.937261 write 0x0106a3 thru 0x016796 (0x60f4 bytes) 000903 1788541875.943381 read 0x014b81 thru 0x01cba2 (0x8022 bytes) 000904 1788541875.962948 punch from 0x00c6ce to 0x018bc8 (0xc4fb bytes) 000905 1788541875.984379 trunc from 0x018ad3 to 0x022f0e (0xa43c bytes) 000906 1788541876.003102 punch from 0x02d650 to 0x032989 (0x533a bytes) 000907 1788541876.009686 trunc from 0x03298a to 0x00a5a2 (0x283e7 bytes) 000909 1788541876.034001 MIRROR_RESYNC 4 mirrors 000910 1788541877.138736 mapread 0x004fd6 thru 0x00a5a2 (0x55cd bytes) 000911 1788541877.174437 read 0x0061e0 thru 0x00a5a2 (0x43c3 bytes) 000912 1788541877.174763 mapwrite 0x01741f thru 0x01b1e9 (0x3dcb bytes) 000913 1788541877.251034 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1b1ea): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b1ea): Mirror 3: 0xfa867e4a (stale) Mirror 4: 0xfa867e4a (stale) Mirror 5: 0xfa867e4a (stale) CRC-32 checksum value for chunk [0, 0x1b1ea): Mirror 1: 0x2d8d6625 Mirror 6: 0x2d8d6625 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b1ea: skipped 000914 1788541878.093442 punch from 0x014fe6 to 0x01b1e9 (0x6204 bytes) 000916 1788541878.143730 mapwrite 0x03972e thru 0x03ffff (0x68d2 bytes) 000919 1788541878.288477 mapwrite 0x031fa8 thru 0x0345bd (0x2616 bytes) 000920 1788541878.335038 trunc from 0x02cf22 to 0x02ec6e (0x1d4d bytes) 000921 1788541878.342363 punch from 0x03c32d to 0x03ffff (0x3cd3 bytes) 000922 1788541878.352029 trunc from 0x040000 to 0x03ba62 (0x459d bytes) 000923 1788541878.372348 mapwrite 0x024b8d thru 0x026229 (0x169d bytes) 000924 1788541878.410830 punch from 0x02afc9 to 0x03911c (0xe154 bytes) 000925 1788541878.418606 mapread 0x02a2f9 thru 0x02c014 (0x1d1c bytes) 000926 1788541878.441596 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ba63): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba63): Mirror 3: 0x68ea126c (stale) Mirror 4: 0x68ea126c (stale) Mirror 5: 0x68ea126c (stale) Mirror 6: 0x5870e526 (stale) CRC-32 checksum value for chunk [0, 0x3ba63): Mirror 1: 0x98928c5 Mirror 7: 0x98928c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba63: skipped 000927 1788541879.473602 mapwrite 0x00b91d thru 0x00eb9c (0x3280 bytes) 000928 1788541879.643808 punch from 0x013ac8 to 0x0221d6 (0xe70f bytes) 000929 1788541879.656841 MIRROR_RESYNC 6 mirrors 000930 1788541881.330110 mapwrite 0x00f987 thru 0x0144ce (0x4b48 bytes) 000931 1788541881.433298 fallocate 0x03d224 thru 0x03ffff (0x2ddc bytes) 000932 1788541881.438646 write 0x033edb thru 0x038839 (0x495f bytes) 000933 1788541881.449111 trunc from 0x008184 to 0x0169eb (0xe868 bytes) 000934 1788541881.455984 write 0x003eed thru 0x011371 (0xd485 bytes) 000935 1788541881.478417 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe6bb3418 (stale) Mirror 4: 0xe6bb3418 (stale) Mirror 5: 0xe6bb3418 (stale) Mirror 6: 0xe6bb3418 (stale) Mirror 7: 0xe6bb3418 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39a57ca5 Mirror 8: 0x39a57ca5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000936 1788541882.380953 write 0x008ff9 thru 0x00b2b3 (0x22bb bytes) 000937 1788541882.472662 punch from 0x010d25 to 0x011088 (0x364 bytes) 000938 1788541882.476582 mapread 0x01adf5 thru 0x02351b (0x8727 bytes) 000941 1788541882.499673 read 0x03a976 thru 0x03ffff (0x568a bytes) 000942 1788541882.514366 punch from 0x035b15 to 0x03a39d (0x4889 bytes) 000943 1788541882.520419 mapread 0x03d264 thru 0x03ffff (0x2d9c bytes) 000945 1788541882.522993 trunc from 0x040000 to 0x022fb8 (0x1d047 bytes) 000946 1788541882.548347 write 0x014354 thru 0x01d68b (0x9338 bytes) 000947 1788541882.555708 mapread 0x002f31 thru 0x00ec19 (0xbce9 bytes) 000948 1788541882.581370 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22fb9): Mirror 3: 0x1d93aed7 (stale) Mirror 4: 0x1d93aed7 (stale) Mirror 5: 0x1d93aed7 (stale) Mirror 7: 0x1d93aed7 (stale) Mirror 8: 0xd39b5008 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22fb9): Mirror 3: 0x1d93aed7 (stale) Mirror 4: 0x1d93aed7 (stale) Mirror 5: 0x1d93aed7 (stale) Mirror 7: 0x1d93aed7 (stale) Mirror 8: 0xd39b5008 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22fb9: skipped 000949 1788541882.972665 mapwrite 0x03bd92 thru 0x03ffff (0x426e bytes) 000950 1788541883.021068 punch from 0x01a9b8 to 0x0230d0 (0x8719 bytes) 000951 1788541883.026418 trunc from 0x033766 to 0x03480c (0x10a7 bytes) 000952 1788541883.030656 read 0x00521a thru 0x008c2e (0x3a15 bytes) 000953 1788541883.036434 trunc from 0x040000 to 0x034f4b (0xb0b4 bytes) 000954 1788541883.053554 trunc from 0x02cc49 to 0x034f4b (0x8303 bytes) 000955 1788541883.062472 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x34f4c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f4c): Mirror 3: 0x2c953077 (stale) Mirror 4: 0x2c953077 (stale) Mirror 5: 0x2c953077 (stale) Mirror 7: 0x2c953077 (stale) Mirror 8: 0xfc96ab50 (stale) CRC-32 checksum value for chunk [0, 0x34f4c): Mirror 1: 0x61e8a6d2 Mirror 9: 0x61e8a6d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f4c: skipped 000956 1788541883.872740 punch from 0x00f956 to 0x00ffc3 (0x66e bytes) 000957 1788541883.926498 trunc from 0x034f4c to 0x03dd6a (0x8e1f bytes) 000958 1788541884.028663 trunc from 0x036199 to 0x03d137 (0x6f9f bytes) 000959 1788541884.040005 punch from 0x02e45a to 0x030eaf (0x2a56 bytes) 000960 1788541884.044703 read 0x012eb1 thru 0x01a8a1 (0x79f1 bytes) 000961 1788541884.057938 read 0x0207b5 thru 0x022b4c (0x2398 bytes) 000962 1788541884.072398 punch from 0x02c3a7 to 0x02c42e (0x88 bytes) 000963 1788541884.079668 write 0x017686 thru 0x023d12 (0xc68d bytes) 000965 1788541884.087458 trunc from 0x03dd6b to 0x01c208 (0x21b62 bytes) 000967 1788541884.111434 fallocate 0x029df9 thru 0x02b5cf (0x17d7 bytes) 000968 1788541884.117236 trunc from 0x01b2b9 to 0x029819 (0xe561 bytes) 000970 1788541884.132565 fallocate 0x020fec thru 0x0217cb (0x7e0 bytes) 000971 1788541884.137302 write 0x01a5b2 thru 0x029944 (0xf393 bytes) 000972 1788541884.142788 fallocate 0x015bcd thru 0x018d93 (0x31c7 bytes) 000973 1788541884.147735 write 0x00b18b thru 0x00d8af (0x2725 bytes) 000974 1788541884.149270 write 0x01151c thru 0x01efb0 (0xda95 bytes) 000975 1788541884.153480 fallocate 0x02d1a4 thru 0x036596 (0x93f3 bytes) 000976 1788541884.167056 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36597): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36597): Mirror 3: 0xc5a2bd19 (stale) Mirror 4: 0xc5a2bd19 (stale) Mirror 5: 0xc5a2bd19 (stale) Mirror 7: 0xc5a2bd19 (stale) Mirror 8: 0xb40e20ce (stale) Mirror 9: 0x92a6879b (stale) CRC-32 checksum value for chunk [0, 0x36597): Mirror 1: 0xf0b58951 Mirror 10: 0xf0b58951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36597: skipped 000978 1788541884.986252 trunc from 0x036597 to 0x01c588 (0x1a00e bytes) 000979 1788541885.081568 write 0x00f3ec thru 0x013686 (0x429b bytes) 000980 1788541885.092008 trunc from 0x01c589 to 0x03ad01 (0x1e779 bytes) 000981 1788541885.121686 read 0x0260b6 thru 0x02bf11 (0x5e5c bytes) 000982 1788541885.139321 fallocate 0x02124a thru 0x02d417 (0xc1ce bytes) 000983 1788541885.143738 fallocate 0x016b08 thru 0x0253e2 (0xe8db bytes) 000984 1788541885.160744 mapwrite 0x00f4b6 thru 0x0132b4 (0x3dff bytes) 000986 1788541885.188264 read 0x02201a thru 0x02795d (0x5944 bytes) 000987 1788541885.198447 read 0x01fb69 thru 0x020aee (0xf86 bytes) 000988 1788541885.199008 read 0x020ca0 thru 0x02b5d0 (0xa931 bytes) 000990 1788541885.203683 mapwrite 0x01815e thru 0x021739 (0x95dc bytes) 000991 1788541885.232144 trunc from 0x03ad02 to 0x02f816 (0xb4eb bytes) 000992 1788541885.245470 read 0x01c7d9 thru 0x026dc0 (0xa5e8 bytes) 000993 1788541885.246076 fallocate 0x021ecc thru 0x0257d2 (0x3907 bytes) 000994 1788541885.254253 mapwrite 0x032d03 thru 0x03c519 (0x9817 bytes) 000995 1788541885.302331 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3c51a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c51a): Mirror 3: 0x295438d (stale) Mirror 4: 0x295438d (stale) Mirror 5: 0x295438d (stale) Mirror 7: 0x295438d (stale) Mirror 8: 0xbff2acb5 (stale) Mirror 9: 0xbd9bf510 (stale) Mirror 10: 0x6fa2b83e (stale) CRC-32 checksum value for chunk [0, 0x3c51a): Mirror 1: 0x1c3f02c4 Mirror 11: 0x1c3f02c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c51a: skipped 000996 1788541886.342941 write 0x0300ff thru 0x03693a (0x683c bytes) 000998 1788541886.395883 MIRROR_RESYNC 9 mirrors 000999 1788541888.975387 MIRROR_RESYNC 9 mirrors 001000 1788541889.582775 trunc from 0x014c7b to 0x020243 (0xb5c9 bytes) All operations completed A-OK! PASS 70a (117s) == sanity-flr test 71: check mirror extend parent fid ==== 13:11:35 (1788541895) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x20:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 13:11:43 (1788541903) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x385:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143098 s, 14.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x367:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x385:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11303 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x367:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x386:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x385:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x367:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x386:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x385:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x368:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 13:11:51 (1788541911) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre2 Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3423584 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1702464 .. Extending file size to 6120256 .. Extending file size to 7055360 .. Extending file size to 8392960 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7198016 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7604928 .. Extending file size to 7787040 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8922240 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8775040 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61609 61611 61612 61613 61616 failed 192.168.206.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre2 (opts:) 192.168.206.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:13:09 (1788541989) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre2 Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1499616 .. Extending file size to 4790720 .. done Extending file size to 5797728 .. Extending file size to 8151072 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5913760 .. Extending file size to 6190304 .. Extending file size to 7721920 .. Extending file size to 8444384 .. Extending file size to 8342272 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8502432 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8714880 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9215264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63379 63380 63381 63385 63386 done 192.168.206.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre2 (opts:) 192.168.206.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:14:28 (1788542068) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2736 1268568 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17800 3589220 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25772 3581248 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43572 7170468 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2736 1268568 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17800 3589220 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25772 3581248 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43572 7170468 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0963117 s, 21.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163595 s, 12.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe863f7b4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:14:44 (1788542084) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106909 s, 19.6 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 13:14:50 (1788542090) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x266:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:14:59 (1788542099) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121241 s, 17.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:15:07 (1788542107) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.076805 s, 27.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x380:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x380:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x381:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:15:15 (1788542115) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0754963 s, 27.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x382:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x382:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x383:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:15:46 (1788542146) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg601-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795792 s, 13.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:16:29 (1788542189) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg601-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.089106 s, 23.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x384:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x384:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:17:18 (1788542238) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg601-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg601-server: Pool lustre.test_204f created Adding targets to pool oleg601-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg601-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.091786 s, 22.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg601-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg601-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg601-server: Pool lustre.test_204f destroyed PASS 204f (67s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:18:25 (1788542305) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:18:32 (1788542312) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:18:40 (1788542320) oleg601-server: Pool lustre.test_206 created Adding targets to pool oleg601-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg601-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg601-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg601-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg601-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:19:06 (1788542346) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866852 s, 12.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054646 s, 19.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x766cc2ed Mirror 3: 0x766cc2ed PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:19:15 (1788542355) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:19:18 (1788542358) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:19:22 (1788542362) Starting client: oleg601-client.virtnet: -o user_xattr,flock 192.168.206.101@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0636696 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1f00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2620000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2620000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2620000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1420000 Read this (4 bytes): BBBB 192.168.206.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg601-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:20:00 (1788542400) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0116093 s, 706 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:20:07 (1788542407) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329169 s, 31.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x287 lmm_fid: [0x200000405:0x287:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x287 lmm_fid: [0x200000405:0x287:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3ae:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611783 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 13:20:09.000000000 -0400 Modify: 2026-09-04 13:20:09.000000000 -0400 Change: 2026-09-04 13:20:10.000000000 -0400 Birth: 2026-09-04 13:20:09.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:20:15 (1788542415) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318056 s, 33.0 MB/s oleg601-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg601-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg601-server: O/280000401/d15/3af: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:20:36 (1788542436) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.229641 s, 178 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0146328 s, 280 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 13:20 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x28c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 13:20 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:20:43 (1788542443) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:20:51 (1788542451) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1718 sec =========== 13:20:59 (1788542459) === sanity-flr: start cleanup 13:21:01 (1788542461) === === sanity-flr: finish cleanup 13:21:08 (1788542468) ===