-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 08:58:24 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg651-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:58:44 (1788526724) === oleg651-client.virtnet: executing check_config_client /mnt/lustre oleg651-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg651-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b3c4817d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b3c4817d800.idle_timeout=debug disable quota as required oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:59:17 (1788526757) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7607] == sanity-flr test 0a: lfs mirror create with -N option == 08:59:20 (1788526760) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:59:39 (1788526779) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:59:42 (1788526782) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:59:46 (1788526786) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:00:10 (1788526810) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:00:14 (1788526814) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:00:17 (1788526817) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b3c4817d800.stats=clear osc.lustre-OST0001-osc-ffff9b3c4817d800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:00:25 (1788526825) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:00:37 (1788526837) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:00:45 (1788526845) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:00:49 (1788526849) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:00:56 (1788526856) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 09:01:06 (1788526866) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:01:13 (1788526873) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:01:17 (1788526877) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:01:23 (1788526883) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:01:33 (1788526893) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:01:39 (1788526899) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 2.05736 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982735 s, 10.7 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:01:49 (1788526909) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493382 s, 21.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:01:57 (1788526917) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:02:07 (1788526927) Stopping /mnt/lustre-ost1 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b3c4817d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b3c4817d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg651-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg651-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b3c4817d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b3c4817d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg651-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (128s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:04:15 (1788527055) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.289593 s, 14.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b3c4817d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b3c4817d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg651-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:04:59 (1788527099) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b3c4817d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b3c4817d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg651-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b3c4817d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b3c4817d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg651-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:06:19 (1788527179) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.12873 s, 10.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 09:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.311s user 0m0.001s sys 0m0.185s Stopping /mnt/lustre-ost1 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b3c4817d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b3c4817d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.292s user 0m0.001s sys 0m0.191s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg651-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg651-server oleg651-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b3c4817d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b3c4817d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.531s user 0m0.001s sys 0m0.205s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg651-server: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg651-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg651-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b3c4817d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b3c4817d800.ost_server_uuid in FULL state after 0 sec PASS 33b (109s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:08:08 (1788527288) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:08:11 (1788527291) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:08:15 (1788527295) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 09:08:19 (1788527299) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 09:08:27 (1788527307) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.707338 s, 17.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:08:41 (1788527321) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254361 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281634 s, 37.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:08:49 (1788527329) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:08:59 (1788527339) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00128131 s, 799 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00625729 s, 164 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00160486 s, 638 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0041545 s, 246 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 09:09:12 (1788527352) create test files with size 15749395 .. 21cda94b560ce89a8091979148b88798 - 6b726d0c4a5d595a58d0178cd9633c76 - d9c6e3cc1f6fd4707eb958afa3de5d38 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg651-client.virtnet /mnt/lustre (opts:) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 09:09:41 (1788527381) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:09:56 (1788527396) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:10:03 (1788527403) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107035 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258843 s, 40.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 09:10:13 (1788527413) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180226 s, 11.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.471845 s, 8.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171074 s, 12.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 09:10:30 (1788527430) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:10:34 (1788527434) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:10:38 (1788527438) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284665 s, 36.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 09:10:45 (1788527445) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:10:49 (1788527449) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:10:57 (1788527457) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.172841 s, 60.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg651-client.virtnet /mnt/lustre (opts:) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:11:11 (1788527471) llite.lustre-ffff9b3c456a1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.355725 s, 29.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg651-client.virtnet /mnt/lustre (opts:) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b3c46593000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:11:26 (1788527486) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:12:11 (1788527531) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:12:51 (1788527571) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:12:54 (1788527574) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.24282 s, 13.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.204448 s, 15.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1f276d37 (nosync,stale) Mirror 2: 0x1f276d37 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 979106395afa7945606874168e374841 - mirror 1 checksum: eba8dc4c18d48e5d6d05933a4bddb7d7 - mirror 2 checksum: 979106395afa7945606874168e374841 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T09:12:56-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T09:12:56-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 979106395afa7945606874168e374841 - mirror 1 checksum: eba8dc4c18d48e5d6d05933a4bddb7d7 - mirror 2 checksum: eba8dc4c18d48e5d6d05933a4bddb7d7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T09:12:56-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T09:12:56-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:13:06 (1788527586) Failing mds1 on oleg651-server Stopping /mnt/lustre-mds1 (opts:) on oleg651-server 09:13:17 (1788527597) shut down facet: mds1 facet_host: oleg651-server facet_failover_host: oleg651-server Failover mds1 to oleg651-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg651-server: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-MDT0000 09:13:38 (1788527618) targets are mounted 09:13:38 (1788527618) facet_failover done oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381277 s, 27.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:14:06 (1788527646) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:14:09 (1788527649) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:14:13 (1788527653) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00679444 s, 2.4 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:14:20 (1788527660) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:14:23 (1788527663) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0124579 s, 329 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.166787 s, 24.6 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:14:32 (1788527672) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:14:55 (1788527695) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:15:23 (1788527723) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:15:46 (1788527746) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:16:09 (1788527769) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 09:16:16 (1788527776) mirror create pid 54324 mirror split pid 54326 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54324 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54326 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 09:17:23 (1788527843) Seed set to 7845 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x3bc22 000002 1788527845.430214 trunc from 00000000 to 0x03bc21 (0x3bc22 bytes) 000004 1788527845.460991 read 0x0036b9 thru 0x00a455 (0x6d9d bytes) 000006 1788527845.476767 mapread 0x00aa41 thru 0x019edd (0xf49d bytes) 000008 1788527845.518446 write 0x0215b8 thru 0x02b669 (0xa0b2 bytes) 000009 1788527845.534745 mapwrite 0x02390d thru 0x02448f (0xb83 bytes) 000011 1788527845.595936 write 0x009ed4 thru 0x01382d (0x995a bytes) 000012 1788527845.597065 mapwrite 0x00872a thru 0x00eb81 (0x6458 bytes) 000013 1788527845.650243 mapwrite 0x000ad8 thru 0x009fb6 (0x94df bytes) 000014 1788527845.695752 mapwrite 0x02104f thru 0x0239a8 (0x295a bytes) 000016 1788527845.741041 mapread 0x0103f5 thru 0x010f0c (0xb18 bytes) 000017 1788527845.741953 read 0x00ebb4 thru 0x01c50f (0xd95c bytes) 000019 1788527845.743154 trunc from 0x03bc22 to 0x025f25 (0x15cfc bytes) 000020 1788527845.763157 trunc from 0x025f26 to 0x010bf9 (0x1532c bytes) 000022 1788527845.780648 punch from 0x00bfc1 to 0x00e7d9 (0x2819 bytes) 000023 1788527845.792970 mapread 0x010a19 thru 0x010bf9 (0x1e1 bytes) 000024 1788527845.793873 punch from 0x0075d5 to 0x010bf9 (0x9625 bytes) 000026 1788527845.804488 punch from 0x008bab to 0x010bf9 (0x804f bytes) 000028 1788527845.812292 punch from 0x001ec4 to 0x00d0ca (0xb207 bytes) 000030 1788527845.821536 write 0x003507 thru 0x00354f (0x49 bytes) 000032 1788527845.836221 read 0x008d42 thru 0x010705 (0x79c4 bytes) 000034 1788527845.838500 write 0x01b5a8 thru 0x0257c4 (0xa21d bytes) 000035 1788527845.840368 trunc from 0x0257c5 to 0x03ab6b (0x153a7 bytes) 000037 1788527845.862254 write 0x02e86c thru 0x030fcb (0x2760 bytes) 000038 1788527845.888319 punch from 0x02b5a6 to 0x035c23 (0xa67e bytes) 000039 1788527845.901683 write 0x028f50 thru 0x03738a (0xe43b bytes) 000040 1788527845.906417 write 0x02d5eb thru 0x039bd3 (0xc5e9 bytes) 000041 1788527845.913196 mapwrite 0x004a71 thru 0x00e398 (0x9928 bytes) 000042 1788527846.041022 mapwrite 0x00ea6d thru 0x011683 (0x2c17 bytes) 000045 1788527846.113196 mapwrite 0x01e579 thru 0x026b00 (0x8588 bytes) 000046 1788527846.189751 trunc from 0x03ab6c to 0x017555 (0x23616 bytes) 000047 1788527846.205553 mapwrite 0x03b664 thru 0x03ffff (0x499c bytes) 000048 1788527846.292337 MIRROR_EXTEND to 2 mirrors 000049 1788527847.157044 punch from 0x00c703 to 0x017e6d (0xb76b bytes) 000051 1788527847.259036 mapread 0x0342cf thru 0x03ffff (0xbd31 bytes) 000052 1788527847.288185 read 0x012e50 thru 0x01bdb7 (0x8f68 bytes) 000053 1788527847.299504 write 0x02963e thru 0x02b4aa (0x1e6d bytes) 000054 1788527847.315945 read 0x029a50 thru 0x02dfc6 (0x4577 bytes) 000055 1788527847.327071 trunc from 0x040000 to 0x01946b (0x26b94 bytes) 000059 1788527847.374960 mapread 0x011123 thru 0x01620e (0x50ec bytes) 000061 1788527847.380347 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1946c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1946c): Mirror 2: 0x3025e303 (stale) CRC-32 checksum value for chunk [0, 0x1946c): Mirror 1: 0x6276c629 Mirror 3: 0x6276c629 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1946c: skipped 000063 1788527848.070151 trunc from 0x01946c to 0x02680c (0xd3a1 bytes) 000064 1788527848.187630 punch from 0x007255 to 0x0122a8 (0xb054 bytes) 000065 1788527848.196988 mapwrite 0x013c70 thru 0x01bf37 (0x82c8 bytes) 000067 1788527848.264166 trunc from 0x02680d to 0x024542 (0x22ca bytes) 000068 1788527848.280339 read 0x0042c6 thru 0x012499 (0xe1d4 bytes) 000074 1788527848.291012 punch from 0x01d07c to 0x021a64 (0x49e9 bytes) 000075 1788527848.297031 trunc from 0x024543 to 0x025b64 (0x1622 bytes) 000076 1788527848.311988 write 0x01cf02 thru 0x02a7dd (0xd8dc bytes) 000077 1788527848.313669 write 0x03d840 thru 0x03ffff (0x27c0 bytes) 000078 1788527848.316058 punch from 0x034a53 to 0x0360d6 (0x1684 bytes) 000079 1788527848.339482 write 0x032553 thru 0x03ea91 (0xc53f bytes) 000081 1788527848.347366 trunc from 0x040000 to 0x01e772 (0x2188d bytes) 000082 1788527848.366062 trunc from 0x01e773 to 0x034a9e (0x1632c bytes) 000083 1788527848.387972 read 0x002e19 thru 0x00c48f (0x9677 bytes) 000085 1788527848.402519 mapwrite 0x00d9e0 thru 0x01473b (0x6d5c bytes) 000086 1788527848.459906 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34a9f): Mirror 3: 0x9f9e8f07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a9f): Mirror 3: 0x9f9e8f07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a9f: skipped 000087 1788527848.744020 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) 000088 1788527848.884336 MIRROR_EXTEND to 2 mirrors 000092 1788527849.888087 trunc from 0x034a9f to 0x012131 (0x2296d bytes) 000093 1788527850.066429 mapwrite 0x002843 thru 0x011dec (0xf5aa bytes) 000094 1788527850.313389 mapread 0x007fdc thru 0x00ee9f (0x6ec4 bytes) 000095 1788527850.319720 MIRROR_RESYNC 2 mirrors 000096 1788527851.289754 write 0x01bbc5 thru 0x02b7f8 (0xfc34 bytes) 000098 1788527851.422158 MIRROR_RESYNC 2 mirrors 000099 1788527852.309569 trunc from 0x02b7f9 to 0x014a12 (0x16de6 bytes) 000101 1788527852.370503 punch from 0x013ae0 to 0x014a12 (0xf33 bytes) 000102 1788527852.375755 mapread 0x00b048 thru 0x00fa08 (0x49c1 bytes) 000103 1788527852.399547 write 0x00eb23 thru 0x0179fd (0x8edb bytes) 000104 1788527852.400990 mapwrite 0x01c75f thru 0x02240d (0x5caf bytes) 000106 1788527852.523024 MIRROR_RESYNC 2 mirrors 000107 1788527853.201959 read 0x01e635 thru 0x021ae8 (0x34b4 bytes) 000108 1788527853.210048 write 0x03f591 thru 0x03ffff (0xa6f bytes) 000109 1788527853.299840 punch from 0x00b63e to 0x012bd3 (0x7596 bytes) 000110 1788527853.317516 punch from 0x03bc43 to 0x03ffff (0x43bd bytes) 000111 1788527853.322373 punch from 0x0306f5 to 0x03c6ff (0xc00b bytes) 000112 1788527853.336310 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5bb843b0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f9946b3 Mirror 3: 0x6f9946b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000113 1788527854.492579 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5bb843b0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f9946b3 Mirror 3: 0x6f9946b3 Mirror 4: 0x6f9946b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000114 1788527855.711691 mapwrite 0x02fd52 thru 0x039734 (0x99e3 bytes) 000115 1788527855.934561 mapread 0x038dd6 thru 0x03bbef (0x2e1a bytes) truncating to largest ever: 0x3fd24 000116 1788527855.937564 trunc from 0x040000 to 0x03fd23 (0x2dc bytes) 000117 1788527855.968567 mapwrite 0x03fc80 thru 0x03ffff (0x380 bytes) 000118 1788527856.056317 read 0x006886 thru 0x00d33d (0x6ab8 bytes) 000121 1788527856.071428 mapread 0x020fef thru 0x021646 (0x658 bytes) 000124 1788527856.090133 read 0x022fda thru 0x025f90 (0x2fb7 bytes) 000126 1788527856.100542 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5bb843b0 (stale) Mirror 3: 0x6f9946b3 (stale) Mirror 4: 0x6f9946b3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbc430e03 Mirror 5: 0xbc430e03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000127 1788527857.126404 mapread 0x00fd90 thru 0x01ea05 (0xec76 bytes) 000128 1788527857.170832 mapwrite 0x027d27 thru 0x033965 (0xbc3f bytes) 000130 1788527857.403689 punch from 0x012942 to 0x017bab (0x526a bytes) 000131 1788527857.410571 trunc from 0x040000 to 0x0005a4 (0x3fa5b bytes) 000133 1788527857.452333 mapwrite 0x01316a thru 0x01410c (0xfa3 bytes) 000134 1788527857.600053 MIRROR_RESYNC 5 mirrors 000135 1788527858.806261 mapwrite 0x03e4e8 thru 0x03ffff (0x1b18 bytes) 000136 1788527858.980679 mapread 0x01d029 thru 0x02359c (0x6574 bytes) 000137 1788527858.994808 write 0x038a37 thru 0x03ffff (0x75c9 bytes) 000138 1788527859.000902 mapread 0x0355f1 thru 0x03ffff (0xaa0f bytes) 000139 1788527859.009678 write 0x034f1c thru 0x03d689 (0x876e bytes) 000141 1788527859.012350 read 0x033ad0 thru 0x036844 (0x2d75 bytes) 000142 1788527859.012982 read 0x02a39f thru 0x03778b (0xd3ed bytes) 000143 1788527859.024907 punch from 0x03e45a to 0x03ffff (0x1ba6 bytes) 000144 1788527859.045961 punch from 0x019ffc to 0x01d291 (0x3296 bytes) 000145 1788527859.050750 punch from 0x03bfc5 to 0x03ffff (0x403b bytes) 000146 1788527859.054948 mapread 0x03bbae thru 0x03f6a5 (0x3af8 bytes) 000147 1788527859.076629 write 0x028eb5 thru 0x02a5df (0x172b bytes) 000150 1788527859.078978 write 0x02e6c1 thru 0x0389d9 (0xa319 bytes) 000152 1788527859.081337 trunc from 0x040000 to 0x024034 (0x1bfcb bytes) 000153 1788527859.114876 mapread 0x000792 thru 0x007fb0 (0x781f bytes) 000154 1788527859.116661 punch from 0x01a7cf to 0x01d0c9 (0x28fb bytes) 000155 1788527859.128443 mapread 0x02190b thru 0x024034 (0x272a bytes) 000156 1788527859.143334 punch from 0x02156b to 0x024034 (0x2aca bytes) 000157 1788527859.159156 mapwrite 0x03e8f1 thru 0x03ffff (0x170f bytes) 000159 1788527859.248561 read 0x00147c thru 0x00333f (0x1ec4 bytes) 000160 1788527859.249170 write 0x01090f thru 0x010c3e (0x330 bytes) 000161 1788527859.250000 trunc from 0x040000 to 0x010884 (0x2f77b bytes) 000162 1788527859.271740 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10885): Mirror 2: 0x4ebc420a (stale) Mirror 3: 0x4ebc420a (stale) Mirror 4: 0x4ebc420a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10885): Mirror 2: 0x4ebc420a (stale) Mirror 3: 0x4ebc420a (stale) Mirror 4: 0x4ebc420a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10885: skipped 000163 1788527859.631251 trunc from 0x010885 to 0x02d38b (0x1cb07 bytes) 000165 1788527859.675310 mapwrite 0x0325f4 thru 0x03f0af (0xcabc bytes) 000166 1788527859.859725 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f0b0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0b0): Mirror 2: 0x97406730 (stale) Mirror 3: 0x97406730 (stale) Mirror 4: 0x97406730 (stale) CRC-32 checksum value for chunk [0, 0x3f0b0): Mirror 1: 0xd2618d8 Mirror 5: 0xd2618d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0b0: skipped 000167 1788527860.880081 read 0x010a4b thru 0x01fe5e (0xf414 bytes) 000169 1788527860.903890 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f0b0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0b0): Mirror 2: 0x97406730 (stale) Mirror 3: 0x97406730 (stale) Mirror 4: 0x97406730 (stale) CRC-32 checksum value for chunk [0, 0x3f0b0): Mirror 1: 0xd2618d8 Mirror 5: 0xd2618d8 Mirror 6: 0xd2618d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0b0: skipped 000171 1788527862.227432 trunc from 0x03f0b0 to 0x007777 (0x37938 bytes) 000172 1788527862.397472 read 0x005aba thru 0x007777 (0x1cbe bytes) 000173 1788527862.412138 mapwrite 0x014199 thru 0x01c173 (0x7fdb bytes) 000175 1788527862.527819 trunc from 0x01c174 to 0x008b32 (0x13641 bytes) 000176 1788527862.551204 write 0x0302ba thru 0x0352b2 (0x4ff9 bytes) 000177 1788527862.552388 mapwrite 0x016de9 thru 0x017a5c (0xc74 bytes) 000178 1788527862.650406 punch from 0x0092b8 to 0x017056 (0xdd9f bytes) 000179 1788527862.656383 punch from 0x0149e8 to 0x0248f9 (0xff12 bytes) 000180 1788527862.662563 write 0x03f4f5 thru 0x03ffff (0xb0b bytes) 000181 1788527862.665292 mapread 0x03f6ed thru 0x03ffff (0x913 bytes) 000185 1788527862.671078 punch from 0x00d4a1 to 0x018048 (0xaba8 bytes) 000186 1788527862.694424 mapwrite 0x01ce72 thru 0x026052 (0x91e1 bytes) 000192 1788527862.866200 read 0x00b569 thru 0x00e2a6 (0x2d3e bytes) 000193 1788527862.887675 MIRROR_RESYNC 6 mirrors 000194 1788527864.288521 mapread 0x01fea1 thru 0x023697 (0x37f7 bytes) 000197 1788527864.331886 trunc from 0x040000 to 0x02cc61 (0x1339e bytes) 000198 1788527864.498083 read 0x008a28 thru 0x0153ec (0xc9c5 bytes) 000199 1788527864.520961 read 0x012d35 thru 0x01af6e (0x823a bytes) 000202 1788527864.537197 punch from 0x025f92 to 0x026cfe (0xd6d bytes) 000204 1788527864.543456 trunc from 0x02cc62 to 0x028c9c (0x3fc5 bytes) 000206 1788527864.574060 mapread 0x01114e thru 0x020c6c (0xfb1f bytes) 000207 1788527864.576116 read 0x004e13 thru 0x010e9e (0xc08c bytes) 000208 1788527864.578143 mapwrite 0x027d19 thru 0x0322e5 (0xa5cd bytes) 000209 1788527864.722210 punch from 0x018f89 to 0x0245b9 (0xb631 bytes) 000210 1788527864.728898 trunc from 0x0322e6 to 0x03c009 (0x9d24 bytes) 000211 1788527864.750958 write 0x01dc5a thru 0x02d526 (0xf8cd bytes) 000212 1788527864.771798 write 0x025598 thru 0x034e3b (0xf8a4 bytes) 000214 1788527864.779093 mapread 0x00867f thru 0x0109e2 (0x8364 bytes) 000215 1788527864.787104 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c00a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c00a): Mirror 2: 0x4e222951 (stale) Mirror 3: 0x4e222951 (stale) Mirror 4: 0x4e222951 (stale) Mirror 5: 0x4e222951 (stale) Mirror 6: 0x4e222951 (stale) CRC-32 checksum value for chunk [0, 0x3c00a): Mirror 1: 0x61bcec0c Mirror 7: 0x61bcec0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c00a: skipped 000216 1788527866.298539 mapwrite 0x025f33 thru 0x02c89c (0x696a bytes) 000218 1788527866.577659 mapwrite 0x02edc1 thru 0x03a6c3 (0xb903 bytes) 000219 1788527866.662936 punch from 0x01dca1 to 0x02c770 (0xead0 bytes) 000220 1788527866.687047 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c00a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c00a): Mirror 2: 0x4e222951 (stale) Mirror 3: 0x4e222951 (stale) Mirror 4: 0x4e222951 (stale) Mirror 5: 0x4e222951 (stale) Mirror 6: 0x4e222951 (stale) Mirror 7: 0x61bcec0c (stale) CRC-32 checksum value for chunk [0, 0x3c00a): Mirror 1: 0xe7d070dd Mirror 8: 0xe7d070dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c00a: skipped 000221 1788527867.618544 mapwrite 0x013076 thru 0x019b59 (0x6ae4 bytes) 000222 1788527867.813015 mapwrite 0x015d72 thru 0x01a63f (0x48ce bytes) 000223 1788527867.869707 mapread 0x03accd thru 0x03c009 (0x133d bytes) 000224 1788527867.881445 write 0x01f755 thru 0x026dd6 (0x7682 bytes) 000225 1788527867.891142 trunc from 0x03c00a to 0x03fcd7 (0x3cce bytes) 000227 1788527867.913020 trunc from 0x03fcd8 to 0x021466 (0x1e871 bytes) 000229 1788527867.933060 read 0x01cd0f thru 0x021466 (0x4758 bytes) 000230 1788527867.944032 MIRROR_RESYNC 8 mirrors 000232 1788527869.247082 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) 000234 1788527869.650398 trunc from 0x021467 to 0x038588 (0x17122 bytes) 000235 1788527869.735209 trunc from 0x038589 to 0x03d87a (0x52f2 bytes) 000236 1788527869.769483 read 0x01cdc3 thru 0x0217b7 (0x49f5 bytes) 000237 1788527869.786515 read 0x0328ca thru 0x03d87a (0xafb1 bytes) 000238 1788527869.799175 trunc from 0x03d87b to 0x0198c0 (0x23fba bytes) 000241 1788527869.816390 mapwrite 0x02519f thru 0x029663 (0x44c5 bytes) 000242 1788527869.940405 read 0x014fbd thru 0x021da4 (0xcde8 bytes) 000243 1788527869.956497 write 0x00ac03 thru 0x017d31 (0xd12f bytes) 000245 1788527869.958851 write 0x01bb84 thru 0x0229de (0x6e5b bytes) 000246 1788527869.961112 write 0x02641f thru 0x02822c (0x1e0e bytes) 000247 1788527869.963499 mapread 0x023112 thru 0x0251a0 (0x208f bytes) 000248 1788527869.964658 mapread 0x001c26 thru 0x008ac0 (0x6e9b bytes) 000249 1788527869.965644 trunc from 0x029664 to 0x02728e (0x23d5 bytes) 000251 1788527869.985550 punch from 0x002929 to 0x00a75c (0x7e34 bytes) 000252 1788527870.015798 MIRROR_RESYNC 7 mirrors 000253 1788527871.549660 read 0x0175e4 thru 0x02356f (0xbf8c bytes) 000254 1788527871.561221 trunc from 0x02728f to 0x02f50f (0x8281 bytes) 000255 1788527871.654104 mapwrite 0x03c51d thru 0x03ffff (0x3ae3 bytes) 000256 1788527871.783754 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbda31a06 (stale) Mirror 3: 0xbda31a06 (stale) Mirror 5: 0xbda31a06 (stale) Mirror 6: 0xbda31a06 (stale) Mirror 7: 0xbda31a06 (stale) Mirror 8: 0xbda31a06 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa529e75e Mirror 9: 0xa529e75e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000257 1788527872.887637 mapread 0x0120dd thru 0x01a4e5 (0x8409 bytes) 000258 1788527872.928226 trunc from 0x040000 to 0x0389b6 (0x7649 bytes) 000259 1788527873.017548 trunc from 0x0389b7 to 0x029864 (0xf152 bytes) 000260 1788527873.052936 read 0x027ee4 thru 0x029864 (0x1981 bytes) 000261 1788527873.059297 read 0x00acf1 thru 0x01341e (0x872e bytes) 000262 1788527873.074722 write 0x010c76 thru 0x01e215 (0xd5a0 bytes) 000263 1788527873.076301 trunc from 0x029865 to 0x014a0b (0x14e59 bytes) 000264 1788527873.116792 mapread 0x01110f thru 0x014a0b (0x38fd bytes) 000265 1788527873.118019 write 0x006101 thru 0x011b14 (0xba14 bytes) 000266 1788527873.122205 write 0x039e65 thru 0x03f054 (0x51f0 bytes) 000267 1788527873.125497 mapwrite 0x03da52 thru 0x03ffff (0x25ae bytes) 000269 1788527873.237507 mapread 0x03260a thru 0x03ead4 (0xc4cb bytes) 000270 1788527873.265347 mapread 0x00747e thru 0x0166d7 (0xf25a bytes) 000271 1788527873.277492 read 0x004329 thru 0x00a071 (0x5d49 bytes) 000272 1788527873.278491 punch from 0x00160f to 0x004f76 (0x3968 bytes) 000273 1788527873.284400 write 0x005cd1 thru 0x00ee6d (0x919d bytes) 000274 1788527873.287093 mapwrite 0x006d08 thru 0x008978 (0x1c71 bytes) 000275 1788527873.401055 read 0x0169ac thru 0x0266cd (0xfd22 bytes) 000276 1788527873.417378 write 0x0210bf thru 0x02dfaa (0xceec bytes) 000277 1788527873.418822 mapwrite 0x0136dc thru 0x01956f (0x5e94 bytes) 000279 1788527873.554776 read 0x007b7e thru 0x014c7e (0xd101 bytes) 000281 1788527873.557108 mapwrite 0x02aa2e thru 0x039067 (0xe63a bytes) 000282 1788527873.643625 read 0x015bf6 thru 0x01ce39 (0x7244 bytes) 000284 1788527873.646452 punch from 0x03f4f7 to 0x03ffff (0xb09 bytes) 000285 1788527873.656381 read 0x02fc1f thru 0x03709d (0x747f bytes) 000286 1788527873.658069 read 0x00c9f7 thru 0x01569c (0x8ca6 bytes) 000287 1788527873.661692 write 0x02bbe9 thru 0x039ed3 (0xe2eb bytes) 000288 1788527873.664151 trunc from 0x040000 to 0x008d92 (0x3726d bytes) 000292 1788527873.709582 mapread 0x000f9e thru 0x001a52 (0xab5 bytes) 000294 1788527873.713002 write 0x03534c thru 0x03da2a (0x86df bytes) 000296 1788527873.720802 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3da2b): Mirror 3: 0x2c795c97 (stale) Mirror 5: 0x2c795c97 (stale) Mirror 6: 0x2c795c97 (stale) Mirror 7: 0x2c795c97 (stale) Mirror 8: 0x2c795c97 (stale) Mirror 9: 0x21ecb26f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da2b): Mirror 3: 0x2c795c97 (stale) Mirror 5: 0x2c795c97 (stale) Mirror 6: 0x2c795c97 (stale) Mirror 7: 0x2c795c97 (stale) Mirror 8: 0x2c795c97 (stale) Mirror 9: 0x21ecb26f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da2b: skipped 000297 1788527874.415740 mapread 0x002f01 thru 0x012a8c (0xfb8c bytes) 000298 1788527874.492730 mapread 0x01d136 thru 0x02375a (0x6625 bytes) 000300 1788527874.511475 write 0x00052c thru 0x0045c5 (0x409a bytes) 000301 1788527874.527580 MIRROR_RESYNC 7 mirrors 000303 1788527876.911850 mapread 0x039809 thru 0x03da2a (0x4222 bytes) 000304 1788527876.944651 mapwrite 0x034d86 thru 0x0399d0 (0x4c4b bytes) 000305 1788527877.135908 mapwrite 0x030eb4 thru 0x031e5e (0xfab bytes) 000306 1788527877.235427 write 0x00e1ef thru 0x01cf5a (0xed6c bytes) 000307 1788527877.247119 read 0x02c9e7 thru 0x03ad94 (0xe3ae bytes) 000308 1788527877.256156 mapread 0x03286e thru 0x03ab79 (0x830c bytes) 000309 1788527877.257446 mapwrite 0x005f9a thru 0x00fce1 (0x9d48 bytes) 000310 1788527877.346582 punch from 0x011df6 to 0x016971 (0x4b7c bytes) 000315 1788527877.358584 punch from 0x03944e to 0x03da2a (0x45dd bytes) 000316 1788527877.370470 mapread 0x014f2a thru 0x02252b (0xd602 bytes) 000320 1788527877.415049 punch from 0x03b2a7 to 0x03d16b (0x1ec5 bytes) 000322 1788527877.439645 mapread 0x0046b5 thru 0x00e78a (0xa0d6 bytes) 000323 1788527877.441238 read 0x00f6ca thru 0x01c793 (0xd0ca bytes) 000325 1788527877.448734 trunc from 0x03da2b to 0x02c85f (0x111cb bytes) 000326 1788527877.521879 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c860): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c860): Mirror 3: 0xa7caf553 (stale) Mirror 5: 0xa7caf553 (stale) Mirror 6: 0xa7caf553 (stale) Mirror 7: 0xa7caf553 (stale) Mirror 8: 0xa7caf553 (stale) Mirror 9: 0xa7caf553 (stale) CRC-32 checksum value for chunk [0, 0x2c860): Mirror 1: 0x7ac074b6 Mirror 10: 0x7ac074b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c860: skipped 000328 1788527879.000791 punch from 0x01865b to 0x0276b3 (0xf059 bytes) 000329 1788527879.088447 write 0x0374c7 thru 0x03ffff (0x8b39 bytes) 000330 1788527879.089908 mapwrite 0x028ecf thru 0x03481e (0xb950 bytes) 000331 1788527879.182926 read 0x0200f9 thru 0x022d9c (0x2ca4 bytes) 000333 1788527879.188667 write 0x00bce0 thru 0x011b20 (0x5e41 bytes) 000334 1788527879.197566 mapread 0x0053be thru 0x00eec0 (0x9b03 bytes) 000335 1788527879.207272 read 0x006851 thru 0x00c166 (0x5916 bytes) 000336 1788527879.209166 write 0x02ea7c thru 0x03e846 (0xfdcb bytes) 000337 1788527879.216723 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6b0fc8f3 (stale) Mirror 6: 0x6b0fc8f3 (stale) Mirror 7: 0x6b0fc8f3 (stale) Mirror 8: 0x6b0fc8f3 (stale) Mirror 9: 0x6b0fc8f3 (stale) Mirror 10: 0x5da89f95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6b0fc8f3 (stale) Mirror 6: 0x6b0fc8f3 (stale) Mirror 7: 0x6b0fc8f3 (stale) Mirror 8: 0x6b0fc8f3 (stale) Mirror 9: 0x6b0fc8f3 (stale) Mirror 10: 0x5da89f95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000339 1788527879.769320 read 0x034dc8 thru 0x039ce2 (0x4f1b bytes) 000340 1788527879.781418 read 0x003a12 thru 0x00dca1 (0xa290 bytes) 000341 1788527879.804421 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6b0fc8f3 (stale) Mirror 7: 0x6b0fc8f3 (stale) Mirror 8: 0x6b0fc8f3 (stale) Mirror 9: 0x6b0fc8f3 (stale) Mirror 10: 0x5da89f95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6b0fc8f3 (stale) Mirror 7: 0x6b0fc8f3 (stale) Mirror 8: 0x6b0fc8f3 (stale) Mirror 9: 0x6b0fc8f3 (stale) Mirror 10: 0x5da89f95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000343 1788527880.257104 mapwrite 0x01e8ce thru 0x02e19e (0xf8d1 bytes) 000344 1788527880.332352 mapwrite 0x033764 thru 0x03ffff (0xc89c bytes) 000345 1788527880.412773 MIRROR_RESYNC 6 mirrors 000348 1788527882.178552 mapwrite 0x02671f thru 0x0300f3 (0x99d5 bytes) 000350 1788527882.511512 punch from 0x00bce0 to 0x0124ee (0x680f bytes) 000351 1788527882.517095 punch from 0x034c67 to 0x039026 (0x43c0 bytes) 000352 1788527882.521357 mapwrite 0x01ff94 thru 0x029566 (0x95d3 bytes) 000353 1788527882.610055 mapwrite 0x01735a thru 0x01bcc4 (0x496b bytes) 000355 1788527882.685454 mapwrite 0x036f93 thru 0x03ffff (0x906d bytes) 000356 1788527882.753529 trunc from 0x040000 to 0x023a27 (0x1c5d8 bytes) 000357 1788527882.782979 read 0x00f233 thru 0x013406 (0x41d4 bytes) 000359 1788527882.801228 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23a28): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23a28): Mirror 6: 0x336673c8 (stale) Mirror 7: 0x336673c8 (stale) Mirror 8: 0x336673c8 (stale) Mirror 9: 0x336673c8 (stale) Mirror 10: 0x336673c8 (stale) CRC-32 checksum value for chunk [0, 0x23a28): Mirror 1: 0x82ddeb73 Mirror 11: 0x82ddeb73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23a28: skipped 000360 1788527884.101420 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23a28): Mirror 7: 0x336673c8 (stale) Mirror 8: 0x336673c8 (stale) Mirror 9: 0x336673c8 (stale) Mirror 10: 0x336673c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23a28): Mirror 7: 0x336673c8 (stale) Mirror 8: 0x336673c8 (stale) Mirror 9: 0x336673c8 (stale) Mirror 10: 0x336673c8 (stale) CRC-32 checksum value for chunk [0, 0x23a28): Mirror 1: 0x82ddeb73 Mirror 11: 0x82ddeb73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23a28: skipped 000363 1788527884.994005 trunc from 0x023a28 to 0x03cd4c (0x19325 bytes) 000364 1788527885.264412 read 0x01403a thru 0x018a23 (0x49ea bytes) 000365 1788527885.300804 trunc from 0x03cd4d to 0x00a014 (0x32d38 bytes) 000366 1788527885.332942 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa015): Mirror 7: 0x2598ede1 (stale) Mirror 8: 0x2598ede1 (stale) Mirror 9: 0x2598ede1 (stale) Mirror 10: 0x2598ede1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa015): Mirror 7: 0x2598ede1 (stale) Mirror 8: 0x2598ede1 (stale) Mirror 9: 0x2598ede1 (stale) Mirror 10: 0x2598ede1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa015: skipped 000367 1788527885.828481 write 0x02a2dc thru 0x03209c (0x7dc1 bytes) 000370 1788527885.838212 mapwrite 0x02f97b thru 0x037b89 (0x820f bytes) 000371 1788527886.030435 read 0x028a38 thru 0x029864 (0xe2d bytes) 000372 1788527886.041441 read 0x019b73 thru 0x01b195 (0x1623 bytes) 000374 1788527886.080510 read 0x0106d2 thru 0x01c70c (0xc03b bytes) 000376 1788527886.088680 punch from 0x0189c4 to 0x02470a (0xbd47 bytes) 000377 1788527886.107024 mapread 0x01e3be thru 0x0250b6 (0x6cf9 bytes) 000378 1788527886.133416 write 0x016e38 thru 0x01bb4a (0x4d13 bytes) 000379 1788527886.134638 write 0x03553f thru 0x03c1d8 (0x6c9a bytes) 000380 1788527886.142561 write 0x031c89 thru 0x032d60 (0x10d8 bytes) 000381 1788527886.153170 mapread 0x01b377 thru 0x025d91 (0xaa1b bytes) 000382 1788527886.157138 write 0x029b4c thru 0x029ef8 (0x3ad bytes) 000383 1788527886.159446 trunc from 0x03c1d9 to 0x0084f7 (0x33ce1 bytes) 000384 1788527886.202103 MIRROR_RESYNC 5 mirrors 000385 1788527887.338125 trunc from 0x0084f8 to 0x0075ec (0xf0b bytes) 000386 1788527887.520826 write 0x030ced thru 0x03e73b (0xda4f bytes) 000387 1788527887.529969 mapwrite 0x01b90e thru 0x01e692 (0x2d85 bytes) 000388 1788527887.635921 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e73c): Mirror 7: 0xe80dbb46 (stale) Mirror 8: 0xe80dbb46 (stale) Mirror 9: 0xe80dbb46 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e73c): Mirror 7: 0xe80dbb46 (stale) Mirror 8: 0xe80dbb46 (stale) Mirror 9: 0xe80dbb46 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e73c: skipped 000389 1788527888.183590 mapwrite 0x0177d8 thru 0x026cdb (0xf504 bytes) 000390 1788527888.336892 write 0x011fe7 thru 0x01bacd (0x9ae7 bytes) 000391 1788527888.353081 write 0x0398f1 thru 0x03d59e (0x3cae bytes) 000392 1788527888.365241 trunc from 0x03e73c to 0x00d4ab (0x31290 bytes) 000393 1788527888.389106 punch from 0x00b236 to 0x00d4ab (0x2276 bytes) 000395 1788527888.401973 mapread 0x0070f0 thru 0x00d4ab (0x63bc bytes) 000396 1788527888.415418 punch from 0x008dd6 to 0x008f1b (0x146 bytes) 000397 1788527888.432572 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd4ac): Mirror 8: 0x7a76956e (stale) Mirror 9: 0x7a76956e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd4ac): Mirror 8: 0x7a76956e (stale) Mirror 9: 0x7a76956e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd4ac: skipped 000398 1788527888.792716 read 0x002c64 thru 0x004672 (0x1a0f bytes) 000399 1788527888.802111 mapread 0x006654 thru 0x00d4ab (0x6e58 bytes) 000400 1788527888.847488 write 0x03ca9c thru 0x03ffff (0x3564 bytes) 000402 1788527888.854998 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc674e2c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc674e2c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000404 1788527889.195454 mapwrite 0x00339e thru 0x00b16f (0x7dd2 bytes) 000406 1788527889.269852 write 0x011803 thru 0x01ebf6 (0xd3f4 bytes) 000407 1788527889.282368 read 0x005035 thru 0x0129a0 (0xd96c bytes) 000409 1788527889.284099 trunc from 0x040000 to 0x03b779 (0x4886 bytes) 000410 1788527889.317710 mapwrite 0x01c081 thru 0x028852 (0xc7d2 bytes) 000411 1788527889.467360 mapread 0x00cd63 thru 0x019547 (0xc7e5 bytes) 000412 1788527889.468410 read 0x0132f6 thru 0x01a3d7 (0x70e2 bytes) 000413 1788527889.469019 punch from 0x01e87f to 0x026533 (0x7cb5 bytes) 000414 1788527889.485876 mapread 0x0290aa thru 0x02dda2 (0x4cf9 bytes) 000415 1788527889.491064 mapwrite 0x000add thru 0x006eee (0x6412 bytes) 000416 1788527889.592257 write 0x0192e0 thru 0x024bf9 (0xb91a bytes) 000417 1788527889.594223 punch from 0x01ce6f to 0x022230 (0x53c2 bytes) 000418 1788527889.633209 mapwrite 0x005571 thru 0x009ab0 (0x4540 bytes) 000419 1788527889.740433 trunc from 0x03b77a to 0x0381b5 (0x35c4 bytes) 000420 1788527889.783219 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x381b6): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381b6): Mirror 9: 0xcbd492ac (stale) CRC-32 checksum value for chunk [0, 0x381b6): Mirror 1: 0x7c74a3fd Mirror 10: 0x7c74a3fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381b6: skipped 000421 1788527890.992180 read 0x0283ef thru 0x02e717 (0x6329 bytes) 000422 1788527890.999915 write 0x0111aa thru 0x017c72 (0x6ac9 bytes) 000423 1788527891.163043 mapread 0x01fe7a thru 0x020d6f (0xef6 bytes) 000424 1788527891.188670 write 0x0164ff thru 0x0238f6 (0xd3f8 bytes) 000425 1788527891.202540 write 0x01556b thru 0x017220 (0x1cb6 bytes) 000426 1788527891.204861 read 0x037adb thru 0x0381b5 (0x6db bytes) 000428 1788527891.220103 punch from 0x01d092 to 0x0241c5 (0x7134 bytes) 000429 1788527891.247874 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x381b6): Mirror 10: 0x7c74a3fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381b6): Mirror 10: 0x7c74a3fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381b6: skipped 000430 1788527891.705000 mapwrite 0x02cc8b thru 0x02d5bc (0x932 bytes) 000431 1788527891.849469 mapwrite 0x0071c1 thru 0x00a5ad (0x33ed bytes) 000432 1788527891.892467 read 0x0209cf thru 0x02d09c (0xc6ce bytes) 000433 1788527891.906127 read 0x00902c thru 0x009ee6 (0xebb bytes) 000434 1788527891.907127 punch from 0x00a51f to 0x00d808 (0x32ea bytes) 000435 1788527891.918771 MIRROR_RESYNC 2 mirrors 000436 1788527892.601493 MIRROR_RESYNC 2 mirrors 000437 1788527893.071346 punch from 0x016a0f to 0x0186d6 (0x1cc8 bytes) 000438 1788527893.163325 mapwrite 0x001e9d thru 0x010f06 (0xf06a bytes) 000439 1788527893.408736 punch from 0x0379d5 to 0x0381b5 (0x7e1 bytes) 000440 1788527893.416278 write 0x020ceb thru 0x02937c (0x8692 bytes) 000443 1788527893.445109 write 0x035546 thru 0x036e0c (0x18c7 bytes) 000444 1788527893.466091 write 0x01e1a2 thru 0x021c6d (0x3acc bytes) 000445 1788527893.476787 read 0x00c54f thru 0x010602 (0x40b4 bytes) 000446 1788527893.477412 trunc from 0x0381b6 to 0x0175f1 (0x20bc4 bytes) 000447 1788527893.522005 mapwrite 0x00e718 thru 0x01c649 (0xdf32 bytes) 000448 1788527893.624022 write 0x00ce68 thru 0x01b9b8 (0xeb51 bytes) 000449 1788527893.640165 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) 000450 1788527893.879203 MIRROR_EXTEND to 2 mirrors 000452 1788527894.766994 punch from 0x01adb6 to 0x01c649 (0x1894 bytes) 000454 1788527894.865497 trunc from 0x01c64a to 0x01bca0 (0x9a9 bytes) 000455 1788527894.890469 trunc from 0x01bca1 to 0x01edbd (0x311d bytes) 000458 1788527894.908058 write 0x02d40c thru 0x03bdbb (0xe9b0 bytes) 000459 1788527894.915553 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) 000460 1788527895.125971 trunc from 0x03bdbc to 0x0147b7 (0x27604 bytes) 000462 1788527895.145370 read 0x001297 thru 0x005a36 (0x47a0 bytes) 000464 1788527895.155030 mapwrite 0x01d660 thru 0x020964 (0x3305 bytes) 000465 1788527895.238379 write 0x011c39 thru 0x01a0cc (0x8494 bytes) 000466 1788527895.244356 mapwrite 0x03ec69 thru 0x03ffff (0x1397 bytes) 000467 1788527895.386045 punch from 0x036f25 to 0x03d7ed (0x68c9 bytes) 000468 1788527895.393729 punch from 0x000a3e to 0x008677 (0x7c3a bytes) 000469 1788527895.399728 write 0x0394b7 thru 0x03c32c (0x2e76 bytes) 000470 1788527895.409294 read 0x027514 thru 0x031e05 (0xa8f2 bytes) 000471 1788527895.425360 mapwrite 0x0375f5 thru 0x03ffff (0x8a0b bytes) 000472 1788527895.481971 punch from 0x033911 to 0x033fff (0x6ef bytes) 000474 1788527895.489359 trunc from 0x040000 to 0x03bfa3 (0x405c bytes) 000476 1788527895.512220 mapwrite 0x01a997 thru 0x0273c7 (0xca31 bytes) 000478 1788527895.585364 punch from 0x0306ae to 0x03a65a (0x9fad bytes) 000482 1788527895.592529 trunc from 0x03bfa4 to 0x031b0c (0xa497 bytes) 000483 1788527895.617465 mapread 0x012121 thru 0x01a385 (0x8265 bytes) 000484 1788527895.621718 write 0x02d151 thru 0x033fc3 (0x6e73 bytes) 000487 1788527895.626543 read 0x01d15a thru 0x02a92b (0xd7d2 bytes) 000490 1788527895.631297 trunc from 0x033fc4 to 0x016aac (0x1d517 bytes) 000492 1788527895.672214 trunc from 0x016aad to 0x00b52a (0xb582 bytes) 000493 1788527895.699477 write 0x00a213 thru 0x0158d9 (0xb6c7 bytes) 000494 1788527895.703876 mapwrite 0x015f3e thru 0x01b4bf (0x5582 bytes) 000495 1788527895.868051 mapread 0x000bd8 thru 0x00d2ac (0xc6d5 bytes) 000496 1788527895.871026 mapread 0x01496c thru 0x01b4bf (0x6b54 bytes) 000497 1788527895.873003 write 0x01d5dc thru 0x02b9b3 (0xe3d8 bytes) 000499 1788527895.881912 mapwrite 0x032c8e thru 0x03c521 (0x9894 bytes) 000504 1788527896.045335 mapread 0x0013a3 thru 0x00678e (0x53ec bytes) 000507 1788527896.049542 trunc from 0x03c522 to 0x02054c (0x1bfd5 bytes) 000509 1788527896.088747 read 0x00207c thru 0x00600b (0x3f90 bytes) 000511 1788527896.091471 write 0x0020c6 thru 0x006ca2 (0x4bdd bytes) 000512 1788527896.095574 write 0x02be26 thru 0x03ba04 (0xfbdf bytes) 000514 1788527896.098382 punch from 0x02d274 to 0x039e83 (0xcc10 bytes) 000515 1788527896.129221 read 0x0287ba thru 0x02d855 (0x509c bytes) 000516 1788527896.146932 punch from 0x026246 to 0x02c6aa (0x6465 bytes) 000517 1788527896.152095 trunc from 0x03ba05 to 0x00f24f (0x2c7b5 bytes) 000518 1788527896.178433 mapread 0x00cfe9 thru 0x00f24f (0x2267 bytes) 000519 1788527896.179397 punch from 0x00bae6 to 0x00f24f (0x376a bytes) 000522 1788527896.196870 read 0x00ba15 thru 0x00f24f (0x383b bytes) 000523 1788527896.220258 punch from 0x00eb99 to 0x00f24f (0x6b7 bytes) 000524 1788527896.233345 punch from 0x009ec5 to 0x00f24f (0x538b bytes) 000526 1788527896.251839 punch from 0x00e0c8 to 0x00f24f (0x1188 bytes) 000527 1788527896.266136 read 0x00ed20 thru 0x00f24f (0x530 bytes) 000528 1788527896.282590 read 0x005cd6 thru 0x00f24f (0x957a bytes) 000529 1788527896.283826 punch from 0x00c5db to 0x00f24f (0x2c75 bytes) 000530 1788527896.289404 trunc from 0x00f250 to 0x01c293 (0xd044 bytes) 000533 1788527896.316378 mapwrite 0x01c0c2 thru 0x01fc74 (0x3bb3 bytes) 000534 1788527896.443047 write 0x02e525 thru 0x032912 (0x43ee bytes) 000535 1788527896.446494 punch from 0x0180f7 to 0x018a43 (0x94d bytes) 000536 1788527896.461875 write 0x021711 thru 0x022da0 (0x1690 bytes) 000538 1788527896.473749 mapwrite 0x023f03 thru 0x03139b (0xd499 bytes) 000539 1788527896.531142 write 0x038cbe thru 0x0399d2 (0xd15 bytes) 000541 1788527896.534217 mapread 0x01074d thru 0x01ae8c (0xa740 bytes) 000542 1788527896.545489 MIRROR_EXTEND to 2 mirrors 000543 1788527897.379182 mapread 0x0121d0 thru 0x017d0a (0x5b3b bytes) 000544 1788527897.409419 trunc from 0x0399d3 to 0x02eeeb (0xaae7 bytes) 000545 1788527897.491278 mapread 0x014e57 thru 0x01ff25 (0xb0cf bytes) 000546 1788527897.509326 trunc from 0x02eeec to 0x02a2b3 (0x4c38 bytes) 000547 1788527897.531078 punch from 0x00342d to 0x00bf2f (0x8b03 bytes) 000548 1788527897.541442 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a2b4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2b4): Mirror 2: 0xe38aaf63 (stale) CRC-32 checksum value for chunk [0, 0x2a2b4): Mirror 1: 0xf415a01 Mirror 3: 0xf415a01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2b4: skipped 000549 1788527898.652957 read 0x006df6 thru 0x0094a7 (0x26b2 bytes) 000551 1788527898.660709 read 0x0235ea thru 0x02a2b3 (0x6cca bytes) 000552 1788527898.671017 mapread 0x009902 thru 0x013d2e (0xa42d bytes) 000553 1788527898.672028 read 0x016197 thru 0x02407e (0xdee8 bytes) 000554 1788527898.677288 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a2b4): Mirror 2: 0xe38aaf63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2b4): Mirror 2: 0xe38aaf63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2b4: skipped 000555 1788527898.992699 write 0x013fce thru 0x014922 (0x955 bytes) 000556 1788527899.012160 write 0x00df0b thru 0x00e7d2 (0x8c8 bytes) 000557 1788527899.031241 mapread 0x024cb7 thru 0x02a2b3 (0x55fd bytes) 000558 1788527899.066881 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) 000559 1788527899.261445 read 0x00668e thru 0x011898 (0xb20b bytes) 000561 1788527899.284213 mapwrite 0x01d49d thru 0x027134 (0x9c98 bytes) 000562 1788527899.412912 punch from 0x00be65 to 0x012ed0 (0x706c bytes) 000563 1788527899.419073 read 0x0244f2 thru 0x0282b8 (0x3dc7 bytes) 000564 1788527899.445989 punch from 0x005a3c to 0x00cfcc (0x7591 bytes) 000567 1788527899.454475 write 0x007932 thru 0x012f63 (0xb632 bytes) 000568 1788527899.463244 trunc from 0x02a2b4 to 0x02843f (0x1e74 bytes) 000569 1788527899.499285 MIRROR_EXTEND to 2 mirrors 000570 1788527900.501517 MIRROR_EXTEND to 3 mirrors 000571 1788527901.369064 trunc from 0x028440 to 0x02b9be (0x357f bytes) 000573 1788527901.484355 mapread 0x01a3c5 thru 0x023af8 (0x9734 bytes) 000575 1788527901.527864 mapwrite 0x0102a4 thru 0x013e06 (0x3b63 bytes) 000576 1788527901.619959 mapwrite 0x018808 thru 0x01b131 (0x292a bytes) 000577 1788527901.685732 trunc from 0x02b9bf to 0x0123db (0x195e3 bytes) 000578 1788527901.702791 punch from 0x0068f2 to 0x0123db (0xbaea bytes) 000579 1788527901.707425 trunc from 0x0123dc to 0x02f8c6 (0x1d4eb bytes) 000581 1788527901.729467 mapread 0x010ff5 thru 0x013113 (0x211f bytes) 000582 1788527901.746518 punch from 0x028b07 to 0x02f8c6 (0x6dc0 bytes) 000583 1788527901.758043 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f8c7): Mirror 2: 0x82157b7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f8c7): Mirror 2: 0x82157b7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f8c7: skipped 000591 1788527902.038186 mapread 0x000527 thru 0x007069 (0x6b43 bytes) 000593 1788527902.079100 mapread 0x02e715 thru 0x02f8c6 (0x11b2 bytes) 000594 1788527902.087934 mapwrite 0x035cbf thru 0x037d8c (0x20ce bytes) 000595 1788527902.151465 mapwrite 0x02bf4b thru 0x02cd03 (0xdb9 bytes) 000597 1788527902.207514 write 0x004c3e thru 0x00926a (0x462d bytes) 000598 1788527902.211105 trunc from 0x037d8d to 0x02a303 (0xda89 bytes) 000599 1788527902.237044 mapwrite 0x030342 thru 0x03adb2 (0xaa71 bytes) 000600 1788527902.365157 mapwrite 0x0191dd thru 0x0265b2 (0xd3d6 bytes) 000601 1788527902.457659 mapwrite 0x00625f thru 0x00df1c (0x7cbe bytes) 000602 1788527902.546913 mapread 0x023d8e thru 0x02fe5e (0xc0d1 bytes) 000603 1788527902.548233 read 0x016f23 thru 0x024413 (0xd4f1 bytes) 000604 1788527902.549846 write 0x0274fa thru 0x031a6b (0xa572 bytes) 000606 1788527902.552167 trunc from 0x03adb3 to 0x019cfd (0x210b5 bytes) 000610 1788527902.578580 write 0x02cd1a thru 0x036fdc (0xa2c3 bytes) 000611 1788527902.580470 mapread 0x021b6e thru 0x02b1c5 (0x9658 bytes) 000614 1788527902.602042 trunc from 0x036fdd to 0x03abdd (0x3c01 bytes) 000615 1788527902.622113 write 0x02ee26 thru 0x03014a (0x1325 bytes) 000616 1788527902.623231 punch from 0x0292ae to 0x0332ae (0xa001 bytes) 000617 1788527902.643327 write 0x007d38 thru 0x010258 (0x8521 bytes) 000618 1788527902.651756 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3abde): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3abde): Mirror 2: 0x31811773 (stale) CRC-32 checksum value for chunk [0, 0x3abde): Mirror 1: 0x392c2397 Mirror 3: 0x392c2397 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3abde: skipped 000619 1788527904.282324 trunc from 0x03abde to 0x03a7c6 (0x417 bytes) 000621 1788527904.428143 write 0x032e03 thru 0x034337 (0x1535 bytes) 000622 1788527904.460949 mapread 0x00eb71 thru 0x014332 (0x57c2 bytes) 000623 1788527904.552511 punch from 0x01b889 to 0x026e3b (0xb5b3 bytes) 000624 1788527904.573911 read 0x02cacb thru 0x0371a7 (0xa6dd bytes) 000625 1788527904.598271 mapwrite 0x035d5b thru 0x03ffff (0xa2a5 bytes) 000626 1788527904.826499 write 0x025e5c thru 0x02fd32 (0x9ed7 bytes) 000627 1788527904.835041 mapwrite 0x007b3f thru 0x012eb0 (0xb372 bytes) 000628 1788527904.896332 mapwrite 0x008401 thru 0x012b1c (0xa71c bytes) 000630 1788527904.970035 mapread 0x008c5e thru 0x00e911 (0x5cb4 bytes) 000634 1788527904.980826 trunc from 0x040000 to 0x02f210 (0x10def bytes) 000635 1788527905.026130 read 0x01b0dc thru 0x01c7f1 (0x1716 bytes) 000636 1788527905.043625 trunc from 0x02f211 to 0x00162d (0x2dbe3 bytes) 000637 1788527905.080867 trunc from 0x00162e to 0x0085b0 (0x6f83 bytes) 000638 1788527905.119769 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x85b1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x85b1): Mirror 2: 0x478eebee (stale) Mirror 3: 0xc27fad9c (stale) CRC-32 checksum value for chunk [0, 0x85b1): Mirror 1: 0x7a8f11d6 Mirror 4: 0x7a8f11d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x85b1: skipped 000639 1788527906.160208 trunc from 0x0085b1 to 0x02d0a4 (0x24af4 bytes) 000640 1788527906.278924 write 0x03a7d4 thru 0x03ffff (0x582c bytes) 000641 1788527906.280894 trunc from 0x040000 to 0x03684c (0x97b3 bytes) 000642 1788527906.311078 read 0x02c0c8 thru 0x033fc8 (0x7f01 bytes) 000643 1788527906.321844 write 0x00021e thru 0x0094a1 (0x9284 bytes) 000645 1788527906.332365 trunc from 0x03684d to 0x02a9ae (0xbe9e bytes) 000646 1788527906.346452 mapread 0x02333f thru 0x02a9ae (0x7670 bytes) 000648 1788527906.364546 mapread 0x00f3f9 thru 0x015102 (0x5d0a bytes) 000649 1788527906.375178 write 0x037b0a thru 0x03ffff (0x84f6 bytes) 000651 1788527906.378137 mapread 0x02a6f3 thru 0x039aa9 (0xf3b7 bytes) 000652 1788527906.392360 write 0x0386ed thru 0x03ffff (0x7913 bytes) 000653 1788527906.393546 trunc from 0x040000 to 0x0246b3 (0x1b94c bytes) 000654 1788527906.411944 mapwrite 0x00d4f5 thru 0x01653b (0x9047 bytes) 000655 1788527906.463288 write 0x0086b0 thru 0x0101a9 (0x7afa bytes) 000658 1788527906.470343 MIRROR_RESYNC 4 mirrors 000659 1788527908.254714 trunc from 0x0246b4 to 0x039ab9 (0x15406 bytes) 000660 1788527908.450800 mapread 0x014078 thru 0x01db40 (0x9ac9 bytes) 000661 1788527908.489867 mapread 0x0285aa thru 0x0337ad (0xb204 bytes) 000662 1788527908.525142 punch from 0x035b57 to 0x039ab9 (0x3f63 bytes) 000663 1788527908.535574 mapread 0x008269 thru 0x00e46c (0x6204 bytes) 000665 1788527908.554579 MIRROR_RESYNC 4 mirrors 000667 1788527910.212409 mapread 0x010457 thru 0x013d04 (0x38ae bytes) 000669 1788527910.245919 write 0x0260e6 thru 0x029f10 (0x3e2b bytes) 000670 1788527910.370815 mapwrite 0x004864 thru 0x009769 (0x4f06 bytes) 000671 1788527910.559554 write 0x0380c8 thru 0x03f4c8 (0x7401 bytes) 000672 1788527910.572288 trunc from 0x03f4c9 to 0x0169fc (0x28acc bytes) 000673 1788527910.629156 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x169fd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169fd): Mirror 2: 0xf71cf2f2 (stale) Mirror 3: 0xf71cf2f2 (stale) Mirror 4: 0xf71cf2f2 (stale) CRC-32 checksum value for chunk [0, 0x169fd): Mirror 1: 0x4b9e6dd5 Mirror 5: 0x4b9e6dd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169fd: skipped 000674 1788527911.496003 punch from 0x003e9b to 0x00ff4b (0xc0b1 bytes) 000675 1788527911.669390 read 0x00931f thru 0x00992b (0x60d bytes) 000676 1788527911.687031 mapread 0x003159 thru 0x0071de (0x4086 bytes) 000677 1788527911.742335 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x169fd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169fd): Mirror 2: 0xf71cf2f2 (stale) Mirror 3: 0xf71cf2f2 (stale) Mirror 4: 0xf71cf2f2 (stale) Mirror 5: 0x4b9e6dd5 (stale) CRC-32 checksum value for chunk [0, 0x169fd): Mirror 1: 0xc392f012 Mirror 6: 0xc392f012 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169fd: skipped 000678 1788527912.873127 mapread 0x0105a0 thru 0x01666b (0x60cc bytes) 000680 1788527912.966773 read 0x00ba03 thru 0x0169fc (0xaffa bytes) 000681 1788527912.985159 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x169fd): Mirror 2: 0xf71cf2f2 (stale) Mirror 4: 0xf71cf2f2 (stale) Mirror 5: 0x4b9e6dd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169fd): Mirror 2: 0xf71cf2f2 (stale) Mirror 4: 0xf71cf2f2 (stale) Mirror 5: 0x4b9e6dd5 (stale) CRC-32 checksum value for chunk [0, 0x169fd): Mirror 1: 0xc392f012 Mirror 6: 0xc392f012 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169fd: skipped 000682 1788527914.073355 mapwrite 0x037804 thru 0x03ffff (0x87fc bytes) 000684 1788527914.268382 write 0x03595e thru 0x03ffff (0xa6a2 bytes) 000685 1788527914.270799 trunc from 0x040000 to 0x02a74a (0x158b5 bytes) 000686 1788527914.308905 punch from 0x00c73b to 0x01234a (0x5c10 bytes) 000687 1788527914.315996 read 0x0021b2 thru 0x00e5be (0xc40d bytes) 000688 1788527914.322816 read 0x014fe3 thru 0x022ff2 (0xe010 bytes) 000689 1788527914.332021 read 0x01d556 thru 0x028cbb (0xb766 bytes) 000691 1788527914.333243 mapwrite 0x01523e thru 0x01ccbb (0x7a7e bytes) 000692 1788527914.399279 trunc from 0x02a74b to 0x028502 (0x2248 bytes) 000693 1788527914.426441 read 0x018583 thru 0x02001c (0x7a9a bytes) 000694 1788527914.427966 punch from 0x01d376 to 0x0270d1 (0x9d5c bytes) 000696 1788527914.439366 trunc from 0x028503 to 0x01fba7 (0x895b bytes) 000698 1788527914.470962 read 0x01df10 thru 0x01fba7 (0x1c98 bytes) 000699 1788527914.501042 mapread 0x01ed68 thru 0x01fba7 (0xe40 bytes) 000700 1788527914.511510 mapwrite 0x01a98e thru 0x01d556 (0x2bc9 bytes) 000702 1788527914.632834 write 0x035d03 thru 0x03ffff (0xa2fd bytes) 000704 1788527914.640389 punch from 0x0264b6 to 0x02c73c (0x6287 bytes) 000705 1788527914.672449 read 0x001197 thru 0x0063a3 (0x520d bytes) 000706 1788527914.673466 write 0x0067d5 thru 0x0136fd (0xcf29 bytes) 000707 1788527914.685121 punch from 0x003df9 to 0x009548 (0x5750 bytes) 000708 1788527914.719933 mapwrite 0x007eb1 thru 0x00ef56 (0x70a6 bytes) 000710 1788527914.892361 mapwrite 0x002613 thru 0x009de6 (0x77d4 bytes) 000715 1788527915.030457 mapread 0x015fe0 thru 0x021043 (0xb064 bytes) 000717 1788527915.036317 mapread 0x026ce5 thru 0x0296ca (0x29e6 bytes) 000720 1788527915.075814 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8ce6d15f (stale) Mirror 4: 0x8ce6d15f (stale) Mirror 5: 0x35adf621 (stale) Mirror 6: 0xc92c4dca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3233a2b6 Mirror 7: 0x3233a2b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000721 1788527917.570525 mapread 0x0062ed thru 0x00ca0c (0x6720 bytes) 000722 1788527917.723631 read 0x0000ac thru 0x006fd3 (0x6f28 bytes) 000723 1788527917.770366 mapread 0x038fa0 thru 0x03d7d7 (0x4838 bytes) 000725 1788527917.870198 mapwrite 0x018dc3 thru 0x01a71c (0x195a bytes) 000729 1788527918.180986 mapread 0x035b4e thru 0x03ffff (0xa4b2 bytes) 000730 1788527918.206285 mapread 0x025f24 thru 0x0345ed (0xe6ca bytes) 000731 1788527918.224402 trunc from 0x040000 to 0x001804 (0x3e7fb bytes) 000733 1788527918.307085 read 0x000e52 thru 0x001804 (0x9b3 bytes) 000735 1788527918.318028 read 0x0011dc thru 0x001804 (0x629 bytes) 000736 1788527918.318733 trunc from 0x001805 to 0x00469e (0x2e9a bytes) 000737 1788527918.382879 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x469f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x469f): Mirror 2: 0x84a5b9d8 (stale) Mirror 4: 0x84a5b9d8 (stale) Mirror 5: 0x84a5b9d8 (stale) Mirror 6: 0x3546b566 (stale) Mirror 7: 0xbbd86d01 (stale) CRC-32 checksum value for chunk [0, 0x469f): Mirror 1: 0x515b340 Mirror 8: 0x515b340 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x469f: skipped 000738 1788527920.287331 read 0x003a06 thru 0x00469e (0xc99 bytes) 000739 1788527920.302988 mapread 0x0016fc thru 0x00469e (0x2fa3 bytes) 000741 1788527920.342160 punch from 0x002df8 to 0x00469e (0x18a7 bytes) 000742 1788527920.430395 mapread 0x0002e2 thru 0x00469e (0x43bd bytes) 000743 1788527920.437389 punch from 0x0036e8 to 0x00469e (0xfb7 bytes) 000745 1788527920.446884 write 0x00f19e thru 0x011c1d (0x2a80 bytes) 000746 1788527920.448904 mapread 0x006c45 thru 0x011c1d (0xafd9 bytes) 000747 1788527920.460266 punch from 0x00b17f to 0x011c1d (0x6a9f bytes) 000748 1788527920.474799 trunc from 0x011c1e to 0x02bb87 (0x19f6a bytes) 000749 1788527920.499331 mapread 0x022f0a thru 0x02bb87 (0x8c7e bytes) 000751 1788527920.518581 mapread 0x01a9f8 thru 0x023b2b (0x9134 bytes) 000753 1788527920.548167 punch from 0x020efb to 0x02a4cc (0x95d2 bytes) 000754 1788527920.554215 write 0x01dbe7 thru 0x02b9d1 (0xddeb bytes) 000756 1788527920.566384 read 0x0050b7 thru 0x005a33 (0x97d bytes) 000758 1788527920.567456 mapwrite 0x035bad thru 0x03ee0a (0x925e bytes) 000759 1788527920.720783 trunc from 0x03ee0b to 0x03f007 (0x1fd bytes) 000760 1788527920.769119 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f008): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f008): Mirror 2: 0xaa2de9bf (stale) Mirror 4: 0xaa2de9bf (stale) Mirror 5: 0xa5e520db (stale) Mirror 6: 0x177aff1f (stale) Mirror 7: 0xacf95100 (stale) Mirror 8: 0x45b748 (stale) CRC-32 checksum value for chunk [0, 0x3f008): Mirror 1: 0x5a36f242 Mirror 9: 0x5a36f242 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f008: skipped 000761 1788527922.446669 punch from 0x00801e to 0x00ca9f (0x4a82 bytes) 000764 1788527922.640286 mapread 0x01e76a thru 0x01f975 (0x120c bytes) 000765 1788527922.660566 read 0x01a114 thru 0x0221dd (0x80ca bytes) 000766 1788527922.681050 write 0x02fda0 thru 0x030b6d (0xdce bytes) 000767 1788527922.705340 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f008): Mirror 2: 0xaa2de9bf (stale) Mirror 5: 0xa5e520db (stale) Mirror 6: 0x177aff1f (stale) Mirror 7: 0xacf95100 (stale) Mirror 8: 0x45b748 (stale) Mirror 9: 0x5a36f242 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f008): Mirror 2: 0xaa2de9bf (stale) Mirror 5: 0xa5e520db (stale) Mirror 6: 0x177aff1f (stale) Mirror 7: 0xacf95100 (stale) Mirror 8: 0x45b748 (stale) Mirror 9: 0x5a36f242 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f008: skipped 000768 1788527923.256753 read 0x02a24e thru 0x02b9a6 (0x1759 bytes) 000771 1788527923.263678 punch from 0x00372d to 0x00d781 (0xa055 bytes) 000772 1788527923.279058 mapwrite 0x03c131 thru 0x03ffff (0x3ecf bytes) 000774 1788527923.369069 mapread 0x00ed6e thru 0x01007e (0x1311 bytes) 000776 1788527923.380267 punch from 0x00e9b0 to 0x01319e (0x47ef bytes) 000777 1788527923.385420 punch from 0x00d5b2 to 0x010f4e (0x399d bytes) 000778 1788527923.393887 trunc from 0x040000 to 0x014b54 (0x2b4ab bytes) 000780 1788527923.420882 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14b55): Mirror 2: 0x7be424c5 (stale) Mirror 5: 0xff21e356 (stale) Mirror 6: 0x38ab23e7 (stale) Mirror 8: 0x23d51d9a (stale) Mirror 9: 0x23d51d9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14b55): Mirror 2: 0x7be424c5 (stale) Mirror 5: 0xff21e356 (stale) Mirror 6: 0x38ab23e7 (stale) Mirror 8: 0x23d51d9a (stale) Mirror 9: 0x23d51d9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14b55: skipped 000781 1788527924.097143 mapwrite 0x03019c thru 0x03b923 (0xb788 bytes) 000782 1788527924.283585 read 0x002b29 thru 0x0077f5 (0x4ccd bytes) 000784 1788527924.299172 read 0x021226 thru 0x02f023 (0xddfe bytes) 000786 1788527924.314765 mapwrite 0x01a5e8 thru 0x025841 (0xb25a bytes) 000787 1788527924.403548 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b924): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b924): Mirror 2: 0xfd181560 (stale) Mirror 5: 0xb1d6f271 (stale) Mirror 6: 0x661af073 (stale) Mirror 8: 0x36d39a03 (stale) Mirror 9: 0x8c8502de (stale) CRC-32 checksum value for chunk [0, 0x3b924): Mirror 1: 0x3e07089f Mirror 10: 0x3e07089f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b924: skipped 000789 1788527925.563532 mapwrite 0x00f035 thru 0x01669a (0x7666 bytes) 000793 1788527925.823946 trunc from 0x03b924 to 0x03757b (0x43a8 bytes) 000796 1788527925.876028 mapread 0x034ede thru 0x03757b (0x269e bytes) 000798 1788527925.885110 punch from 0x0311f6 to 0x03757b (0x6386 bytes) 000799 1788527925.892562 MIRROR_RESYNC 7 mirrors 000801 1788527927.719347 read 0x018dc6 thru 0x0193ff (0x63a bytes) 000802 1788527927.735822 MIRROR_RESYNC 7 mirrors 000803 1788527928.441940 trunc from 0x03757c to 0x0031e0 (0x3439b bytes) 000804 1788527928.523756 read 0x000a2c thru 0x0031e0 (0x27b5 bytes) 000805 1788527928.531560 write 0x017598 thru 0x021c21 (0xa68a bytes) 000806 1788527928.534061 mapwrite 0x0049ad thru 0x00889c (0x3ef0 bytes) 000807 1788527928.619923 mapwrite 0x02ca03 thru 0x0376ee (0xacec bytes) 000808 1788527928.773876 mapwrite 0x00a8d9 thru 0x01807d (0xd7a5 bytes) 000811 1788527928.940096 trunc from 0x0376ef to 0x0323b7 (0x5337 bytes) 000812 1788527928.966811 mapread 0x00bd16 thru 0x01a8f6 (0xebe1 bytes) 000814 1788527928.970565 mapwrite 0x0200f7 thru 0x0229ec (0x28f6 bytes) 000815 1788527929.082487 trunc from 0x0323b8 to 0x01607e (0x1c339 bytes) 000816 1788527929.101782 read 0x010025 thru 0x01607e (0x605a bytes) 000817 1788527929.102754 trunc from 0x01607f to 0x01166d (0x4a11 bytes) 000819 1788527929.144919 write 0x03fc24 thru 0x03ffff (0x3dc bytes) 000820 1788527929.150221 mapread 0x03382c thru 0x0350db (0x18b0 bytes) 000821 1788527929.170382 trunc from 0x040000 to 0x03bad4 (0x452b bytes) 000822 1788527929.217748 write 0x016171 thru 0x018547 (0x23d7 bytes) 000823 1788527929.218902 mapread 0x0215d7 thru 0x026763 (0x518d bytes) 000824 1788527929.227518 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3bad5): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bad5): Mirror 2: 0xa4d349c3 (stale) Mirror 5: 0xa4d349c3 (stale) Mirror 6: 0xa4d349c3 (stale) Mirror 8: 0xa4d349c3 (stale) Mirror 9: 0xa4d349c3 (stale) Mirror 10: 0xa4d349c3 (stale) CRC-32 checksum value for chunk [0, 0x3bad5): Mirror 1: 0x36b020b Mirror 11: 0x36b020b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bad5: skipped 000825 1788527930.597225 write 0x013613 thru 0x0173e1 (0x3dcf bytes) 000827 1788527930.712931 read 0x03522d thru 0x03928f (0x4063 bytes) 000828 1788527930.724487 trunc from 0x03bad5 to 0x021b6b (0x19f69 bytes) 000829 1788527930.785796 mapread 0x00a657 thru 0x01535e (0xad08 bytes) 000830 1788527930.822332 MIRROR_RESYNC 8 mirrors 000831 1788527932.993217 MIRROR_EXTEND to 9 mirrors 000832 1788527934.345418 trunc from 0x021b6c to 0x02c1a2 (0xa637 bytes) 000835 1788527934.475240 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c1a3): Mirror 2: 0xeb1db0de (stale) Mirror 5: 0xeb1db0de (stale) Mirror 6: 0xeb1db0de (stale) Mirror 9: 0xeb1db0de (stale) Mirror 10: 0xeb1db0de (stale) Mirror 11: 0xeb1db0de (stale) Mirror 12: 0xeb1db0de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c1a3): Mirror 2: 0xeb1db0de (stale) Mirror 5: 0xeb1db0de (stale) Mirror 6: 0xeb1db0de (stale) Mirror 9: 0xeb1db0de (stale) Mirror 10: 0xeb1db0de (stale) Mirror 11: 0xeb1db0de (stale) Mirror 12: 0xeb1db0de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c1a3: skipped 000836 1788527935.405177 punch from 0x010136 to 0x01e8e0 (0xe7ab bytes) 000837 1788527935.419072 punch from 0x017cd4 to 0x01f192 (0x74bf bytes) 000840 1788527935.435130 trunc from 0x02c1a3 to 0x00eb3f (0x1d663 bytes) 000842 1788527935.479936 write 0x0237dc thru 0x02b76f (0x7f94 bytes) 000843 1788527935.492080 MIRROR_RESYNC 8 mirrors 000844 1788527938.213294 write 0x0117ec thru 0x018e3f (0x7654 bytes) 000846 1788527938.313255 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b770): Mirror 5: 0xeeec604b (stale) Mirror 6: 0xeeec604b (stale) Mirror 9: 0xeeec604b (stale) Mirror 10: 0xeeec604b (stale) Mirror 11: 0xeeec604b (stale) Mirror 12: 0xeeec604b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b770): Mirror 5: 0xeeec604b (stale) Mirror 6: 0xeeec604b (stale) Mirror 9: 0xeeec604b (stale) Mirror 10: 0xeeec604b (stale) Mirror 11: 0xeeec604b (stale) Mirror 12: 0xeeec604b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b770: skipped 000847 1788527938.818478 mapread 0x010ddb thru 0x0117d5 (0x9fb bytes) 000848 1788527938.829943 write 0x033e01 thru 0x03e92c (0xab2c bytes) 000849 1788527938.831809 read 0x022b20 thru 0x02dfad (0xb48e bytes) 000850 1788527938.839912 trunc from 0x03e92d to 0x00aaf6 (0x33e36 bytes) 000851 1788527938.862240 trunc from 0x00aaf7 to 0x015662 (0xab6c bytes) 000852 1788527938.887755 mapwrite 0x034603 thru 0x03ffff (0xb9fd bytes) 000853 1788527938.978298 MIRROR_RESYNC 7 mirrors 000854 1788527941.016160 trunc from 0x040000 to 0x011e7a (0x2e185 bytes) 000856 1788527941.088143 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11e7b): Mirror 5: 0xa6ba9d11 (stale) Mirror 6: 0xa6ba9d11 (stale) Mirror 9: 0xa6ba9d11 (stale) Mirror 10: 0xa6ba9d11 (stale) Mirror 12: 0xa6ba9d11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11e7b): Mirror 5: 0xa6ba9d11 (stale) Mirror 6: 0xa6ba9d11 (stale) Mirror 9: 0xa6ba9d11 (stale) Mirror 10: 0xa6ba9d11 (stale) Mirror 12: 0xa6ba9d11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11e7b: skipped 000857 1788527941.568455 mapread 0x006903 thru 0x011e7a (0xb578 bytes) 000858 1788527941.601936 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x11e7b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11e7b): Mirror 5: 0xa6ba9d11 (stale) Mirror 6: 0xa6ba9d11 (stale) Mirror 9: 0xa6ba9d11 (stale) Mirror 10: 0xa6ba9d11 (stale) Mirror 12: 0xa6ba9d11 (stale) CRC-32 checksum value for chunk [0, 0x11e7b): Mirror 1: 0xa6ba9d11 Mirror 13: 0xa6ba9d11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11e7b: skipped 000859 1788527943.304367 mapread 0x00979a thru 0x009958 (0x1bf bytes) 000860 1788527943.332587 trunc from 0x011e7b to 0x02eba7 (0x1cd2d bytes) 000863 1788527943.491762 read 0x013707 thru 0x017dce (0x46c8 bytes) 000865 1788527943.501780 punch from 0x02bfbe to 0x02d03a (0x107d bytes) 000866 1788527943.510117 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2eba8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eba8): Mirror 5: 0xd6f5869c (stale) Mirror 6: 0xd6f5869c (stale) Mirror 9: 0xd6f5869c (stale) Mirror 10: 0xd6f5869c (stale) Mirror 12: 0xd6f5869c (stale) Mirror 13: 0xd6f5869c (stale) CRC-32 checksum value for chunk [0, 0x2eba8): Mirror 1: 0xd6f5869c Mirror 14: 0xd6f5869c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eba8: skipped 000867 1788527945.177402 trunc from 0x02eba8 to 0x023d25 (0xae82 bytes) 000868 1788527945.358147 MIRROR_RESYNC 8 mirrors 000871 1788527947.371237 mapwrite 0x00cc78 thru 0x016733 (0x9abc bytes) 000873 1788527947.749354 trunc from 0x023d26 to 0x009d22 (0x1a003 bytes) 000875 1788527947.777627 mapread 0x0016e6 thru 0x004f10 (0x382b bytes) 000876 1788527947.795237 punch from 0x007e94 to 0x008ea9 (0x1016 bytes) 000877 1788527947.801460 trunc from 0x009d23 to 0x03cf46 (0x33224 bytes) 000878 1788527947.825289 punch from 0x01b82c to 0x0298d9 (0xe0ae bytes) 000879 1788527947.839834 mapwrite 0x002b5c thru 0x00a454 (0x78f9 bytes) 000880 1788527947.917943 read 0x0171c7 thru 0x0220bb (0xaef5 bytes) 000881 1788527947.937013 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cf47): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf47): Mirror 5: 0x4b4f3e6 (stale) Mirror 6: 0x4b4f3e6 (stale) Mirror 9: 0x4b4f3e6 (stale) Mirror 10: 0x4b4f3e6 (stale) Mirror 12: 0x4b4f3e6 (stale) Mirror 13: 0x4b4f3e6 (stale) Mirror 14: 0x4b4f3e6 (stale) CRC-32 checksum value for chunk [0, 0x3cf47): Mirror 1: 0xe592e12 Mirror 15: 0xe592e12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf47: skipped 000882 1788527949.315422 punch from 0x028ec7 to 0x037830 (0xe96a bytes) 000883 1788527949.391887 read 0x001c29 thru 0x00d73b (0xbb13 bytes) 000884 1788527949.401359 mapread 0x02f870 thru 0x03030f (0xaa0 bytes) 000885 1788527949.414712 read 0x01cb2d thru 0x02b4df (0xe9b3 bytes) 000886 1788527949.425147 mapread 0x00747c thru 0x00c43c (0x4fc1 bytes) 000887 1788527949.426309 mapread 0x014eb8 thru 0x0216c1 (0xc80a bytes) 000889 1788527949.432169 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3cf47): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf47): Mirror 5: 0x4b4f3e6 (stale) Mirror 6: 0x4b4f3e6 (stale) Mirror 9: 0x4b4f3e6 (stale) Mirror 10: 0x4b4f3e6 (stale) Mirror 12: 0x4b4f3e6 (stale) Mirror 13: 0x4b4f3e6 (stale) Mirror 14: 0x4b4f3e6 (stale) Mirror 15: 0xe592e12 (stale) CRC-32 checksum value for chunk [0, 0x3cf47): Mirror 1: 0xe592e12 Mirror 16: 0xe592e12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf47: skipped 000893 1788527950.840888 MIRROR_RESYNC 10 mirrors 000895 1788527952.800905 mapwrite 0x03df57 thru 0x03ffff (0x20a9 bytes) 000896 1788527953.039275 write 0x0117dc thru 0x01b5b2 (0x9dd7 bytes) 000899 1788527953.083043 trunc from 0x040000 to 0x02d5bc (0x12a43 bytes) 000900 1788527953.115550 read 0x015ee2 thru 0x0257ce (0xf8ed bytes) 000901 1788527953.128185 write 0x010c45 thru 0x01f5c9 (0xe985 bytes) 000902 1788527953.139439 read 0x0137f6 thru 0x0214d2 (0xdcdd bytes) 000904 1788527953.142032 mapwrite 0x03a9d2 thru 0x03ffff (0x562e bytes) 000905 1788527953.238525 punch from 0x020353 to 0x02f2ba (0xef68 bytes) 000907 1788527953.246902 read 0x0388cb thru 0x03ffff (0x7735 bytes) 000908 1788527953.252952 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3ddf5ed8 (stale) Mirror 6: 0x3ddf5ed8 (stale) Mirror 9: 0x3ddf5ed8 (stale) Mirror 10: 0x3ddf5ed8 (stale) Mirror 12: 0x3ddf5ed8 (stale) Mirror 13: 0x3ddf5ed8 (stale) Mirror 14: 0x3ddf5ed8 (stale) Mirror 15: 0x3ddf5ed8 (stale) Mirror 16: 0x3ddf5ed8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x219ee340 Mirror 17: 0x219ee340 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000912 1788527955.022387 mapread 0x00e97e thru 0x00fb4d (0x11d0 bytes) 000916 1788527955.104855 mapwrite 0x0197b4 thru 0x01dcca (0x4517 bytes) 000918 1788527955.322172 mapwrite 0x020db2 thru 0x030917 (0xfb66 bytes) 000919 1788527955.461928 mapwrite 0x0286d9 thru 0x035dd0 (0xd6f8 bytes) 000920 1788527955.578319 read 0x03e1c5 thru 0x03ffff (0x1e3b bytes) 000921 1788527955.591790 trunc from 0x040000 to 0x024fbe (0x1b041 bytes) 000922 1788527955.647856 trunc from 0x024fbf to 0x01137b (0x13c43 bytes) 000924 1788527955.666648 mapwrite 0x02ce69 thru 0x032ca2 (0x5e3a bytes) 000925 1788527955.740009 punch from 0x02c828 to 0x02ce2b (0x604 bytes) 000927 1788527955.747651 write 0x0258f8 thru 0x033bcf (0xe2d8 bytes) 000928 1788527955.748974 read 0x01dc48 thru 0x01e397 (0x750 bytes) 000929 1788527955.764575 trunc from 0x033bd0 to 0x029669 (0xa566 bytes) 000931 1788527955.798428 trunc from 0x02966a to 0x01a35d (0xf30c bytes) 000933 1788527955.834135 mapread 0x010187 thru 0x013a37 (0x38b1 bytes) 000934 1788527955.837195 write 0x0021a0 thru 0x00f1ca (0xd02b bytes) 000936 1788527955.845793 punch from 0x00ebaf to 0x015643 (0x6a95 bytes) 000938 1788527955.867977 trunc from 0x01a35e to 0x00beb9 (0xe4a4 bytes) 000939 1788527955.921316 trunc from 0x00beba to 0x0255db (0x19722 bytes) 000940 1788527955.960701 trunc from 0x0255dc to 0x0059d4 (0x1fc07 bytes) 000941 1788527956.005223 mapwrite 0x03b656 thru 0x03ffff (0x49aa bytes) 000942 1788527956.210416 trunc from 0x040000 to 0x007b9d (0x38462 bytes) 000943 1788527956.242796 mapread 0x0067cb thru 0x006a0c (0x242 bytes) 000944 1788527956.247847 read 0x002e53 thru 0x007b9d (0x4d4b bytes) 000945 1788527956.252862 mapwrite 0x02c6bc thru 0x0337c9 (0x710e bytes) 000946 1788527956.335583 punch from 0x009871 to 0x014f8e (0xb71e bytes) 000948 1788527956.354848 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x337ca): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337ca): Mirror 5: 0xba3704ea (stale) Mirror 6: 0xba3704ea (stale) Mirror 9: 0xba3704ea (stale) Mirror 10: 0xba3704ea (stale) Mirror 12: 0xba3704ea (stale) Mirror 13: 0xba3704ea (stale) Mirror 14: 0xba3704ea (stale) Mirror 15: 0xba3704ea (stale) Mirror 16: 0xba3704ea (stale) Mirror 17: 0x65c37037 (stale) CRC-32 checksum value for chunk [0, 0x337ca): Mirror 1: 0xc2b5f8e9 Mirror 18: 0xc2b5f8e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337ca: skipped 000949 1788527957.574091 mapwrite 0x00d04d thru 0x0184c2 (0xb476 bytes) 000950 1788527957.794066 punch from 0x00a987 to 0x00fe9f (0x5519 bytes) 000951 1788527957.799329 punch from 0x005703 to 0x00f0a8 (0x99a6 bytes) 000952 1788527957.804449 read 0x00c1fe thru 0x015546 (0x9349 bytes) 000953 1788527957.810812 punch from 0x024d9b to 0x02769b (0x2901 bytes) 000955 1788527957.827182 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x337ca): Mirror 5: 0xba3704ea (stale) Mirror 6: 0xba3704ea (stale) Mirror 9: 0xba3704ea (stale) Mirror 10: 0xba3704ea (stale) Mirror 12: 0xba3704ea (stale) Mirror 13: 0xba3704ea (stale) Mirror 14: 0xba3704ea (stale) Mirror 15: 0xba3704ea (stale) Mirror 16: 0xba3704ea (stale) Mirror 18: 0xc2b5f8e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337ca): Mirror 5: 0xba3704ea (stale) Mirror 6: 0xba3704ea (stale) Mirror 9: 0xba3704ea (stale) Mirror 10: 0xba3704ea (stale) Mirror 12: 0xba3704ea (stale) Mirror 13: 0xba3704ea (stale) Mirror 14: 0xba3704ea (stale) Mirror 15: 0xba3704ea (stale) Mirror 16: 0xba3704ea (stale) Mirror 18: 0xc2b5f8e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337ca: skipped 000956 1788527958.693487 read 0x02601c thru 0x0287f0 (0x27d5 bytes) 000957 1788527958.707196 read 0x01f31d thru 0x024ac1 (0x57a5 bytes) 000958 1788527958.720166 read 0x01fe9b thru 0x0235ae (0x3714 bytes) 000959 1788527958.732061 mapwrite 0x01bb6b thru 0x022d0f (0x71a5 bytes) 000960 1788527958.834654 MIRROR_RESYNC 11 mirrors 000962 1788527961.427360 punch from 0x023e15 to 0x02659e (0x278a bytes) 000963 1788527961.502985 mapread 0x028f1a thru 0x0337c9 (0xa8b0 bytes) 000965 1788527961.540283 mapwrite 0x01c078 thru 0x01ddb1 (0x1d3a bytes) 000966 1788527961.623146 write 0x02bef7 thru 0x0379d3 (0xbadd bytes) 000967 1788527961.624444 punch from 0x022a31 to 0x02ef45 (0xc515 bytes) 000970 1788527961.642149 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x379d4): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x379d4): Mirror 5: 0x9cff2df6 (stale) Mirror 6: 0x9cff2df6 (stale) Mirror 9: 0x9cff2df6 (stale) Mirror 10: 0x9cff2df6 (stale) Mirror 12: 0x9cff2df6 (stale) Mirror 13: 0x9cff2df6 (stale) Mirror 14: 0x9cff2df6 (stale) Mirror 15: 0x9cff2df6 (stale) Mirror 16: 0x9cff2df6 (stale) Mirror 18: 0x9cff2df6 (stale) CRC-32 checksum value for chunk [0, 0x379d4): Mirror 1: 0xd6e60e57 Mirror 19: 0xd6e60e57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x379d4: skipped 000971 1788527963.116137 read 0x033f0e thru 0x0340c0 (0x1b3 bytes) 000972 1788527963.128371 read 0x005ece thru 0x00c04e (0x6181 bytes) 000973 1788527963.143420 read 0x014805 thru 0x02322c (0xea28 bytes) 000974 1788527963.149911 write 0x033713 thru 0x03ffff (0xc8ed bytes) 000976 1788527963.220239 write 0x035f53 thru 0x03ffff (0xa0ad bytes) 000978 1788527963.221786 punch from 0x0381fc to 0x03ffff (0x7e04 bytes) 000980 1788527963.232562 trunc from 0x040000 to 0x00ef3d (0x310c2 bytes) 000982 1788527963.254780 mapwrite 0x038c31 thru 0x03dc51 (0x5021 bytes) 000984 1788527963.346479 mapread 0x015d5d thru 0x0194ca (0x376e bytes) 000985 1788527963.373059 mapread 0x01049b thru 0x01bc3c (0xb7a2 bytes) 000986 1788527963.417329 write 0x01f38d thru 0x026284 (0x6ef8 bytes) 000987 1788527963.437963 mapread 0x0352ef thru 0x03dc51 (0x8963 bytes) 000988 1788527963.453625 mapwrite 0x02bca9 thru 0x039545 (0xd89d bytes) 000989 1788527963.558657 write 0x01e022 thru 0x02b6c1 (0xd6a0 bytes) 000990 1788527963.567886 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3dc52): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc52): Mirror 5: 0xfeb337b6 (stale) Mirror 6: 0xfeb337b6 (stale) Mirror 9: 0xfeb337b6 (stale) Mirror 10: 0xfeb337b6 (stale) Mirror 12: 0xfeb337b6 (stale) Mirror 13: 0xfeb337b6 (stale) Mirror 14: 0xfeb337b6 (stale) Mirror 15: 0xfeb337b6 (stale) Mirror 16: 0xfeb337b6 (stale) Mirror 18: 0xfeb337b6 (stale) Mirror 19: 0xdddcd0a0 (stale) CRC-32 checksum value for chunk [0, 0x3dc52): Mirror 1: 0x5a387471 Mirror 20: 0x5a387471 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc52: skipped 000991 1788527964.487083 mapread 0x006b8c thru 0x012c34 (0xc0a9 bytes) 000992 1788527964.527900 punch from 0x028621 to 0x02e043 (0x5a23 bytes) 000993 1788527964.616749 read 0x0157cc thru 0x024c65 (0xf49a bytes) 000994 1788527964.626169 trunc from 0x03dc52 to 0x030c20 (0xd031 bytes) 000995 1788527964.670484 mapread 0x01bd4c thru 0x022115 (0x63ca bytes) 000997 1788527964.673486 read 0x012f9d thru 0x014a5e (0x1ac2 bytes) 000998 1788527964.684964 punch from 0x00f1a8 to 0x019537 (0xa390 bytes) 001000 1788527964.691539 read 0x00031e thru 0x00d86a (0xd54d bytes) All operations completed A-OK! PASS 70a (126s) == sanity-flr test 71: check mirror extend parent fid ==== 09:19:29 (1788527969) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 09:19:32 (1788527972) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.288861 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235568 s, 4.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 09:19:40 (1788527980) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre2 Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 657536 .. Extending file size to 5996832 .. failed Extending file size to 5251680 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7622176 .. Extending file size to 7978880 .. Extending file size to 9016288 .. Extending file size to 8528096 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8993376 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9195936 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 57410 57412 57413 57414 57416 192.168.206.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre2 (opts:) 192.168.206.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:21:00 (1788528060) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre2 Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1489344 .. Extending file size to 4221440 .. failed Extending file size to 7138976 .. Extending file size to 8859456 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8047360 .. Extending file size to 9030400 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8196928 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9267776 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58853 58854 58857 58858 58859 done 192.168.206.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre2 (opts:) 192.168.206.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (89s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:22:29 (1788528149) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123993 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.479715 s, 4.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xeef21b98 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:22:52 (1788528172) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118656 s, 17.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 09:22:59 (1788528179) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:23:07 (1788528187) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.221695 s, 9.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:23:16 (1788528196) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203707 s, 10.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:23:25 (1788528205) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237085 s, 8.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:24:00 (1788528240) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg651-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837881 s, 12.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:25:01 (1788528301) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg651-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.272049 s, 7.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:25:57 (1788528357) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg651-server mds-ost sync done. Creating new pool test_204f oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: Waiting 90s for '' Adding targets to pool oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.454553 s, 4.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: Waiting 90s for 'foo' Updated after 3s: want 'foo' got 'foo' PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:27:15 (1788528435) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:27:23 (1788528443) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:27:32 (1788528452) oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: Waiting 90s for '' Adding targets to pool oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: oleg651-server: Warning, standalone MGS for "lustre", unable to check pool updates on clients. oleg651-server: Verify if pools are updated on a client (client sync delay: 1-2s). oleg651-server: Waiting 90s for 'foo' PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:27:53 (1788528473) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10851 s, 9.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357681 s, 29.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6d694153 Mirror 3: 0x6d694153 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:28:04 (1788528484) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:28:07 (1788528487) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:28:11 (1788528491) Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.108572 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1980000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xaf0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xaf0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xaf0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x850000 Read this (4 bytes): BBBB 192.168.206.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg651-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:28:46 (1788528526) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0230915 s, 355 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:28:52 (1788528532) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642019 s, 16.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x21a lmm_fid: [0x200000404:0x21a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x21a lmm_fid: [0x200000404:0x21a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834458 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 09:28:54.000000000 -0400 Modify: 2026-09-04 09:28:54.000000000 -0400 Change: 2026-09-04 09:28:56.000000000 -0400 Birth: 2026-09-04 09:28:53.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:29:01 (1788528541) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:29:04 (1788528544) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.483546 s, 84.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0379974 s, 108 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 09:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 09:29 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:29:12 (1788528552) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:29:22 (1788528562) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1865 sec =========== 09:29:31 (1788528571) === sanity-flr: start cleanup 09:29:33 (1788528573) === === sanity-flr: finish cleanup 09:29:41 (1788528581) ===