-----============= acceptance-small: sanity-flr ============----- Tue Sep 1 10:08:27 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg207-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:08:45 (1788271725) === oleg207-client.virtnet: executing check_config_client /mnt/lustre oleg207-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg207-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91a389ea4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91a389ea4800.idle_timeout=debug disable quota as required oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:09:18 (1788271758) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7628] == sanity-flr test 0a: lfs mirror create with -N option == 10:09:20 (1788271760) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:09:34 (1788271774) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:09:38 (1788271778) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:09:41 (1788271781) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:10:02 (1788271802) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:10:05 (1788271805) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 10:10:08 (1788271808) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff91a389ea4800.stats=clear osc.lustre-OST0001-osc-ffff91a389ea4800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:10:15 (1788271815) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:10:25 (1788271825) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:10:33 (1788271833) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:10:37 (1788271837) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:10:44 (1788271844) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 10:10:53 (1788271853) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:11:00 (1788271860) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:11:03 (1788271863) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:11:10 (1788271870) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:11:20 (1788271880) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:11:25 (1788271885) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.79371 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145666 s, 7.2 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:11:36 (1788271896) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.420793 s, 44.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:11:44 (1788271904) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:11:55 (1788271915) Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a389ea4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a389ea4800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg207-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a389ea4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a389ea4800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:14:12 (1788272052) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.115435 s, 27.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a389ea4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a389ea4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:14:55 (1788272095) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a389ea4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a389ea4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a389ea4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a389ea4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:16:14 (1788272174) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.0134 s, 13.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 1 10:16 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.900s user 0m0.001s sys 0m0.170s Stopping /mnt/lustre-ost1 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91a389ea4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91a389ea4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.006s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg207-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg207-server oleg207-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91a389ea4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a389ea4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.050s user 0m0.002s sys 0m0.167s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg207-server: oleg207-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg207-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg207-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91a389ea4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91a389ea4800.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:17:48 (1788272268) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:17:53 (1788272273) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:17:57 (1788272277) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 10:18:01 (1788272281) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 10:18:10 (1788272290) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.952838 s, 13.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (22s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:18:32 (1788272312) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907588 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617499 s, 17.0 MB/s PASS 36b (13s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:18:46 (1788272326) PASS 36c (12s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:18:57 (1788272337) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0016801 s, 609 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00154499 s, 663 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00195423 s, 524 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00100366 s, 1.0 MB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 10:19:11 (1788272351) create test files with size 15729284 .. af307bf019b32f3a5092401db040054b - 9fc29115e3de13854d21903bb1c6cc66 - 675001608ce79bd049a5ae94cc409d7f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 10:19:35 (1788272375) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:19:50 (1788272390) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:19:56 (1788272396) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11186 s, 9.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603503 s, 17.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 10:20:05 (1788272405) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0991083 s, 21.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.358962 s, 11.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286468 s, 7.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 10:20:24 (1788272424) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 10:20:28 (1788272428) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:20:33 (1788272433) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372437 s, 28.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 10:20:44 (1788272444) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:20:49 (1788272449) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:21:02 (1788272462) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.389011 s, 27.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:21:13 (1788272473) llite.lustre-ffff91a389d5d800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.245726 s, 42.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg207-client.virtnet /mnt/lustre (opts:) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff91a39c8f5800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:21:25 (1788272485) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:22:03 (1788272523) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:22:40 (1788272560) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:22:44 (1788272564) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.354173 s, 8.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.221562 s, 14.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x69aa66c0 (nosync,stale) Mirror 2: 0x69aa66c0 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2e8c01aa4ba4751b7a08354a16b1b7e3 - mirror 1 checksum: 54ec622739bbe33140405f993cbd9026 - mirror 2 checksum: 2e8c01aa4ba4751b7a08354a16b1b7e3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T10:22:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T10:22:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2e8c01aa4ba4751b7a08354a16b1b7e3 - mirror 1 checksum: 54ec622739bbe33140405f993cbd9026 - mirror 2 checksum: 54ec622739bbe33140405f993cbd9026 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T10:22:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T10:22:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:22:59 (1788272579) Failing mds1 on oleg207-server Stopping /mnt/lustre-mds1 (opts:) on oleg207-server 10:23:05 (1788272585) shut down facet: mds1 facet_host: oleg207-server facet_failover_host: oleg207-server Failover mds1 to oleg207-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 10:23:29 (1788272609) targets are mounted 10:23:29 (1788272609) facet_failover done oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323544 s, 32.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:23:46 (1788272626) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:23:50 (1788272630) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:23:54 (1788272634) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0158022 s, 1.0 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:24:03 (1788272643) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:24:08 (1788272648) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0189363 s, 216 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.172426 s, 23.8 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:24:18 (1788272658) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:24:45 (1788272685) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:25:13 (1788272713) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:25:37 (1788272737) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:26:03 (1788272763) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 10:26:11 (1788272771) mirror create pid 54336 mirror split pid 54337 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54336 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54337 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 10:27:19 (1788272839) Seed set to 2840 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005 1788272841.150999 mapwrite 0x02a68e thru 0x02c638 (0x1fab bytes) 000006 1788272841.224140 mapread 0x001a5f thru 0x007976 (0x5f18 bytes) 000007 1788272841.234964 write 0x039bb1 thru 0x03ffff (0x644f bytes) 000009 1788272841.236260 mapread 0x02f84a thru 0x03eda8 (0xf55f bytes) 000010 1788272841.245937 mapread 0x012e57 thru 0x014807 (0x19b1 bytes) 000012 1788272841.255542 read 0x00194d thru 0x003304 (0x19b8 bytes) 000013 1788272841.257071 read 0x02cf94 thru 0x0383f1 (0xb45e bytes) 000014 1788272841.257868 read 0x011288 thru 0x018630 (0x73a9 bytes) 000016 1788272841.258962 mapwrite 0x03c76e thru 0x03ffff (0x3892 bytes) truncating to largest ever: 0x1a59a 000017 1788272841.303877 trunc from 0x040000 to 0x01a599 (0x25a66 bytes) 000019 1788272841.321184 write 0x02cc2f thru 0x0331fa (0x65cc bytes) 000021 1788272841.323527 mapwrite 0x0019f7 thru 0x00a3ff (0x8a09 bytes) 000022 1788272841.368938 read 0x00ab77 thru 0x010db0 (0x623a bytes) 000023 1788272841.369843 write 0x019f92 thru 0x01b22b (0x129a bytes) truncating to largest ever: 0x1c52d 000024 1788272841.377841 trunc from 0x0331fb to 0x01c52c (0x16cce bytes) 000031 1788272841.391783 trunc from 0x01c52d to 0x00abc5 (0x11967 bytes) 000032 1788272841.401976 mapread 0x003f55 thru 0x00abc5 (0x6c71 bytes) 000033 1788272841.402568 write 0x00ea05 thru 0x0194b7 (0xaab3 bytes) 000034 1788272841.403451 trunc from 0x0194b8 to 0x015186 (0x4331 bytes) 000035 1788272841.418916 mapread 0x012da1 thru 0x015186 (0x23e6 bytes) 000038 1788272841.422213 read 0x003f04 thru 0x00a2b9 (0x63b6 bytes) 000042 1788272841.425638 read 0x00228e thru 0x010a37 (0xe7aa bytes) 000047 1788272841.434521 write 0x021d12 thru 0x028fd7 (0x72c6 bytes) 000048 1788272841.440643 MIRROR_EXTEND to 2 mirrors 000049 1788272841.989120 mapread 0x008684 thru 0x012ca0 (0xa61d bytes) 000050 1788272842.001252 read 0x005065 thru 0x010be0 (0xbb7c bytes) 000053 1788272842.002244 trunc from 0x028fd8 to 0x01bddc (0xd1fb bytes) 000054 1788272842.092265 mapwrite 0x00cd79 thru 0x0184c9 (0xb751 bytes) 000059 1788272842.235734 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1bddd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bddd): Mirror 2: 0xf5362625 (stale) CRC-32 checksum value for chunk [0, 0x1bddd): Mirror 1: 0xeef38baa Mirror 3: 0xeef38baa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bddd: skipped 000060 1788272842.963529 mapwrite 0x033596 thru 0x03ae3d (0x78a8 bytes) 000061 1788272843.204727 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae3e): Mirror 3: 0x6c63dd07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae3e): Mirror 3: 0x6c63dd07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae3e: skipped 000062 1788272843.549332 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ae3e): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae3e): Mirror 3: 0x6c63dd07 (stale) CRC-32 checksum value for chunk [0, 0x3ae3e): Mirror 1: 0x2719d504 Mirror 4: 0x2719d504 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae3e: skipped 000065 1788272844.440195 write 0x018e7c thru 0x02124d (0x83d2 bytes) 000066 1788272844.533491 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae3e): Mirror 4: 0x2719d504 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae3e): Mirror 4: 0x2719d504 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae3e: skipped 000067 1788272844.787240 trunc from 0x03ae3e to 0x01a0a7 (0x20d96 bytes) 000069 1788272844.804545 trunc from 0x01a0a8 to 0x011c85 (0x8422 bytes) 000070 1788272844.821740 write 0x031250 thru 0x03e350 (0xd101 bytes) 000071 1788272844.824458 mapread 0x0071a3 thru 0x014046 (0xcea4 bytes) 000073 1788272844.850091 mapwrite 0x023c44 thru 0x028861 (0x4c1e bytes) 000074 1788272844.910713 write 0x038001 thru 0x03ffff (0x7fff bytes) 000075 1788272844.912781 mapwrite 0x00dd69 thru 0x01450f (0x67a7 bytes) truncating to largest ever: 0x39aee 000076 1788272844.966106 trunc from 0x040000 to 0x039aed (0x6512 bytes) 000077 1788272844.981145 write 0x01bcdd thru 0x0250c4 (0x93e8 bytes) 000079 1788272844.991306 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) 000082 1788272845.148689 write 0x004418 thru 0x0056c9 (0x12b2 bytes) 000084 1788272845.160315 mapwrite 0x038379 thru 0x03fd6b (0x79f3 bytes) 000089 1788272845.249452 mapwrite 0x032cd5 thru 0x03b97f (0x8cab bytes) 000090 1788272845.290686 trunc from 0x03fd6c to 0x0270a8 (0x18cc3 bytes) 000091 1788272845.304343 mapread 0x0128af thru 0x013e78 (0x15ca bytes) 000098 1788272845.324445 mapwrite 0x00cb10 thru 0x016076 (0x9567 bytes) 000099 1788272845.371713 trunc from 0x0270a9 to 0x0264f3 (0xbb5 bytes) 000100 1788272845.387972 write 0x01b63d thru 0x01e21c (0x2be0 bytes) 000105 1788272845.404042 mapwrite 0x039d55 thru 0x03ce3f (0x30eb bytes) 000107 1788272845.471668 mapread 0x00cfc2 thru 0x01a02a (0xd069 bytes) 000109 1788272845.473064 trunc from 0x03ce40 to 0x00214d (0x3acf2 bytes) 000110 1788272845.493862 mapwrite 0x018874 thru 0x025655 (0xcde2 bytes) 000113 1788272845.566659 write 0x00a95d thru 0x012cf9 (0x839d bytes) 000114 1788272845.573097 write 0x01195e thru 0x01db4c (0xc1ef bytes) 000115 1788272845.574454 write 0x00ced8 thru 0x01b91c (0xea45 bytes) 000122 1788272845.581497 mapwrite 0x028f1c thru 0x03321e (0xa303 bytes) 000123 1788272845.648275 mapread 0x02e0af thru 0x03321e (0x5170 bytes) 000124 1788272845.649286 read 0x01c3e6 thru 0x020dc3 (0x49de bytes) 000125 1788272845.651488 mapwrite 0x02bf1c thru 0x02fe6e (0x3f53 bytes) 000127 1788272845.695186 read 0x00977b thru 0x00e916 (0x519c bytes) 000133 1788272845.707093 mapread 0x016ac1 thru 0x01f465 (0x89a5 bytes) 000134 1788272845.708406 mapwrite 0x03021e thru 0x036905 (0x66e8 bytes) 000137 1788272845.776494 mapread 0x01b455 thru 0x0283dd (0xcf89 bytes) 000138 1788272845.778111 mapwrite 0x0050e2 thru 0x0141fa (0xf119 bytes) 000139 1788272845.823296 write 0x022dd2 thru 0x030025 (0xd254 bytes) 000141 1788272845.825896 read 0x0259bb thru 0x0329b0 (0xcff6 bytes) 000144 1788272845.827918 mapwrite 0x02ad15 thru 0x02f67e (0x496a bytes) 000146 1788272845.875447 mapread 0x011b3b thru 0x016014 (0x44da bytes) 000148 1788272845.876777 mapwrite 0x01fec5 thru 0x025255 (0x5391 bytes) 000149 1788272845.918195 read 0x0258a0 thru 0x028fd0 (0x3731 bytes) 000150 1788272845.919248 mapread 0x030aa8 thru 0x0366ba (0x5c13 bytes) 000151 1788272845.920127 read 0x0279fd thru 0x02ba5e (0x4062 bytes) 000153 1788272845.922953 trunc from 0x036906 to 0x006cc5 (0x2fc40 bytes) 000157 1788272845.944823 write 0x02ccbf thru 0x02e60d (0x194f bytes) 000160 1788272845.950210 mapwrite 0x013472 thru 0x019cb1 (0x6840 bytes) 000165 1788272846.024628 MIRROR_EXTEND to 2 mirrors 000166 1788272846.850667 MIRROR_RESYNC 2 mirrors 000168 1788272847.111791 MIRROR_EXTEND to 3 mirrors 000170 1788272847.990272 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) 000172 1788272848.338875 write 0x03c0e8 thru 0x03d559 (0x1472 bytes) 000173 1788272848.412768 read 0x00193a thru 0x009e8b (0x8552 bytes) 000174 1788272848.418001 mapread 0x0152ed thru 0x01a583 (0x5297 bytes) 000176 1788272848.438099 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d55a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d55a): Mirror 3: 0x8e995fee (stale) CRC-32 checksum value for chunk [0, 0x3d55a): Mirror 1: 0xf8ab11e9 Mirror 4: 0xf8ab11e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d55a: skipped 000177 1788272849.335783 trunc from 0x03d55a to 0x01f43d (0x1e11c bytes) 000179 1788272849.443247 read 0x01ead3 thru 0x01f43d (0x96b bytes) truncating to largest ever: 0x3b13c 000180 1788272849.449704 trunc from 0x01f43e to 0x03b13b (0x1bcfe bytes) 000183 1788272849.472105 write 0x02c893 thru 0x03a2d1 (0xda3f bytes) 000185 1788272849.483007 mapwrite 0x002b3c thru 0x00cc99 (0xa15e bytes) 000186 1788272849.605920 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b13c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b13c): Mirror 3: 0x144dbb92 (stale) Mirror 4: 0x144dbb92 (stale) CRC-32 checksum value for chunk [0, 0x3b13c): Mirror 1: 0x2a8d2fa4 Mirror 5: 0x2a8d2fa4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b13c: skipped 000187 1788272850.737849 mapwrite 0x01f3ee thru 0x0244d7 (0x50ea bytes) 000188 1788272851.026815 trunc from 0x03b13c to 0x00930f (0x31e2c bytes) 000189 1788272851.100847 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9310): Mirror 3: 0xbfdd77c0 (stale) Mirror 5: 0x68851b41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9310): Mirror 3: 0xbfdd77c0 (stale) Mirror 5: 0x68851b41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9310: skipped 000190 1788272851.458628 mapread 0x000db7 thru 0x00577d (0x49c7 bytes) 000192 1788272851.482460 write 0x0016d2 thru 0x00938a (0x7cb9 bytes) 000193 1788272851.484475 trunc from 0x00938b to 0x0323a0 (0x29016 bytes) 000194 1788272851.502145 write 0x02b4d3 thru 0x038953 (0xd481 bytes) 000195 1788272851.511088 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38954): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38954): Mirror 3: 0xd0460bcd (stale) Mirror 5: 0x692b9361 (stale) CRC-32 checksum value for chunk [0, 0x38954): Mirror 1: 0x4d7c50cb Mirror 6: 0x4d7c50cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38954: skipped 000197 1788272852.863741 MIRROR_RESYNC 4 mirrors 000198 1788272854.053085 write 0x0022bf thru 0x008c93 (0x69d5 bytes) 000199 1788272854.164353 mapwrite 0x03172c thru 0x0346d1 (0x2fa6 bytes) 000200 1788272854.258560 mapwrite 0x01704a thru 0x023449 (0xc400 bytes) 000202 1788272854.313682 mapwrite 0x01642f thru 0x019442 (0x3014 bytes) 000203 1788272854.361054 mapread 0x00a2c1 thru 0x00bf37 (0x1c77 bytes) 000204 1788272854.368303 trunc from 0x038954 to 0x0246a5 (0x142ae bytes) 000205 1788272854.389047 write 0x0299ce thru 0x03163d (0x7c70 bytes) 000208 1788272854.392709 mapwrite 0x01b425 thru 0x022637 (0x7213 bytes) 000209 1788272854.447337 write 0x025f0e thru 0x02b25d (0x5350 bytes) 000210 1788272854.452546 mapwrite 0x033740 thru 0x034303 (0xbc4 bytes) 000212 1788272854.502026 mapwrite 0x03cba8 thru 0x03ffff (0x3458 bytes) 000213 1788272854.563105 mapwrite 0x017d73 thru 0x01b97a (0x3c08 bytes) 000219 1788272854.605824 write 0x0075aa thru 0x00e0a3 (0x6afa bytes) 000221 1788272854.609565 read 0x01345f thru 0x020a22 (0xd5c4 bytes) 000223 1788272854.612676 trunc from 0x040000 to 0x0134cb (0x2cb34 bytes) 000224 1788272854.631541 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x134cc): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x134cc): Mirror 3: 0xe0232aca (stale) Mirror 5: 0xe0232aca (stale) Mirror 6: 0xe0232aca (stale) CRC-32 checksum value for chunk [0, 0x134cc): Mirror 1: 0xc96c69f8 Mirror 7: 0xc96c69f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x134cc: skipped 000225 1788272855.604530 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x134cc): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x134cc): Mirror 3: 0xe0232aca (stale) Mirror 5: 0xe0232aca (stale) Mirror 6: 0xe0232aca (stale) CRC-32 checksum value for chunk [0, 0x134cc): Mirror 1: 0xc96c69f8 Mirror 7: 0xc96c69f8 Mirror 8: 0xc96c69f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x134cc: skipped 000228 1788272856.322540 mapread 0x00caea thru 0x00f951 (0x2e68 bytes) 000229 1788272856.343183 write 0x0216ff thru 0x023b88 (0x248a bytes) 000233 1788272856.411565 mapwrite 0x022836 thru 0x02a41e (0x7be9 bytes) 000234 1788272856.474061 write 0x02e5ca thru 0x034056 (0x5a8d bytes) 000235 1788272856.475361 write 0x010fc0 thru 0x01e70e (0xd74f bytes) 000239 1788272856.488520 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x34057): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34057): Mirror 3: 0x7645a90c (stale) Mirror 5: 0x7645a90c (stale) Mirror 6: 0x7645a90c (stale) Mirror 7: 0xacfe8162 (stale) Mirror 8: 0xacfe8162 (stale) CRC-32 checksum value for chunk [0, 0x34057): Mirror 1: 0x22d312bc Mirror 9: 0x22d312bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34057: skipped 000240 1788272857.203161 MIRROR_RESYNC 7 mirrors 000244 1788272858.293006 MIRROR_RESYNC 7 mirrors 000247 1788272858.651788 mapwrite 0x011150 thru 0x013bf9 (0x2aaa bytes) 000248 1788272858.794102 write 0x00bc3d thru 0x01b1c9 (0xf58d bytes) 000251 1788272858.801322 mapwrite 0x01c5b1 thru 0x02581f (0x926f bytes) 000252 1788272858.865566 trunc from 0x034057 to 0x02b8aa (0x87ac bytes) 000253 1788272858.911375 read 0x0271b4 thru 0x02b8aa (0x46f7 bytes) 000254 1788272858.912059 read 0x01d4e6 thru 0x01fe70 (0x298b bytes) 000256 1788272858.913069 mapwrite 0x036316 thru 0x03f025 (0x8d10 bytes) 000257 1788272859.033525 mapread 0x029a90 thru 0x03960f (0xfb80 bytes) 000259 1788272859.035620 trunc from 0x03f026 to 0x036b10 (0x8515 bytes) 000260 1788272859.058316 mapread 0x000a1c thru 0x00b45e (0xaa43 bytes) 000264 1788272859.062991 read 0x00ee23 thru 0x010a40 (0x1c1e bytes) 000265 1788272859.063749 mapread 0x007eb1 thru 0x0154e1 (0xd631 bytes) 000266 1788272859.065658 write 0x000287 thru 0x000bdc (0x956 bytes) 000267 1788272859.067105 mapwrite 0x01e522 thru 0x026c4d (0x872c bytes) 000269 1788272859.131493 write 0x036c86 thru 0x03be68 (0x51e3 bytes) 000270 1788272859.138348 MIRROR_RESYNC 7 mirrors 000272 1788272861.209456 read 0x01f936 thru 0x02ead3 (0xf19e bytes) 000273 1788272861.219411 read 0x0160e0 thru 0x01b973 (0x5894 bytes) 000274 1788272861.228258 mapwrite 0x00cf8d thru 0x01b288 (0xe2fc bytes) 000275 1788272861.456117 mapread 0x0273b0 thru 0x0318f7 (0xa548 bytes) 000276 1788272861.510268 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3be69): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be69): Mirror 3: 0xa6c29c27 (stale) Mirror 5: 0xa6c29c27 (stale) Mirror 6: 0xa6c29c27 (stale) Mirror 7: 0xa6c29c27 (stale) Mirror 8: 0xa6c29c27 (stale) Mirror 9: 0xa6c29c27 (stale) CRC-32 checksum value for chunk [0, 0x3be69): Mirror 1: 0xd401c65f Mirror 10: 0xd401c65f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be69: skipped 000277 1788272862.414361 trunc from 0x03be69 to 0x02d185 (0xece3 bytes) 000278 1788272862.576897 write 0x00fa42 thru 0x017282 (0x7841 bytes) 000279 1788272862.587943 mapread 0x01f21e thru 0x020668 (0x144b bytes) 000282 1788272862.604211 mapwrite 0x03a92c thru 0x03ffff (0x56d4 bytes) 000283 1788272862.701715 write 0x0070a3 thru 0x00eba8 (0x7b06 bytes) truncating to largest ever: 0x3c88c 000285 1788272862.715560 trunc from 0x040000 to 0x03c88b (0x3774 bytes) 000286 1788272862.745753 MIRROR_RESYNC 8 mirrors 000287 1788272864.653228 read 0x02cf2f thru 0x0396ca (0xc79c bytes) 000288 1788272864.667035 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) 000292 1788272865.108329 MIRROR_EXTEND to 8 mirrors 000293 1788272865.789330 read 0x034ecc thru 0x03c88b (0x79c0 bytes) 000294 1788272865.797043 trunc from 0x03c88c to 0x03affc (0x188f bytes) 000295 1788272865.912550 mapread 0x02f1af thru 0x034a5f (0x58b1 bytes) 000298 1788272865.937525 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3affd): Mirror 5: 0x8b3e98aa (stale) Mirror 6: 0x8b3e98aa (stale) Mirror 7: 0x8b3e98aa (stale) Mirror 8: 0x8b3e98aa (stale) Mirror 10: 0x8b3e98aa (stale) Mirror 11: 0x8b3e98aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3affd): Mirror 5: 0x8b3e98aa (stale) Mirror 6: 0x8b3e98aa (stale) Mirror 7: 0x8b3e98aa (stale) Mirror 8: 0x8b3e98aa (stale) Mirror 10: 0x8b3e98aa (stale) Mirror 11: 0x8b3e98aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3affd: skipped 000299 1788272866.360078 mapwrite 0x022d95 thru 0x02947e (0x66ea bytes) 000300 1788272866.427554 trunc from 0x03affd to 0x0062a2 (0x34d5a bytes) 000302 1788272866.447928 trunc from 0x0062a3 to 0x01a13a (0x13e98 bytes) 000305 1788272866.472076 mapwrite 0x0009af thru 0x0057ed (0x4e3f bytes) 000306 1788272866.531159 mapwrite 0x00cdb3 thru 0x00f3aa (0x25f8 bytes) 000309 1788272866.565529 mapwrite 0x00bbc7 thru 0x0115be (0x59f8 bytes) 000311 1788272866.621873 trunc from 0x01a13b to 0x022322 (0x81e8 bytes) 000312 1788272866.641986 MIRROR_RESYNC 7 mirrors 000313 1788272867.973249 read 0x01228e thru 0x0179c7 (0x573a bytes) 000315 1788272867.981166 read 0x021da3 thru 0x022322 (0x580 bytes) 000316 1788272868.000221 mapread 0x0134b0 thru 0x0161fb (0x2d4c bytes) 000317 1788272868.001281 mapread 0x000312 thru 0x009480 (0x916f bytes) 000319 1788272868.002204 mapwrite 0x026cd1 thru 0x02da49 (0x6d79 bytes) 000323 1788272868.164975 mapread 0x016a7a thru 0x01db1b (0x70a2 bytes) 000324 1788272868.208451 write 0x001d3f thru 0x009966 (0x7c28 bytes) 000325 1788272868.215662 write 0x008800 thru 0x00c9be (0x41bf bytes) 000326 1788272868.232744 MIRROR_RESYNC 7 mirrors 000327 1788272869.961020 mapwrite 0x0063f3 thru 0x00ce2e (0x6a3c bytes) 000329 1788272870.129498 write 0x039f18 thru 0x03ffff (0x60e8 bytes) 000330 1788272870.138997 MIRROR_RESYNC 7 mirrors 000331 1788272872.324893 mapread 0x0349e2 thru 0x038c5d (0x427c bytes) 000332 1788272872.348046 trunc from 0x040000 to 0x022cc0 (0x1d33f bytes) 000334 1788272872.417866 trunc from 0x022cc1 to 0x032df5 (0x10135 bytes) 000335 1788272872.432634 trunc from 0x032df6 to 0x01ac92 (0x18163 bytes) 000337 1788272872.447709 MIRROR_RESYNC 7 mirrors 000342 1788272873.767515 write 0x030d69 thru 0x038ac5 (0x7d5d bytes) 000343 1788272873.868651 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ac6): Mirror 5: 0x290bea9c (stale) Mirror 6: 0x290bea9c (stale) Mirror 7: 0x290bea9c (stale) Mirror 8: 0x290bea9c (stale) Mirror 11: 0x290bea9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ac6): Mirror 5: 0x290bea9c (stale) Mirror 6: 0x290bea9c (stale) Mirror 7: 0x290bea9c (stale) Mirror 8: 0x290bea9c (stale) Mirror 11: 0x290bea9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ac6: skipped 000344 1788272874.300468 read 0x00c68b thru 0x00e4c6 (0x1e3c bytes) 000346 1788272874.308843 read 0x008db1 thru 0x015eab (0xd0fb bytes) 000347 1788272874.318212 mapread 0x01aeb0 thru 0x01b47f (0x5d0 bytes) 000349 1788272874.323115 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ac6): Mirror 5: 0x290bea9c (stale) Mirror 6: 0x290bea9c (stale) Mirror 7: 0x290bea9c (stale) Mirror 8: 0x290bea9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ac6): Mirror 5: 0x290bea9c (stale) Mirror 6: 0x290bea9c (stale) Mirror 7: 0x290bea9c (stale) Mirror 8: 0x290bea9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ac6: skipped 000352 1788272874.705549 write 0x03d92c thru 0x03f429 (0x1afe bytes) 000353 1788272874.707102 mapwrite 0x0119ec thru 0x014431 (0x2a46 bytes) 000357 1788272874.784317 write 0x00e119 thru 0x017257 (0x913f bytes) 000358 1788272874.795273 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f42a): Mirror 5: 0xf643f399 (stale) Mirror 6: 0xf643f399 (stale) Mirror 8: 0xf643f399 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f42a): Mirror 5: 0xf643f399 (stale) Mirror 6: 0xf643f399 (stale) Mirror 8: 0xf643f399 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f42a: skipped 000360 1788272875.272370 mapwrite 0x01eb3c thru 0x026d34 (0x81f9 bytes) 000361 1788272875.346453 mapwrite 0x002436 thru 0x00fae0 (0xd6ab bytes) 000364 1788272875.408483 read 0x038572 thru 0x03f429 (0x6eb8 bytes) 000365 1788272875.419351 MIRROR_RESYNC 4 mirrors 000367 1788272876.757382 mapread 0x00396d thru 0x011717 (0xddab bytes) 000370 1788272876.784171 mapread 0x006c4b thru 0x00f524 (0x88da bytes) truncating to largest ever: 0x3f095 000371 1788272876.784925 trunc from 0x03f42a to 0x03f094 (0x395 bytes) 000372 1788272876.874947 MIRROR_RESYNC 4 mirrors 000374 1788272878.227814 write 0x034f2b thru 0x03a2c2 (0x5398 bytes) 000375 1788272878.315506 read 0x0189ce thru 0x01a3bf (0x19f2 bytes) 000377 1788272878.323215 mapread 0x029992 thru 0x037a66 (0xe0d5 bytes) 000378 1788272878.346154 MIRROR_RESYNC 4 mirrors 000379 1788272879.891620 read 0x038482 thru 0x03f094 (0x6c13 bytes) 000380 1788272879.902029 read 0x00692c thru 0x013146 (0xc81b bytes) 000381 1788272879.919545 mapwrite 0x03289e thru 0x038187 (0x58ea bytes) 000382 1788272880.182407 mapwrite 0x00f526 thru 0x0107aa (0x1285 bytes) 000383 1788272880.248948 read 0x010154 thru 0x016ee8 (0x6d95 bytes) 000384 1788272880.262011 mapread 0x008b41 thru 0x0124cb (0x998b bytes) 000385 1788272880.263048 read 0x037787 thru 0x03f094 (0x790e bytes) 000386 1788272880.263644 write 0x021297 thru 0x021e12 (0xb7c bytes) 000387 1788272880.265195 mapwrite 0x031ee0 thru 0x0328e7 (0xa08 bytes) 000388 1788272880.313184 MIRROR_RESYNC 4 mirrors 000390 1788272881.888783 write 0x0175bd thru 0x025b28 (0xe56c bytes) 000391 1788272882.015629 read 0x038c06 thru 0x03a7a2 (0x1b9d bytes) 000393 1788272882.023705 mapwrite 0x02d120 thru 0x02f30c (0x21ed bytes) 000395 1788272882.143416 trunc from 0x03f095 to 0x00d4fa (0x31b9a bytes) 000398 1788272882.165085 trunc from 0x00d4fb to 0x034ca6 (0x277ac bytes) 000399 1788272882.180545 mapwrite 0x001d24 thru 0x004d12 (0x2fef bytes) 000400 1788272882.226015 write 0x022575 thru 0x031292 (0xed1e bytes) 000401 1788272882.235285 write 0x0372f3 thru 0x03ffff (0x8d0d bytes) 000404 1788272882.249828 MIRROR_RESYNC 4 mirrors 000405 1788272883.852019 mapwrite 0x021cd9 thru 0x02551b (0x3843 bytes) 000406 1788272884.028977 trunc from 0x040000 to 0x0240e5 (0x1bf1a bytes) 000407 1788272884.062769 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x240e6): Mirror 5: 0x78cff95e (stale) Mirror 6: 0x78cff95e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x240e6): Mirror 5: 0x78cff95e (stale) Mirror 6: 0x78cff95e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x240e6: skipped 000409 1788272884.351868 trunc from 0x0240e6 to 0x02e3f0 (0xa30b bytes) 000410 1788272884.371485 write 0x03a17a thru 0x03ffff (0x5e86 bytes) 000411 1788272884.372864 mapwrite 0x00f357 thru 0x01456d (0x5217 bytes) 000412 1788272884.431474 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe1a2135e (stale) Mirror 6: 0xe1a2135e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd551c651 Mirror 7: 0xd551c651 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000413 1788272885.254751 write 0x0325f5 thru 0x03ffff (0xda0b bytes) 000416 1788272885.332272 read 0x02ae37 thru 0x02f859 (0x4a23 bytes) 000417 1788272885.339854 mapread 0x0021a0 thru 0x006c13 (0x4a74 bytes) 000418 1788272885.356260 read 0x031881 thru 0x03ffff (0xe77f bytes) 000419 1788272885.365073 mapread 0x00b0cd thru 0x015bda (0xab0e bytes) 000421 1788272885.374094 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe1a2135e (stale) Mirror 6: 0xe1a2135e (stale) Mirror 7: 0xd551c651 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbd35d5eb Mirror 8: 0xbd35d5eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000423 1788272886.719871 write 0x00dcad thru 0x0110c9 (0x341d bytes) 000428 1788272886.883266 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe1a2135e (stale) Mirror 6: 0xe1a2135e (stale) Mirror 7: 0xd551c651 (stale) Mirror 8: 0xbd35d5eb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd3efe10b Mirror 9: 0xd3efe10b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000430 1788272888.343861 trunc from 0x040000 to 0x009ad2 (0x3652d bytes) 000434 1788272888.510508 read 0x007024 thru 0x009ad2 (0x2aaf bytes) 000437 1788272888.516425 read 0x0021e7 thru 0x009ad2 (0x78ec bytes) truncating to largest ever: 0x3f1ad 000438 1788272888.525377 trunc from 0x009ad3 to 0x03f1ac (0x356da bytes) 000439 1788272888.544874 mapread 0x01d643 thru 0x025201 (0x7bbf bytes) 000443 1788272888.566965 write 0x0034ec thru 0x00882d (0x5342 bytes) 000445 1788272888.575084 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f1ad): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1ad): Mirror 5: 0x8da4c167 (stale) Mirror 6: 0x8da4c167 (stale) Mirror 7: 0x77741344 (stale) Mirror 8: 0x74defb17 (stale) Mirror 9: 0x7e1b5a47 (stale) CRC-32 checksum value for chunk [0, 0x3f1ad): Mirror 1: 0xf62879ba Mirror 10: 0xf62879ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1ad: skipped 000446 1788272889.813423 write 0x02edfe thru 0x035cd8 (0x6edb bytes) 000447 1788272889.883005 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f1ad): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1ad): Mirror 5: 0x8da4c167 (stale) Mirror 6: 0x8da4c167 (stale) Mirror 7: 0x77741344 (stale) Mirror 8: 0x74defb17 (stale) Mirror 9: 0x7e1b5a47 (stale) Mirror 10: 0xf62879ba (stale) CRC-32 checksum value for chunk [0, 0x3f1ad): Mirror 1: 0x564a4c3a Mirror 11: 0x564a4c3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1ad: skipped 000452 1788272891.470781 mapwrite 0x00a851 thru 0x00f07a (0x482a bytes) 000453 1788272891.698447 mapwrite 0x00589b thru 0x00bd8c (0x64f2 bytes) 000454 1788272891.759124 write 0x026bb7 thru 0x02cfc1 (0x640b bytes) 000455 1788272891.782251 mapread 0x007227 thru 0x016e89 (0xfc63 bytes) 000456 1788272891.795806 trunc from 0x03f1ad to 0x02960e (0x15b9e bytes) 000458 1788272891.827121 mapread 0x01d693 thru 0x02472a (0x7098 bytes) 000459 1788272891.838827 mapread 0x002f56 thru 0x0074f5 (0x45a0 bytes) 000461 1788272891.846088 mapread 0x028aac thru 0x02960e (0xb63 bytes) 000462 1788272891.846962 write 0x011878 thru 0x01935b (0x7ae4 bytes) 000463 1788272891.848021 mapwrite 0x0245be thru 0x02c7f3 (0x8236 bytes) 000469 1788272891.950615 read 0x002cd1 thru 0x00bb9a (0x8eca bytes) 000470 1788272891.953863 mapread 0x004586 thru 0x008180 (0x3bfb bytes) 000472 1788272891.956158 read 0x012114 thru 0x01efd0 (0xcebd bytes) 000474 1788272891.957545 trunc from 0x02c7f4 to 0x006796 (0x2605d bytes) 000476 1788272891.983023 MIRROR_RESYNC 8 mirrors 000477 1788272893.418754 mapwrite 0x0354e9 thru 0x03982c (0x4344 bytes) 000481 1788272893.622813 mapread 0x025638 thru 0x033535 (0xdefe bytes) 000483 1788272893.657728 read 0x0143b4 thru 0x016020 (0x1c6d bytes) 000485 1788272893.671568 MIRROR_RESYNC 8 mirrors 000488 1788272895.970374 trunc from 0x03982d to 0x01a67d (0x1f1af bytes) 000491 1788272896.060282 read 0x002e9a thru 0x006c60 (0x3dc7 bytes) 000493 1788272896.066538 mapwrite 0x0068a4 thru 0x00ee74 (0x85d1 bytes) 000494 1788272896.142324 trunc from 0x01a67e to 0x01bd0f (0x1692 bytes) 000495 1788272896.165069 write 0x013a20 thru 0x014a9b (0x107c bytes) 000496 1788272896.184562 MIRROR_RESYNC 8 mirrors 000499 1788272897.791360 mapwrite 0x01d57a thru 0x02d435 (0xfebc bytes) 000500 1788272897.980205 write 0x019933 thru 0x01a037 (0x705 bytes) 000503 1788272897.984515 mapwrite 0x02f86e thru 0x03ec08 (0xf39b bytes) 000504 1788272898.114540 mapwrite 0x02d962 thru 0x03225e (0x48fd bytes) 000508 1788272898.175997 mapread 0x039c45 thru 0x03d05b (0x3417 bytes) 000511 1788272898.178736 read 0x015c3a thru 0x021dfc (0xc1c3 bytes) 000512 1788272898.184923 mapwrite 0x03dd56 thru 0x03ffff (0x22aa bytes) 000514 1788272898.278715 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe5076a2 (stale) Mirror 7: 0xe5076a2 (stale) Mirror 8: 0xe5076a2 (stale) Mirror 9: 0xe5076a2 (stale) Mirror 10: 0xe5076a2 (stale) Mirror 11: 0xe5076a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe5076a2 (stale) Mirror 7: 0xe5076a2 (stale) Mirror 8: 0xe5076a2 (stale) Mirror 9: 0xe5076a2 (stale) Mirror 10: 0xe5076a2 (stale) Mirror 11: 0xe5076a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1788272898.749729 trunc from 0x040000 to 0x01c334 (0x23ccb bytes) 000518 1788272898.794924 write 0x016a37 thru 0x01be1c (0x53e6 bytes) 000521 1788272898.805210 trunc from 0x01c335 to 0x02a3c9 (0xe095 bytes) 000522 1788272898.822618 mapwrite 0x0166f9 thru 0x0206af (0x9fb7 bytes) 000523 1788272898.925924 trunc from 0x02a3ca to 0x007d3f (0x2268a bytes) 000524 1788272898.942413 mapread 0x0029ad thru 0x007d3f (0x5393 bytes) 000525 1788272898.949569 write 0x039149 thru 0x03a222 (0x10da bytes) 000526 1788272898.955021 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a223): Mirror 5: 0x25c920cf (stale) Mirror 7: 0x25c920cf (stale) Mirror 8: 0x25c920cf (stale) Mirror 10: 0x25c920cf (stale) Mirror 11: 0x25c920cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a223): Mirror 5: 0x25c920cf (stale) Mirror 7: 0x25c920cf (stale) Mirror 8: 0x25c920cf (stale) Mirror 10: 0x25c920cf (stale) Mirror 11: 0x25c920cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a223: skipped 000528 1788272899.358784 mapread 0x015bdc thru 0x01d1de (0x7603 bytes) 000532 1788272899.374344 trunc from 0x03a223 to 0x006701 (0x33b21 bytes) 000535 1788272899.397352 mapread 0x00483c thru 0x006701 (0x1ec6 bytes) 000536 1788272899.405605 trunc from 0x006702 to 0x00154b (0x51b6 bytes) 000538 1788272899.434952 MIRROR_RESYNC 6 mirrors 000539 1788272900.294243 trunc from 0x00154c to 0x00ce61 (0xb916 bytes) 000540 1788272900.392362 mapwrite 0x031847 thru 0x0323a4 (0xb5e bytes) 000541 1788272900.486322 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x323a5): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323a5): Mirror 5: 0x4aade2fa (stale) Mirror 7: 0x4aade2fa (stale) Mirror 8: 0x4aade2fa (stale) Mirror 10: 0x4aade2fa (stale) Mirror 11: 0x4aade2fa (stale) CRC-32 checksum value for chunk [0, 0x323a5): Mirror 1: 0x3b0b757e Mirror 12: 0x3b0b757e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323a5: skipped 000543 1788272901.434818 write 0x00f6f3 thru 0x0131ff (0x3b0d bytes) 000546 1788272901.523031 write 0x03d196 thru 0x03ffff (0x2e6a bytes) 000552 1788272901.528028 trunc from 0x040000 to 0x01b84d (0x247b2 bytes) 000553 1788272901.573990 write 0x005263 thru 0x012e5d (0xdbfb bytes) 000554 1788272901.584862 MIRROR_RESYNC 7 mirrors 000556 1788272903.039855 write 0x01077f thru 0x0191bc (0x8a3e bytes) 000558 1788272903.135678 trunc from 0x01b84e to 0x027c26 (0xc3d9 bytes) 000559 1788272903.162122 trunc from 0x027c27 to 0x00bb52 (0x1c0d4 bytes) 000560 1788272903.195299 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xbb53): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbb53): Mirror 5: 0xbb6e7e75 (stale) Mirror 7: 0xbb6e7e75 (stale) Mirror 8: 0xbb6e7e75 (stale) Mirror 10: 0xbb6e7e75 (stale) Mirror 11: 0xbb6e7e75 (stale) Mirror 12: 0xbb6e7e75 (stale) CRC-32 checksum value for chunk [0, 0xbb53): Mirror 1: 0xbb6e7e75 Mirror 13: 0xbb6e7e75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbb53: skipped 000561 1788272903.972545 mapread 0x006a8b thru 0x00bb52 (0x50c8 bytes) 000563 1788272903.989408 mapread 0x004811 thru 0x00aac3 (0x62b3 bytes) 000564 1788272903.990158 mapwrite 0x000789 thru 0x009700 (0x8f78 bytes) 000565 1788272904.227506 MIRROR_RESYNC 8 mirrors 000567 1788272905.741793 read 0x001ba2 thru 0x00924d (0x76ac bytes) 000568 1788272905.753534 mapread 0x0064a0 thru 0x007a29 (0x158a bytes) 000569 1788272905.754641 trunc from 0x00bb53 to 0x01b4d9 (0xf987 bytes) 000570 1788272905.865891 mapwrite 0x039a6d thru 0x03ffff (0x6593 bytes) 000574 1788272905.974503 read 0x024e9f thru 0x029a5b (0x4bbd bytes) 000577 1788272905.986051 trunc from 0x040000 to 0x027967 (0x18698 bytes) 000578 1788272906.010402 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x27968): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27968): Mirror 5: 0x7a576ce8 (stale) Mirror 7: 0x7a576ce8 (stale) Mirror 8: 0x7a576ce8 (stale) Mirror 10: 0x7a576ce8 (stale) Mirror 11: 0x7a576ce8 (stale) Mirror 12: 0x7a576ce8 (stale) Mirror 13: 0x7a576ce8 (stale) CRC-32 checksum value for chunk [0, 0x27968): Mirror 1: 0x7a576ce8 Mirror 14: 0x7a576ce8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27968: skipped 000579 1788272907.425139 read 0x01b5ef thru 0x02364b (0x805d bytes) 000580 1788272907.475098 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x27968): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27968): Mirror 5: 0x7a576ce8 (stale) Mirror 7: 0x7a576ce8 (stale) Mirror 8: 0x7a576ce8 (stale) Mirror 10: 0x7a576ce8 (stale) Mirror 11: 0x7a576ce8 (stale) Mirror 12: 0x7a576ce8 (stale) Mirror 13: 0x7a576ce8 (stale) CRC-32 checksum value for chunk [0, 0x27968): Mirror 1: 0x7a576ce8 Mirror 14: 0x7a576ce8 Mirror 15: 0x7a576ce8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27968: skipped 000581 1788272908.411312 mapread 0x0199bb thru 0x027967 (0xdfad bytes) 000586 1788272908.443191 mapwrite 0x00c25b thru 0x00d570 (0x1316 bytes) 000587 1788272908.629335 mapread 0x00086a thru 0x00336d (0x2b04 bytes) 000588 1788272908.661166 mapread 0x00f6ba thru 0x015a70 (0x63b7 bytes) 000589 1788272908.675309 mapread 0x002473 thru 0x005416 (0x2fa4 bytes) 000590 1788272908.679480 write 0x0396d7 thru 0x03c8a0 (0x31ca bytes) 000591 1788272908.685977 write 0x030a2e thru 0x03dd55 (0xd328 bytes) 000592 1788272908.708761 read 0x011826 thru 0x016b86 (0x5361 bytes) 000593 1788272908.711726 mapwrite 0x032cce thru 0x03ffff (0xd332 bytes) 000594 1788272908.872831 mapwrite 0x03e2ee thru 0x03ffff (0x1d12 bytes) 000595 1788272908.947694 mapread 0x02663c thru 0x0357a5 (0xf16a bytes) 000596 1788272908.981268 mapread 0x03e13f thru 0x03ffff (0x1ec1 bytes) 000597 1788272908.982267 mapwrite 0x0378ca thru 0x03ffff (0x8736 bytes) 000599 1788272909.096941 mapwrite 0x011cfa thru 0x01acac (0x8fb3 bytes) 000601 1788272909.179476 write 0x0389a0 thru 0x03ffff (0x7660 bytes) 000602 1788272909.184045 read 0x00aa9f thru 0x01540b (0xa96d bytes) 000603 1788272909.197664 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x537dd379 (stale) Mirror 7: 0x537dd379 (stale) Mirror 8: 0x537dd379 (stale) Mirror 10: 0x537dd379 (stale) Mirror 11: 0x537dd379 (stale) Mirror 12: 0x537dd379 (stale) Mirror 13: 0x537dd379 (stale) Mirror 14: 0x537dd379 (stale) Mirror 15: 0x537dd379 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x576a3870 Mirror 16: 0x576a3870 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000604 1788272910.244335 write 0x02ee37 thru 0x035f02 (0x70cc bytes) 000607 1788272910.338114 mapwrite 0x00ff0b thru 0x01e941 (0xea37 bytes) 000608 1788272910.411039 mapwrite 0x02801f thru 0x033515 (0xb4f7 bytes) 000609 1788272910.461911 read 0x00ae34 thru 0x018805 (0xd9d2 bytes) 000612 1788272910.463517 mapwrite 0x00be9d thru 0x016842 (0xa9a6 bytes) 000614 1788272910.522260 mapwrite 0x00062c thru 0x0069be (0x6393 bytes) 000615 1788272910.560806 trunc from 0x040000 to 0x01351a (0x2cae5 bytes) 000616 1788272910.584140 mapread 0x003855 thru 0x00fe45 (0xc5f1 bytes) 000621 1788272910.586446 write 0x034288 thru 0x03ffff (0xbd78 bytes) 000622 1788272910.588167 write 0x015339 thru 0x01ef0b (0x9bd3 bytes) 000623 1788272910.597944 trunc from 0x040000 to 0x038e66 (0x7199 bytes) 000624 1788272910.615562 mapwrite 0x03cc4e thru 0x03ffff (0x33b2 bytes) 000625 1788272910.681285 mapwrite 0x024089 thru 0x026eff (0x2e77 bytes) 000626 1788272910.733665 mapread 0x000293 thru 0x00fb49 (0xf8b7 bytes) 000628 1788272910.736042 trunc from 0x040000 to 0x014d6c (0x2b293 bytes) 000631 1788272910.753481 mapread 0x00d826 thru 0x0131f7 (0x59d2 bytes) 000632 1788272910.754990 trunc from 0x014d6d to 0x005243 (0xfb29 bytes) 000634 1788272910.774352 mapwrite 0x0395c5 thru 0x03ffff (0x6a3b bytes) 000635 1788272910.907485 write 0x02f5df thru 0x02f8e8 (0x30a bytes) 000636 1788272910.921157 write 0x038545 thru 0x03e5da (0x6096 bytes) 000637 1788272910.922346 mapread 0x020d03 thru 0x02e498 (0xd796 bytes) 000638 1788272910.975918 trunc from 0x040000 to 0x00e4ff (0x31b00 bytes) 000641 1788272911.020748 mapread 0x000b11 thru 0x001f18 (0x1408 bytes) 000642 1788272911.022550 mapwrite 0x01ad27 thru 0x01f7eb (0x4ac5 bytes) 000644 1788272911.245853 write 0x026b2d thru 0x033273 (0xc747 bytes) 000645 1788272911.247654 trunc from 0x033274 to 0x017580 (0x1bcf3 bytes) 000646 1788272911.303085 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17581): Mirror 5: 0x6252b36e (stale) Mirror 8: 0x6252b36e (stale) Mirror 10: 0x6252b36e (stale) Mirror 11: 0x6252b36e (stale) Mirror 12: 0x6252b36e (stale) Mirror 13: 0x6252b36e (stale) Mirror 14: 0x6252b36e (stale) Mirror 15: 0x6252b36e (stale) Mirror 16: 0x441c4fb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17581): Mirror 5: 0x6252b36e (stale) Mirror 8: 0x6252b36e (stale) Mirror 10: 0x6252b36e (stale) Mirror 11: 0x6252b36e (stale) Mirror 12: 0x6252b36e (stale) Mirror 13: 0x6252b36e (stale) Mirror 14: 0x6252b36e (stale) Mirror 15: 0x6252b36e (stale) Mirror 16: 0x441c4fb4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17581: skipped 000647 1788272911.866039 mapwrite 0x000066 thru 0x00bbe6 (0xbb81 bytes) 000649 1788272911.972618 mapwrite 0x016f55 thru 0x01d1ad (0x6259 bytes) 000652 1788272912.054959 write 0x0138f7 thru 0x01f0ff (0xb809 bytes) 000655 1788272912.064285 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f100): Mirror 5: 0xb03fda18 (stale) Mirror 8: 0xb03fda18 (stale) Mirror 10: 0xb03fda18 (stale) Mirror 11: 0xb03fda18 (stale) Mirror 12: 0xb03fda18 (stale) Mirror 14: 0xb03fda18 (stale) Mirror 15: 0xb03fda18 (stale) Mirror 16: 0xa1e89272 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f100): Mirror 5: 0xb03fda18 (stale) Mirror 8: 0xb03fda18 (stale) Mirror 10: 0xb03fda18 (stale) Mirror 11: 0xb03fda18 (stale) Mirror 12: 0xb03fda18 (stale) Mirror 14: 0xb03fda18 (stale) Mirror 15: 0xb03fda18 (stale) Mirror 16: 0xa1e89272 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f100: skipped 000657 1788272912.544568 MIRROR_RESYNC 9 mirrors 000660 1788272914.226099 mapwrite 0x0202d9 thru 0x023963 (0x368b bytes) 000662 1788272914.360437 read 0x01e12d thru 0x020e83 (0x2d57 bytes) 000664 1788272914.362407 write 0x018e91 thru 0x021d4e (0x8ebe bytes) 000667 1788272914.369798 mapwrite 0x014f87 thru 0x01c429 (0x74a3 bytes) 000669 1788272914.439652 write 0x0285a4 thru 0x0292a5 (0xd02 bytes) 000670 1788272914.445508 MIRROR_RESYNC 9 mirrors 000672 1788272917.518104 trunc from 0x0292a6 to 0x030b57 (0x78b2 bytes) 000676 1788272917.643784 write 0x00cc11 thru 0x0122d2 (0x56c2 bytes) 000677 1788272917.657997 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30b58): Mirror 5: 0xaaa73219 (stale) Mirror 10: 0xaaa73219 (stale) Mirror 11: 0xaaa73219 (stale) Mirror 12: 0xaaa73219 (stale) Mirror 14: 0xaaa73219 (stale) Mirror 15: 0xaaa73219 (stale) Mirror 16: 0xaaa73219 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b58): Mirror 5: 0xaaa73219 (stale) Mirror 10: 0xaaa73219 (stale) Mirror 11: 0xaaa73219 (stale) Mirror 12: 0xaaa73219 (stale) Mirror 14: 0xaaa73219 (stale) Mirror 15: 0xaaa73219 (stale) Mirror 16: 0xaaa73219 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b58: skipped 000679 1788272918.219021 mapwrite 0x03bd2c thru 0x03fd3f (0x4014 bytes) 000680 1788272918.356106 write 0x02ebbc thru 0x03a51c (0xb961 bytes) 000681 1788272918.363633 read 0x032d48 thru 0x03fd3f (0xcff8 bytes) 000683 1788272918.364729 read 0x013dff thru 0x01d5dc (0x97de bytes) 000685 1788272918.378381 read 0x032a78 thru 0x037fbc (0x5545 bytes) 000686 1788272918.379246 read 0x018966 thru 0x02256f (0x9c0a bytes) 000687 1788272918.382205 mapwrite 0x0388e6 thru 0x03dbaa (0x52c5 bytes) 000688 1788272918.453128 trunc from 0x03fd40 to 0x029428 (0x16917 bytes) 000690 1788272918.484787 mapread 0x01018a thru 0x01be88 (0xbcff bytes) 000691 1788272918.490262 read 0x01faa1 thru 0x021a16 (0x1f76 bytes) 000694 1788272918.497153 write 0x015d27 thru 0x016850 (0xb2a bytes) 000696 1788272918.500346 trunc from 0x029429 to 0x01b946 (0xdae2 bytes) 000697 1788272918.532147 mapread 0x009341 thru 0x01825f (0xef1f bytes) 000698 1788272918.535467 mapwrite 0x00adad thru 0x00fb05 (0x4d59 bytes) 000701 1788272918.709539 write 0x019849 thru 0x02897f (0xf137 bytes) 000704 1788272918.718958 trunc from 0x028980 to 0x0061c7 (0x227b8 bytes) 000705 1788272918.749901 trunc from 0x0061c8 to 0x0001a1 (0x6026 bytes) 000706 1788272918.793862 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a2): Mirror 5: 0x8de3fbae (stale) Mirror 10: 0x8de3fbae (stale) Mirror 11: 0x8de3fbae (stale) Mirror 12: 0x8de3fbae (stale) Mirror 14: 0x8de3fbae (stale) Mirror 15: 0x8de3fbae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a2): Mirror 5: 0x8de3fbae (stale) Mirror 10: 0x8de3fbae (stale) Mirror 11: 0x8de3fbae (stale) Mirror 12: 0x8de3fbae (stale) Mirror 14: 0x8de3fbae (stale) Mirror 15: 0x8de3fbae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a2: skipped 000708 1788272919.483915 mapwrite 0x0162db thru 0x017080 (0xda6 bytes) 000709 1788272919.646550 write 0x03badf thru 0x03ffff (0x4521 bytes) 000710 1788272919.648078 mapwrite 0x00306c thru 0x0056e5 (0x267a bytes) 000712 1788272919.819797 read 0x02d927 thru 0x03a191 (0xc86b bytes) 000713 1788272919.860387 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x757e43f8 (stale) Mirror 10: 0x757e43f8 (stale) Mirror 11: 0x757e43f8 (stale) Mirror 12: 0x757e43f8 (stale) Mirror 14: 0x757e43f8 (stale) Mirror 15: 0x757e43f8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd2b21bd4 Mirror 16: 0xd2b21bd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000715 1788272921.801016 trunc from 0x040000 to 0x01f945 (0x206ba bytes) 000716 1788272921.964576 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1f946): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f946): Mirror 5: 0xfdbe7204 (stale) Mirror 10: 0xfdbe7204 (stale) Mirror 11: 0xfdbe7204 (stale) Mirror 12: 0xfdbe7204 (stale) Mirror 14: 0xfdbe7204 (stale) Mirror 15: 0xfdbe7204 (stale) Mirror 16: 0x1081f28a (stale) CRC-32 checksum value for chunk [0, 0x1f946): Mirror 1: 0x1081f28a Mirror 17: 0x1081f28a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f946: skipped 000717 1788272923.416359 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1f946): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f946): Mirror 5: 0xfdbe7204 (stale) Mirror 10: 0xfdbe7204 (stale) Mirror 11: 0xfdbe7204 (stale) Mirror 12: 0xfdbe7204 (stale) Mirror 14: 0xfdbe7204 (stale) Mirror 15: 0xfdbe7204 (stale) Mirror 16: 0x1081f28a (stale) CRC-32 checksum value for chunk [0, 0x1f946): Mirror 1: 0x1081f28a Mirror 17: 0x1081f28a Mirror 18: 0x1081f28a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f946: skipped 000718 1788272924.567292 read 0x002c38 thru 0x00c928 (0x9cf1 bytes) 000719 1788272924.579617 MIRROR_RESYNC 10 mirrors 000721 1788272926.882973 mapwrite 0x008e24 thru 0x013782 (0xa95f bytes) 000722 1788272927.013843 mapread 0x00491d thru 0x010f03 (0xc5e7 bytes) 000723 1788272927.014902 mapread 0x003172 thru 0x010ccb (0xdb5a bytes) 000724 1788272927.016676 write 0x03c30a thru 0x03dbc8 (0x18bf bytes) 000725 1788272927.021266 MIRROR_RESYNC 10 mirrors 000727 1788272930.860175 MIRROR_RESYNC 10 mirrors 000728 1788272931.664907 write 0x038ea9 thru 0x03ffff (0x7157 bytes) 000729 1788272931.786350 write 0x037ef6 thru 0x03b11b (0x3226 bytes) 000730 1788272931.795243 trunc from 0x040000 to 0x01389a (0x2c765 bytes) 000733 1788272931.825100 mapread 0x00d8cb thru 0x0108ff (0x3035 bytes) 000736 1788272931.845684 read 0x0078ea thru 0x00d6d2 (0x5de9 bytes) 000737 1788272931.854235 mapwrite 0x00b38a thru 0x0190ea (0xdd61 bytes) 000738 1788272931.956698 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x190eb): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x190eb): Mirror 5: 0xf995c5ac (stale) Mirror 10: 0xf995c5ac (stale) Mirror 11: 0xf995c5ac (stale) Mirror 12: 0xf995c5ac (stale) Mirror 14: 0xf995c5ac (stale) Mirror 15: 0xf995c5ac (stale) Mirror 16: 0xf995c5ac (stale) Mirror 17: 0xf995c5ac (stale) Mirror 18: 0xf995c5ac (stale) CRC-32 checksum value for chunk [0, 0x190eb): Mirror 1: 0xb994ebf5 Mirror 19: 0xb994ebf5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x190eb: skipped 000740 1788272932.861563 write 0x027a88 thru 0x02cee8 (0x5461 bytes) truncating to largest ever: 0x3fefd 000745 1788272932.955610 trunc from 0x02cee9 to 0x03fefc (0x13014 bytes) 000746 1788272932.986954 mapread 0x0374ae thru 0x03fefc (0x8a4f bytes) 000747 1788272933.010308 write 0x000473 thru 0x002d5b (0x28e9 bytes) 000751 1788272933.020660 read 0x03a0e3 thru 0x03fefc (0x5e1a bytes) 000753 1788272933.022008 mapread 0x00196a thru 0x00966e (0x7d05 bytes) 000754 1788272933.030686 write 0x007b5d thru 0x0153d0 (0xd874 bytes) 000756 1788272933.036614 mapwrite 0x024b55 thru 0x0347e6 (0xfc92 bytes) 000757 1788272933.125863 write 0x01b6dc thru 0x01b9f7 (0x31c bytes) 000758 1788272933.135496 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3fefd): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fefd): Mirror 5: 0x24b7c0c0 (stale) Mirror 10: 0x24b7c0c0 (stale) Mirror 11: 0x24b7c0c0 (stale) Mirror 12: 0x24b7c0c0 (stale) Mirror 14: 0x24b7c0c0 (stale) Mirror 15: 0x24b7c0c0 (stale) Mirror 16: 0x24b7c0c0 (stale) Mirror 17: 0x24b7c0c0 (stale) Mirror 18: 0x24b7c0c0 (stale) Mirror 19: 0x6ad17157 (stale) CRC-32 checksum value for chunk [0, 0x3fefd): Mirror 1: 0xbc2619d6 Mirror 20: 0xbc2619d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fefd: skipped 000759 1788272934.406257 MIRROR_RESYNC 12 mirrors 000760 1788272937.867279 trunc from 0x03fefd to 0x01a70e (0x257ee bytes) 000762 1788272937.961160 trunc from 0x01a70f to 0x002ec4 (0x1784a bytes) 000763 1788272937.980785 trunc from 0x002ec5 to 0x023ae6 (0x20c22 bytes) 000767 1788272938.004902 write 0x03d738 thru 0x03ffff (0x28c8 bytes) 000769 1788272938.008920 mapwrite 0x02db96 thru 0x034179 (0x65e4 bytes) 000772 1788272938.089112 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xadc10d6a (stale) Mirror 10: 0xadc10d6a (stale) Mirror 11: 0xadc10d6a (stale) Mirror 12: 0xadc10d6a (stale) Mirror 14: 0xadc10d6a (stale) Mirror 15: 0xadc10d6a (stale) Mirror 16: 0xadc10d6a (stale) Mirror 17: 0xadc10d6a (stale) Mirror 18: 0xadc10d6a (stale) Mirror 19: 0xadc10d6a (stale) Mirror 20: 0xadc10d6a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x65fdaded Mirror 21: 0x65fdaded lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000773 1788272939.126149 mapread 0x010967 thru 0x015164 (0x47fe bytes) 000774 1788272939.157840 write 0x02dab8 thru 0x032dfe (0x5347 bytes) 000775 1788272939.240981 read 0x008bdb thru 0x00d6fb (0x4b21 bytes) 000776 1788272939.248243 mapwrite 0x02131b thru 0x028154 (0x6e3a bytes) 000777 1788272939.312007 write 0x021d91 thru 0x026981 (0x4bf1 bytes) 000778 1788272939.313253 write 0x00faed thru 0x019095 (0x95a9 bytes) 000780 1788272939.322229 trunc from 0x040000 to 0x015354 (0x2acab bytes) 000781 1788272939.363640 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15355): Mirror 5: 0xf2e572b3 (stale) Mirror 10: 0xf2e572b3 (stale) Mirror 11: 0xf2e572b3 (stale) Mirror 12: 0xf2e572b3 (stale) Mirror 14: 0xf2e572b3 (stale) Mirror 15: 0xf2e572b3 (stale) Mirror 16: 0xf2e572b3 (stale) Mirror 17: 0xf2e572b3 (stale) Mirror 18: 0xf2e572b3 (stale) Mirror 20: 0xf2e572b3 (stale) Mirror 21: 0x568261c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15355): Mirror 5: 0xf2e572b3 (stale) Mirror 10: 0xf2e572b3 (stale) Mirror 11: 0xf2e572b3 (stale) Mirror 12: 0xf2e572b3 (stale) Mirror 14: 0xf2e572b3 (stale) Mirror 15: 0xf2e572b3 (stale) Mirror 16: 0xf2e572b3 (stale) Mirror 17: 0xf2e572b3 (stale) Mirror 18: 0xf2e572b3 (stale) Mirror 20: 0xf2e572b3 (stale) Mirror 21: 0x568261c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15355: skipped 000782 1788272940.050785 mapwrite 0x002f27 thru 0x005062 (0x213c bytes) 000783 1788272940.150299 mapread 0x010575 thru 0x015354 (0x4de0 bytes) 000785 1788272940.171996 read 0x00ca53 thru 0x015354 (0x8902 bytes) 000788 1788272940.181605 mapwrite 0x0242d8 thru 0x03055d (0xc286 bytes) 000789 1788272940.347072 read 0x00e518 thru 0x01a172 (0xbc5b bytes) 000790 1788272940.358892 read 0x004724 thru 0x00567c (0xf59 bytes) 000791 1788272940.359567 read 0x009ca2 thru 0x017602 (0xd961 bytes) 000793 1788272940.366981 write 0x00e3c4 thru 0x01bdd1 (0xda0e bytes) 000794 1788272940.369316 mapread 0x0268ef thru 0x03055d (0x9c6f bytes) 000796 1788272940.390514 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3055e): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3055e): Mirror 5: 0xf9da5218 (stale) Mirror 10: 0xf9da5218 (stale) Mirror 11: 0xf9da5218 (stale) Mirror 12: 0xf9da5218 (stale) Mirror 14: 0xf9da5218 (stale) Mirror 15: 0xf9da5218 (stale) Mirror 16: 0xf9da5218 (stale) Mirror 17: 0xf9da5218 (stale) Mirror 18: 0xf9da5218 (stale) Mirror 20: 0xf9da5218 (stale) Mirror 21: 0x9c010af5 (stale) CRC-32 checksum value for chunk [0, 0x3055e): Mirror 1: 0xd42bfd5d Mirror 22: 0xd42bfd5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3055e: skipped 000800 1788272942.207368 mapread 0x002837 thru 0x00a985 (0x814f bytes) 000802 1788272942.245339 write 0x02a517 thru 0x033848 (0x9332 bytes) 000803 1788272942.343851 read 0x00e769 thru 0x01ca58 (0xe2f0 bytes) 000805 1788272942.371154 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x33849): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33849): Mirror 5: 0xf6f1a29 (stale) Mirror 10: 0xf6f1a29 (stale) Mirror 11: 0xf6f1a29 (stale) Mirror 12: 0xf6f1a29 (stale) Mirror 14: 0xf6f1a29 (stale) Mirror 15: 0xf6f1a29 (stale) Mirror 16: 0xf6f1a29 (stale) Mirror 17: 0xf6f1a29 (stale) Mirror 18: 0xf6f1a29 (stale) Mirror 20: 0xf6f1a29 (stale) Mirror 21: 0x6778be2b (stale) Mirror 22: 0xe6859a25 (stale) CRC-32 checksum value for chunk [0, 0x33849): Mirror 1: 0xb632aafc Mirror 23: 0xb632aafc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33849: skipped 000807 1788272943.850690 trunc from 0x033849 to 0x02b073 (0x87d5 bytes) 000810 1788272943.954859 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2b074): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b074): Mirror 5: 0xc28b8d (stale) Mirror 10: 0xc28b8d (stale) Mirror 11: 0xc28b8d (stale) Mirror 12: 0xc28b8d (stale) Mirror 14: 0xc28b8d (stale) Mirror 15: 0xc28b8d (stale) Mirror 16: 0xc28b8d (stale) Mirror 17: 0xc28b8d (stale) Mirror 18: 0xc28b8d (stale) Mirror 20: 0xc28b8d (stale) Mirror 21: 0x49254864 (stale) Mirror 22: 0x4fc49341 (stale) Mirror 23: 0xe0646cc (stale) CRC-32 checksum value for chunk [0, 0x2b074): Mirror 1: 0xe0646cc Mirror 24: 0xe0646cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b074: skipped 000811 1788272945.073697 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2b074): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b074): Mirror 5: 0xc28b8d (stale) Mirror 10: 0xc28b8d (stale) Mirror 11: 0xc28b8d (stale) Mirror 12: 0xc28b8d (stale) Mirror 14: 0xc28b8d (stale) Mirror 15: 0xc28b8d (stale) Mirror 16: 0xc28b8d (stale) Mirror 17: 0xc28b8d (stale) Mirror 18: 0xc28b8d (stale) Mirror 20: 0xc28b8d (stale) Mirror 21: 0x49254864 (stale) Mirror 22: 0x4fc49341 (stale) Mirror 23: 0xe0646cc (stale) CRC-32 checksum value for chunk [0, 0x2b074): Mirror 1: 0xe0646cc Mirror 24: 0xe0646cc Mirror 25: 0xe0646cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b074: skipped 000813 1788272946.053334 read 0x001c59 thru 0x00f33d (0xd6e5 bytes) 000816 1788272946.064836 write 0x022516 thru 0x027c52 (0x573d bytes) 000821 1788272946.159148 mapread 0x012225 thru 0x015db0 (0x3b8c bytes) 000823 1788272946.175218 write 0x0199ca thru 0x01b925 (0x1f5c bytes) 000824 1788272946.176160 mapwrite 0x011691 thru 0x020afa (0xf46a bytes) 000826 1788272946.240899 write 0x0032fe thru 0x00af68 (0x7c6b bytes) 000827 1788272946.250441 mapwrite 0x0391b2 thru 0x03ffff (0x6e4e bytes) 000828 1788272946.337673 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xadc10d6a (stale) Mirror 10: 0xadc10d6a (stale) Mirror 11: 0xadc10d6a (stale) Mirror 12: 0xadc10d6a (stale) Mirror 14: 0xadc10d6a (stale) Mirror 15: 0xadc10d6a (stale) Mirror 16: 0xadc10d6a (stale) Mirror 17: 0xadc10d6a (stale) Mirror 18: 0xadc10d6a (stale) Mirror 20: 0xadc10d6a (stale) Mirror 22: 0x7b019c92 (stale) Mirror 23: 0xc03c6dc4 (stale) Mirror 24: 0x1896fa4a (stale) Mirror 25: 0x1896fa4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xadc10d6a (stale) Mirror 10: 0xadc10d6a (stale) Mirror 11: 0xadc10d6a (stale) Mirror 12: 0xadc10d6a (stale) Mirror 14: 0xadc10d6a (stale) Mirror 15: 0xadc10d6a (stale) Mirror 16: 0xadc10d6a (stale) Mirror 17: 0xadc10d6a (stale) Mirror 18: 0xadc10d6a (stale) Mirror 20: 0xadc10d6a (stale) Mirror 22: 0x7b019c92 (stale) Mirror 23: 0xc03c6dc4 (stale) Mirror 24: 0x1896fa4a (stale) Mirror 25: 0x1896fa4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000830 1788272946.928414 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xadc10d6a (stale) Mirror 10: 0xadc10d6a (stale) Mirror 11: 0xadc10d6a (stale) Mirror 12: 0xadc10d6a (stale) Mirror 14: 0xadc10d6a (stale) Mirror 15: 0xadc10d6a (stale) Mirror 16: 0xadc10d6a (stale) Mirror 17: 0xadc10d6a (stale) Mirror 18: 0xadc10d6a (stale) Mirror 20: 0xadc10d6a (stale) Mirror 22: 0x7b019c92 (stale) Mirror 23: 0xc03c6dc4 (stale) Mirror 24: 0x1896fa4a (stale) Mirror 25: 0x1896fa4a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2946909 Mirror 26: 0x2946909 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000831 1788272948.250336 mapread 0x03fd46 thru 0x03ffff (0x2ba bytes) 000833 1788272948.257159 trunc from 0x040000 to 0x0283df (0x17c20 bytes) 000835 1788272948.386263 trunc from 0x0283e0 to 0x03b190 (0x12db1 bytes) 000836 1788272948.424447 read 0x00c749 thru 0x0141c9 (0x7a81 bytes) 000837 1788272948.433933 mapwrite 0x031c89 thru 0x038e64 (0x71dc bytes) 000838 1788272948.545278 read 0x003af2 thru 0x00d9d4 (0x9ee3 bytes) 000839 1788272948.559945 mapread 0x028f75 thru 0x029055 (0xe1 bytes) 000840 1788272948.561021 read 0x029246 thru 0x02bc89 (0x2a44 bytes) 000841 1788272948.562293 write 0x01e5bc thru 0x024891 (0x62d6 bytes) 000843 1788272948.566849 trunc from 0x03b191 to 0x01258f (0x28c01 bytes) 000847 1788272948.607453 write 0x00f4e8 thru 0x01a7c0 (0xb2d9 bytes) 000848 1788272948.609553 mapwrite 0x0117c8 thru 0x015697 (0x3ed0 bytes) 000853 1788272948.652339 trunc from 0x01a7c1 to 0x00beef (0xe8d1 bytes) 000856 1788272948.679843 mapread 0x00b369 thru 0x00b9f2 (0x68a bytes) 000859 1788272948.682562 mapread 0x009774 thru 0x00beef (0x277c bytes) 000861 1788272948.685142 read 0x003fee thru 0x00beef (0x7f02 bytes) 000863 1788272948.691733 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbef0): Mirror 10: 0xb1e5763a (stale) Mirror 11: 0xb1e5763a (stale) Mirror 12: 0xb1e5763a (stale) Mirror 14: 0xb1e5763a (stale) Mirror 15: 0xb1e5763a (stale) Mirror 16: 0xb1e5763a (stale) Mirror 17: 0xb1e5763a (stale) Mirror 18: 0xb1e5763a (stale) Mirror 20: 0xb1e5763a (stale) Mirror 22: 0x98d962dd (stale) Mirror 23: 0x98d962dd (stale) Mirror 24: 0x98d962dd (stale) Mirror 25: 0x98d962dd (stale) Mirror 26: 0xcbc4b90d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbef0): Mirror 10: 0xb1e5763a (stale) Mirror 11: 0xb1e5763a (stale) Mirror 12: 0xb1e5763a (stale) Mirror 14: 0xb1e5763a (stale) Mirror 15: 0xb1e5763a (stale) Mirror 16: 0xb1e5763a (stale) Mirror 17: 0xb1e5763a (stale) Mirror 18: 0xb1e5763a (stale) Mirror 20: 0xb1e5763a (stale) Mirror 22: 0x98d962dd (stale) Mirror 23: 0x98d962dd (stale) Mirror 24: 0x98d962dd (stale) Mirror 25: 0x98d962dd (stale) Mirror 26: 0xcbc4b90d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbef0: skipped 000864 1788272949.374045 read 0x002b8b thru 0x00646a (0x38e0 bytes) 000866 1788272949.386198 read 0x0089f8 thru 0x00af60 (0x2569 bytes) 000867 1788272949.397082 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbef0): Mirror 11: 0xb1e5763a (stale) Mirror 12: 0xb1e5763a (stale) Mirror 14: 0xb1e5763a (stale) Mirror 15: 0xb1e5763a (stale) Mirror 16: 0xb1e5763a (stale) Mirror 17: 0xb1e5763a (stale) Mirror 18: 0xb1e5763a (stale) Mirror 20: 0xb1e5763a (stale) Mirror 22: 0x98d962dd (stale) Mirror 23: 0x98d962dd (stale) Mirror 24: 0x98d962dd (stale) Mirror 25: 0x98d962dd (stale) Mirror 26: 0xcbc4b90d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbef0): Mirror 11: 0xb1e5763a (stale) Mirror 12: 0xb1e5763a (stale) Mirror 14: 0xb1e5763a (stale) Mirror 15: 0xb1e5763a (stale) Mirror 16: 0xb1e5763a (stale) Mirror 17: 0xb1e5763a (stale) Mirror 18: 0xb1e5763a (stale) Mirror 20: 0xb1e5763a (stale) Mirror 22: 0x98d962dd (stale) Mirror 23: 0x98d962dd (stale) Mirror 24: 0x98d962dd (stale) Mirror 25: 0x98d962dd (stale) Mirror 26: 0xcbc4b90d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbef0: skipped 000868 1788272950.033906 write 0x03aead thru 0x03ffff (0x5153 bytes) 000869 1788272950.041884 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xadc10d6a (stale) Mirror 14: 0xadc10d6a (stale) Mirror 15: 0xadc10d6a (stale) Mirror 16: 0xadc10d6a (stale) Mirror 17: 0xadc10d6a (stale) Mirror 18: 0xadc10d6a (stale) Mirror 20: 0xadc10d6a (stale) Mirror 22: 0x7b019c92 (stale) Mirror 23: 0xc03c6dc4 (stale) Mirror 24: 0x1896fa4a (stale) Mirror 25: 0x1896fa4a (stale) Mirror 26: 0x2946909 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xadc10d6a (stale) Mirror 14: 0xadc10d6a (stale) Mirror 15: 0xadc10d6a (stale) Mirror 16: 0xadc10d6a (stale) Mirror 17: 0xadc10d6a (stale) Mirror 18: 0xadc10d6a (stale) Mirror 20: 0xadc10d6a (stale) Mirror 22: 0x7b019c92 (stale) Mirror 23: 0xc03c6dc4 (stale) Mirror 24: 0x1896fa4a (stale) Mirror 25: 0x1896fa4a (stale) Mirror 26: 0x2946909 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000872 1788272950.768653 trunc from 0x040000 to 0x010f4b (0x2f0b4 bytes) 000874 1788272950.806856 mapread 0x007b62 thru 0x010e7a (0x9319 bytes) 000876 1788272950.822683 mapwrite 0x00dbf5 thru 0x01802b (0xa437 bytes) 000878 1788272950.946134 trunc from 0x01802c to 0x03a01d (0x21ff2 bytes) 000879 1788272950.961701 read 0x037d52 thru 0x03a01d (0x22cc bytes) 000880 1788272950.969425 read 0x022918 thru 0x02a2e0 (0x79c9 bytes) 000881 1788272950.978715 mapread 0x029b6d thru 0x0364ef (0xc983 bytes) 000882 1788272950.980206 mapread 0x01b8ee thru 0x026ed9 (0xb5ec bytes) 000884 1788272950.982307 read 0x01fe0c thru 0x021730 (0x1925 bytes) 000887 1788272950.984508 read 0x03061a thru 0x031142 (0xb29 bytes) 000889 1788272950.985881 read 0x01c0f4 thru 0x01cde6 (0xcf3 bytes) 000890 1788272950.986861 mapread 0x004408 thru 0x0055ce (0x11c7 bytes) 000891 1788272950.988406 mapwrite 0x013a4b thru 0x013db1 (0x367 bytes) 000892 1788272951.017814 write 0x002fe4 thru 0x003ba9 (0xbc6 bytes) 000893 1788272951.019079 write 0x01b9bd thru 0x022a89 (0x70cd bytes) 000895 1788272951.020833 read 0x016da0 thru 0x0207e2 (0x9a43 bytes) 000896 1788272951.021748 mapread 0x020fb7 thru 0x028d29 (0x7d73 bytes) 000897 1788272951.022863 mapwrite 0x009645 thru 0x00fe53 (0x680f bytes) 000903 1788272951.058455 trunc from 0x03a01e to 0x021df4 (0x18229 bytes) 000904 1788272951.069686 mapread 0x004de9 thru 0x0138d6 (0xeaee bytes) 000907 1788272951.072084 read 0x01a24f thru 0x021df4 (0x7ba6 bytes) 000908 1788272951.073060 mapread 0x019f72 thru 0x021df4 (0x7e83 bytes) 000909 1788272951.074310 read 0x01ec96 thru 0x021df4 (0x315f bytes) 000910 1788272951.075075 trunc from 0x021df5 to 0x01a577 (0x787d bytes) 000911 1788272951.087403 write 0x01b404 thru 0x023a40 (0x863d bytes) 000915 1788272951.090646 trunc from 0x023a41 to 0x01aefb (0x8b45 bytes) 000916 1788272951.105990 read 0x0020a3 thru 0x00a5ad (0x850b bytes) 000920 1788272951.108488 mapread 0x001e84 thru 0x00af01 (0x907e bytes) 000921 1788272951.109539 read 0x019410 thru 0x01a6e6 (0x12d7 bytes) 000922 1788272951.110453 mapread 0x006603 thru 0x0096a9 (0x30a7 bytes) 000923 1788272951.120010 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1aefc): Mirror 12: 0xd5d3e5d5 (stale) Mirror 14: 0xd5d3e5d5 (stale) Mirror 15: 0xd5d3e5d5 (stale) Mirror 16: 0xd5d3e5d5 (stale) Mirror 17: 0xd5d3e5d5 (stale) Mirror 18: 0xd5d3e5d5 (stale) Mirror 20: 0xd5d3e5d5 (stale) Mirror 22: 0xd3fc8f31 (stale) Mirror 23: 0xd3fc8f31 (stale) Mirror 25: 0xd3fc8f31 (stale) Mirror 26: 0x13fa5f47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aefc): Mirror 12: 0xd5d3e5d5 (stale) Mirror 14: 0xd5d3e5d5 (stale) Mirror 15: 0xd5d3e5d5 (stale) Mirror 16: 0xd5d3e5d5 (stale) Mirror 17: 0xd5d3e5d5 (stale) Mirror 18: 0xd5d3e5d5 (stale) Mirror 20: 0xd5d3e5d5 (stale) Mirror 22: 0xd3fc8f31 (stale) Mirror 23: 0xd3fc8f31 (stale) Mirror 25: 0xd3fc8f31 (stale) Mirror 26: 0x13fa5f47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aefc: skipped 000925 1788272951.770455 trunc from 0x01aefc to 0x02085c (0x5961 bytes) 000926 1788272951.785615 mapread 0x002427 thru 0x0103e6 (0xdfc0 bytes) 000931 1788272951.809895 trunc from 0x02085d to 0x010e1a (0xfa42 bytes) 000932 1788272951.823910 read 0x005372 thru 0x00d439 (0x80c8 bytes) 000933 1788272951.824740 read 0x00536a thru 0x00a42e (0x50c5 bytes) 000934 1788272951.830548 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x10e1b): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10e1b): Mirror 12: 0x9cf8aa43 (stale) Mirror 14: 0x9cf8aa43 (stale) Mirror 15: 0x9cf8aa43 (stale) Mirror 16: 0x9cf8aa43 (stale) Mirror 17: 0x9cf8aa43 (stale) Mirror 18: 0x9cf8aa43 (stale) Mirror 20: 0x9cf8aa43 (stale) Mirror 22: 0xbc506917 (stale) Mirror 23: 0xbc506917 (stale) Mirror 25: 0xbc506917 (stale) Mirror 26: 0x5ca33dac (stale) CRC-32 checksum value for chunk [0, 0x10e1b): Mirror 1: 0xc3e9502b Mirror 27: 0xc3e9502b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10e1b: skipped 000935 1788272952.926635 write 0x03a2b9 thru 0x03ffff (0x5d47 bytes) 000937 1788272953.008775 read 0x0022a2 thru 0x004960 (0x26bf bytes) 000941 1788272953.017441 mapread 0x00ac00 thru 0x00f40a (0x480b bytes) 000943 1788272953.034088 write 0x03d89b thru 0x03ffff (0x2765 bytes) 000948 1788272953.037810 mapread 0x0372bf thru 0x03ffff (0x8d41 bytes) 000950 1788272953.045372 trunc from 0x040000 to 0x03ca0f (0x35f0 bytes) 000951 1788272953.068660 read 0x01565f thru 0x01ba24 (0x63c6 bytes) 000952 1788272953.077576 mapread 0x00ac84 thru 0x016a32 (0xbdaf bytes) 000954 1788272953.080079 mapwrite 0x017271 thru 0x01c9da (0x576a bytes) 000955 1788272953.132285 write 0x03b188 thru 0x03ffff (0x4e78 bytes) 000962 1788272953.139196 trunc from 0x040000 to 0x01cca0 (0x2335f bytes) 000963 1788272953.160666 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cca1): Mirror 12: 0xaa02da7e (stale) Mirror 14: 0xaa02da7e (stale) Mirror 16: 0xaa02da7e (stale) Mirror 17: 0xaa02da7e (stale) Mirror 18: 0xaa02da7e (stale) Mirror 20: 0xaa02da7e (stale) Mirror 22: 0x31318875 (stale) Mirror 23: 0x31318875 (stale) Mirror 25: 0x31318875 (stale) Mirror 26: 0x878659ae (stale) Mirror 27: 0x67854ea1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cca1): Mirror 12: 0xaa02da7e (stale) Mirror 14: 0xaa02da7e (stale) Mirror 16: 0xaa02da7e (stale) Mirror 17: 0xaa02da7e (stale) Mirror 18: 0xaa02da7e (stale) Mirror 20: 0xaa02da7e (stale) Mirror 22: 0x31318875 (stale) Mirror 23: 0x31318875 (stale) Mirror 25: 0x31318875 (stale) Mirror 26: 0x878659ae (stale) Mirror 27: 0x67854ea1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cca1: skipped 000964 1788272953.690281 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1cca1): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cca1): Mirror 12: 0xaa02da7e (stale) Mirror 14: 0xaa02da7e (stale) Mirror 16: 0xaa02da7e (stale) Mirror 17: 0xaa02da7e (stale) Mirror 18: 0xaa02da7e (stale) Mirror 20: 0xaa02da7e (stale) Mirror 22: 0x31318875 (stale) Mirror 23: 0x31318875 (stale) Mirror 25: 0x31318875 (stale) Mirror 26: 0x878659ae (stale) Mirror 27: 0x67854ea1 (stale) CRC-32 checksum value for chunk [0, 0x1cca1): Mirror 1: 0x379edfbc Mirror 28: 0x379edfbc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cca1: skipped 000965 1788272954.556212 MIRROR_RESYNC 13 mirrors 000966 1788272956.022415 mapread 0x011810 thru 0x0119b3 (0x1a4 bytes) 000968 1788272956.032295 MIRROR_RESYNC 13 mirrors 000969 1788272956.752476 MIRROR_RESYNC 13 mirrors 000970 1788272957.355765 trunc from 0x01cca1 to 0x023443 (0x67a3 bytes) 000973 1788272957.450811 read 0x017b41 thru 0x023443 (0xb903 bytes) 000976 1788272957.460542 read 0x004c0d thru 0x0091bc (0x45b0 bytes) 000978 1788272957.474435 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x23444): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23444): Mirror 12: 0x12afcaa7 (stale) Mirror 14: 0x12afcaa7 (stale) Mirror 16: 0x12afcaa7 (stale) Mirror 17: 0x12afcaa7 (stale) Mirror 18: 0x12afcaa7 (stale) Mirror 20: 0x12afcaa7 (stale) Mirror 22: 0x12afcaa7 (stale) Mirror 23: 0x12afcaa7 (stale) Mirror 25: 0x12afcaa7 (stale) Mirror 26: 0x12afcaa7 (stale) Mirror 27: 0x12afcaa7 (stale) Mirror 28: 0x12afcaa7 (stale) CRC-32 checksum value for chunk [0, 0x23444): Mirror 1: 0x12afcaa7 Mirror 29: 0x12afcaa7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23444: skipped 000979 1788272958.498452 write 0x02b9c2 thru 0x02dc6e (0x22ad bytes) 000980 1788272958.577347 write 0x00ed83 thru 0x014a82 (0x5d00 bytes) 000981 1788272958.587440 mapread 0x01659b thru 0x019b45 (0x35ab bytes) 000982 1788272958.607145 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2dc6f): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dc6f): Mirror 12: 0x468477df (stale) Mirror 14: 0x468477df (stale) Mirror 16: 0x468477df (stale) Mirror 17: 0x468477df (stale) Mirror 18: 0x468477df (stale) Mirror 20: 0x468477df (stale) Mirror 22: 0x468477df (stale) Mirror 23: 0x468477df (stale) Mirror 25: 0x468477df (stale) Mirror 26: 0x468477df (stale) Mirror 27: 0x468477df (stale) Mirror 28: 0x468477df (stale) Mirror 29: 0x468477df (stale) CRC-32 checksum value for chunk [0, 0x2dc6f): Mirror 1: 0x2a71f77b Mirror 30: 0x2a71f77b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dc6f: skipped 000983 1788272959.495263 trunc from 0x02dc6f to 0x03c2b1 (0xe643 bytes) 000984 1788272959.575726 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c2b2): Mirror 12: 0xae94ac3e (stale) Mirror 14: 0xae94ac3e (stale) Mirror 17: 0xae94ac3e (stale) Mirror 18: 0xae94ac3e (stale) Mirror 20: 0xae94ac3e (stale) Mirror 22: 0xae94ac3e (stale) Mirror 23: 0xae94ac3e (stale) Mirror 25: 0xae94ac3e (stale) Mirror 26: 0xae94ac3e (stale) Mirror 27: 0xae94ac3e (stale) Mirror 28: 0xae94ac3e (stale) Mirror 29: 0xae94ac3e (stale) Mirror 30: 0x1431f1af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2b2): Mirror 12: 0xae94ac3e (stale) Mirror 14: 0xae94ac3e (stale) Mirror 17: 0xae94ac3e (stale) Mirror 18: 0xae94ac3e (stale) Mirror 20: 0xae94ac3e (stale) Mirror 22: 0xae94ac3e (stale) Mirror 23: 0xae94ac3e (stale) Mirror 25: 0xae94ac3e (stale) Mirror 26: 0xae94ac3e (stale) Mirror 27: 0xae94ac3e (stale) Mirror 28: 0xae94ac3e (stale) Mirror 29: 0xae94ac3e (stale) Mirror 30: 0x1431f1af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2b2: skipped 000986 1788272960.130704 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c2b2): Mirror 14: 0xae94ac3e (stale) Mirror 17: 0xae94ac3e (stale) Mirror 18: 0xae94ac3e (stale) Mirror 20: 0xae94ac3e (stale) Mirror 22: 0xae94ac3e (stale) Mirror 23: 0xae94ac3e (stale) Mirror 25: 0xae94ac3e (stale) Mirror 26: 0xae94ac3e (stale) Mirror 27: 0xae94ac3e (stale) Mirror 28: 0xae94ac3e (stale) Mirror 29: 0xae94ac3e (stale) Mirror 30: 0x1431f1af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2b2): Mirror 14: 0xae94ac3e (stale) Mirror 17: 0xae94ac3e (stale) Mirror 18: 0xae94ac3e (stale) Mirror 20: 0xae94ac3e (stale) Mirror 22: 0xae94ac3e (stale) Mirror 23: 0xae94ac3e (stale) Mirror 25: 0xae94ac3e (stale) Mirror 26: 0xae94ac3e (stale) Mirror 27: 0xae94ac3e (stale) Mirror 28: 0xae94ac3e (stale) Mirror 29: 0xae94ac3e (stale) Mirror 30: 0x1431f1af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2b2: skipped 000987 1788272960.701256 read 0x02fa61 thru 0x03c2b1 (0xc851 bytes) 000990 1788272960.709619 write 0x03f95b thru 0x03ffff (0x6a5 bytes) 000991 1788272960.710880 read 0x00f4bf thru 0x014726 (0x5268 bytes) 000992 1788272960.721617 mapread 0x007c96 thru 0x0130ca (0xb435 bytes) 000994 1788272960.723486 mapwrite 0x023b0d thru 0x026eac (0x33a0 bytes) 000999 1788272960.773001 mapwrite 0x029706 thru 0x03610e (0xca09 bytes) All operations completed A-OK! PASS 70a (127s) == sanity-flr test 71: check mirror extend parent fid ==== 10:29:26 (1788272966) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 10:29:29 (1788272969) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313979 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141469 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 10:29:39 (1788272979) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1117728 .. Extending file size to 4543680 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4837344 .. Extending file size to 6550432 .. Extending file size to 7100192 .. Extending file size to 7785760 .. Extending file size to 7352992 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8143456 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8403872 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8455456 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8495776 .. Extending file size to 8815360 .. Extending file size to 8629376 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8828128 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8930816 .. Waiting 57505 57506 57507 57508 57509 done 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) 192.168.202.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (87s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:31:06 (1788273066) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6643264 .. Extending file size to 3539200 .. Extending file size to 5566688 .. Extending file size to 7991392 .. Extending file size to 6103872 .. Extending file size to 8536512 .. Extending file size to 8196160 .. Extending file size to 8262016 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8466432 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8685856 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8603008 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59137 59139 59140 59141 59142 done 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) 192.168.202.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:32:27 (1788273147) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121546 s, 17.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.285855 s, 7.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6b2b5ba0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:32:45 (1788273165) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182951 s, 11.5 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 10:32:54 (1788273174) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x216:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:33:02 (1788273182) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.310666 s, 6.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:33:12 (1788273192) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10555 s, 19.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:33:20 (1788273200) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.364652 s, 5.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:33:53 (1788273233) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161364 s, 6.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:34:50 (1788273290) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169977 s, 12.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:35:46 (1788273346) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg207-server mds-ost sync done. Creating new pool test_204f oleg207-server: Pool lustre.test_204f created Adding targets to pool oleg207-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg207-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251658 s, 8.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg207-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg207-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg207-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:37:01 (1788273421) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:37:07 (1788273427) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:37:14 (1788273434) oleg207-server: Pool lustre.test_206 created Adding targets to pool oleg207-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg207-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg207-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg207-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg207-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:37:36 (1788273456) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825742 s, 12.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655242 s, 16.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2a1beec8 Mirror 3: 0x2a1beec8 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:37:45 (1788273465) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:37:48 (1788273468) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:37:51 (1788273471) Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.107111 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x730000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ad0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ad0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ad0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2480000 Read this (4 bytes): BBBB 192.168.202.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg207-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:38:24 (1788273504) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0400619 s, 204 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:38:31 (1788273511) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443807 s, 23.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23b lmm_fid: [0x200000404:0x23b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x23b lmm_fid: [0x200000404:0x23b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x325:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834491 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 10:38:33.000000000 -0400 Modify: 2026-09-01 10:38:33.000000000 -0400 Change: 2026-09-01 10:38:34.000000000 -0400 Birth: 2026-09-01 10:38:33.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:38:39 (1788273519) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:38:42 (1788273522) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.469685 s, 87.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0496179 s, 82.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 1 10:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 1 10:38 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:38:49 (1788273529) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:38:57 (1788273537) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1838 sec =========== 10:39:06 (1788273546) === sanity-flr: start cleanup 10:39:08 (1788273548) === === sanity-flr: finish cleanup 10:39:14 (1788273554) ===