-----============= acceptance-small: sanity-flr ============----- Wed Sep 16 14:39:27 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg442-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:39:49 (1789583989) === oleg442-client.virtnet: executing check_config_client /mnt/lustre oleg442-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg442-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94ed03dff000.idle_timeout=debug osc.lustre-OST0001-osc-ffff94ed03dff000.idle_timeout=debug disable quota as required oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:40:29 (1789584029) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7629] == sanity-flr test 0a: lfs mirror create with -N option == 14:40:32 (1789584032) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:40:53 (1789584053) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:40:58 (1789584058) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:41:02 (1789584062) PASS 0d (33s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:41:35 (1789584095) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:41:39 (1789584099) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:41:42 (1789584102) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff94ed03dff000.stats=clear osc.lustre-OST0001-osc-ffff94ed03dff000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:41:51 (1789584111) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:42:05 (1789584125) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:42:15 (1789584135) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:42:19 (1789584139) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:42:28 (1789584148) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 14:42:38 (1789584158) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:42:47 (1789584167) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:42:51 (1789584171) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:43:00 (1789584180) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:43:11 (1789584191) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:43:18 (1789584198) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.473564 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100059 s, 10.5 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:43:28 (1789584208) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.142258 s, 66.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:43:36 (1789584216) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:43:46 (1789584226) Stopping /mnt/lustre-ost1 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94ed03dff000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94ed03dff000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg442-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg442-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94ed03dff000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94ed03dff000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg442-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:46:03 (1789584363) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.250617 s, 16.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94ed03dff000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94ed03dff000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg442-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:46:45 (1789584405) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94ed03dff000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94ed03dff000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg442-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94ed03dff000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94ed03dff000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg442-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:48:03 (1789584483) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.0425 s, 10.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 16 14:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.220s user 0m0.000s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94ed03dff000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94ed03dff000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.251s user 0m0.000s sys 0m0.214s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg442-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg442-server oleg442-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94ed03dff000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94ed03dff000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.410s user 0m0.000s sys 0m0.186s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg442-server: oleg442-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg442-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94ed03dff000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94ed03dff000.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:49:44 (1789584584) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:49:48 (1789584588) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:49:52 (1789584592) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 14:49:55 (1789584595) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 14:50:05 (1789584605) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.818221 s, 15.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:50:21 (1789584621) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364409 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031264 s, 33.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:50:29 (1789584629) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:50:38 (1789584638) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000961344 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00148733 s, 688 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000934556 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00522887 s, 196 kB/s flv=10 olv=10 PASS 36d (15s) == sanity-flr test 37: mirror I/O API verification ======= 14:50:53 (1789584653) create test files with size 15739599 .. be7c8e998d092f2bbfea2c44b356b09e - d470884313091776acc2b91cd7f03390 - c90a85bafc21581a4ec6e00e3b9a95fa - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 14:51:23 (1789584683) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:51:42 (1789584702) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:51:48 (1789584708) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116251 s, 9.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047332 s, 22.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 14:51:57 (1789584717) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206515 s, 10.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.543869 s, 7.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259147 s, 8.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 14:52:16 (1789584736) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:52:20 (1789584740) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:52:24 (1789584744) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565017 s, 18.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 14:52:34 (1789584754) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:52:38 (1789584758) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:52:50 (1789584770) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.424347 s, 24.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:53:05 (1789584785) llite.lustre-ffff94ed03df9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.476708 s, 22.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg442-client.virtnet /mnt/lustre (opts:) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff94ed0522c000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:53:22 (1789584802) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (51s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:54:13 (1789584853) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (45s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:54:58 (1789584898) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 14:55:02 (1789584902) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.304193 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.183988 s, 17.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x76598f7c (nosync,stale) Mirror 2: 0x76598f7c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 43c309565bcdbd2b5d17d7782dd14125 - mirror 1 checksum: 8dcd8ac45f421062ee643f32edc0e6ea - mirror 2 checksum: 43c309565bcdbd2b5d17d7782dd14125 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T14:55:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T14:55:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 43c309565bcdbd2b5d17d7782dd14125 - mirror 1 checksum: 8dcd8ac45f421062ee643f32edc0e6ea - mirror 2 checksum: 8dcd8ac45f421062ee643f32edc0e6ea - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T14:55:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T14:55:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:55:15 (1789584915) Failing mds1 on oleg442-server Stopping /mnt/lustre-mds1 (opts:) on oleg442-server 14:55:21 (1789584921) shut down facet: mds1 facet_host: oleg442-server facet_failover_host: oleg442-server Failover mds1 to oleg442-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 14:55:45 (1789584945) targets are mounted 14:55:45 (1789584945) facet_failover done oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357004 s, 29.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:56:03 (1789584963) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:56:06 (1789584966) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:56:10 (1789584970) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00595021 s, 2.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:56:18 (1789584978) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:56:21 (1789584981) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0161611 s, 253 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0698019 s, 58.7 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:56:30 (1789584990) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:56:55 (1789585015) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:57:26 (1789585046) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:57:50 (1789585070) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:58:17 (1789585097) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 14:58:25 (1789585105) mirror create pid 54355 mirror split pid 54356 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54355 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54356 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 14:59:33 (1789585173) Seed set to 5176 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole truncating to largest ever: 0xec4b 000002 1789585176.380389 trunc from 00000000 to 0x00ec4a (0xec4b bytes) 000008 1789585176.420599 mapread 0x00bca2 thru 0x00ec4a (0x2fa9 bytes) 000010 1789585176.452723 trunc from 0x00ec4b to 0x007a7d (0x71cd bytes) 000012 1789585176.480859 write 0x000fe1 thru 0x00386a (0x288a bytes) 000015 1789585176.495724 read 0x006a17 thru 0x007a7d (0x1067 bytes) 000017 1789585176.497816 punch from 0x006bb3 to 0x007a7d (0xecb bytes) 000019 1789585176.513745 read 0x007271 thru 0x007a7d (0x80d bytes) 000020 1789585176.523766 mapwrite 0x03bf42 thru 0x03ffff (0x40be bytes) 000021 1789585176.691916 MIRROR_EXTEND to 2 mirrors 000022 1789585178.148489 read 0x03ece2 thru 0x03ffff (0x131e bytes) 000023 1789585178.172591 read 0x0387e5 thru 0x03bac2 (0x32de bytes) 000024 1789585178.222142 punch from 0x01829c to 0x01bb3a (0x389f bytes) 000025 1789585178.304429 punch from 0x016696 to 0x024554 (0xdebf bytes) 000027 1789585178.321604 mapread 0x033e78 thru 0x03ffff (0xc188 bytes) 000028 1789585178.439592 punch from 0x038fac to 0x03ffff (0x7054 bytes) 000032 1789585178.447872 punch from 0x0093e3 to 0x01066a (0x7288 bytes) 000034 1789585178.455707 mapread 0x020841 thru 0x02e967 (0xe127 bytes) truncating to largest ever: 0x33256 000035 1789585178.556660 trunc from 0x040000 to 0x033255 (0xcdaa bytes) 000037 1789585178.616587 write 0x024c03 thru 0x02f005 (0xa403 bytes) 000038 1789585178.653765 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000039 1789585178.886025 mapread 0x014cb2 thru 0x01e102 (0x9451 bytes) 000041 1789585178.915329 write 0x039fe9 thru 0x03f5a9 (0x55c1 bytes) 000042 1789585178.917498 mapread 0x03423a thru 0x03f5a9 (0xb370 bytes) 000045 1789585178.932662 read 0x031515 thru 0x03559a (0x4086 bytes) 000046 1789585178.933462 trunc from 0x03f5aa to 0x01d99e (0x21c0b bytes) 000048 1789585178.961788 write 0x030296 thru 0x03108d (0xdf8 bytes) 000049 1789585178.964724 read 0x01e9f1 thru 0x029a0b (0xb01b bytes) 000051 1789585178.990517 read 0x0131c7 thru 0x01c12e (0x8f68 bytes) truncating to largest ever: 0x3a751 000053 1789585178.991624 trunc from 0x03108e to 0x03a750 (0x96c3 bytes) 000056 1789585179.028688 write 0x00cb3e thru 0x019c49 (0xd10c bytes) 000057 1789585179.034662 trunc from 0x03a751 to 0x007637 (0x33119 bytes) 000058 1789585179.067002 punch from 0x002c14 to 0x007637 (0x4a24 bytes) 000059 1789585179.083545 mapwrite 0x0212ff thru 0x02b95a (0xa65c bytes) 000060 1789585179.259784 punch from 0x0263eb to 0x02b95a (0x5570 bytes) 000062 1789585179.286524 read 0x000575 thru 0x00cd02 (0xc78e bytes) 000064 1789585179.290130 punch from 0x01550e to 0x01b1c8 (0x5cbb bytes) 000066 1789585179.309593 punch from 0x000675 to 0x000e1f (0x7ab bytes) 000067 1789585179.324576 mapwrite 0x012a60 thru 0x01cb9d (0xa13e bytes) 000068 1789585179.466103 read 0x029c79 thru 0x02b95a (0x1ce2 bytes) 000069 1789585179.469467 read 0x01041c thru 0x01edf1 (0xe9d6 bytes) 000070 1789585179.473890 punch from 0x013dd8 to 0x01d95c (0x9b85 bytes) 000071 1789585179.483556 write 0x02e0d2 thru 0x035316 (0x7245 bytes) 000073 1789585179.496372 write 0x00f2df thru 0x01e9e9 (0xf70b bytes) 000075 1789585179.507274 write 0x002591 thru 0x00869b (0x610b bytes) 000078 1789585179.517258 punch from 0x007e7e to 0x013e4b (0xbfce bytes) 000079 1789585179.658074 mapread 0x004c8f thru 0x00bb4a (0x6ebc bytes) 000080 1789585179.677134 mapwrite 0x008a1f thru 0x00d6fd (0x4cdf bytes) 000081 1789585179.788456 trunc from 0x035317 to 0x007deb (0x2d52b bytes) 000082 1789585179.829550 punch from 0x004605 to 0x007deb (0x37e7 bytes) 000083 1789585179.837541 mapwrite 0x009fde thru 0x013c39 (0x9c5c bytes) 000085 1789585180.090574 punch from 0x008839 to 0x013c39 (0xb401 bytes) 000087 1789585180.110711 trunc from 0x013c3a to 0x037246 (0x2360d bytes) 000090 1789585180.185661 punch from 0x0114ea to 0x020f33 (0xfa4a bytes) 000091 1789585180.195429 mapwrite 0x0114f1 thru 0x0201ae (0xecbe bytes) 000092 1789585180.417757 mapwrite 0x03d05a thru 0x03ffff (0x2fa6 bytes) 000093 1789585180.510080 mapwrite 0x03d723 thru 0x03ffff (0x28dd bytes) 000096 1789585180.573842 punch from 0x016ee0 to 0x01dffe (0x711f bytes) 000098 1789585180.582556 punch from 0x010e22 to 0x015771 (0x4950 bytes) 000099 1789585180.590329 read 0x011d1a thru 0x0194a0 (0x7787 bytes) 000101 1789585180.643321 MIRROR_EXTEND to 2 mirrors 000102 1789585181.779484 write 0x014bbe thru 0x019624 (0x4a67 bytes) 000103 1789585181.898262 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000106 1789585182.115112 trunc from 0x040000 to 0x00399e (0x3c661 bytes) 000108 1789585182.145171 mapwrite 0x00c556 thru 0x01bae9 (0xf594 bytes) 000113 1789585182.325462 MIRROR_EXTEND to 2 mirrors 000114 1789585183.252660 write 0x020997 thru 0x023017 (0x2681 bytes) 000115 1789585183.383822 write 0x00d153 thru 0x00fedb (0x2d89 bytes) 000116 1789585183.435974 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23018): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23018): Mirror 2: 0x15870f67 (stale) CRC-32 checksum value for chunk [0, 0x23018): Mirror 1: 0xf71b281c Mirror 3: 0xf71b281c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23018: skipped 000118 1789585184.952510 punch from 0x002be4 to 0x00e4af (0xb8cc bytes) 000119 1789585185.057629 trunc from 0x023018 to 0x0253be (0x23a7 bytes) 000120 1789585185.103601 mapread 0x0252ea thru 0x0253be (0xd5 bytes) 000121 1789585185.119053 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x253bf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x253bf): Mirror 2: 0xa44da6ee (stale) Mirror 3: 0x9f1483ce (stale) CRC-32 checksum value for chunk [0, 0x253bf): Mirror 1: 0xa0b0be29 Mirror 4: 0xa0b0be29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x253bf: skipped 000122 1789585186.412448 write 0x0213d1 thru 0x02a70c (0x933c bytes) 000124 1789585186.498258 read 0x0294f4 thru 0x02a70c (0x1219 bytes) 000125 1789585186.510967 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a70d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a70d): Mirror 2: 0x5e447d7a (stale) Mirror 3: 0x815ce962 (stale) Mirror 4: 0x35e01133 (stale) CRC-32 checksum value for chunk [0, 0x2a70d): Mirror 1: 0x2e448356 Mirror 5: 0x2e448356 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a70d: skipped 000127 1789585187.971460 mapwrite 0x01f22b thru 0x02adcb (0xbba1 bytes) 000128 1789585188.204882 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2adcc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adcc): Mirror 2: 0x4c6fc8df (stale) Mirror 3: 0x2fe15327 (stale) Mirror 4: 0x5190c4c4 (stale) Mirror 5: 0xb3f24a8e (stale) CRC-32 checksum value for chunk [0, 0x2adcc): Mirror 1: 0x237c6502 Mirror 6: 0x237c6502 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adcc: skipped 000129 1789585189.390294 write 0x00ddcb thru 0x018546 (0xa77c bytes) 000130 1789585189.533304 mapread 0x01c9e1 thru 0x0251ad (0x87cd bytes) 000132 1789585189.578530 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2adcc): Mirror 2: 0x4c6fc8df (stale) Mirror 3: 0x2fe15327 (stale) Mirror 4: 0x5190c4c4 (stale) Mirror 5: 0xb3f24a8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adcc): Mirror 2: 0x4c6fc8df (stale) Mirror 3: 0x2fe15327 (stale) Mirror 4: 0x5190c4c4 (stale) Mirror 5: 0xb3f24a8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adcc: skipped 000133 1789585190.150440 mapread 0x025938 thru 0x02adcb (0x5494 bytes) 000134 1789585190.198598 punch from 0x01d717 to 0x01efb4 (0x189e bytes) 000135 1789585190.221805 punch from 0x01654f to 0x020c20 (0xa6d2 bytes) 000136 1789585190.233574 mapread 0x00dafa thru 0x0129d6 (0x4edd bytes) 000137 1789585190.288071 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2adcc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adcc): Mirror 2: 0x4c6fc8df (stale) Mirror 3: 0x2fe15327 (stale) Mirror 4: 0x5190c4c4 (stale) Mirror 5: 0xb3f24a8e (stale) CRC-32 checksum value for chunk [0, 0x2adcc): Mirror 1: 0x6168ecfe Mirror 6: 0x6168ecfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adcc: skipped 000139 1789585192.102986 MIRROR_RESYNC 6 mirrors 000140 1789585194.031804 write 0x017b42 thru 0x01da19 (0x5ed8 bytes) 000142 1789585194.136936 mapwrite 0x028e9c thru 0x034651 (0xb7b6 bytes) 000143 1789585194.309711 punch from 0x02a245 to 0x02cab0 (0x286c bytes) 000146 1789585194.318738 read 0x007913 thru 0x008ed7 (0x15c5 bytes) 000148 1789585194.333665 read 0x00fd6a thru 0x013055 (0x32ec bytes) 000150 1789585194.351043 mapread 0x01b9a8 thru 0x0265d0 (0xac29 bytes) 000151 1789585194.354026 punch from 0x02f639 to 0x034651 (0x5019 bytes) 000152 1789585194.362213 mapread 0x0005f0 thru 0x007d35 (0x7746 bytes) 000153 1789585194.364156 mapwrite 0x0294c9 thru 0x035f99 (0xcad1 bytes) 000154 1789585194.497166 write 0x00e00b thru 0x0174aa (0x94a0 bytes) 000159 1789585194.501786 mapwrite 0x0365f4 thru 0x036c01 (0x60e bytes) 000160 1789585194.634252 mapread 0x013cf1 thru 0x0155f3 (0x1903 bytes) 000161 1789585194.651826 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36c02): Mirror 2: 0xe08be29b (stale) Mirror 3: 0xe08be29b (stale) Mirror 4: 0xe08be29b (stale) Mirror 5: 0xe08be29b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c02): Mirror 2: 0xe08be29b (stale) Mirror 3: 0xe08be29b (stale) Mirror 4: 0xe08be29b (stale) Mirror 5: 0xe08be29b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c02: skipped 000162 1789585195.221110 trunc from 0x036c02 to 0x00e826 (0x283db bytes) 000163 1789585195.246887 punch from 0x00856c to 0x00e826 (0x62bb bytes) 000165 1789585195.255722 read 0x004830 thru 0x00bf19 (0x76ea bytes) 000166 1789585195.273566 mapwrite 0x000742 thru 0x002e65 (0x2724 bytes) 000168 1789585195.355458 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe827): Mirror 3: 0xa505e8ca (stale) Mirror 4: 0xa505e8ca (stale) Mirror 5: 0xa505e8ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe827): Mirror 3: 0xa505e8ca (stale) Mirror 4: 0xa505e8ca (stale) Mirror 5: 0xa505e8ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe827: skipped 000170 1789585195.791232 mapread 0x008017 thru 0x00e826 (0x6810 bytes) 000171 1789585195.845501 read 0x007b54 thru 0x00e826 (0x6cd3 bytes) 000172 1789585195.847130 punch from 0x003f35 to 0x00e7a1 (0xa86d bytes) 000173 1789585195.869633 write 0x0367fe thru 0x03ffff (0x9802 bytes) 000174 1789585195.874097 write 0x0264a4 thru 0x03023e (0x9d9b bytes) 000175 1789585195.887248 mapwrite 0x03e68f thru 0x03ffff (0x1971 bytes) 000176 1789585195.999130 punch from 0x03a30e to 0x03ffff (0x5cf2 bytes) 000178 1789585196.010907 punch from 0x016515 to 0x021ba2 (0xb68e bytes) 000179 1789585196.021927 mapwrite 0x02cd37 thru 0x03b165 (0xe42f bytes) 000180 1789585196.199338 MIRROR_RESYNC 4 mirrors 000181 1789585198.282597 mapread 0x01eacb thru 0x025920 (0x6e56 bytes) 000183 1789585198.321277 mapread 0x027fd1 thru 0x02933a (0x136a bytes) 000184 1789585198.340169 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000185 1789585198.792046 write 0x0237c4 thru 0x024ec8 (0x1705 bytes) 000186 1789585198.941304 punch from 0x005a35 to 0x010e31 (0xb3fd bytes) 000187 1789585198.959953 mapwrite 0x014ac4 thru 0x019f9f (0x54dc bytes) 000188 1789585199.179127 punch from 0x00a886 to 0x0131e0 (0x895b bytes) 000189 1789585199.193951 read 0x031657 thru 0x03373d (0x20e7 bytes) 000190 1789585199.209847 write 0x01e09f thru 0x027766 (0x96c8 bytes) 000192 1789585199.229800 mapwrite 0x02f9ea thru 0x030900 (0xf17 bytes) 000193 1789585199.294828 mapwrite 0x0380ea thru 0x03ffff (0x7f16 bytes) 000194 1789585199.368988 write 0x024ad1 thru 0x029a99 (0x4fc9 bytes) 000195 1789585199.370885 mapread 0x031bcf thru 0x03e006 (0xc438 bytes) 000197 1789585199.381357 trunc from 0x040000 to 0x035b1b (0xa4e4 bytes) 000199 1789585199.464487 read 0x006cd9 thru 0x00aa1c (0x3d44 bytes) 000200 1789585199.514908 read 0x0220ca thru 0x029c3c (0x7b73 bytes) 000203 1789585199.522612 mapwrite 0x02e64b thru 0x031498 (0x2e4e bytes) 000204 1789585199.582536 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35b1c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b1c): Mirror 4: 0xa586b601 (stale) Mirror 5: 0xa586b601 (stale) CRC-32 checksum value for chunk [0, 0x35b1c): Mirror 1: 0x259f7b7b Mirror 6: 0x259f7b7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b1c: skipped 000208 1789585201.033989 punch from 0x00c236 to 0x017d07 (0xbad2 bytes) 000210 1789585201.166108 trunc from 0x035b1c to 0x01454b (0x215d0 bytes) 000211 1789585201.233309 mapwrite 0x00a008 thru 0x01796d (0xd966 bytes) 000212 1789585201.450802 mapread 0x0169c2 thru 0x01796d (0xfac bytes) 000213 1789585201.452954 mapwrite 0x014e0f thru 0x021da5 (0xcf97 bytes) 000214 1789585201.594732 trunc from 0x021da6 to 0x02ce07 (0xb062 bytes) 000215 1789585201.614131 mapwrite 0x02064e thru 0x029b33 (0x94e6 bytes) 000217 1789585201.721781 mapwrite 0x00ae43 thru 0x01624a (0xb408 bytes) 000218 1789585201.812813 mapwrite 0x0254cc thru 0x02fa5b (0xa590 bytes) 000219 1789585201.924734 punch from 0x027d07 to 0x02fa5b (0x7d55 bytes) 000220 1789585201.930877 trunc from 0x02fa5c to 0x028809 (0x7252 bytes) 000221 1789585201.959616 mapwrite 0x0059d5 thru 0x00dc86 (0x82b2 bytes) 000222 1789585202.063555 MIRROR_RESYNC 4 mirrors 000223 1789585203.411045 trunc from 0x02880a to 0x02cf9b (0x4792 bytes) 000226 1789585203.487847 mapread 0x00f960 thru 0x013068 (0x3709 bytes) 000228 1789585203.512129 read 0x01d587 thru 0x02945c (0xbed6 bytes) 000229 1789585203.522630 mapread 0x020cca thru 0x02cf9b (0xc2d2 bytes) 000232 1789585203.535954 write 0x008095 thru 0x008514 (0x480 bytes) 000233 1789585203.554731 punch from 0x008989 to 0x00fa25 (0x709d bytes) 000237 1789585203.580065 trunc from 0x02cf9c to 0x01fc33 (0xd368 bytes) 000238 1789585203.631404 mapread 0x009362 thru 0x011a23 (0x86c2 bytes) 000240 1789585203.648706 mapread 0x01a2b0 thru 0x01fc33 (0x5984 bytes) 000243 1789585203.665103 write 0x02ee8d thru 0x030689 (0x17fd bytes) 000246 1789585203.671720 trunc from 0x03068a to 0x0315bf (0xf36 bytes) 000247 1789585203.696334 read 0x03153f thru 0x0315bf (0x81 bytes) 000248 1789585203.719537 trunc from 0x0315c0 to 0x01bb0a (0x15ab5 bytes) 000251 1789585203.763499 read 0x008567 thru 0x00a6a8 (0x2142 bytes) 000254 1789585203.767591 trunc from 0x01bb0b to 0x01f2b7 (0x37ad bytes) 000258 1789585203.802900 read 0x0198b2 thru 0x01f2b7 (0x5a06 bytes) 000260 1789585203.828134 MIRROR_RESYNC 4 mirrors 000263 1789585205.160581 write 0x010ea9 thru 0x01b216 (0xa36e bytes) 000265 1789585205.276192 read 0x011ad4 thru 0x01f2b7 (0xd7e4 bytes) 000267 1789585205.297382 mapread 0x0141e1 thru 0x017a2b (0x384b bytes) 000268 1789585205.300224 read 0x019582 thru 0x01f2b7 (0x5d36 bytes) 000269 1789585205.305021 punch from 0x00899d to 0x01720c (0xe870 bytes) 000271 1789585205.333201 mapread 0x0060fc thru 0x01041a (0xa31f bytes) 000272 1789585205.366474 MIRROR_RESYNC 4 mirrors 000273 1789585206.402977 mapwrite 0x00e8db thru 0x01666b (0x7d91 bytes) 000274 1789585206.623337 write 0x02b111 thru 0x02c4eb (0x13db bytes) 000275 1789585206.628999 read 0x010ccf thru 0x01bc4f (0xaf81 bytes) 000276 1789585206.638588 mapread 0x001b51 thru 0x00833f (0x67ef bytes) 000277 1789585206.650961 write 0x0180db thru 0x026c52 (0xeb78 bytes) 000278 1789585206.657961 trunc from 0x02c4ec to 0x038889 (0xc39e bytes) 000279 1789585206.736059 punch from 0x00a107 to 0x017188 (0xd082 bytes) 000280 1789585206.777545 mapwrite 0x028899 thru 0x02f0fa (0x6862 bytes) 000281 1789585206.875963 read 0x0065c4 thru 0x011345 (0xad82 bytes) 000282 1789585206.900229 write 0x0120fb thru 0x01fad9 (0xd9df bytes) 000283 1789585206.903040 read 0x0244df thru 0x02cd59 (0x887b bytes) 000284 1789585206.905823 mapread 0x009607 thru 0x017278 (0xdc72 bytes) 000285 1789585206.906992 write 0x0085f3 thru 0x0172a0 (0xecae bytes) 000286 1789585206.909194 write 0x000ce6 thru 0x00c564 (0xb87f bytes) 000287 1789585206.911334 write 0x0300d7 thru 0x03b16f (0xb099 bytes) 000288 1789585206.913966 write 0x038f55 thru 0x03ffff (0x70ab bytes) 000289 1789585206.916212 trunc from 0x040000 to 0x01db94 (0x2246b bytes) 000291 1789585206.943319 punch from 0x00f97a to 0x0156c2 (0x5d49 bytes) 000294 1789585206.967995 mapread 0x002642 thru 0x00a3e5 (0x7da4 bytes) 000295 1789585206.970274 read 0x001a78 thru 0x00c878 (0xae01 bytes) 000296 1789585206.971984 read 0x016288 thru 0x01db94 (0x790d bytes) 000297 1789585206.974688 punch from 0x013c12 to 0x013db3 (0x1a2 bytes) 000298 1789585206.989004 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1db95): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1db95): Mirror 4: 0xd3fc0c76 (stale) Mirror 5: 0xd3fc0c76 (stale) Mirror 6: 0xd3fc0c76 (stale) CRC-32 checksum value for chunk [0, 0x1db95): Mirror 1: 0xbaf55fd8 Mirror 7: 0xbaf55fd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1db95: skipped 000300 1789585208.225956 trunc from 0x01db95 to 0x02c4a3 (0xe90f bytes) 000301 1789585208.321783 punch from 0x011833 to 0x01b907 (0xa0d5 bytes) 000303 1789585208.328853 read 0x029e7b thru 0x02c4a3 (0x2629 bytes) 000304 1789585208.338046 trunc from 0x02c4a4 to 0x0314d6 (0x5033 bytes) 000306 1789585208.366016 trunc from 0x0314d7 to 0x00327c (0x2e25a bytes) 000308 1789585208.403305 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x327d): Mirror 5: 0xaff09b2e (stale) Mirror 6: 0xaff09b2e (stale) Mirror 7: 0x127c22ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x327d): Mirror 5: 0xaff09b2e (stale) Mirror 6: 0xaff09b2e (stale) Mirror 7: 0x127c22ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x327d: skipped 000309 1789585208.845742 read 0x002f1a thru 0x00327c (0x363 bytes) 000311 1789585208.858121 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x327d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x327d): Mirror 5: 0xaff09b2e (stale) Mirror 6: 0xaff09b2e (stale) Mirror 7: 0x127c22ac (stale) CRC-32 checksum value for chunk [0, 0x327d): Mirror 1: 0x127c22ac Mirror 8: 0x127c22ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x327d: skipped 000312 1789585210.465803 read 0x002610 thru 0x00327c (0xc6d bytes) 000313 1789585210.503689 mapread 0x002bca thru 0x00327c (0x6b3 bytes) 000314 1789585210.526799 write 0x0191bb thru 0x0290d6 (0xff1c bytes) 000317 1789585210.647155 punch from 0x00cc06 to 0x01187d (0x4c78 bytes) 000320 1789585210.663985 mapread 0x0011b5 thru 0x002061 (0xead bytes) 000322 1789585210.678159 mapread 0x024389 thru 0x0290d6 (0x4d4e bytes) 000323 1789585210.681947 mapread 0x01780d thru 0x020ec6 (0x96ba bytes) 000325 1789585210.705629 trunc from 0x0290d7 to 0x0178be (0x11818 bytes) 000326 1789585210.741738 write 0x02dce2 thru 0x039f63 (0xc282 bytes) 000328 1789585210.746469 read 0x00713b thru 0x0113e7 (0xa2ad bytes) 000330 1789585210.759947 write 0x009364 thru 0x014de2 (0xba7f bytes) 000332 1789585210.765763 write 0x0321b4 thru 0x03ddfa (0xbc47 bytes) 000334 1789585210.775686 write 0x022c12 thru 0x02853c (0x592b bytes) 000335 1789585210.782260 mapwrite 0x00bea7 thru 0x00da28 (0x1b82 bytes) 000337 1789585210.886690 mapwrite 0x021ce3 thru 0x02cd94 (0xb0b2 bytes) 000338 1789585210.971107 write 0x00062f thru 0x00c746 (0xc118 bytes) 000343 1789585210.982842 MIRROR_RESYNC 5 mirrors 000344 1789585212.891206 write 0x028dbb thru 0x03410e (0xb354 bytes) 000345 1789585212.993075 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ddfb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddfb): Mirror 5: 0x1e96cb33 (stale) Mirror 6: 0x1e96cb33 (stale) Mirror 7: 0x1e96cb33 (stale) Mirror 8: 0x1e96cb33 (stale) CRC-32 checksum value for chunk [0, 0x3ddfb): Mirror 1: 0x94adf911 Mirror 9: 0x94adf911 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddfb: skipped 000346 1789585214.593747 read 0x02ef1e thru 0x0354ef (0x65d2 bytes) 000348 1789585214.624335 write 0x01f3a1 thru 0x02e213 (0xee73 bytes) 000350 1789585214.708389 write 0x037f0e thru 0x03ffff (0x80f2 bytes) 000351 1789585214.722043 MIRROR_RESYNC 6 mirrors 000352 1789585216.562057 mapwrite 0x022af1 thru 0x0323ea (0xf8fa bytes) 000355 1789585216.815824 write 0x00f9ed thru 0x01ee0b (0xf41f bytes) 000357 1789585216.831936 MIRROR_RESYNC 6 mirrors 000358 1789585218.357466 write 0x01baff thru 0x022772 (0x6c74 bytes) 000359 1789585218.527030 mapwrite 0x02a8c8 thru 0x0316ad (0x6de6 bytes) 000360 1789585218.649169 read 0x02bb21 thru 0x0389af (0xce8f bytes) 000361 1789585218.657894 trunc from 0x040000 to 0x01c78c (0x23873 bytes) 000364 1789585218.688290 punch from 0x00c522 to 0x00e7ed (0x22cc bytes) 000365 1789585218.694390 punch from 0x00da63 to 0x00e263 (0x801 bytes) 000366 1789585218.699492 write 0x015df3 thru 0x016c51 (0xe5f bytes) 000367 1789585218.709674 read 0x0011f0 thru 0x0070cd (0x5ede bytes) 000368 1789585218.719983 mapwrite 0x02131c thru 0x02aebc (0x9ba1 bytes) 000372 1789585218.801624 mapwrite 0x029860 thru 0x030865 (0x7006 bytes) 000375 1789585218.885907 punch from 0x021741 to 0x02e827 (0xd0e7 bytes) 000376 1789585218.891887 mapread 0x0097f5 thru 0x011ea5 (0x86b1 bytes) 000378 1789585218.894836 mapread 0x02f009 thru 0x030865 (0x185d bytes) 000379 1789585218.896113 punch from 0x022f42 to 0x02b4eb (0x85aa bytes) 000381 1789585218.901071 trunc from 0x030866 to 0x02ca43 (0x3e22 bytes) 000382 1789585218.928936 write 0x036b2a thru 0x03ffff (0x94d6 bytes) 000383 1789585218.931985 read 0x006293 thru 0x00f7dc (0x954a bytes) 000385 1789585218.933895 mapread 0x00e52f thru 0x018706 (0xa1d8 bytes) 000388 1789585218.937893 write 0x00e1ad thru 0x018b50 (0xa9a4 bytes) truncating to largest ever: 0x3b98d 000389 1789585218.941926 trunc from 0x040000 to 0x03b98c (0x4673 bytes) 000390 1789585218.966263 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 5: 0xc5a8109b (stale) Mirror 6: 0xc5a8109b (stale) Mirror 7: 0xc5a8109b (stale) Mirror 8: 0xc5a8109b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 5: 0xc5a8109b (stale) Mirror 6: 0xc5a8109b (stale) Mirror 7: 0xc5a8109b (stale) Mirror 8: 0xc5a8109b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b98d: skipped 000391 1789585219.490030 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 5: 0xc5a8109b (stale) Mirror 6: 0xc5a8109b (stale) Mirror 7: 0xc5a8109b (stale) Mirror 8: 0xc5a8109b (stale) CRC-32 checksum value for chunk [0, 0x3b98d): Mirror 1: 0x8ad8bec6 Mirror 9: 0x8ad8bec6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b98d: skipped 000392 1789585220.573257 mapwrite 0x02fb08 thru 0x039683 (0x9b7c bytes) 000394 1789585220.846009 mapwrite 0x000645 thru 0x00ca54 (0xc410 bytes) 000395 1789585220.931828 write 0x028b2d thru 0x036053 (0xd527 bytes) 000396 1789585220.945066 punch from 0x00e701 to 0x01d1cf (0xeacf bytes) 000397 1789585220.974625 read 0x00c85c thru 0x01273b (0x5ee0 bytes) 000398 1789585220.998824 trunc from 0x03b98d to 0x0132de (0x286ae bytes) 000399 1789585221.033527 mapwrite 0x03718e thru 0x03dfde (0x6e51 bytes) 000402 1789585221.143948 trunc from 0x03dfdf to 0x010908 (0x2d6d6 bytes) 000403 1789585221.172715 punch from 0x00dfbe to 0x0104e9 (0x252c bytes) 000405 1789585221.183830 read 0x00c8b9 thru 0x00c9fb (0x143 bytes) 000406 1789585221.185035 mapwrite 0x003feb thru 0x010b4e (0xcb64 bytes) 000407 1789585221.274044 write 0x035b1a thru 0x03838c (0x2873 bytes) truncating to largest ever: 0x3feb9 000409 1789585221.277068 trunc from 0x03838d to 0x03feb8 (0x7b2c bytes) 000411 1789585221.297565 trunc from 0x03feb9 to 0x00bcd3 (0x341e5 bytes) 000412 1789585221.318565 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbcd4): Mirror 5: 0xff1eea9 (stale) Mirror 6: 0xff1eea9 (stale) Mirror 7: 0xff1eea9 (stale) Mirror 8: 0xff1eea9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbcd4): Mirror 5: 0xff1eea9 (stale) Mirror 6: 0xff1eea9 (stale) Mirror 7: 0xff1eea9 (stale) Mirror 8: 0xff1eea9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbcd4: skipped 000413 1789585221.876799 trunc from 0x00bcd4 to 0x0362f4 (0x2a621 bytes) 000414 1789585221.903806 trunc from 0x0362f5 to 0x024424 (0x11ed0 bytes) 000415 1789585221.931562 punch from 0x00315b to 0x00cf0e (0x9db4 bytes) 000416 1789585221.947258 read 0x021985 thru 0x024424 (0x2aa0 bytes) 000417 1789585221.958336 read 0x02144d thru 0x024424 (0x2fd8 bytes) 000418 1789585221.971269 MIRROR_RESYNC 5 mirrors 000419 1789585223.700909 mapwrite 0x02c3a6 thru 0x034434 (0x808f bytes) 000420 1789585223.875988 mapwrite 0x030c85 thru 0x03faf7 (0xee73 bytes) 000422 1789585223.986107 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3faf8): Mirror 6: 0x95673c3b (stale) Mirror 7: 0x95673c3b (stale) Mirror 8: 0x95673c3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3faf8): Mirror 6: 0x95673c3b (stale) Mirror 7: 0x95673c3b (stale) Mirror 8: 0x95673c3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3faf8: skipped 000423 1789585224.735071 punch from 0x00024a to 0x003dff (0x3bb6 bytes) 000427 1789585224.752628 MIRROR_RESYNC 4 mirrors 000429 1789585226.195679 MIRROR_EXTEND to 5 mirrors 000431 1789585227.153702 trunc from 0x03faf8 to 0x0009e6 (0x3f111 bytes) 000432 1789585227.234638 mapwrite 0x039100 thru 0x03a918 (0x1819 bytes) 000433 1789585227.334599 punch from 0x018372 to 0x0191df (0xe6e bytes) 000434 1789585227.339893 mapread 0x00a5bc thru 0x012fda (0x8a1f bytes) 000435 1789585227.358867 write 0x00311b thru 0x004f24 (0x1e0a bytes) 000440 1789585227.364803 read 0x00497e thru 0x00d061 (0x86e4 bytes) 000442 1789585227.366933 read 0x00987a thru 0x00ea01 (0x5188 bytes) 000443 1789585227.367930 punch from 0x02dad2 to 0x03a918 (0xce47 bytes) 000444 1789585227.379576 punch from 0x02c1b3 to 0x037818 (0xb666 bytes) 000445 1789585227.387193 mapread 0x001feb thru 0x004970 (0x2986 bytes) 000446 1789585227.406090 punch from 0x01d8d9 to 0x02b752 (0xde7a bytes) 000447 1789585227.415879 mapwrite 0x036452 thru 0x03fe98 (0x9a47 bytes) 000448 1789585227.518151 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe99): Mirror 7: 0xb1622f6a (stale) Mirror 8: 0xb1622f6a (stale) Mirror 9: 0xb1622f6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe99): Mirror 7: 0xb1622f6a (stale) Mirror 8: 0xb1622f6a (stale) Mirror 9: 0xb1622f6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe99: skipped 000449 1789585228.089385 mapwrite 0x00e9da thru 0x010b54 (0x217b bytes) 000450 1789585228.176572 write 0x00a6c5 thru 0x00c8c1 (0x21fd bytes) 000451 1789585228.189607 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fe99): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe99): Mirror 7: 0xb1622f6a (stale) Mirror 8: 0xb1622f6a (stale) Mirror 9: 0xb1622f6a (stale) CRC-32 checksum value for chunk [0, 0x3fe99): Mirror 1: 0x4119a112 Mirror 10: 0x4119a112 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe99: skipped 000453 1789585229.486708 MIRROR_RESYNC 5 mirrors 000454 1789585230.914298 write 0x0284b6 thru 0x03665b (0xe1a6 bytes) 000455 1789585231.000817 trunc from 0x03fe99 to 0x00a3c8 (0x35ad0 bytes) 000457 1789585231.045309 punch from 0x0049e2 to 0x00a3c8 (0x59e7 bytes) 000458 1789585231.050607 punch from 0x00677d to 0x00a3c8 (0x3c4c bytes) 000459 1789585231.058563 punch from 0x006378 to 0x00a3c8 (0x4051 bytes) 000462 1789585231.065840 write 0x03451b thru 0x03c31d (0x7e03 bytes) 000464 1789585231.070200 mapread 0x0173fa thru 0x026d0a (0xf911 bytes) 000465 1789585231.100179 mapread 0x0314b3 thru 0x03bcaa (0xa7f8 bytes) 000466 1789585231.111090 mapwrite 0x02d178 thru 0x03b318 (0xe1a1 bytes) 000467 1789585231.187008 read 0x017459 thru 0x022aa5 (0xb64d bytes) 000468 1789585231.187912 trunc from 0x03c31e to 0x0252f5 (0x17028 bytes) 000469 1789585231.211563 trunc from 0x0252f6 to 0x0122db (0x1301a bytes) 000470 1789585231.233068 mapwrite 0x03ae8c thru 0x03b63b (0x7b0 bytes) 000471 1789585231.323281 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b63c): Mirror 7: 0xf627ee3a (stale) Mirror 8: 0xf627ee3a (stale) Mirror 9: 0xf627ee3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b63c): Mirror 7: 0xf627ee3a (stale) Mirror 8: 0xf627ee3a (stale) Mirror 9: 0xf627ee3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b63c: skipped 000473 1789585231.735121 mapwrite 0x004537 thru 0x00c5f1 (0x80bb bytes) 000474 1789585231.830079 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b63c): Mirror 7: 0xf627ee3a (stale) Mirror 9: 0xf627ee3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b63c): Mirror 7: 0xf627ee3a (stale) Mirror 9: 0xf627ee3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b63c: skipped 000475 1789585232.180100 MIRROR_RESYNC 3 mirrors 000476 1789585233.291296 mapwrite 0x027370 thru 0x027894 (0x525 bytes) 000477 1789585233.471925 trunc from 0x03b63c to 0x027b5f (0x13adc bytes) 000480 1789585233.506878 read 0x009b87 thru 0x015c88 (0xc102 bytes) 000481 1789585233.516198 mapwrite 0x008e9a thru 0x012dcb (0x9f32 bytes) 000482 1789585233.564575 read 0x022241 thru 0x027b5f (0x591f bytes) 000484 1789585233.575121 read 0x00f581 thru 0x0191e4 (0x9c64 bytes) 000487 1789585233.577281 mapread 0x00eca4 thru 0x01673b (0x7a98 bytes) 000488 1789585233.579255 trunc from 0x027b60 to 0x035522 (0xd9c3 bytes) 000489 1789585233.607833 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35523): Mirror 7: 0x37e9e27c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35523): Mirror 7: 0x37e9e27c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35523: skipped 000492 1789585233.956336 mapwrite 0x00b9ec thru 0x00c6ab (0xcc0 bytes) 000493 1789585234.054731 mapwrite 0x00ff30 thru 0x014c4f (0x4d20 bytes) 000494 1789585234.185046 mapread 0x0195cd thru 0x0203bd (0x6df1 bytes) 000495 1789585234.223114 MIRROR_RESYNC 2 mirrors 000496 1789585235.360241 read 0x00ee1b thru 0x01d302 (0xe4e8 bytes) 000497 1789585235.386659 mapread 0x030142 thru 0x035522 (0x53e1 bytes) 000498 1789585235.405126 mapwrite 0x004eeb thru 0x0117f1 (0xc907 bytes) 000499 1789585235.696257 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35523): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35523): Mirror 7: 0xfab81b81 (stale) CRC-32 checksum value for chunk [0, 0x35523): Mirror 1: 0x37a057d1 Mirror 8: 0x37a057d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35523: skipped 000501 1789585237.304454 trunc from 0x035523 to 0x02d116 (0x840c bytes) 000502 1789585237.481762 read 0x029878 thru 0x02d116 (0x389f bytes) 000503 1789585237.520635 trunc from 0x02d117 to 0x003076 (0x2a0a0 bytes) 000506 1789585237.593051 trunc from 0x003077 to 0x034a6c (0x319f6 bytes) 000507 1789585237.621483 punch from 0x005b25 to 0x00df88 (0x8464 bytes) 000508 1789585237.628204 mapwrite 0x03aa1c thru 0x03ffff (0x55e4 bytes) 000511 1789585237.752963 mapread 0x007385 thru 0x00922c (0x1ea8 bytes) 000512 1789585237.774522 MIRROR_RESYNC 3 mirrors 000513 1789585239.122278 MIRROR_RESYNC 3 mirrors 000514 1789585239.699326 MIRROR_EXTEND to 4 mirrors 000517 1789585240.646421 mapwrite 0x03eedc thru 0x03ffff (0x1124 bytes) 000518 1789585240.883410 write 0x008e3f thru 0x00d900 (0x4ac2 bytes) 000519 1789585240.907213 write 0x00fc6d thru 0x017d62 (0x80f6 bytes) 000521 1789585240.941261 MIRROR_RESYNC 4 mirrors 000522 1789585242.404978 mapwrite 0x0233d0 thru 0x023812 (0x443 bytes) 000523 1789585242.629644 mapread 0x02d941 thru 0x03054d (0x2c0d bytes) 000524 1789585242.648993 punch from 0x016eb1 to 0x0176a5 (0x7f5 bytes) 000525 1789585242.657056 trunc from 0x040000 to 0x017b71 (0x2848e bytes) 000526 1789585242.688218 write 0x021b3b thru 0x02b94b (0x9e11 bytes) 000529 1789585242.696120 mapread 0x005ef6 thru 0x013814 (0xd91f bytes) 000530 1789585242.743948 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b94c): Mirror 8: 0x8d985d35 (stale) Mirror 9: 0x8d985d35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b94c): Mirror 8: 0x8d985d35 (stale) Mirror 9: 0x8d985d35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b94c: skipped 000531 1789585243.214112 MIRROR_RESYNC 3 mirrors 000534 1789585244.450992 write 0x007f92 thru 0x00e43e (0x64ad bytes) 000535 1789585244.515957 write 0x01a5e9 thru 0x01cd8a (0x27a2 bytes) 000536 1789585244.526472 mapwrite 0x0378d2 thru 0x03ccc8 (0x53f7 bytes) 000537 1789585244.624339 MIRROR_RESYNC 3 mirrors 000538 1789585245.558093 read 0x023978 thru 0x028533 (0x4bbc bytes) 000539 1789585245.579957 MIRROR_EXTEND to 4 mirrors 000540 1789585246.572931 mapwrite 0x00e6bb thru 0x019af7 (0xb43d bytes) 000541 1789585246.801272 mapread 0x017024 thru 0x01b4ea (0x44c7 bytes) 000542 1789585246.802447 punch from 0x02254d to 0x022b37 (0x5eb bytes) 000546 1789585246.813754 trunc from 0x03ccc9 to 0x00949d (0x3382b bytes) 000547 1789585246.843827 MIRROR_RESYNC 4 mirrors 000548 1789585247.900334 punch from 0x006232 to 0x00949d (0x326c bytes) 000549 1789585247.967136 mapwrite 0x0033ef thru 0x0118aa (0xe4bc bytes) 000550 1789585248.159615 mapread 0x00f0b4 thru 0x0118aa (0x27f7 bytes) 000552 1789585248.162079 trunc from 0x0118ab to 0x03183b (0x1ff91 bytes) 000556 1789585248.186151 trunc from 0x03183c to 0x028d56 (0x8ae5 bytes) 000557 1789585248.211768 mapwrite 0x02670d thru 0x032b56 (0xc44a bytes) 000558 1789585248.389645 punch from 0x006e1e to 0x01527b (0xe45e bytes) 000559 1789585248.398818 mapread 0x002f9f thru 0x00a47f (0x74e1 bytes) 000560 1789585248.425300 mapwrite 0x03dc4b thru 0x03ffff (0x23b5 bytes) 000561 1789585248.573606 trunc from 0x040000 to 0x000629 (0x3f9d6 bytes) 000562 1789585248.601873 punch from 0x0001df to 0x000629 (0x44b bytes) 000563 1789585248.611409 trunc from 0x00062a to 0x009e02 (0x97d9 bytes) 000565 1789585248.632933 punch from 0x0051ae to 0x009e02 (0x4c55 bytes) 000567 1789585248.639708 mapwrite 0x01d621 thru 0x026ebf (0x989f bytes) 000570 1789585248.735963 mapwrite 0x03e298 thru 0x03ffff (0x1d68 bytes) 000571 1789585248.811009 read 0x024209 thru 0x02b66f (0x7467 bytes) 000573 1789585248.819463 read 0x00eaae thru 0x01ae1c (0xc36f bytes) 000574 1789585248.832148 mapread 0x01f282 thru 0x0229fa (0x3779 bytes) 000576 1789585248.833635 mapwrite 0x00bbd9 thru 0x01562f (0x9a57 bytes) 000577 1789585248.882422 read 0x00eef5 thru 0x011d38 (0x2e44 bytes) 000578 1789585248.883090 read 0x024044 thru 0x02e3c0 (0xa37d bytes) 000580 1789585248.889971 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe7cee4a2 (stale) Mirror 9: 0xe7cee4a2 (stale) Mirror 10: 0xe7cee4a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb26f448 Mirror 11: 0xb26f448 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000581 1789585250.104247 mapread 0x02d9a4 thru 0x03cba5 (0xf202 bytes) 000582 1789585250.200850 mapread 0x0273e8 thru 0x035987 (0xe5a0 bytes) 000583 1789585250.245079 trunc from 0x040000 to 0x01fad7 (0x20528 bytes) 000584 1789585250.359529 mapread 0x003b40 thru 0x004a50 (0xf11 bytes) 000585 1789585250.384247 mapwrite 0x00feb9 thru 0x01b46b (0xb5b3 bytes) 000587 1789585250.496834 write 0x00b560 thru 0x00d73f (0x21e0 bytes) 000588 1789585250.499207 trunc from 0x01fad8 to 0x0362f5 (0x1681e bytes) 000589 1789585250.534660 trunc from 0x0362f6 to 0x015352 (0x20fa3 bytes) 000590 1789585250.566512 write 0x031dc8 thru 0x03ffff (0xe238 bytes) 000591 1789585250.573429 mapwrite 0x003d67 thru 0x006c53 (0x2eed bytes) 000592 1789585250.633906 mapwrite 0x005dfa thru 0x007c6e (0x1e75 bytes) 000593 1789585250.711166 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe7cee4a2 (stale) Mirror 9: 0xe7cee4a2 (stale) Mirror 10: 0xe7cee4a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe7cee4a2 (stale) Mirror 9: 0xe7cee4a2 (stale) Mirror 10: 0xe7cee4a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000594 1789585251.169274 punch from 0x019594 to 0x020450 (0x6ebd bytes) 000596 1789585251.175872 punch from 0x022d5f to 0x02a737 (0x79d9 bytes) 000597 1789585251.181036 write 0x0342d7 thru 0x03554a (0x1274 bytes) 000598 1789585251.196021 mapread 0x002d34 thru 0x003d9d (0x106a bytes) 000599 1789585251.210327 punch from 0x018c57 to 0x028311 (0xf6bb bytes) 000600 1789585251.226149 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe7cee4a2 (stale) Mirror 9: 0xe7cee4a2 (stale) Mirror 10: 0xe7cee4a2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf1c15c4a Mirror 11: 0xf1c15c4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000602 1789585252.506038 mapread 0x0233f0 thru 0x026b0b (0x371c bytes) 000604 1789585252.580540 punch from 0x03c5e0 to 0x03ffff (0x3a20 bytes) 000605 1789585252.686897 mapwrite 0x01c149 thru 0x029b37 (0xd9ef bytes) 000606 1789585252.775075 trunc from 0x040000 to 0x03d28f (0x2d70 bytes) 000607 1789585252.834989 punch from 0x02e30c to 0x034652 (0x6347 bytes) 000608 1789585252.842867 read 0x001c1e thru 0x01193a (0xfd1d bytes) 000609 1789585252.868050 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d290): Mirror 8: 0xa362ae19 (stale) Mirror 9: 0xa362ae19 (stale) Mirror 11: 0x95e98010 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d290): Mirror 8: 0xa362ae19 (stale) Mirror 9: 0xa362ae19 (stale) Mirror 11: 0x95e98010 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d290: skipped 000612 1789585253.471707 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d290): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d290): Mirror 8: 0xa362ae19 (stale) Mirror 9: 0xa362ae19 (stale) Mirror 11: 0x95e98010 (stale) CRC-32 checksum value for chunk [0, 0x3d290): Mirror 1: 0x541ce72f Mirror 12: 0x541ce72f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d290: skipped 000615 1789585254.996196 MIRROR_RESYNC 5 mirrors 000616 1789585256.153644 read 0x026600 thru 0x032cd3 (0xc6d4 bytes) 000617 1789585256.169119 punch from 0x0304f8 to 0x03ca06 (0xc50f bytes) 000618 1789585256.241972 mapwrite 0x025d1b thru 0x02adf5 (0x50db bytes) 000621 1789585256.348693 mapwrite 0x017a8e thru 0x01e68c (0x6bff bytes) 000622 1789585256.449176 mapread 0x00340f thru 0x003d62 (0x954 bytes) 000623 1789585256.469054 read 0x03276c thru 0x0398d9 (0x716e bytes) 000624 1789585256.483079 trunc from 0x03d290 to 0x022ca0 (0x1a5ef bytes) 000626 1789585256.503214 read 0x022364 thru 0x022ca0 (0x93d bytes) 000627 1789585256.503935 mapwrite 0x0213c1 thru 0x0265dd (0x521d bytes) 000630 1789585256.641779 write 0x01271f thru 0x01b221 (0x8b03 bytes) 000631 1789585256.644892 read 0x01dbe8 thru 0x0265dd (0x89f6 bytes) 000632 1789585256.648034 trunc from 0x0265de to 0x00ed8e (0x1784f bytes) 000633 1789585256.687895 punch from 0x0069ad to 0x00796e (0xfc2 bytes) 000635 1789585256.696894 read 0x006057 thru 0x00ed8e (0x8d38 bytes) 000636 1789585256.704258 punch from 0x00ae8f to 0x00ed8e (0x3f00 bytes) 000637 1789585256.711419 read 0x000dff thru 0x00b009 (0xa20b bytes) 000639 1789585256.731824 punch from 0x00c66f to 0x00ed8e (0x2720 bytes) 000640 1789585256.738649 read 0x00b3b7 thru 0x00ed8e (0x39d8 bytes) 000641 1789585256.744826 punch from 0x004332 to 0x00a94e (0x661d bytes) 000642 1789585256.749840 read 0x003adc thru 0x007702 (0x3c27 bytes) 000643 1789585256.760104 mapwrite 0x0041f9 thru 0x008363 (0x416b bytes) 000644 1789585256.815133 trunc from 0x00ed8f to 0x006530 (0x885e bytes) 000648 1789585256.845170 punch from 0x005e27 to 0x006530 (0x70a bytes) 000650 1789585256.854321 mapwrite 0x01f1af thru 0x02dc06 (0xea58 bytes) 000654 1789585256.982537 mapwrite 0x0338d7 thru 0x03541a (0x1b44 bytes) 000655 1789585257.060190 write 0x03656e thru 0x03f70d (0x91a0 bytes) 000656 1789585257.063021 write 0x01649e thru 0x024816 (0xe379 bytes) 000657 1789585257.064692 mapwrite 0x024c58 thru 0x02929c (0x4645 bytes) 000658 1789585257.118720 mapwrite 0x00ab92 thru 0x00e85f (0x3cce bytes) 000659 1789585257.212708 read 0x026b72 thru 0x027862 (0xcf1 bytes) 000660 1789585257.213944 write 0x01253f thru 0x01981d (0x72df bytes) 000662 1789585257.249685 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 8: 0x4893832b (stale) Mirror 9: 0x4893832b (stale) Mirror 11: 0x4893832b (stale) Mirror 12: 0x4893832b (stale) CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 1: 0xee39e1ea Mirror 13: 0xee39e1ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f70e: skipped 000663 1789585258.304187 mapwrite 0x018584 thru 0x0213c0 (0x8e3d bytes) 000664 1789585258.564666 punch from 0x01477e to 0x0226f8 (0xdf7b bytes) 000667 1789585258.580332 mapwrite 0x0132ff thru 0x01ecfd (0xb9ff bytes) 000668 1789585258.660598 write 0x02e57a thru 0x03c4d0 (0xdf57 bytes) 000669 1789585258.673210 read 0x01262d thru 0x01bce3 (0x96b7 bytes) 000672 1789585258.675065 write 0x032d75 thru 0x03d529 (0xa7b5 bytes) 000673 1789585258.688183 MIRROR_RESYNC 6 mirrors 000674 1789585259.913745 punch from 0x03bed5 to 0x03f70d (0x3839 bytes) 000675 1789585259.987934 write 0x02b0a0 thru 0x036289 (0xb1ea bytes) 000680 1789585260.006080 mapwrite 0x020878 thru 0x02f599 (0xed22 bytes) 000681 1789585260.105080 mapread 0x034671 thru 0x03f70d (0xb09d bytes) 000682 1789585260.121732 punch from 0x00cf91 to 0x011d01 (0x4d71 bytes) 000683 1789585260.133247 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 8: 0x6d1a6688 (stale) Mirror 9: 0x6d1a6688 (stale) Mirror 11: 0x6d1a6688 (stale) Mirror 12: 0x6d1a6688 (stale) Mirror 13: 0x6d1a6688 (stale) CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 1: 0xe86a015f Mirror 14: 0xe86a015f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f70e: skipped 000684 1789585261.535399 write 0x005239 thru 0x00c785 (0x754d bytes) 000686 1789585261.667473 punch from 0x02738d to 0x02e174 (0x6de8 bytes) 000688 1789585261.721391 write 0x014c0a thru 0x015d9d (0x1194 bytes) 000689 1789585261.766860 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 8: 0x6d1a6688 (stale) Mirror 9: 0x6d1a6688 (stale) Mirror 11: 0x6d1a6688 (stale) Mirror 12: 0x6d1a6688 (stale) Mirror 13: 0x6d1a6688 (stale) Mirror 14: 0xe86a015f (stale) CRC-32 checksum value for chunk [0, 0x3f70e): Mirror 1: 0x829377ef Mirror 15: 0x829377ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f70e: skipped 000690 1789585263.478893 write 0x03ce43 thru 0x03ffff (0x31bd bytes) 000691 1789585263.624297 write 0x00b3b1 thru 0x00cd98 (0x19e8 bytes) 000694 1789585263.670583 MIRROR_RESYNC 8 mirrors 000695 1789585265.848502 trunc from 0x040000 to 0x00c9dd (0x33622 bytes) 000697 1789585265.931813 trunc from 0x00c9de to 0x005cdb (0x6d02 bytes) 000698 1789585265.964062 punch from 0x0029d6 to 0x005cdb (0x3306 bytes) 000700 1789585265.970922 punch from 0x0025be to 0x005cdb (0x371e bytes) 000701 1789585265.977210 read 0x0036ec thru 0x005cdb (0x25f0 bytes) 000702 1789585265.993896 MIRROR_RESYNC 8 mirrors 000703 1789585267.414062 punch from 0x000380 to 0x005cdb (0x595c bytes) 000704 1789585267.486139 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5cdc): Mirror 8: 0x5e00f571 (stale) Mirror 9: 0x5e00f571 (stale) Mirror 11: 0x5e00f571 (stale) Mirror 13: 0x5e00f571 (stale) Mirror 14: 0x5e00f571 (stale) Mirror 15: 0x5e00f571 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5cdc): Mirror 8: 0x5e00f571 (stale) Mirror 9: 0x5e00f571 (stale) Mirror 11: 0x5e00f571 (stale) Mirror 13: 0x5e00f571 (stale) Mirror 14: 0x5e00f571 (stale) Mirror 15: 0x5e00f571 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5cdc: skipped 000705 1789585267.894070 punch from 0x003b4a to 0x005cdb (0x2192 bytes) 000708 1789585267.901581 read 0x005401 thru 0x005cdb (0x8db bytes) 000709 1789585267.906559 write 0x020cb2 thru 0x02937b (0x86ca bytes) 000710 1789585267.910163 punch from 0x0001c7 to 0x009a97 (0x98d1 bytes) 000711 1789585267.924694 mapread 0x006f33 thru 0x015887 (0xe955 bytes) 000712 1789585267.953416 write 0x038334 thru 0x03ffff (0x7ccc bytes) 000713 1789585267.955174 write 0x03e51f thru 0x03ffff (0x1ae1 bytes) 000714 1789585267.956978 mapwrite 0x02ee50 thru 0x036449 (0x75fa bytes) 000715 1789585268.036027 MIRROR_RESYNC 7 mirrors 000716 1789585270.606431 mapwrite 0x016920 thru 0x01fece (0x95af bytes) 000717 1789585270.801364 punch from 0x0068bb to 0x00d7a6 (0x6eec bytes) 000718 1789585270.813928 MIRROR_RESYNC 7 mirrors 000723 1789585272.948450 write 0x03130e thru 0x034dde (0x3ad1 bytes) 000724 1789585273.070482 punch from 0x01f24f to 0x02ac93 (0xba45 bytes) 000725 1789585273.096220 trunc from 0x040000 to 0x003aba (0x3c545 bytes) 000726 1789585273.136908 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3abb): Mirror 8: 0x98c17265 (stale) Mirror 9: 0x98c17265 (stale) Mirror 11: 0x98c17265 (stale) Mirror 13: 0x98c17265 (stale) Mirror 14: 0x98c17265 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3abb): Mirror 8: 0x98c17265 (stale) Mirror 9: 0x98c17265 (stale) Mirror 11: 0x98c17265 (stale) Mirror 13: 0x98c17265 (stale) Mirror 14: 0x98c17265 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3abb: skipped 000727 1789585273.569137 write 0x015c0b thru 0x01bba7 (0x5f9d bytes) 000729 1789585273.573442 read 0x011cde thru 0x013a27 (0x1d4a bytes) 000730 1789585273.583882 mapread 0x00e6da thru 0x013444 (0x4d6b bytes) 000731 1789585273.627885 read 0x000d1f thru 0x010a59 (0xfd3b bytes) 000732 1789585273.642497 read 0x00693b thru 0x012487 (0xbb4d bytes) 000733 1789585273.643506 mapread 0x0067ea thru 0x00e00a (0x7821 bytes) 000734 1789585273.650873 MIRROR_RESYNC 6 mirrors 000735 1789585275.618227 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000736 1789585276.236608 read 0x00558d thru 0x007b01 (0x2575 bytes) 000737 1789585276.255262 read 0x0119b4 thru 0x01bba7 (0xa1f4 bytes) 000738 1789585276.278236 write 0x0302c0 thru 0x037734 (0x7475 bytes) 000739 1789585276.365588 mapwrite 0x001909 thru 0x00ebb1 (0xd2a9 bytes) 000740 1789585276.528603 read 0x010992 thru 0x01bce9 (0xb358 bytes) 000741 1789585276.550807 mapread 0x01821c thru 0x019071 (0xe56 bytes) 000742 1789585276.554441 mapwrite 0x03b5a7 thru 0x03ffff (0x4a59 bytes) 000743 1789585276.673739 mapread 0x00ce6f thru 0x015a81 (0x8c13 bytes) 000744 1789585276.674853 read 0x01d506 thru 0x02c472 (0xef6d bytes) 000745 1789585276.677428 trunc from 0x040000 to 0x015037 (0x2afc8 bytes) 000746 1789585276.703535 write 0x03ec48 thru 0x03ffff (0x13b8 bytes) 000749 1789585276.707930 mapread 0x023939 thru 0x030033 (0xc6fb bytes) 000750 1789585276.731692 punch from 0x00f45a to 0x0119ed (0x2594 bytes) 000751 1789585276.752417 read 0x039bda thru 0x03ffff (0x6426 bytes) 000752 1789585276.754817 mapwrite 0x03211e thru 0x03c81d (0xa700 bytes) 000753 1789585276.838523 read 0x00d0e8 thru 0x017989 (0xa8a2 bytes) 000756 1789585276.851577 mapread 0x03c00d thru 0x03ffff (0x3ff3 bytes) 000757 1789585276.853313 trunc from 0x040000 to 0x028b65 (0x1749a bytes) 000759 1789585276.882300 write 0x02c5aa thru 0x033082 (0x6ad9 bytes) 000760 1789585276.885439 write 0x033189 thru 0x03a86b (0x76e3 bytes) 000761 1789585276.887839 write 0x011720 thru 0x02130e (0xfbef bytes) 000763 1789585276.892554 write 0x01bd43 thru 0x023fde (0x829c bytes) 000764 1789585276.895603 trunc from 0x03a86c to 0x02e969 (0xbf02 bytes) 000765 1789585276.918614 mapread 0x00e2d3 thru 0x0169f1 (0x871f bytes) 000768 1789585276.922816 trunc from 0x02e96a to 0x017a9a (0x16ecf bytes) 000770 1789585276.951245 punch from 0x00ce38 to 0x012460 (0x5629 bytes) 000771 1789585276.970265 mapwrite 0x037d4b thru 0x03ffff (0x82b5 bytes) 000773 1789585277.175339 write 0x035d88 thru 0x03ffff (0xa278 bytes) 000776 1789585277.184878 trunc from 0x040000 to 0x03f927 (0x6d8 bytes) 000777 1789585277.208912 read 0x023326 thru 0x02c4a3 (0x917e bytes) 000779 1789585277.225903 write 0x0184b7 thru 0x027bbe (0xf708 bytes) 000780 1789585277.228591 mapread 0x01d0a7 thru 0x02c4cf (0xf429 bytes) 000782 1789585277.230603 mapwrite 0x00eb1f thru 0x019744 (0xac26 bytes) 000783 1789585277.288751 mapwrite 0x016f18 thru 0x02572a (0xe813 bytes) 000784 1789585277.344535 mapwrite 0x032b0e thru 0x03ffff (0xd4f2 bytes) 000785 1789585277.425507 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xcbeadfc0 (stale) Mirror 9: 0xcbeadfc0 (stale) Mirror 13: 0xcbeadfc0 (stale) Mirror 14: 0xcbeadfc0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6caca6b6 Mirror 15: 0x6caca6b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1789585278.976723 trunc from 0x040000 to 0x03985a (0x67a5 bytes) 000789 1789585279.089032 mapread 0x038135 thru 0x03985a (0x1726 bytes) 000790 1789585279.102880 read 0x001adc thru 0x0053fb (0x3920 bytes) 000791 1789585279.116070 read 0x021b97 thru 0x02e504 (0xc96e bytes) 000792 1789585279.133958 MIRROR_RESYNC 6 mirrors 000793 1789585281.038911 mapread 0x00b4f5 thru 0x013b5d (0x8669 bytes) 000794 1789585281.085526 punch from 0x004c39 to 0x00a47d (0x5845 bytes) 000795 1789585281.212473 write 0x02934b thru 0x02ff99 (0x6c4f bytes) 000796 1789585281.224330 punch from 0x0109f1 to 0x0120ac (0x16bc bytes) 000797 1789585281.238338 read 0x009020 thru 0x018694 (0xf675 bytes) 000798 1789585281.248457 trunc from 0x03985b to 0x032c1b (0x6c3f bytes) 000799 1789585281.271384 mapread 0x02aa47 thru 0x02ccc0 (0x227a bytes) 000800 1789585281.272511 trunc from 0x032c1c to 0x001a0a (0x31211 bytes) 000802 1789585281.291481 read 0x0006e5 thru 0x001a0a (0x1326 bytes) 000803 1789585281.302839 trunc from 0x001a0b to 0x0030d8 (0x16ce bytes) 000804 1789585281.360890 punch from 0x0013a6 to 0x0030d8 (0x1d33 bytes) 000805 1789585281.378800 trunc from 0x0030d9 to 0x03d804 (0x3a72c bytes) 000807 1789585281.434427 MIRROR_RESYNC 6 mirrors 000808 1789585283.110527 mapread 0x0335c2 thru 0x0343f9 (0xe38 bytes) 000809 1789585283.127210 write 0x029485 thru 0x037138 (0xdcb4 bytes) 000811 1789585283.231050 read 0x03191a thru 0x03d804 (0xbeeb bytes) 000812 1789585283.242065 read 0x0190ff thru 0x01d36b (0x426d bytes) 000813 1789585283.258612 mapread 0x027323 thru 0x035b09 (0xe7e7 bytes) 000814 1789585283.259872 trunc from 0x03d805 to 0x01998c (0x23e78 bytes) 000816 1789585283.281299 mapread 0x00ff7a thru 0x01998c (0x9a13 bytes) 000818 1789585283.290456 mapread 0x009e62 thru 0x010fb4 (0x7153 bytes) 000819 1789585283.291708 punch from 0x008c75 to 0x00d3e8 (0x4774 bytes) 000820 1789585283.314998 read 0x00ea61 thru 0x0135f8 (0x4b98 bytes) 000821 1789585283.315975 write 0x024a11 thru 0x031054 (0xc644 bytes) 000822 1789585283.325511 punch from 0x006852 to 0x00812b (0x18da bytes) 000823 1789585283.358623 read 0x02da83 thru 0x031054 (0x35d2 bytes) 000825 1789585283.363697 write 0x00f165 thru 0x010a73 (0x190f bytes) 000827 1789585283.374109 write 0x03482a thru 0x03d059 (0x8830 bytes) 000828 1789585283.378432 mapwrite 0x021066 thru 0x02e3fe (0xd399 bytes) 000829 1789585283.487611 punch from 0x00875b to 0x01475a (0xc000 bytes) 000830 1789585283.499428 mapwrite 0x01a62a thru 0x02556f (0xaf46 bytes) 000832 1789585283.604091 write 0x01de17 thru 0x02946b (0xb655 bytes) 000833 1789585283.605526 trunc from 0x03d05a to 0x01122e (0x2be2b bytes) 000834 1789585283.667956 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1122f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1122f): Mirror 8: 0x15689d1a (stale) Mirror 9: 0x15689d1a (stale) Mirror 13: 0x15689d1a (stale) Mirror 14: 0x15689d1a (stale) Mirror 15: 0x15689d1a (stale) CRC-32 checksum value for chunk [0, 0x1122f): Mirror 1: 0x15689d1a Mirror 16: 0x15689d1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1122f: skipped 000837 1789585285.219250 punch from 0x00b9a4 to 0x00dc83 (0x22e0 bytes) 000838 1789585285.305794 trunc from 0x01122f to 0x013b58 (0x292a bytes) 000839 1789585285.331269 write 0x015b09 thru 0x022f94 (0xd48c bytes) 000841 1789585285.334205 punch from 0x01dd9a to 0x022f94 (0x51fb bytes) 000845 1789585285.363616 read 0x00fa52 thru 0x015d3b (0x62ea bytes) 000847 1789585285.379190 mapread 0x0205b1 thru 0x022f94 (0x29e4 bytes) 000848 1789585285.405062 read 0x013ca0 thru 0x017289 (0x35ea bytes) 000849 1789585285.406128 mapread 0x011763 thru 0x01fddb (0xe679 bytes) 000850 1789585285.408259 write 0x023df9 thru 0x02f78d (0xb995 bytes) 000851 1789585285.413053 punch from 0x02f3e3 to 0x02f78d (0x3ab bytes) 000852 1789585285.429293 write 0x02f8c2 thru 0x03bf59 (0xc698 bytes) 000853 1789585285.445202 write 0x0290e7 thru 0x036c79 (0xdb93 bytes) 000854 1789585285.448281 punch from 0x03554b to 0x03bf59 (0x6a0f bytes) 000857 1789585285.465483 mapwrite 0x026b4e thru 0x02c4da (0x598d bytes) 000858 1789585285.541205 mapwrite 0x00caa1 thru 0x019f3d (0xd49d bytes) 000859 1789585285.699682 write 0x006267 thru 0x00a23d (0x3fd7 bytes) 000860 1789585285.700937 mapwrite 0x020ab8 thru 0x0305be (0xfb07 bytes) 000861 1789585285.808222 mapwrite 0x014dc3 thru 0x01cd87 (0x7fc5 bytes) 000863 1789585285.901631 trunc from 0x03bf5a to 0x02235c (0x19bfd bytes) 000864 1789585285.947277 mapread 0x005197 thru 0x00d328 (0x8192 bytes) 000865 1789585285.954548 read 0x014d7f thru 0x01e1aa (0x942c bytes) 000866 1789585285.960657 mapread 0x0012bf thru 0x00cf6d (0xbcaf bytes) 000868 1789585285.979054 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2235d): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2235d): Mirror 8: 0x601a78f9 (stale) Mirror 9: 0x601a78f9 (stale) Mirror 13: 0x601a78f9 (stale) Mirror 14: 0x601a78f9 (stale) Mirror 15: 0x601a78f9 (stale) Mirror 16: 0x601a78f9 (stale) CRC-32 checksum value for chunk [0, 0x2235d): Mirror 1: 0xc211190c Mirror 17: 0xc211190c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2235d: skipped 000869 1789585287.256722 read 0x018f8d thru 0x020dfa (0x7e6e bytes) 000870 1789585287.283884 mapread 0x011a85 thru 0x0153c4 (0x3940 bytes) 000871 1789585287.372573 punch from 0x00d7b3 to 0x00ffcf (0x281d bytes) 000874 1789585287.576543 punch from 0x0094b0 to 0x00b81d (0x236e bytes) 000876 1789585287.603878 mapread 0x005f93 thru 0x012922 (0xc990 bytes) 000877 1789585287.636605 read 0x005274 thru 0x00aac8 (0x5855 bytes) 000879 1789585287.637749 mapwrite 0x02178b thru 0x023af2 (0x2368 bytes) 000881 1789585287.789073 mapread 0x00658e thru 0x008bbd (0x2630 bytes) 000884 1789585287.796330 MIRROR_RESYNC 8 mirrors 000885 1789585289.498266 punch from 0x01462e to 0x017964 (0x3337 bytes) 000886 1789585289.578184 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23af3): Mirror 8: 0x37e0caa8 (stale) Mirror 9: 0x37e0caa8 (stale) Mirror 13: 0x37e0caa8 (stale) Mirror 14: 0x37e0caa8 (stale) Mirror 15: 0x37e0caa8 (stale) Mirror 17: 0x37e0caa8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23af3): Mirror 8: 0x37e0caa8 (stale) Mirror 9: 0x37e0caa8 (stale) Mirror 13: 0x37e0caa8 (stale) Mirror 14: 0x37e0caa8 (stale) Mirror 15: 0x37e0caa8 (stale) Mirror 17: 0x37e0caa8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23af3: skipped 000887 1789585290.286418 mapread 0x00acbc thru 0x00af17 (0x25c bytes) 000888 1789585290.301671 mapread 0x01c84d thru 0x023af2 (0x72a6 bytes) 000890 1789585290.332957 mapread 0x019763 thru 0x01be4f (0x26ed bytes) 000891 1789585290.334110 trunc from 0x023af3 to 0x02e48c (0xa99a bytes) 000892 1789585290.357727 read 0x015d08 thru 0x02513b (0xf434 bytes) 000893 1789585290.374284 mapwrite 0x01275d thru 0x015573 (0x2e17 bytes) 000894 1789585290.465625 trunc from 0x02e48d to 0x03d13b (0xecaf bytes) 000895 1789585290.507545 punch from 0x030e2a to 0x03d13b (0xc312 bytes) 000897 1789585290.521049 write 0x01ac1f thru 0x01c431 (0x1813 bytes) 000898 1789585290.523604 mapwrite 0x03f845 thru 0x03ffff (0x7bb bytes) 000900 1789585290.650308 mapwrite 0x022eb3 thru 0x024ec0 (0x200e bytes) 000901 1789585290.696910 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x4b43ddf1 (stale) Mirror 9: 0x4b43ddf1 (stale) Mirror 13: 0x4b43ddf1 (stale) Mirror 14: 0x4b43ddf1 (stale) Mirror 15: 0x4b43ddf1 (stale) Mirror 17: 0x4b43ddf1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc587c6b3 Mirror 18: 0xc587c6b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000902 1789585292.233611 punch from 0x027ebe to 0x033d54 (0xbe97 bytes) 000904 1789585292.380907 write 0x03176b thru 0x03fd58 (0xe5ee bytes) 000905 1789585292.398457 punch from 0x01757d to 0x01fb6d (0x85f1 bytes) 000908 1789585292.414474 mapwrite 0x018f49 thru 0x020277 (0x732f bytes) 000910 1789585292.502449 mapread 0x037219 thru 0x03ffff (0x8de7 bytes) 000911 1789585292.504519 mapwrite 0x022228 thru 0x03038a (0xe163 bytes) 000912 1789585292.592570 mapwrite 0x03ea7b thru 0x03ffff (0x1585 bytes) 000913 1789585292.636214 write 0x031e24 thru 0x031e40 (0x1d bytes) 000914 1789585292.638308 punch from 0x02bc0e to 0x02ef4f (0x3342 bytes) 000915 1789585292.665978 MIRROR_RESYNC 8 mirrors 000916 1789585294.498277 punch from 0x00b8f9 to 0x0197bd (0xdec5 bytes) 000917 1789585294.608968 MIRROR_RESYNC 8 mirrors 000918 1789585297.476363 mapwrite 0x0115b4 thru 0x01c195 (0xabe2 bytes) 000922 1789585297.707116 write 0x01e4c1 thru 0x023b28 (0x5668 bytes) 000923 1789585297.709256 mapread 0x01e98b thru 0x026278 (0x78ee bytes) 000924 1789585297.727468 trunc from 0x040000 to 0x00ba3a (0x345c5 bytes) 000925 1789585297.758964 mapwrite 0x03a14a thru 0x03ffff (0x5eb6 bytes) 000926 1789585297.886463 write 0x02b0ab thru 0x03abe8 (0xfb3e bytes) 000928 1789585297.898834 punch from 0x034863 to 0x03c7d1 (0x7f6f bytes) 000930 1789585297.917266 write 0x02a634 thru 0x02dc50 (0x361d bytes) 000932 1789585297.925049 mapwrite 0x0195a0 thru 0x019d6d (0x7ce bytes) 000933 1789585298.000059 read 0x017306 thru 0x01f76f (0x846a bytes) 000934 1789585298.001588 mapwrite 0x02ef7c thru 0x03b45f (0xc4e4 bytes) 000935 1789585298.089865 mapwrite 0x012808 thru 0x01cfcf (0xa7c8 bytes) 000936 1789585298.180790 mapread 0x0191b0 thru 0x01c362 (0x31b3 bytes) 000937 1789585298.181789 mapread 0x036712 thru 0x03d9ec (0x72db bytes) 000938 1789585298.206953 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x11c5eee5 (stale) Mirror 9: 0x11c5eee5 (stale) Mirror 14: 0x11c5eee5 (stale) Mirror 15: 0x11c5eee5 (stale) Mirror 17: 0x11c5eee5 (stale) Mirror 18: 0x11c5eee5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x11c5eee5 (stale) Mirror 9: 0x11c5eee5 (stale) Mirror 14: 0x11c5eee5 (stale) Mirror 15: 0x11c5eee5 (stale) Mirror 17: 0x11c5eee5 (stale) Mirror 18: 0x11c5eee5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000939 1789585298.983149 punch from 0x02bb66 to 0x02e9a3 (0x2e3e bytes) 000940 1789585299.003342 MIRROR_RESYNC 7 mirrors 000941 1789585301.531794 punch from 0x015f15 to 0x01ddb8 (0x7ea4 bytes) 000942 1789585301.610328 mapwrite 0x00e5d7 thru 0x00e762 (0x18c bytes) 000943 1789585301.774175 write 0x02b4c2 thru 0x02db17 (0x2656 bytes) 000945 1789585301.806517 trunc from 0x040000 to 0x035096 (0xaf69 bytes) 000946 1789585301.828981 mapread 0x00d9e5 thru 0x01559b (0x7bb7 bytes) 000948 1789585301.846962 write 0x021c64 thru 0x02c0f4 (0xa491 bytes) 000949 1789585301.855115 trunc from 0x035097 to 0x03f8b4 (0xa81e bytes) 000950 1789585301.882181 mapwrite 0x01eb86 thru 0x021f13 (0x338e bytes) 000951 1789585301.939125 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f8b5): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8b5): Mirror 8: 0x76ed3805 (stale) Mirror 9: 0x76ed3805 (stale) Mirror 14: 0x76ed3805 (stale) Mirror 15: 0x76ed3805 (stale) Mirror 17: 0x76ed3805 (stale) Mirror 18: 0x76ed3805 (stale) CRC-32 checksum value for chunk [0, 0x3f8b5): Mirror 1: 0x7c806928 Mirror 19: 0x7c806928 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8b5: skipped 000952 1789585303.436732 punch from 0x020ab8 to 0x02a6a2 (0x9beb bytes) 000953 1789585303.542203 mapread 0x0165f1 thru 0x018afb (0x250b bytes) 000956 1789585303.577289 punch from 0x00da42 to 0x0116a0 (0x3c5f bytes) 000959 1789585303.594897 mapwrite 0x002a97 thru 0x004464 (0x19ce bytes) 000960 1789585303.684326 mapwrite 0x00228a thru 0x0110c3 (0xee3a bytes) 000961 1789585303.797017 read 0x0108bc thru 0x018795 (0x7eda bytes) 000962 1789585303.802429 mapwrite 0x0329de thru 0x032cf9 (0x31c bytes) 000964 1789585303.911858 mapwrite 0x0327cb thru 0x03a4d2 (0x7d08 bytes) 000965 1789585304.006441 trunc from 0x03f8b5 to 0x00e08d (0x31827 bytes) 000966 1789585304.038985 mapread 0x005b76 thru 0x00e08d (0x8518 bytes) 000967 1789585304.041265 trunc from 0x00e08e to 0x02724d (0x191c0 bytes) 000968 1789585304.062188 write 0x0389ea thru 0x03ffff (0x7616 bytes) 000969 1789585304.066030 trunc from 0x040000 to 0x0218c5 (0x1e73a bytes) 000970 1789585304.085224 write 0x0051db thru 0x007ec9 (0x2cef bytes) 000972 1789585304.087333 trunc from 0x0218c6 to 0x020bb2 (0xd13 bytes) 000973 1789585304.135576 read 0x002803 thru 0x006e42 (0x4640 bytes) 000974 1789585304.136736 trunc from 0x020bb3 to 0x03fa95 (0x1eee3 bytes) 000975 1789585304.178523 write 0x02c6a9 thru 0x030c93 (0x45eb bytes) 000976 1789585304.227296 punch from 0x024910 to 0x02ec0c (0xa2fd bytes) 000977 1789585304.264328 mapwrite 0x02dd42 thru 0x02f41b (0x16da bytes) 000978 1789585304.369316 punch from 0x02d6a3 to 0x02eb0e (0x146c bytes) 000979 1789585304.391554 read 0x03acf7 thru 0x03fa95 (0x4d9f bytes) 000980 1789585304.392298 read 0x014fcc thru 0x024488 (0xf4bd bytes) 000983 1789585304.414238 write 0x00083b thru 0x002e39 (0x25ff bytes) 000984 1789585304.416811 punch from 0x00d5da to 0x015543 (0x7f6a bytes) 000985 1789585304.434792 mapwrite 0x015546 thru 0x0171cf (0x1c8a bytes) 000988 1789585304.560602 mapread 0x01feec thru 0x02c0dd (0xc1f2 bytes) 000989 1789585304.564555 trunc from 0x03fa96 to 0x00ebe2 (0x30eb3 bytes) 000990 1789585304.627977 read 0x00a180 thru 0x00af84 (0xe05 bytes) 000991 1789585304.630016 mapread 0x001514 thru 0x00ebe2 (0xd6cf bytes) 000992 1789585304.645340 punch from 0x005af2 to 0x0066bf (0xbce bytes) 000994 1789585304.666294 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xebe3): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xebe3): Mirror 8: 0x66d6f7a0 (stale) Mirror 9: 0x66d6f7a0 (stale) Mirror 14: 0x66d6f7a0 (stale) Mirror 15: 0x66d6f7a0 (stale) Mirror 17: 0x66d6f7a0 (stale) Mirror 18: 0x66d6f7a0 (stale) Mirror 19: 0x507fec30 (stale) CRC-32 checksum value for chunk [0, 0xebe3): Mirror 1: 0x1aa935a9 Mirror 20: 0x1aa935a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xebe3: skipped 000996 1789585306.018776 punch from 0x00de3a to 0x00ebe2 (0xda9 bytes) 000997 1789585306.130233 mapread 0x003284 thru 0x003c1d (0x99a bytes) 001000 1789585306.150259 MIRROR_RESYNC 9 mirrors All operations completed A-OK! PASS 70a (141s) == sanity-flr test 71: check mirror extend parent fid ==== 15:01:54 (1789585314) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 15:01:58 (1789585318) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.302218 s, 6.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17054 s, 6.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 15:02:08 (1789585328) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7944960 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1464576 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3496544 .. failed Extending file size to 8990240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8496736 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9164512 .. Extending file size to 8506400 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9278816 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 57376 57377 57380 57381 57384 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) 192.168.204.142@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:03:35 (1789585415) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1260448 .. Extending file size to 6015808 .. done Extending file size to 3282080 .. Extending file size to 6667584 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7939136 .. Extending file size to 8204992 .. Extending file size to 8590720 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8896480 .. done Extending file size to 9002304 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8597792 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9325376 .. Waiting 58633 58634 58636 58638 58639 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) 192.168.204.142@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:04:53 (1789585493) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170088 s, 12.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.359039 s, 5.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcdfdf2d1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:05:15 (1789585515) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121272 s, 17.3 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 15:05:24 (1789585524) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:05:34 (1789585534) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.295797 s, 7.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:05:44 (1789585544) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18867 s, 11.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:05:52 (1789585552) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2632 s, 8.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:06:26 (1789585586) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg442-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124025 s, 8.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:07:26 (1789585646) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg442-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.276322 s, 7.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:08:32 (1789585712) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg442-server mds-ost sync done. Creating new pool test_204f oleg442-server: Pool lustre.test_204f created Adding targets to pool oleg442-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg442-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.45116 s, 4.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg442-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg442-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg442-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:09:57 (1789585797) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:10:05 (1789585805) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:10:13 (1789585813) oleg442-server: Pool lustre.test_206 created Adding targets to pool oleg442-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg442-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg442-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg442-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg442-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:10:40 (1789585840) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953503 s, 11.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393904 s, 26.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x459ede81 Mirror 3: 0x459ede81 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:10:51 (1789585851) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:10:54 (1789585854) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:10:58 (1789585858) Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0922155 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2120000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x930000 Read this (4 bytes): BBBB 192.168.204.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg442-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:11:35 (1789585895) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0243687 s, 336 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:11:41 (1789585901) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430222 s, 24.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f8 lmm_fid: [0x200000404:0x1f8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f8 lmm_fid: [0x200000404:0x1f8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834424 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 15:11:43.000000000 -0400 Modify: 2026-09-16 15:11:43.000000000 -0400 Change: 2026-09-16 15:11:44.000000000 -0400 Birth: 2026-09-16 15:11:44.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:11:49 (1789585909) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:11:52 (1789585912) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.735887 s, 55.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0712652 s, 57.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 16 15:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 16 15:11 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:12:01 (1789585921) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:12:08 (1789585928) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1969 sec =========== 15:12:18 (1789585938) === sanity-flr: start cleanup 15:12:20 (1789585940) === === sanity-flr: finish cleanup 15:12:27 (1789585947) ===