-----============= acceptance-small: sanity-flr ============----- Wed Sep 9 12:06:04 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg239-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:06:24 (1788969984) === oleg239-client.virtnet: executing check_config_client /mnt/lustre oleg239-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg239-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f0d08e4d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f0d08e4d000.idle_timeout=debug disable quota as required oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:06:57 (1788970017) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7609] == sanity-flr test 0a: lfs mirror create with -N option == 12:07:00 (1788970020) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:07:17 (1788970037) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:07:20 (1788970040) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:07:24 (1788970044) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:07:48 (1788970068) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:07:51 (1788970071) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:07:55 (1788970075) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f0d08e4d000.stats=clear osc.lustre-OST0001-osc-ffff9f0d08e4d000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:08:02 (1788970082) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:08:14 (1788970094) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:08:23 (1788970103) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:08:28 (1788970108) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (11s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:08:39 (1788970119) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (13s) == sanity-flr test 2: create components from existing files ========================================================== 12:08:52 (1788970132) PASS 2 (11s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:09:03 (1788970143) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:09:07 (1788970147) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:09:16 (1788970156) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:09:27 (1788970167) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:09:35 (1788970175) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.654333 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103695 s, 10.1 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:09:47 (1788970187) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.30427 s, 34.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:09:57 (1788970197) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:10:08 (1788970208) Stopping /mnt/lustre-ost1 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0d08e4d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0d08e4d000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg239-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg239-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0d08e4d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0d08e4d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg239-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (139s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:12:28 (1788970348) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.227282 s, 18.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0d08e4d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0d08e4d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg239-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:13:11 (1788970391) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0d08e4d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0d08e4d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg239-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0d08e4d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0d08e4d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg239-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (73s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:14:24 (1788970464) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.47027 s, 9.6 MB/s -rw-r--r-- 1 root root 52428800 Sep 9 12:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.138s user 0m0.000s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0d08e4d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0d08e4d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.112s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg239-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg239-server oleg239-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0d08e4d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0d08e4d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.054s user 0m0.000s sys 0m0.160s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg239-server: oleg239-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg239-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg239-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0d08e4d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0d08e4d000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:16:00 (1788970560) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:16:03 (1788970563) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:16:07 (1788970567) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:16:10 (1788970570) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 12:16:18 (1788970578) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.393742 s, 32.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:16:32 (1788970592) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173106 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171287 s, 61.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:16:40 (1788970600) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:16:48 (1788970608) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00136578 s, 750 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00209218 s, 489 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011328 s, 904 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113673 s, 901 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 12:16:57 (1788970617) create test files with size 15747083 .. 01ae40c4ffb8e9a37c55c25f4f3f82c2 - 953c364b2d97a10b36d6b4705895c4c7 - 53c452b9433dca71dd14ad21ee389d95 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg239-client.virtnet /mnt/lustre (opts:) Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 12:17:24 (1788970644) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:17:39 (1788970659) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:17:46 (1788970666) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887548 s, 11.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245924 s, 42.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 12:17:54 (1788970674) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161775 s, 13.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.363705 s, 11.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114537 s, 18.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 12:18:10 (1788970690) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 12:18:13 (1788970693) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:18:16 (1788970696) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219259 s, 47.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 12:18:25 (1788970705) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:18:29 (1788970709) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:18:40 (1788970720) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.332674 s, 31.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg239-client.virtnet /mnt/lustre (opts:) Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:18:52 (1788970732) llite.lustre-ffff9f0d2c1c9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.278296 s, 37.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg239-client.virtnet /mnt/lustre (opts:) Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9f0d07802800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:19:05 (1788970745) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:19:48 (1788970788) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:20:23 (1788970823) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 12:20:27 (1788970827) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.184643 s, 17.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.185576 s, 17.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x36b27c71 (nosync,stale) Mirror 2: 0x36b27c71 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 36bd32b261cba3de68fd3c496e50e26e - mirror 1 checksum: 0db59e3056c580cf111d79873db67410 - mirror 2 checksum: 36bd32b261cba3de68fd3c496e50e26e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-09T12:20:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-09T12:20:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 36bd32b261cba3de68fd3c496e50e26e - mirror 1 checksum: 0db59e3056c580cf111d79873db67410 - mirror 2 checksum: 0db59e3056c580cf111d79873db67410 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-09T12:20:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-09T12:20:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:20:39 (1788970839) Failing mds1 on oleg239-server Stopping /mnt/lustre-mds1 (opts:) on oleg239-server 12:20:45 (1788970845) shut down facet: mds1 facet_host: oleg239-server facet_failover_host: oleg239-server Failover mds1 to oleg239-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 12:21:18 (1788970878) targets are mounted 12:21:18 (1788970878) facet_failover done oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224232 s, 46.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (55s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:21:34 (1788970894) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:21:37 (1788970897) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:21:40 (1788970900) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00976615 s, 1.7 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:21:47 (1788970907) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:21:50 (1788970910) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00737151 s, 556 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.081459 s, 50.3 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:21:57 (1788970917) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:22:19 (1788970939) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:22:48 (1788970968) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:23:12 (1788970992) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:23:37 (1788971017) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 12:23:44 (1788971024) mirror create pid 54318 mirror split pid 54319 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54318 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54319 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (71s) == sanity-flr test 70a: flr mode fsx test ================ 12:24:55 (1788971095) Seed set to 1098 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000005 1788971098.682490 mapwrite 0x008f95 thru 0x009ba5 (0xc11 bytes) 000007 1788971098.847425 mapwrite 0x03f1e7 thru 0x03ffff (0xe19 bytes) 000008 1788971099.007900 mapwrite 0x0248fc thru 0x024b3f (0x244 bytes) 000009 1788971099.104615 punch from 0x02ed77 to 0x0347ca (0x5a54 bytes) truncating to largest ever: 0x1239e 000010 1788971099.111449 trunc from 0x040000 to 0x01239d (0x2dc62 bytes) 000011 1788971099.142912 write 0x00ba0f thru 0x016f24 (0xb516 bytes) 000013 1788971099.161668 punch from 0x016d26 to 0x016f24 (0x1ff bytes) 000014 1788971099.187866 read 0x0113d1 thru 0x016f24 (0x5b54 bytes) 000015 1788971099.192406 trunc from 0x016f25 to 0x00150a (0x15a1a bytes) 000017 1788971099.227582 punch from 0x000e68 to 0x00150a (0x6a3 bytes) 000018 1788971099.235546 mapread 0x00004e thru 0x000e98 (0xe4b bytes) 000020 1788971099.244876 punch from 0x000ebd to 0x000fdf (0x123 bytes) truncating to largest ever: 0x1eef2 000021 1788971099.255205 trunc from 0x00150b to 0x01eef1 (0x1d9e7 bytes) truncating to largest ever: 0x3bdbd 000022 1788971099.281904 trunc from 0x01eef2 to 0x03bdbc (0x1cecb bytes) 000023 1788971099.313161 trunc from 0x03bdbd to 0x013a27 (0x28395 bytes) 000024 1788971099.329575 write 0x02cda4 thru 0x038e32 (0xc08f bytes) 000025 1788971099.332402 trunc from 0x038e33 to 0x0107de (0x28654 bytes) 000026 1788971099.359374 trunc from 0x0107df to 0x01f877 (0xf099 bytes) 000029 1788971099.392949 mapwrite 0x0167aa thru 0x018204 (0x1a5b bytes) 000030 1788971099.460610 mapwrite 0x037769 thru 0x03ffff (0x8897 bytes) 000031 1788971099.562484 MIRROR_EXTEND to 2 mirrors 000032 1788971100.524568 write 0x02517f thru 0x02bbcb (0x6a4d bytes) 000033 1788971100.635371 MIRROR_RESYNC 2 mirrors 000034 1788971101.883138 MIRROR_RESYNC 2 mirrors 000036 1788971102.362565 mapread 0x02bea8 thru 0x02bfdc (0x135 bytes) 000037 1788971102.369674 punch from 0x01c8e2 to 0x022d6a (0x6489 bytes) 000038 1788971102.458003 mapread 0x03fdc5 thru 0x03ffff (0x23b bytes) 000039 1788971102.473860 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb7014f86 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb7014f86 Mirror 3: 0xb7014f86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000042 1788971103.392263 mapwrite 0x03b4fd thru 0x03ffff (0x4b03 bytes) 000045 1788971103.573151 trunc from 0x040000 to 0x00a051 (0x35fae bytes) 000046 1788971103.602834 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xa052): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa052): Mirror 2: 0xff0fae4d (stale) Mirror 3: 0xff0fae4d (stale) CRC-32 checksum value for chunk [0, 0xa052): Mirror 1: 0xff0fae4d Mirror 4: 0xff0fae4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa052: skipped 000047 1788971104.373498 mapread 0x006d42 thru 0x00a051 (0x3310 bytes) 000049 1788971104.414829 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa052): Mirror 3: 0xff0fae4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa052): Mirror 3: 0xff0fae4d (stale) CRC-32 checksum value for chunk [0, 0xa052): Mirror 1: 0xff0fae4d Mirror 4: 0xff0fae4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa052: skipped 000050 1788971104.819428 mapwrite 0x0066bd thru 0x00ecf5 (0x8639 bytes) 000051 1788971104.996739 trunc from 0x00ecf6 to 0x021951 (0x12c5c bytes) 000054 1788971105.019934 trunc from 0x021952 to 0x034a09 (0x130b8 bytes) 000055 1788971105.042275 mapwrite 0x031971 thru 0x038c20 (0x72b0 bytes) 000056 1788971105.112597 write 0x02b073 thru 0x02b5ae (0x53c bytes) 000058 1788971105.115254 punch from 0x006096 to 0x01342d (0xd398 bytes) 000059 1788971105.124456 mapread 0x0237fd thru 0x02bfbc (0x87c0 bytes) 000060 1788971105.133619 mapread 0x01b5b9 thru 0x02a6dc (0xf124 bytes) 000061 1788971105.145123 MIRROR_RESYNC 3 mirrors 000062 1788971106.298733 MIRROR_EXTEND to 4 mirrors 000063 1788971107.181401 mapwrite 0x02e61d thru 0x03490f (0x62f3 bytes) 000064 1788971107.341419 punch from 0x0345f0 to 0x038c20 (0x4631 bytes) 000065 1788971107.350741 punch from 0x01e617 to 0x025ad5 (0x74bf bytes) 000066 1788971107.355474 punch from 0x038386 to 0x038c20 (0x89b bytes) 000067 1788971107.362756 trunc from 0x038c21 to 0x03a82d (0x1c0d bytes) 000069 1788971107.382088 mapread 0x02766a thru 0x030be8 (0x957f bytes) 000072 1788971107.391534 read 0x038f56 thru 0x03a82d (0x18d8 bytes) 000073 1788971107.400667 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a82e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a82e): Mirror 3: 0x596709c5 (stale) Mirror 4: 0x596709c5 (stale) Mirror 5: 0x596709c5 (stale) CRC-32 checksum value for chunk [0, 0x3a82e): Mirror 1: 0xd4826b39 Mirror 6: 0xd4826b39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a82e: skipped 000074 1788971108.988512 write 0x00f53c thru 0x01d9f4 (0xe4b9 bytes) 000075 1788971109.097235 mapread 0x01ebff thru 0x025731 (0x6b33 bytes) 000076 1788971109.112989 trunc from 0x03a82e to 0x0125d2 (0x2825b bytes) 000078 1788971109.133137 read 0x006f86 thru 0x0119e2 (0xaa5d bytes) 000079 1788971109.141213 mapwrite 0x002901 thru 0x00e2e8 (0xb9e8 bytes) 000080 1788971109.195351 mapwrite 0x01f126 thru 0x02e483 (0xf35e bytes) 000081 1788971109.285668 punch from 0x01bdf2 to 0x028f59 (0xd168 bytes) 000083 1788971109.291547 write 0x00c225 thru 0x0110b7 (0x4e93 bytes) 000084 1788971109.293348 mapwrite 0x004d9f thru 0x009847 (0x4aa9 bytes) 000085 1788971109.346400 mapwrite 0x032129 thru 0x03a90d (0x87e5 bytes) 000086 1788971109.420877 trunc from 0x03a90e to 0x024680 (0x1628d bytes) 000087 1788971109.441785 mapwrite 0x01544c thru 0x021719 (0xc2ce bytes) 000088 1788971109.502728 punch from 0x013ad3 to 0x01f0b7 (0xb5e5 bytes) 000089 1788971109.508408 trunc from 0x024681 to 0x03153e (0xcebe bytes) 000090 1788971109.527991 punch from 0x010305 to 0x01a7a0 (0xa49c bytes) 000094 1788971109.535948 read 0x030d10 thru 0x03153e (0x82f bytes) 000095 1788971109.544427 mapwrite 0x022448 thru 0x030f20 (0xead9 bytes) 000097 1788971109.604223 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3153f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3153f): Mirror 3: 0xe3723ca3 (stale) Mirror 4: 0xe3723ca3 (stale) Mirror 5: 0xe3723ca3 (stale) Mirror 6: 0x1a113353 (stale) CRC-32 checksum value for chunk [0, 0x3153f): Mirror 1: 0x46e8dd2b Mirror 7: 0x46e8dd2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3153f: skipped 000098 1788971110.657313 punch from 0x01ac35 to 0x025ea7 (0xb273 bytes) 000099 1788971110.739013 mapread 0x002f2c thru 0x006e95 (0x3f6a bytes) 000102 1788971110.757430 write 0x01286d thru 0x017a0f (0x51a3 bytes) 000105 1788971110.768462 trunc from 0x03153f to 0x02fca0 (0x189e bytes) 000106 1788971110.790796 read 0x01d7fd thru 0x02bb0e (0xe312 bytes) 000108 1788971110.806678 punch from 0x01172f to 0x01514b (0x3a1d bytes) 000109 1788971110.819607 trunc from 0x02fca1 to 0x0384dc (0x883c bytes) 000110 1788971110.841462 mapread 0x02b91a thru 0x0304e6 (0x4bcd bytes) 000111 1788971110.854735 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x384dd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x384dd): Mirror 3: 0xe2f0128a (stale) Mirror 4: 0xe2f0128a (stale) Mirror 5: 0xe2f0128a (stale) Mirror 6: 0x38fbd874 (stale) Mirror 7: 0x9522cf02 (stale) CRC-32 checksum value for chunk [0, 0x384dd): Mirror 1: 0x1d3147c0 Mirror 8: 0x1d3147c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x384dd: skipped 000112 1788971112.722506 punch from 0x028713 to 0x02e817 (0x6105 bytes) 000113 1788971112.823106 mapwrite 0x03b975 thru 0x03ffff (0x468b bytes) 000114 1788971112.969963 write 0x003e0e thru 0x00eab8 (0xacab bytes) 000115 1788971112.987650 mapread 0x03ecc1 thru 0x03ffff (0x133f bytes) 000116 1788971112.989603 trunc from 0x040000 to 0x00ad44 (0x352bb bytes) 000117 1788971113.031166 punch from 0x0042c9 to 0x00ad44 (0x6a7c bytes) 000118 1788971113.061691 read 0x006548 thru 0x00ad44 (0x47fd bytes) 000121 1788971113.074127 write 0x020ff3 thru 0x02cfb0 (0xbfbe bytes) 000123 1788971113.081563 write 0x03d6d8 thru 0x03ffff (0x2928 bytes) 000124 1788971113.093476 MIRROR_RESYNC 7 mirrors 000125 1788971115.868903 punch from 0x010615 to 0x01d7b2 (0xd19e bytes) 000126 1788971115.944602 trunc from 0x040000 to 0x037466 (0x8b99 bytes) 000127 1788971115.969950 punch from 0x032764 to 0x037466 (0x4d03 bytes) 000129 1788971115.977315 mapread 0x00a8d6 thru 0x00db97 (0x32c2 bytes) 000130 1788971116.007707 mapwrite 0x01ba31 thru 0x020d8a (0x535a bytes) 000131 1788971116.065753 mapread 0x02af79 thru 0x037466 (0xc4ee bytes) 000132 1788971116.083680 trunc from 0x037467 to 0x0113ef (0x26077 bytes) 000135 1788971116.101893 mapread 0x00d775 thru 0x0113ef (0x3c7b bytes) 000136 1788971116.103814 mapread 0x00c594 thru 0x00c846 (0x2b3 bytes) 000137 1788971116.105161 write 0x027a43 thru 0x02beda (0x4498 bytes) 000138 1788971116.106993 mapwrite 0x02414d thru 0x024d46 (0xbfa bytes) 000139 1788971116.161787 punch from 0x00d04c to 0x019908 (0xc8bd bytes) 000140 1788971116.172152 MIRROR_RESYNC 7 mirrors 000141 1788971118.124843 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) 000142 1788971118.625993 read 0x003b9a thru 0x005976 (0x1ddd bytes) 000143 1788971118.631234 read 0x01a0e9 thru 0x02206c (0x7f84 bytes) 000144 1788971118.641743 mapread 0x010eeb thru 0x01897f (0x7a95 bytes) 000145 1788971118.642642 punch from 0x00fc12 to 0x013862 (0x3c51 bytes) 000146 1788971118.735278 MIRROR_RESYNC 6 mirrors 000147 1788971120.793649 read 0x012dfb thru 0x01fb49 (0xcd4f bytes) 000149 1788971120.813908 read 0x0146da thru 0x01757c (0x2ea3 bytes) 000150 1788971120.814432 mapwrite 0x00d55f thru 0x01b967 (0xe409 bytes) 000151 1788971121.085631 mapwrite 0x02b87e thru 0x02fed9 (0x465c bytes) 000152 1788971121.214420 read 0x00091c thru 0x007693 (0x6d78 bytes) 000153 1788971121.230425 punch from 0x0294ef to 0x02fed9 (0x69eb bytes) 000155 1788971121.242687 punch from 0x02d9ea to 0x02fed9 (0x24f0 bytes) 000156 1788971121.251749 trunc from 0x02feda to 0x01d138 (0x12da1 bytes) 000159 1788971121.292768 write 0x03972e thru 0x03fc65 (0x6538 bytes) 000160 1788971121.293982 mapwrite 0x00be1d thru 0x01006a (0x424e bytes) 000161 1788971121.405869 punch from 0x03502d to 0x038608 (0x35dc bytes) 000162 1788971121.414486 punch from 0x038268 to 0x03d79e (0x5537 bytes) 000164 1788971121.428955 mapwrite 0x018a1b thru 0x027a29 (0xf00f bytes) 000165 1788971121.562011 trunc from 0x03fc66 to 0x01e141 (0x21b24 bytes) 000166 1788971121.587717 mapwrite 0x028994 thru 0x02db7d (0x51ea bytes) 000167 1788971121.720672 punch from 0x000d8c to 0x001e84 (0x10f9 bytes) 000168 1788971121.727312 mapread 0x0265a4 thru 0x02db7d (0x75da bytes) 000171 1788971121.739690 write 0x0049c8 thru 0x0126cb (0xdd04 bytes) 000172 1788971121.741458 mapwrite 0x03e76f thru 0x03ffff (0x1891 bytes) 000174 1788971121.843932 mapread 0x027339 thru 0x029055 (0x1d1d bytes) 000176 1788971121.851400 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67c9e1b8 (stale) Mirror 4: 0x67c9e1b8 (stale) Mirror 6: 0x67c9e1b8 (stale) Mirror 7: 0x67c9e1b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67c9e1b8 (stale) Mirror 4: 0x67c9e1b8 (stale) Mirror 6: 0x67c9e1b8 (stale) Mirror 7: 0x67c9e1b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000177 1788971122.226587 trunc from 0x040000 to 0x004b58 (0x3b4a7 bytes) 000178 1788971122.243187 write 0x00e1cb thru 0x00e4ce (0x304 bytes) 000181 1788971122.245129 trunc from 0x00e4cf to 0x01b4ce (0xd000 bytes) 000182 1788971122.276249 trunc from 0x01b4cf to 0x0148cb (0x6c03 bytes) 000183 1788971122.324695 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x148cc): Mirror 4: 0x68be75b3 (stale) Mirror 6: 0x68be75b3 (stale) Mirror 7: 0x68be75b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x148cc): Mirror 4: 0x68be75b3 (stale) Mirror 6: 0x68be75b3 (stale) Mirror 7: 0x68be75b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x148cc: skipped 000185 1788971122.782798 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x148cc): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x148cc): Mirror 4: 0x68be75b3 (stale) Mirror 6: 0x68be75b3 (stale) Mirror 7: 0x68be75b3 (stale) CRC-32 checksum value for chunk [0, 0x148cc): Mirror 1: 0x57271548 Mirror 8: 0x57271548 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x148cc: skipped 000186 1788971124.066752 mapread 0x00dbfa thru 0x0148cb (0x6cd2 bytes) 000187 1788971124.122520 punch from 0x009592 to 0x00b58d (0x1ffc bytes) 000188 1788971124.234918 mapwrite 0x01fb93 thru 0x022df6 (0x3264 bytes) 000189 1788971124.313172 write 0x0340fa thru 0x039258 (0x515f bytes) truncating to largest ever: 0x3cbf4 000192 1788971124.315081 trunc from 0x039259 to 0x03cbf3 (0x399b bytes) 000193 1788971124.338499 MIRROR_RESYNC 5 mirrors 000195 1788971126.991974 write 0x0141e1 thru 0x0208e8 (0xc708 bytes) 000196 1788971127.081476 mapread 0x020b01 thru 0x02a292 (0x9792 bytes) 000197 1788971127.102641 punch from 0x001ac3 to 0x008b9d (0x70db bytes) 000198 1788971127.117913 trunc from 0x03cbf4 to 0x03759d (0x5656 bytes) 000200 1788971127.144638 mapwrite 0x0318c6 thru 0x032e64 (0x159f bytes) 000202 1788971127.206874 write 0x00ccfb thru 0x011ca3 (0x4fa9 bytes) 000203 1788971127.223740 mapread 0x014e20 thru 0x01b59e (0x677f bytes) 000204 1788971127.225336 mapwrite 0x01e8e0 thru 0x021a8b (0x31ac bytes) 000206 1788971127.285622 read 0x014367 thru 0x019f6d (0x5c07 bytes) 000208 1788971127.288007 punch from 0x01f4ad to 0x029b42 (0xa696 bytes) 000211 1788971127.300578 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3759e): Mirror 4: 0xe7927043 (stale) Mirror 7: 0xe7927043 (stale) Mirror 8: 0xe7927043 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3759e): Mirror 4: 0xe7927043 (stale) Mirror 7: 0xe7927043 (stale) Mirror 8: 0xe7927043 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3759e: skipped 000212 1788971127.726303 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3759e): Mirror 4: 0xe7927043 (stale) Mirror 7: 0xe7927043 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3759e): Mirror 4: 0xe7927043 (stale) Mirror 7: 0xe7927043 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3759e: skipped 000213 1788971128.104285 read 0x023307 thru 0x02c5a8 (0x92a2 bytes) 000214 1788971128.113910 mapwrite 0x025f20 thru 0x02aabe (0x4b9f bytes) 000216 1788971128.165828 punch from 0x0149f3 to 0x0154bb (0xac9 bytes) 000217 1788971128.176372 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3759e): Mirror 4: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3759e): Mirror 4: 0xe7927043 (stale) Mirror 7: 0xe7927043 (stale) CRC-32 checksum value for chunk [0, 0x3759e): Mirror 1: 0x2d33fc05 Mirror 8: 0x2d33fc05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3759e: skipped 000218 1788971129.239136 write 0x00f018 thru 0x015590 (0x6579 bytes) 000219 1788971129.359194 mapread 0x01efe1 thru 0x023db5 (0x4dd5 bytes) 000220 1788971129.377557 punch from 0x022ed1 to 0x02861c (0x574c bytes) 000221 1788971129.389865 write 0x013234 thru 0x01f456 (0xc223 bytes) 000222 1788971129.391905 write 0x01179b thru 0x01a02c (0x8892 bytes) 000224 1788971129.399744 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3759e): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3759e): Mirror 4: 0xe7927043 (stale) Mirror 7: 0xe7927043 (stale) Mirror 8: 0x2d33fc05 (stale) CRC-32 checksum value for chunk [0, 0x3759e): Mirror 1: 0x7066ffac Mirror 9: 0x7066ffac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3759e: skipped 000225 1788971130.855677 mapread 0x01869b thru 0x01ea87 (0x63ed bytes) 000226 1788971130.879186 write 0x03c8b9 thru 0x03ffff (0x3747 bytes) 000227 1788971130.982658 MIRROR_RESYNC 5 mirrors 000228 1788971133.704911 mapwrite 0x025202 thru 0x02533f (0x13e bytes) 000229 1788971133.892854 mapwrite 0x021cf1 thru 0x022059 (0x369 bytes) 000232 1788971133.939840 trunc from 0x040000 to 0x0355d2 (0xaa2d bytes) 000233 1788971133.963761 trunc from 0x0355d3 to 0x0334f0 (0x20e2 bytes) 000235 1788971133.980855 mapread 0x001c63 thru 0x00c7c1 (0xab5f bytes) 000236 1788971133.989823 mapread 0x01eb42 thru 0x01f4bc (0x97b bytes) 000237 1788971133.990662 punch from 0x0321dd to 0x0334f0 (0x1314 bytes) 000238 1788971133.995163 trunc from 0x0334f1 to 0x02d1fe (0x62f2 bytes) 000239 1788971134.024248 write 0x00e18d thru 0x01b647 (0xd4bb bytes) 000241 1788971134.026338 punch from 0x0258a9 to 0x02d1fe (0x7956 bytes) 000242 1788971134.055851 read 0x004ff3 thru 0x00c3bf (0x73cd bytes) 000243 1788971134.057254 write 0x02d0bc thru 0x02f468 (0x23ad bytes) 000244 1788971134.065803 trunc from 0x02f469 to 0x028499 (0x6fcf bytes) 000245 1788971134.093144 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2849a): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2849a): Mirror 4: 0xd1d507e8 (stale) Mirror 7: 0xd1d507e8 (stale) Mirror 8: 0xd1d507e8 (stale) Mirror 9: 0xd1d507e8 (stale) CRC-32 checksum value for chunk [0, 0x2849a): Mirror 1: 0x1aeb50b5 Mirror 10: 0x1aeb50b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2849a: skipped 000246 1788971135.440753 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2849a): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2849a): Mirror 4: 0xd1d507e8 (stale) Mirror 7: 0xd1d507e8 (stale) Mirror 8: 0xd1d507e8 (stale) Mirror 9: 0xd1d507e8 (stale) CRC-32 checksum value for chunk [0, 0x2849a): Mirror 1: 0x1aeb50b5 Mirror 10: 0x1aeb50b5 Mirror 11: 0x1aeb50b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2849a: skipped 000250 1788971136.344254 trunc from 0x02849a to 0x02c8ee (0x4455 bytes) 000251 1788971136.413855 trunc from 0x02c8ef to 0x038ec9 (0xc5db bytes) 000254 1788971136.438927 MIRROR_RESYNC 7 mirrors 000255 1788971139.886542 mapwrite 0x013f46 thru 0x021532 (0xd5ed bytes) 000256 1788971140.089535 punch from 0x0231e4 to 0x024374 (0x1191 bytes) 000257 1788971140.096415 read 0x028282 thru 0x029f2d (0x1cac bytes) 000258 1788971140.104808 read 0x026957 thru 0x027132 (0x7dc bytes) 000259 1788971140.116967 punch from 0x01850a to 0x02755b (0xf052 bytes) 000260 1788971140.123270 write 0x0120ab thru 0x0208b1 (0xe807 bytes) 000261 1788971140.135322 punch from 0x01c70e to 0x0273bd (0xacb0 bytes) 000262 1788971140.167248 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38eca): Mirror 4: 0x8209df41 (stale) Mirror 7: 0x8209df41 (stale) Mirror 9: 0x8209df41 (stale) Mirror 10: 0x8209df41 (stale) Mirror 11: 0x8209df41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38eca): Mirror 4: 0x8209df41 (stale) Mirror 7: 0x8209df41 (stale) Mirror 9: 0x8209df41 (stale) Mirror 10: 0x8209df41 (stale) Mirror 11: 0x8209df41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38eca: skipped 000263 1788971140.776511 punch from 0x037bb6 to 0x038ec9 (0x1314 bytes) 000264 1788971140.801167 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38eca): Mirror 7: 0x8209df41 (stale) Mirror 9: 0x8209df41 (stale) Mirror 10: 0x8209df41 (stale) Mirror 11: 0x8209df41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38eca): Mirror 7: 0x8209df41 (stale) Mirror 9: 0x8209df41 (stale) Mirror 10: 0x8209df41 (stale) Mirror 11: 0x8209df41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38eca: skipped 000266 1788971141.445939 MIRROR_RESYNC 5 mirrors 000268 1788971143.696673 read 0x01ea2c thru 0x027429 (0x89fe bytes) 000270 1788971143.704883 mapread 0x0062f0 thru 0x00d225 (0x6f36 bytes) 000273 1788971143.726412 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) 000274 1788971144.117012 write 0x032a67 thru 0x037097 (0x4631 bytes) 000275 1788971144.242426 trunc from 0x038eca to 0x0001aa (0x38d1f bytes) 000276 1788971144.287108 mapread 0x0000d4 thru 0x0001aa (0xd7 bytes) 000277 1788971144.294766 punch from 0x000097 to 0x0001aa (0x114 bytes) 000278 1788971144.304188 punch from 0x000008 to 0x0001aa (0x1a3 bytes) 000279 1788971144.309557 punch from 0x00017d to 0x0001aa (0x2e bytes) 000280 1788971144.315793 trunc from 0x0001ab to 0x025918 (0x2576e bytes) 000282 1788971144.338732 read 0x0118c9 thru 0x01c406 (0xab3e bytes) 000284 1788971144.346753 punch from 0x01313d to 0x0131d8 (0x9c bytes) 000286 1788971144.351527 trunc from 0x025919 to 0x03664c (0x10d34 bytes) 000287 1788971144.370675 write 0x00c9e1 thru 0x014235 (0x7855 bytes) 000288 1788971144.378889 mapwrite 0x02a6fa thru 0x032cd4 (0x85db bytes) 000289 1788971144.491955 punch from 0x01bd88 to 0x01e034 (0x22ad bytes) 000290 1788971144.496697 mapread 0x028ec1 thru 0x030056 (0x7196 bytes) 000291 1788971144.497720 trunc from 0x03664d to 0x02cb0e (0x9b3e bytes) truncating to largest ever: 0x3d321 000292 1788971144.525172 trunc from 0x02cb0f to 0x03d320 (0x10812 bytes) 000294 1788971144.557878 mapread 0x031d73 thru 0x03d320 (0xb5ae bytes) 000295 1788971144.602138 MIRROR_RESYNC 4 mirrors 000297 1788971146.368724 write 0x020c8d thru 0x02e206 (0xd57a bytes) 000300 1788971146.440937 trunc from 0x03d321 to 0x02a954 (0x129cc bytes) 000302 1788971146.458555 mapwrite 0x035385 thru 0x03ffff (0xac7b bytes) 000303 1788971146.604499 mapwrite 0x00da9c thru 0x01a388 (0xc8ed bytes) 000304 1788971146.707767 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x41dd7bbb (stale) Mirror 10: 0x41dd7bbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x41dd7bbb (stale) Mirror 10: 0x41dd7bbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000305 1788971147.141563 read 0x030b1e thru 0x03ffff (0xf4e2 bytes) 000306 1788971147.154883 trunc from 0x040000 to 0x0279aa (0x18655 bytes) 000307 1788971147.175132 mapwrite 0x01ad8c thru 0x026cea (0xbf5f bytes) 000308 1788971147.258224 trunc from 0x0279ab to 0x027ddb (0x431 bytes) 000309 1788971147.282629 read 0x0178ce thru 0x0188f9 (0x102c bytes) 000311 1788971147.295829 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27ddc): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ddc): Mirror 7: 0x6ee3fa7c (stale) Mirror 10: 0x6ee3fa7c (stale) CRC-32 checksum value for chunk [0, 0x27ddc): Mirror 1: 0xf3f58dd8 Mirror 11: 0xf3f58dd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ddc: skipped 000312 1788971148.240827 read 0x019f8d thru 0x01d4f3 (0x3567 bytes) 000314 1788971148.288500 mapwrite 0x011404 thru 0x016316 (0x4f13 bytes) 000316 1788971148.495253 mapread 0x0090fd thru 0x018e3c (0xfd40 bytes) 000317 1788971148.507241 mapread 0x007b8a thru 0x00afc6 (0x343d bytes) 000318 1788971148.508473 read 0x024c9a thru 0x027ddb (0x3142 bytes) 000319 1788971148.517894 trunc from 0x027ddc to 0x038c7a (0x10e9f bytes) 000320 1788971148.568689 mapread 0x00361d thru 0x005ff3 (0x29d7 bytes) 000322 1788971148.574726 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38c7b): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c7b): Mirror 7: 0x232b29c1 (stale) Mirror 10: 0x232b29c1 (stale) Mirror 11: 0xeb7a75ad (stale) CRC-32 checksum value for chunk [0, 0x38c7b): Mirror 1: 0x72f7ca01 Mirror 12: 0x72f7ca01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c7b: skipped 000324 1788971149.943604 mapread 0x037ce8 thru 0x03848c (0x7a5 bytes) 000325 1788971149.967722 mapread 0x00ce3b thru 0x01909d (0xc263 bytes) 000326 1788971150.015459 read 0x0257cb thru 0x02b3ed (0x5c23 bytes) 000327 1788971150.033678 MIRROR_RESYNC 5 mirrors 000328 1788971151.317975 trunc from 0x038c7b to 0x017731 (0x21549 bytes) 000330 1788971151.398546 write 0x01830e thru 0x0252f5 (0xcfe8 bytes) 000331 1788971151.400165 punch from 0x014b51 to 0x01d533 (0x89e3 bytes) 000334 1788971151.419890 mapread 0x007316 thru 0x015d0d (0xe9f8 bytes) 000336 1788971151.447415 MIRROR_RESYNC 5 mirrors 000337 1788971152.986407 write 0x007e95 thru 0x008b26 (0xc92 bytes) 000338 1788971153.068806 mapwrite 0x010007 thru 0x017399 (0x7393 bytes) 000340 1788971153.167643 read 0x00df9c thru 0x01ce81 (0xeee6 bytes) 000341 1788971153.177624 punch from 0x01a3c8 to 0x0252f5 (0xaf2e bytes) 000342 1788971153.184915 punch from 0x0138e2 to 0x01e42b (0xab4a bytes) 000343 1788971153.192553 read 0x010a03 thru 0x01f691 (0xec8f bytes) 000344 1788971153.205970 read 0x024397 thru 0x02513b (0xda5 bytes) 000345 1788971153.223733 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x252f6): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x252f6): Mirror 7: 0xf4ca9d69 (stale) Mirror 10: 0xf4ca9d69 (stale) Mirror 11: 0xf4ca9d69 (stale) Mirror 12: 0xf4ca9d69 (stale) CRC-32 checksum value for chunk [0, 0x252f6): Mirror 1: 0xac7504d0 Mirror 13: 0xac7504d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x252f6: skipped 000346 1788971154.817013 trunc from 0x0252f6 to 0x0192c6 (0xc02f bytes) 000348 1788971154.924184 punch from 0x017ef7 to 0x0192c6 (0x13d0 bytes) 000350 1788971154.935755 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x192c7): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x192c7): Mirror 7: 0x1061c7e (stale) Mirror 10: 0x1061c7e (stale) Mirror 11: 0x1061c7e (stale) Mirror 12: 0x1061c7e (stale) Mirror 13: 0xb20d1f22 (stale) CRC-32 checksum value for chunk [0, 0x192c7): Mirror 1: 0xb20d1f22 Mirror 14: 0xb20d1f22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x192c7: skipped 000351 1788971156.110525 read 0x00defb thru 0x010639 (0x273f bytes) 000352 1788971156.117252 mapwrite 0x014e5b thru 0x016949 (0x1aef bytes) 000356 1788971156.328993 read 0x00b8ee thru 0x00da1d (0x2130 bytes) 000357 1788971156.345093 mapread 0x00e922 thru 0x016751 (0x7e30 bytes) truncating to largest ever: 0x3f3f4 000358 1788971156.346215 trunc from 0x0192c7 to 0x03f3f3 (0x2612d bytes) 000359 1788971156.373930 punch from 0x02aeca to 0x038fce (0xe105 bytes) 000360 1788971156.381349 mapread 0x004534 thru 0x00a29e (0x5d6b bytes) 000362 1788971156.382654 write 0x01c904 thru 0x01f0a4 (0x27a1 bytes) 000363 1788971156.402673 mapwrite 0x02e7d4 thru 0x03c643 (0xde70 bytes) 000364 1788971156.491247 MIRROR_RESYNC 7 mirrors 000365 1788971159.636196 write 0x02f958 thru 0x034e06 (0x54af bytes) 000366 1788971159.761162 trunc from 0x03f3f4 to 0x021310 (0x1e0e3 bytes) 000367 1788971159.789297 read 0x012e98 thru 0x014e30 (0x1f99 bytes) 000368 1788971159.795636 mapwrite 0x0254de thru 0x02b4c3 (0x5fe6 bytes) 000369 1788971159.889261 mapwrite 0x0138ed thru 0x014dc1 (0x14d5 bytes) 000370 1788971159.933157 trunc from 0x02b4c4 to 0x010707 (0x1adbc bytes) 000371 1788971159.952812 MIRROR_RESYNC 7 mirrors 000372 1788971161.718942 mapwrite 0x032428 thru 0x03ffff (0xdbd8 bytes) 000373 1788971161.876861 trunc from 0x040000 to 0x02ca0a (0x135f5 bytes) 000374 1788971161.899666 punch from 0x0024c0 to 0x008482 (0x5fc3 bytes) 000375 1788971161.905699 read 0x0118f8 thru 0x014af9 (0x3202 bytes) 000376 1788971161.915485 punch from 0x022d8d to 0x02b64f (0x88c3 bytes) 000377 1788971161.921538 read 0x00a0df thru 0x00c3d6 (0x22f8 bytes) 000378 1788971161.935454 mapwrite 0x01991d thru 0x026318 (0xc9fc bytes) 000381 1788971162.051081 mapread 0x0220b4 thru 0x0290f3 (0x7040 bytes) 000384 1788971162.056331 read 0x01ddac thru 0x0233e8 (0x563d bytes) 000385 1788971162.067999 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ca0b): Mirror 10: 0xed4309b9 (stale) Mirror 11: 0xed4309b9 (stale) Mirror 12: 0xed4309b9 (stale) Mirror 13: 0xed4309b9 (stale) Mirror 14: 0xed4309b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca0b): Mirror 10: 0xed4309b9 (stale) Mirror 11: 0xed4309b9 (stale) Mirror 12: 0xed4309b9 (stale) Mirror 13: 0xed4309b9 (stale) Mirror 14: 0xed4309b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca0b: skipped 000386 1788971162.488652 mapread 0x022fa3 thru 0x02559a (0x25f8 bytes) 000388 1788971162.507725 punch from 0x00df33 to 0x00e6aa (0x778 bytes) 000390 1788971162.516073 read 0x0063d6 thru 0x00e61f (0x824a bytes) 000392 1788971162.525603 write 0x00d907 thru 0x0167ea (0x8ee4 bytes) 000393 1788971162.531164 read 0x02910d thru 0x02ca0a (0x38fe bytes) 000394 1788971162.531806 mapwrite 0x02ccef thru 0x038345 (0xb657 bytes) 000395 1788971162.652904 trunc from 0x038346 to 0x02be8d (0xc4b8 bytes) 000396 1788971162.669916 mapread 0x010b5b thru 0x0191ba (0x8660 bytes) 000397 1788971162.671637 trunc from 0x02be8e to 0x02b5ad (0x8e0 bytes) 000398 1788971162.689266 write 0x00669a thru 0x00a45b (0x3dc2 bytes) 000402 1788971162.691921 write 0x02fe70 thru 0x03c049 (0xc1da bytes) 000403 1788971162.695442 write 0x0068c1 thru 0x008b41 (0x2281 bytes) 000404 1788971162.696914 mapwrite 0x002cf1 thru 0x00a643 (0x7953 bytes) 000405 1788971162.745170 read 0x015090 thru 0x024984 (0xf8f5 bytes) 000408 1788971162.746941 punch from 0x014b5a to 0x01e678 (0x9b1f bytes) 000410 1788971162.754996 mapread 0x023dfc thru 0x02af03 (0x7108 bytes) 000412 1788971162.756648 trunc from 0x03c04a to 0x02b41b (0x10c2e bytes) 000413 1788971162.795217 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b41c): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b41c): Mirror 10: 0x43963ccd (stale) Mirror 11: 0x43963ccd (stale) Mirror 12: 0x43963ccd (stale) Mirror 13: 0x43963ccd (stale) Mirror 14: 0x43963ccd (stale) CRC-32 checksum value for chunk [0, 0x2b41c): Mirror 1: 0x3524d007 Mirror 15: 0x3524d007 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b41c: skipped 000414 1788971164.136791 trunc from 0x02b41c to 0x031681 (0x6266 bytes) 000415 1788971164.226989 trunc from 0x031682 to 0x0211fe (0x10483 bytes) 000417 1788971164.246215 mapwrite 0x02085b thru 0x02253e (0x1ce4 bytes) 000418 1788971164.323225 punch from 0x0105bc to 0x015105 (0x4b4a bytes) 000419 1788971164.331798 MIRROR_RESYNC 7 mirrors 000420 1788971166.808669 trunc from 0x02253f to 0x01d626 (0x4f18 bytes) 000421 1788971166.879740 trunc from 0x01d627 to 0x031c0e (0x145e8 bytes) 000422 1788971166.896954 trunc from 0x031c0f to 0x018c1a (0x18ff4 bytes) 000423 1788971166.921572 mapread 0x002bd7 thru 0x00ae0e (0x8238 bytes) 000424 1788971166.937573 trunc from 0x018c1b to 0x0372d0 (0x1e6b6 bytes) 000425 1788971166.952555 mapread 0x00e7bc thru 0x00f928 (0x116d bytes) 000428 1788971166.955661 punch from 0x00ec9f to 0x016f17 (0x8279 bytes) 000429 1788971166.960455 read 0x01d0e9 thru 0x0217e7 (0x46ff bytes) 000430 1788971166.966904 read 0x00b2d3 thru 0x01715c (0xbe8a bytes) 000431 1788971166.977841 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x372d1): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x372d1): Mirror 10: 0x5fb927da (stale) Mirror 11: 0x5fb927da (stale) Mirror 12: 0x5fb927da (stale) Mirror 13: 0x5fb927da (stale) Mirror 14: 0x5fb927da (stale) Mirror 15: 0x5fb927da (stale) CRC-32 checksum value for chunk [0, 0x372d1): Mirror 1: 0x3425d75c Mirror 16: 0x3425d75c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x372d1: skipped 000432 1788971168.162639 trunc from 0x0372d1 to 0x034741 (0x2b8f bytes) 000433 1788971168.278532 mapread 0x01ad18 thru 0x025db2 (0xb09b bytes) 000434 1788971168.329168 mapread 0x0136c4 thru 0x013934 (0x271 bytes) 000435 1788971168.340270 write 0x02bb8b thru 0x0383e5 (0xc85b bytes) 000436 1788971168.351490 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x383e6): Mirror 11: 0x3ca45297 (stale) Mirror 12: 0x3ca45297 (stale) Mirror 13: 0x3ca45297 (stale) Mirror 14: 0x3ca45297 (stale) Mirror 15: 0x3ca45297 (stale) Mirror 16: 0xeeb35fa1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x383e6): Mirror 11: 0x3ca45297 (stale) Mirror 12: 0x3ca45297 (stale) Mirror 13: 0x3ca45297 (stale) Mirror 14: 0x3ca45297 (stale) Mirror 15: 0x3ca45297 (stale) Mirror 16: 0xeeb35fa1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x383e6: skipped 000437 1788971168.795505 trunc from 0x0383e6 to 0x008d3e (0x2f6a7 bytes) 000438 1788971168.815666 punch from 0x0023dd to 0x005940 (0x3564 bytes) 000439 1788971168.822729 mapwrite 0x017a6b thru 0x02078b (0x8d21 bytes) 000441 1788971168.892179 write 0x01d184 thru 0x01e1cf (0x104c bytes) 000442 1788971168.892917 write 0x015b06 thru 0x018001 (0x24fc bytes) 000444 1788971168.893987 trunc from 0x02078c to 0x02670e (0x5f83 bytes) 000446 1788971168.906208 mapread 0x00215b thru 0x00d695 (0xb53b bytes) 000447 1788971168.930425 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2670f): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2670f): Mirror 11: 0x5faa2ff6 (stale) Mirror 12: 0x5faa2ff6 (stale) Mirror 13: 0x5faa2ff6 (stale) Mirror 14: 0x5faa2ff6 (stale) Mirror 15: 0x5faa2ff6 (stale) Mirror 16: 0x7af36f9b (stale) CRC-32 checksum value for chunk [0, 0x2670f): Mirror 1: 0xc28f975d Mirror 17: 0xc28f975d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2670f: skipped 000448 1788971170.060146 mapwrite 0x021fe5 thru 0x024441 (0x245d bytes) 000451 1788971170.349335 read 0x00145f thru 0x004693 (0x3235 bytes) 000452 1788971170.372603 read 0x006945 thru 0x00f54e (0x8c0a bytes) 000453 1788971170.397840 mapwrite 0x0309ae thru 0x03f562 (0xebb5 bytes) 000455 1788971170.570370 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f563): Mirror 11: 0xcd8b8584 (stale) Mirror 12: 0xcd8b8584 (stale) Mirror 14: 0xcd8b8584 (stale) Mirror 15: 0xcd8b8584 (stale) Mirror 16: 0x84564060 (stale) Mirror 17: 0x3872d74a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f563): Mirror 11: 0xcd8b8584 (stale) Mirror 12: 0xcd8b8584 (stale) Mirror 14: 0xcd8b8584 (stale) Mirror 15: 0xcd8b8584 (stale) Mirror 16: 0x84564060 (stale) Mirror 17: 0x3872d74a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f563: skipped 000456 1788971171.118981 punch from 0x004fa9 to 0x0113eb (0xc443 bytes) 000457 1788971171.144735 MIRROR_RESYNC 7 mirrors 000458 1788971174.038461 MIRROR_EXTEND to 8 mirrors 000460 1788971175.329850 mapwrite 0x02468f thru 0x027380 (0x2cf2 bytes) 000461 1788971175.791464 trunc from 0x03f563 to 0x017b1e (0x27a44 bytes) 000462 1788971175.834482 mapwrite 0x0275e0 thru 0x0370c2 (0xfae3 bytes) 000463 1788971176.002753 punch from 0x026704 to 0x028740 (0x203d bytes) 000464 1788971176.010863 mapwrite 0x0355a9 thru 0x03c3f4 (0x6e4c bytes) 000465 1788971176.145785 write 0x03a83e thru 0x03ed3a (0x44fd bytes) 000466 1788971176.146991 punch from 0x01bb07 to 0x023f4d (0x8447 bytes) 000467 1788971176.163223 mapread 0x0007cc thru 0x00f89c (0xf0d1 bytes) 000468 1788971176.178924 punch from 0x009556 to 0x00aafb (0x15a6 bytes) 000469 1788971176.195190 mapread 0x00b313 thru 0x00bb08 (0x7f6 bytes) 000470 1788971176.198511 write 0x005203 thru 0x014447 (0xf245 bytes) 000473 1788971176.211712 trunc from 0x03ed3b to 0x02923e (0x15afc bytes) 000474 1788971176.242787 trunc from 0x02923f to 0x03d8a2 (0x14664 bytes) 000475 1788971176.264154 punch from 0x023c27 to 0x027b54 (0x3f2e bytes) 000476 1788971176.279540 mapwrite 0x0313d9 thru 0x03774f (0x6377 bytes) 000478 1788971176.377686 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3d8a3): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d8a3): Mirror 11: 0x52b4659c (stale) Mirror 12: 0x52b4659c (stale) Mirror 14: 0x52b4659c (stale) Mirror 15: 0x52b4659c (stale) Mirror 16: 0x52b4659c (stale) Mirror 17: 0x52b4659c (stale) Mirror 18: 0x52b4659c (stale) CRC-32 checksum value for chunk [0, 0x3d8a3): Mirror 1: 0x22ff8e02 Mirror 19: 0x22ff8e02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d8a3: skipped 000479 1788971177.736740 read 0x025ef1 thru 0x030f35 (0xb045 bytes) 000480 1788971177.753924 mapwrite 0x039fd6 thru 0x03ffff (0x602a bytes) 000484 1788971177.908904 mapread 0x017859 thru 0x0178b3 (0x5b bytes) 000485 1788971177.918591 trunc from 0x040000 to 0x01e24f (0x21db0 bytes) 000488 1788971177.942226 mapwrite 0x014e07 thru 0x01f7d3 (0xa9cd bytes) 000490 1788971178.008556 punch from 0x0185ae to 0x01f7d3 (0x7226 bytes) 000491 1788971178.013296 mapwrite 0x02e375 thru 0x03c8dc (0xe568 bytes) 000492 1788971178.084822 trunc from 0x03c8dd to 0x00884c (0x34090 bytes) 000493 1788971178.103430 trunc from 0x00884d to 0x0073f4 (0x1458 bytes) 000494 1788971178.117906 trunc from 0x0073f5 to 0x02fc6c (0x28878 bytes) 000495 1788971178.148230 trunc from 0x02fc6d to 0x011bad (0x1e0bf bytes) 000499 1788971178.172643 mapread 0x00d3ad thru 0x00ec60 (0x18b4 bytes) 000501 1788971178.190618 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11bae): Mirror 11: 0xb845c150 (stale) Mirror 12: 0xb845c150 (stale) Mirror 14: 0xb845c150 (stale) Mirror 15: 0xb845c150 (stale) Mirror 16: 0xb845c150 (stale) Mirror 17: 0xb845c150 (stale) Mirror 19: 0x4bb79409 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11bae): Mirror 11: 0xb845c150 (stale) Mirror 12: 0xb845c150 (stale) Mirror 14: 0xb845c150 (stale) Mirror 15: 0xb845c150 (stale) Mirror 16: 0xb845c150 (stale) Mirror 17: 0xb845c150 (stale) Mirror 19: 0x4bb79409 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11bae: skipped 000503 1788971178.765498 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11bae): Mirror 11: 0xb845c150 (stale) Mirror 14: 0xb845c150 (stale) Mirror 15: 0xb845c150 (stale) Mirror 16: 0xb845c150 (stale) Mirror 17: 0xb845c150 (stale) Mirror 19: 0x4bb79409 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11bae): Mirror 11: 0xb845c150 (stale) Mirror 14: 0xb845c150 (stale) Mirror 15: 0xb845c150 (stale) Mirror 16: 0xb845c150 (stale) Mirror 17: 0xb845c150 (stale) Mirror 19: 0x4bb79409 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11bae: skipped 000505 1788971179.274255 trunc from 0x011bae to 0x02768f (0x15ae2 bytes) 000506 1788971179.296209 mapwrite 0x01c824 thru 0x024a47 (0x8224 bytes) 000507 1788971179.376988 write 0x0041d6 thru 0x009eb2 (0x5cdd bytes) 000508 1788971179.392431 write 0x02b65d thru 0x0301f8 (0x4b9c bytes) 000509 1788971179.398886 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x301f9): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301f9): Mirror 11: 0xb9d97131 (stale) Mirror 14: 0xb9d97131 (stale) Mirror 15: 0xb9d97131 (stale) Mirror 16: 0xb9d97131 (stale) Mirror 17: 0xb9d97131 (stale) Mirror 19: 0x9f4df7c4 (stale) CRC-32 checksum value for chunk [0, 0x301f9): Mirror 1: 0x1a180dc5 Mirror 20: 0x1a180dc5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301f9: skipped 000510 1788971180.812241 read 0x002d56 thru 0x00c084 (0x932f bytes) 000511 1788971180.838112 write 0x0268ef thru 0x02ed82 (0x8494 bytes) 000512 1788971180.975709 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x301f9): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301f9): Mirror 11: 0xb9d97131 (stale) Mirror 14: 0xb9d97131 (stale) Mirror 15: 0xb9d97131 (stale) Mirror 16: 0xb9d97131 (stale) Mirror 17: 0xb9d97131 (stale) Mirror 19: 0x9f4df7c4 (stale) Mirror 20: 0x1a180dc5 (stale) CRC-32 checksum value for chunk [0, 0x301f9): Mirror 1: 0x8a5d5843 Mirror 21: 0x8a5d5843 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301f9: skipped 000513 1788971183.228503 read 0x014991 thru 0x01ca6f (0x80df bytes) 000515 1788971183.262545 punch from 0x027939 to 0x02b0db (0x37a3 bytes) 000516 1788971183.430009 trunc from 0x0301f9 to 0x0309e9 (0x7f1 bytes) 000518 1788971183.485528 mapread 0x008c03 thru 0x00b7fa (0x2bf8 bytes) 000519 1788971183.546372 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x309ea): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x309ea): Mirror 11: 0xc65cd178 (stale) Mirror 14: 0xc65cd178 (stale) Mirror 15: 0xc65cd178 (stale) Mirror 16: 0xc65cd178 (stale) Mirror 17: 0xc65cd178 (stale) Mirror 19: 0xbe3b816e (stale) Mirror 20: 0x5c9cc5fc (stale) Mirror 21: 0x1bd6b5f7 (stale) CRC-32 checksum value for chunk [0, 0x309ea): Mirror 1: 0x79e92776 Mirror 22: 0x79e92776 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x309ea: skipped 000520 1788971185.997456 read 0x00c747 thru 0x019968 (0xd222 bytes) 000521 1788971186.057722 mapread 0x001ab1 thru 0x0044a0 (0x29f0 bytes) 000522 1788971186.145396 read 0x009c4f thru 0x011e01 (0x81b3 bytes) 000524 1788971186.190120 mapwrite 0x013b0e thru 0x0162f4 (0x27e7 bytes) 000525 1788971186.524221 mapread 0x029f8c thru 0x0309e9 (0x6a5e bytes) 000526 1788971186.527003 read 0x00340e thru 0x00d516 (0xa109 bytes) 000527 1788971186.529146 mapwrite 0x021188 thru 0x023b3b (0x29b4 bytes) 000528 1788971186.578702 trunc from 0x0309ea to 0x01ebb6 (0x11e33 bytes) 000529 1788971186.634077 write 0x03f76a thru 0x03ffff (0x896 bytes) 000530 1788971186.636208 trunc from 0x040000 to 0x0029b1 (0x3d64e bytes) 000531 1788971186.655974 trunc from 0x0029b2 to 0x02d0ac (0x2a6fb bytes) 000533 1788971186.725089 mapwrite 0x03219b thru 0x03acfd (0x8b63 bytes) 000536 1788971186.927704 trunc from 0x03acfe to 0x035d8b (0x4f72 bytes) 000537 1788971186.949188 mapread 0x01ebaa thru 0x02aea4 (0xc2fb bytes) 000538 1788971186.950199 read 0x00499f thru 0x0091d1 (0x4833 bytes) 000539 1788971186.950752 mapwrite 0x03ffd9 thru 0x03ffff (0x27 bytes) 000540 1788971187.031209 write 0x00c4b4 thru 0x01863e (0xc18b bytes) 000542 1788971187.036087 mapwrite 0x010c77 thru 0x01744e (0x67d8 bytes) 000543 1788971187.090589 read 0x0339d3 thru 0x03927a (0x58a8 bytes) 000545 1788971187.091747 mapwrite 0x0198a2 thru 0x01ab71 (0x12d0 bytes) 000548 1788971187.155246 punch from 0x037efb to 0x03b423 (0x3529 bytes) 000550 1788971187.163804 read 0x01fd01 thru 0x027801 (0x7b01 bytes) 000551 1788971187.165556 trunc from 0x040000 to 0x02e386 (0x11c79 bytes) 000552 1788971187.210092 trunc from 0x02e387 to 0x014f84 (0x19402 bytes) 000553 1788971187.245729 trunc from 0x014f85 to 0x025e22 (0x10e9e bytes) 000556 1788971187.290824 read 0x01714f thru 0x025e22 (0xecd4 bytes) 000558 1788971187.316539 mapwrite 0x02948c thru 0x0332c4 (0x9e39 bytes) 000559 1788971187.451801 mapread 0x011725 thru 0x01ae7e (0x975a bytes) 000560 1788971187.453341 trunc from 0x0332c5 to 0x032a0c (0x8b8 bytes) 000561 1788971187.482822 write 0x02e04f thru 0x03bb77 (0xdb29 bytes) 000563 1788971187.488804 trunc from 0x03bb78 to 0x03015e (0xba19 bytes) 000565 1788971187.511419 trunc from 0x03015f to 0x015b68 (0x1a5f6 bytes) 000567 1788971187.530966 read 0x00347d thru 0x00f84f (0xc3d3 bytes) 000568 1788971187.532462 trunc from 0x015b69 to 0x011bf6 (0x3f72 bytes) 000573 1788971187.556207 mapread 0x00b61c thru 0x011bf6 (0x65db bytes) 000574 1788971187.559852 trunc from 0x011bf7 to 0x00a01a (0x7bdc bytes) 000575 1788971187.580289 read 0x0043f7 thru 0x00a01a (0x5c24 bytes) 000577 1788971187.582191 mapwrite 0x0264bc thru 0x030b9b (0xa6e0 bytes) 000578 1788971187.683623 write 0x018eb3 thru 0x024a27 (0xbb75 bytes) 000579 1788971187.691295 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30b9c): Mirror 11: 0x2aae945 (stale) Mirror 14: 0x2aae945 (stale) Mirror 15: 0x2aae945 (stale) Mirror 16: 0x2aae945 (stale) Mirror 17: 0x2aae945 (stale) Mirror 19: 0x8d860974 (stale) Mirror 20: 0xc7ee4283 (stale) Mirror 21: 0x335df523 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b9c): Mirror 11: 0x2aae945 (stale) Mirror 14: 0x2aae945 (stale) Mirror 15: 0x2aae945 (stale) Mirror 16: 0x2aae945 (stale) Mirror 17: 0x2aae945 (stale) Mirror 19: 0x8d860974 (stale) Mirror 20: 0xc7ee4283 (stale) Mirror 21: 0x335df523 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b9c: skipped 000580 1788971188.256378 read 0x02eccc thru 0x030b9b (0x1ed0 bytes) 000584 1788971188.263314 punch from 0x02d887 to 0x02fedd (0x2657 bytes) 000585 1788971188.269142 punch from 0x008f32 to 0x018534 (0xf603 bytes) 000587 1788971188.275149 write 0x00ed28 thru 0x01161a (0x28f3 bytes) 000588 1788971188.288586 MIRROR_RESYNC 9 mirrors 000589 1788971192.849395 punch from 0x00ddc7 to 0x01d8c3 (0xfafd bytes) 000590 1788971192.944683 punch from 0x01eb86 to 0x025fb8 (0x7433 bytes) 000592 1788971192.950926 trunc from 0x030b9c to 0x0381e6 (0x764b bytes) 000593 1788971192.979559 write 0x00877a thru 0x00f178 (0x69ff bytes) 000594 1788971192.991620 mapwrite 0x00c982 thru 0x0136b5 (0x6d34 bytes) 000596 1788971193.093376 read 0x018fe5 thru 0x0257f1 (0xc80d bytes) 000597 1788971193.104218 mapwrite 0x008ffe thru 0x0114db (0x84de bytes) 000598 1788971193.170937 punch from 0x01a439 to 0x01bcc9 (0x1891 bytes) 000599 1788971193.180189 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x381e7): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381e7): Mirror 11: 0xe79c90ae (stale) Mirror 14: 0xe79c90ae (stale) Mirror 15: 0xe79c90ae (stale) Mirror 16: 0xe79c90ae (stale) Mirror 17: 0xe79c90ae (stale) Mirror 19: 0xe79c90ae (stale) Mirror 20: 0xe79c90ae (stale) Mirror 21: 0xe79c90ae (stale) CRC-32 checksum value for chunk [0, 0x381e7): Mirror 1: 0x8e0c7a4a Mirror 22: 0x8e0c7a4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381e7: skipped 000600 1788971194.302279 punch from 0x00d930 to 0x016a92 (0x9163 bytes) 000601 1788971194.383886 read 0x01aa2c thru 0x01c5a1 (0x1b76 bytes) 000603 1788971194.391362 write 0x01362c thru 0x02172a (0xe0ff bytes) 000604 1788971194.405507 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x381e7): Mirror 14: 0xe79c90ae (stale) Mirror 15: 0xe79c90ae (stale) Mirror 16: 0xe79c90ae (stale) Mirror 17: 0xe79c90ae (stale) Mirror 19: 0xe79c90ae (stale) Mirror 20: 0xe79c90ae (stale) Mirror 21: 0xe79c90ae (stale) Mirror 22: 0x8e0c7a4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381e7): Mirror 14: 0xe79c90ae (stale) Mirror 15: 0xe79c90ae (stale) Mirror 16: 0xe79c90ae (stale) Mirror 17: 0xe79c90ae (stale) Mirror 19: 0xe79c90ae (stale) Mirror 20: 0xe79c90ae (stale) Mirror 21: 0xe79c90ae (stale) Mirror 22: 0x8e0c7a4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381e7: skipped 000605 1788971194.865100 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x381e7): Mirror 14: 0xe79c90ae (stale) Mirror 15: 0xe79c90ae (stale) Mirror 16: 0xe79c90ae (stale) Mirror 19: 0xe79c90ae (stale) Mirror 20: 0xe79c90ae (stale) Mirror 21: 0xe79c90ae (stale) Mirror 22: 0x8e0c7a4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381e7): Mirror 14: 0xe79c90ae (stale) Mirror 15: 0xe79c90ae (stale) Mirror 16: 0xe79c90ae (stale) Mirror 19: 0xe79c90ae (stale) Mirror 20: 0xe79c90ae (stale) Mirror 21: 0xe79c90ae (stale) Mirror 22: 0x8e0c7a4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381e7: skipped 000607 1788971195.290975 mapread 0x00d5c2 thru 0x013d9b (0x67da bytes) 000609 1788971195.310159 mapwrite 0x01b5e7 thru 0x0201ee (0x4c08 bytes) 000610 1788971195.371169 MIRROR_RESYNC 8 mirrors 000611 1788971199.608583 trunc from 0x0381e7 to 0x01d1fa (0x1afec bytes) 000615 1788971199.715691 read 0x01a9cf thru 0x01c264 (0x1896 bytes) 000616 1788971199.721862 write 0x01b816 thru 0x029f9d (0xe788 bytes) 000617 1788971199.726236 mapwrite 0x035de7 thru 0x03ffff (0xa219 bytes) 000618 1788971199.883898 MIRROR_RESYNC 8 mirrors 000619 1788971204.244144 mapwrite 0x02ab6e thru 0x0307cc (0x5c5f bytes) 000621 1788971204.389201 trunc from 0x040000 to 0x0296d1 (0x1692e bytes) 000622 1788971204.413186 read 0x013087 thru 0x01e6e0 (0xb65a bytes) 000624 1788971204.421604 mapread 0x00395a thru 0x00bb8c (0x8233 bytes) 000626 1788971204.430336 read 0x0151b4 thru 0x01e051 (0x8e9e bytes) 000627 1788971204.431303 mapwrite 0x00cfa4 thru 0x0115b5 (0x4612 bytes) 000628 1788971204.488918 mapread 0x0241c7 thru 0x0296d1 (0x550b bytes) 000630 1788971204.490367 write 0x01c624 thru 0x029613 (0xcff0 bytes) 000632 1788971204.493554 punch from 0x00ef7e to 0x01208c (0x310f bytes) 000635 1788971204.508589 MIRROR_RESYNC 8 mirrors 000636 1788971208.489874 mapwrite 0x025d11 thru 0x029dfb (0x40eb bytes) 000637 1788971208.674481 write 0x00c9c2 thru 0x0161d8 (0x9817 bytes) 000638 1788971208.685092 mapread 0x00a542 thru 0x017b13 (0xd5d2 bytes) 000639 1788971208.699403 write 0x031c94 thru 0x03c351 (0xa6be bytes) 000641 1788971208.701432 trunc from 0x03c352 to 0x002ea7 (0x394aa bytes) 000642 1788971208.722168 mapread 0x001d32 thru 0x002ea7 (0x1176 bytes) 000643 1788971208.732309 MIRROR_RESYNC 8 mirrors 000644 1788971210.399462 mapwrite 0x010dbe thru 0x01bdb5 (0xaff8 bytes) 000645 1788971210.671791 read 0x008752 thru 0x00fcc8 (0x7577 bytes) 000646 1788971210.680543 read 0x00f84c thru 0x0175ab (0x7d60 bytes) 000647 1788971210.681135 write 0x029be0 thru 0x02cd5e (0x317f bytes) 000648 1788971210.682127 read 0x02b621 thru 0x02cd5e (0x173e bytes) 000649 1788971210.684031 mapread 0x02155a thru 0x02cd5e (0xb805 bytes) 000650 1788971210.699161 trunc from 0x02cd5f to 0x01c76e (0x105f0 bytes) 000651 1788971210.733961 read 0x008f82 thru 0x00eda8 (0x5e27 bytes) 000653 1788971210.735123 trunc from 0x01c76f to 0x00bad2 (0x10c9c bytes) 000654 1788971210.761554 trunc from 0x00bad3 to 0x03ef11 (0x3343f bytes) 000656 1788971210.786476 trunc from 0x03ef12 to 0x00d4ba (0x31a57 bytes) 000657 1788971210.827511 mapwrite 0x0029eb thru 0x00426b (0x1881 bytes) 000660 1788971210.938689 trunc from 0x00d4bb to 0x017208 (0x9d4e bytes) 000661 1788971210.969992 mapwrite 0x019ed6 thru 0x01f2a2 (0x53cd bytes) 000662 1788971211.160863 mapwrite 0x00d732 thru 0x010e05 (0x36d4 bytes) 000664 1788971211.273830 read 0x01489e thru 0x0158d8 (0x103b bytes) 000665 1788971211.282237 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1f2a3): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f2a3): Mirror 14: 0x369fbcf6 (stale) Mirror 15: 0x369fbcf6 (stale) Mirror 16: 0x369fbcf6 (stale) Mirror 19: 0x369fbcf6 (stale) Mirror 20: 0x369fbcf6 (stale) Mirror 21: 0x369fbcf6 (stale) Mirror 22: 0x369fbcf6 (stale) CRC-32 checksum value for chunk [0, 0x1f2a3): Mirror 1: 0x645f442c Mirror 23: 0x645f442c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f2a3: skipped 000667 1788971212.363221 write 0x02ada1 thru 0x02e2fb (0x355b bytes) 000668 1788971212.455515 write 0x02a7a5 thru 0x037125 (0xc981 bytes) 000669 1788971212.468881 mapread 0x00a063 thru 0x00c4b7 (0x2455 bytes) 000670 1788971212.513170 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37126): Mirror 14: 0x511f2cfa (stale) Mirror 15: 0x511f2cfa (stale) Mirror 19: 0x511f2cfa (stale) Mirror 20: 0x511f2cfa (stale) Mirror 21: 0x511f2cfa (stale) Mirror 22: 0x511f2cfa (stale) Mirror 23: 0xbbe3e36d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37126): Mirror 14: 0x511f2cfa (stale) Mirror 15: 0x511f2cfa (stale) Mirror 19: 0x511f2cfa (stale) Mirror 20: 0x511f2cfa (stale) Mirror 21: 0x511f2cfa (stale) Mirror 22: 0x511f2cfa (stale) Mirror 23: 0xbbe3e36d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37126: skipped 000672 1788971212.980449 trunc from 0x037126 to 0x017c24 (0x1f501 bytes) 000673 1788971213.017445 mapwrite 0x022ad3 thru 0x02e5d2 (0xbb00 bytes) 000674 1788971213.129011 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2e5d3): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e5d3): Mirror 14: 0xfad796ea (stale) Mirror 15: 0xfad796ea (stale) Mirror 19: 0xfad796ea (stale) Mirror 20: 0xfad796ea (stale) Mirror 21: 0xfad796ea (stale) Mirror 22: 0xfad796ea (stale) Mirror 23: 0xe665a444 (stale) CRC-32 checksum value for chunk [0, 0x2e5d3): Mirror 1: 0x496c1b3e Mirror 24: 0x496c1b3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e5d3: skipped 000676 1788971214.625469 mapread 0x009011 thru 0x00976e (0x75e bytes) 000677 1788971214.654271 write 0x0351b3 thru 0x0360aa (0xef8 bytes) 000678 1788971214.758012 mapread 0x019e96 thru 0x01c1a1 (0x230c bytes) 000679 1788971214.772154 write 0x021020 thru 0x02310a (0x20eb bytes) 000681 1788971214.785938 write 0x0219c2 thru 0x02aa4c (0x908b bytes) 000682 1788971214.805121 mapread 0x0290fd thru 0x03248e (0x9392 bytes) 000683 1788971214.818110 punch from 0x028a3f to 0x02c3c6 (0x3988 bytes) 000684 1788971214.844565 mapread 0x000d26 thru 0x004872 (0x3b4d bytes) 000686 1788971214.872083 read 0x007f90 thru 0x0099fc (0x1a6d bytes) 000688 1788971214.873261 punch from 0x025386 to 0x030ae1 (0xb75c bytes) 000689 1788971214.881779 read 0x025c7b thru 0x035422 (0xf7a8 bytes) 000691 1788971214.903818 punch from 0x01fd43 to 0x022c5e (0x2f1c bytes) 000692 1788971214.912376 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x360ab): Mirror 14: 0xcb729d99 (stale) Mirror 15: 0xcb729d99 (stale) Mirror 19: 0xcb729d99 (stale) Mirror 21: 0xcb729d99 (stale) Mirror 22: 0xcb729d99 (stale) Mirror 23: 0x61e528bf (stale) Mirror 24: 0x7a346031 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x360ab): Mirror 14: 0xcb729d99 (stale) Mirror 15: 0xcb729d99 (stale) Mirror 19: 0xcb729d99 (stale) Mirror 21: 0xcb729d99 (stale) Mirror 22: 0xcb729d99 (stale) Mirror 23: 0x61e528bf (stale) Mirror 24: 0x7a346031 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x360ab: skipped 000694 1788971215.409934 trunc from 0x0360ab to 0x004415 (0x31c95 bytes) 000695 1788971215.488814 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x4416): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4416): Mirror 14: 0xfc1277 (stale) Mirror 15: 0xfc1277 (stale) Mirror 19: 0xfc1277 (stale) Mirror 21: 0xfc1277 (stale) Mirror 22: 0xfc1277 (stale) Mirror 23: 0xfb13bd97 (stale) Mirror 24: 0xfb13bd97 (stale) CRC-32 checksum value for chunk [0, 0x4416): Mirror 1: 0xfb13bd97 Mirror 25: 0xfb13bd97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4416: skipped 000696 1788971217.167909 read 0x0006c5 thru 0x004415 (0x3d51 bytes) 000697 1788971217.205476 write 0x017aba thru 0x01f739 (0x7c80 bytes) 000702 1788971217.285277 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f73a): Mirror 14: 0xea4ed0da (stale) Mirror 15: 0xea4ed0da (stale) Mirror 19: 0xea4ed0da (stale) Mirror 21: 0xea4ed0da (stale) Mirror 22: 0xea4ed0da (stale) Mirror 23: 0x1fdd4db7 (stale) Mirror 25: 0x38c3deec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f73a): Mirror 14: 0xea4ed0da (stale) Mirror 15: 0xea4ed0da (stale) Mirror 19: 0xea4ed0da (stale) Mirror 21: 0xea4ed0da (stale) Mirror 22: 0xea4ed0da (stale) Mirror 23: 0x1fdd4db7 (stale) Mirror 25: 0x38c3deec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f73a: skipped 000706 1788971217.923302 mapread 0x00d0e4 thru 0x01260c (0x5529 bytes) 000708 1788971217.943710 trunc from 0x01f73a to 0x00895d (0x16ddc bytes) 000709 1788971217.959962 punch from 0x0014cd to 0x00895d (0x7491 bytes) 000710 1788971217.968678 MIRROR_RESYNC 8 mirrors 000711 1788971219.459167 MIRROR_EXTEND to 9 mirrors 000712 1788971220.872723 trunc from 0x00895e to 0x01f0cd (0x16770 bytes) 000714 1788971220.992953 MIRROR_RESYNC 9 mirrors 000716 1788971222.264846 trunc from 0x01f0ce to 0x02c8c8 (0xd7fb bytes) 000719 1788971222.429403 write 0x02a569 thru 0x034dfe (0xa896 bytes) 000720 1788971222.443510 read 0x02454b thru 0x02fe53 (0xb909 bytes) 000721 1788971222.461650 mapwrite 0x010aff thru 0x01f495 (0xe997 bytes) 000723 1788971222.602832 read 0x02fd5d thru 0x034dfe (0x50a2 bytes) 000724 1788971222.605380 read 0x02e6de thru 0x034dfe (0x6721 bytes) 000726 1788971222.608752 mapread 0x022ef9 thru 0x02b8d4 (0x89dc bytes) 000727 1788971222.612811 punch from 0x0297e0 to 0x02ddb5 (0x45d6 bytes) 000728 1788971222.656803 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34dff): Mirror 14: 0xd25b6be2 (stale) Mirror 19: 0xd25b6be2 (stale) Mirror 21: 0xd25b6be2 (stale) Mirror 22: 0xd25b6be2 (stale) Mirror 23: 0xd25b6be2 (stale) Mirror 25: 0xd25b6be2 (stale) Mirror 26: 0xd25b6be2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34dff): Mirror 14: 0xd25b6be2 (stale) Mirror 19: 0xd25b6be2 (stale) Mirror 21: 0xd25b6be2 (stale) Mirror 22: 0xd25b6be2 (stale) Mirror 23: 0xd25b6be2 (stale) Mirror 25: 0xd25b6be2 (stale) Mirror 26: 0xd25b6be2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34dff: skipped 000730 1788971223.172677 punch from 0x01917e to 0x026e12 (0xdc95 bytes) 000733 1788971223.178561 write 0x036df0 thru 0x03facd (0x8cde bytes) 000734 1788971223.179990 read 0x039be1 thru 0x03a047 (0x467 bytes) 000736 1788971223.182367 read 0x02bba2 thru 0x02ec26 (0x3085 bytes) 000737 1788971223.195155 mapwrite 0x0084f6 thru 0x015fcf (0xdada bytes) 000738 1788971223.245937 trunc from 0x03face to 0x00d587 (0x32546 bytes) 000740 1788971223.267172 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xd588): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd588): Mirror 14: 0x22222972 (stale) Mirror 19: 0x22222972 (stale) Mirror 21: 0x22222972 (stale) Mirror 22: 0x22222972 (stale) Mirror 23: 0x22222972 (stale) Mirror 25: 0x22222972 (stale) Mirror 26: 0x22222972 (stale) CRC-32 checksum value for chunk [0, 0xd588): Mirror 1: 0x84064314 Mirror 27: 0x84064314 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd588: skipped 000743 1788971224.126557 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd588): Mirror 19: 0x22222972 (stale) Mirror 21: 0x22222972 (stale) Mirror 22: 0x22222972 (stale) Mirror 23: 0x22222972 (stale) Mirror 25: 0x22222972 (stale) Mirror 26: 0x22222972 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd588): Mirror 19: 0x22222972 (stale) Mirror 21: 0x22222972 (stale) Mirror 22: 0x22222972 (stale) Mirror 23: 0x22222972 (stale) Mirror 25: 0x22222972 (stale) Mirror 26: 0x22222972 (stale) CRC-32 checksum value for chunk [0, 0xd588): Mirror 1: 0x84064314 Mirror 27: 0x84064314 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd588: skipped 000744 1788971224.949390 write 0x016dc3 thru 0x01f6fd (0x893b bytes) 000746 1788971225.054002 trunc from 0x01f6fe to 0x028158 (0x8a5b bytes) 000749 1788971225.104773 mapwrite 0x03e431 thru 0x03ffff (0x1bcf bytes) 000750 1788971225.183481 trunc from 0x040000 to 0x028c37 (0x173c8 bytes) 000752 1788971225.204798 mapread 0x028a1c thru 0x028c37 (0x21c bytes) 000754 1788971225.215602 mapwrite 0x0383f3 thru 0x03ffff (0x7c0d bytes) 000755 1788971225.300720 mapwrite 0x00a89e thru 0x017a44 (0xd1a7 bytes) 000756 1788971225.383364 MIRROR_RESYNC 8 mirrors 000758 1788971227.408119 MIRROR_EXTEND to 9 mirrors 000759 1788971228.241609 mapwrite 0x00636e thru 0x010212 (0x9ea5 bytes) 000760 1788971228.427808 write 0x00b92d thru 0x0106ad (0x4d81 bytes) 000761 1788971228.428939 punch from 0x0165fc to 0x01c971 (0x6376 bytes) 000763 1788971228.444459 mapwrite 0x00d412 thru 0x00ddc8 (0x9b7 bytes) 000764 1788971228.486322 mapread 0x00bc47 thru 0x00db2c (0x1ee6 bytes) 000765 1788971228.487307 punch from 0x033e2c to 0x038843 (0x4a18 bytes) 000767 1788971228.495657 punch from 0x00c19a to 0x012ef9 (0x6d60 bytes) 000768 1788971228.500944 mapread 0x01d012 thru 0x022586 (0x5575 bytes) 000769 1788971228.516793 trunc from 0x040000 to 0x00e1f0 (0x31e0f bytes) 000771 1788971228.553288 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xe1f1): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe1f1): Mirror 19: 0x887bdd61 (stale) Mirror 21: 0x887bdd61 (stale) Mirror 22: 0x887bdd61 (stale) Mirror 23: 0x887bdd61 (stale) Mirror 25: 0x887bdd61 (stale) Mirror 26: 0x887bdd61 (stale) Mirror 27: 0x887bdd61 (stale) Mirror 28: 0x887bdd61 (stale) CRC-32 checksum value for chunk [0, 0xe1f1): Mirror 1: 0x7b3a1d79 Mirror 29: 0x7b3a1d79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe1f1: skipped 000772 1788971230.043844 mapread 0x00488e thru 0x005640 (0xdb3 bytes) 000773 1788971230.105879 mapwrite 0x01f507 thru 0x021b49 (0x2643 bytes) 000776 1788971230.387858 write 0x0004cc thru 0x0072b5 (0x6dea bytes) 000779 1788971230.414794 write 0x00e2c6 thru 0x01ac91 (0xc9cc bytes) 000780 1788971230.429121 mapread 0x018748 thru 0x01f965 (0x721e bytes) 000781 1788971230.437652 trunc from 0x021b4a to 0x014491 (0xd6b8 bytes) 000783 1788971230.487856 mapwrite 0x00fcfa thru 0x013f9d (0x42a4 bytes) 000785 1788971230.595975 trunc from 0x014492 to 0x00a32c (0xa165 bytes) 000787 1788971230.631331 mapread 0x00239f thru 0x003528 (0x118a bytes) 000789 1788971230.642982 mapread 0x004dd7 thru 0x00a32c (0x5556 bytes) 000790 1788971230.653999 mapread 0x0068b0 thru 0x00a32c (0x3a7d bytes) 000791 1788971230.654883 trunc from 0x00a32d to 0x0334b9 (0x2918d bytes) 000792 1788971230.690434 mapread 0x003d39 thru 0x00a6ee (0x69b6 bytes) 000793 1788971230.699760 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x334ba): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334ba): Mirror 19: 0x79c67925 (stale) Mirror 21: 0x79c67925 (stale) Mirror 22: 0x79c67925 (stale) Mirror 23: 0x79c67925 (stale) Mirror 25: 0x79c67925 (stale) Mirror 26: 0x79c67925 (stale) Mirror 27: 0x79c67925 (stale) Mirror 28: 0x79c67925 (stale) Mirror 29: 0x1b55f0c5 (stale) CRC-32 checksum value for chunk [0, 0x334ba): Mirror 1: 0x5f2d5075 Mirror 30: 0x5f2d5075 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334ba: skipped 000794 1788971232.484446 trunc from 0x0334ba to 0x025e40 (0xd679 bytes) 000796 1788971232.627624 punch from 0x013097 to 0x01d29c (0xa206 bytes) 000797 1788971232.642740 mapread 0x01fa14 thru 0x025e40 (0x642d bytes) 000798 1788971232.678206 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25e41): Mirror 19: 0x87160b49 (stale) Mirror 21: 0x87160b49 (stale) Mirror 23: 0x87160b49 (stale) Mirror 25: 0x87160b49 (stale) Mirror 26: 0x87160b49 (stale) Mirror 27: 0x87160b49 (stale) Mirror 28: 0x87160b49 (stale) Mirror 29: 0x7d6ef2fa (stale) Mirror 30: 0x298e2233 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e41): Mirror 19: 0x87160b49 (stale) Mirror 21: 0x87160b49 (stale) Mirror 23: 0x87160b49 (stale) Mirror 25: 0x87160b49 (stale) Mirror 26: 0x87160b49 (stale) Mirror 27: 0x87160b49 (stale) Mirror 28: 0x87160b49 (stale) Mirror 29: 0x7d6ef2fa (stale) Mirror 30: 0x298e2233 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e41: skipped 000799 1788971233.320884 write 0x020a9b thru 0x030682 (0xfbe8 bytes) 000800 1788971233.331812 mapread 0x013769 thru 0x020ee5 (0xd77d bytes) 000803 1788971233.385313 mapwrite 0x03d4f3 thru 0x03ffff (0x2b0d bytes) 000804 1788971233.550437 punch from 0x003ed3 to 0x00bb15 (0x7c43 bytes) 000807 1788971233.557876 write 0x030a82 thru 0x037a2c (0x6fab bytes) 000808 1788971233.559698 trunc from 0x040000 to 0x020b7d (0x1f482 bytes) 000811 1788971233.599965 MIRROR_RESYNC 10 mirrors 000812 1788971236.262286 write 0x03d16c thru 0x03dbf0 (0xa85 bytes) 000814 1788971236.399954 trunc from 0x03dbf1 to 0x00e99b (0x2f255 bytes) 000816 1788971236.434337 mapread 0x0073e4 thru 0x00e99b (0x75b8 bytes) 000817 1788971236.469604 punch from 0x00ce71 to 0x00e99b (0x1b2b bytes) 000818 1788971236.485801 mapwrite 0x03d7f6 thru 0x03ffff (0x280a bytes) 000819 1788971236.681984 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xa0bd0e38 (stale) Mirror 21: 0xa0bd0e38 (stale) Mirror 23: 0xa0bd0e38 (stale) Mirror 25: 0xa0bd0e38 (stale) Mirror 26: 0xa0bd0e38 (stale) Mirror 27: 0xa0bd0e38 (stale) Mirror 28: 0xa0bd0e38 (stale) Mirror 29: 0xa0bd0e38 (stale) Mirror 30: 0xa0bd0e38 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc6f372d3 Mirror 31: 0xc6f372d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000820 1788971237.716657 write 0x00c4cc thru 0x010725 (0x425a bytes) 000822 1788971237.821922 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xa0bd0e38 (stale) Mirror 21: 0xa0bd0e38 (stale) Mirror 23: 0xa0bd0e38 (stale) Mirror 25: 0xa0bd0e38 (stale) Mirror 26: 0xa0bd0e38 (stale) Mirror 27: 0xa0bd0e38 (stale) Mirror 28: 0xa0bd0e38 (stale) Mirror 29: 0xa0bd0e38 (stale) Mirror 30: 0xa0bd0e38 (stale) Mirror 31: 0xc6f372d3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x12e2bc96 Mirror 32: 0x12e2bc96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000823 1788971239.566278 mapwrite 0x0264ff thru 0x028dea (0x28ec bytes) 000829 1788971239.729510 mapwrite 0x006709 thru 0x006a66 (0x35e bytes) 000830 1788971239.772945 punch from 0x025ff3 to 0x034dc7 (0xedd5 bytes) 000833 1788971239.783819 write 0x01f478 thru 0x023035 (0x3bbe bytes) 000835 1788971239.790892 read 0x009cb2 thru 0x019718 (0xfa67 bytes) 000836 1788971239.800574 write 0x01264e thru 0x01a8f5 (0x82a8 bytes) 000841 1788971239.808816 write 0x03eec2 thru 0x03ffff (0x113e bytes) 000842 1788971239.816551 read 0x03956a thru 0x03fd13 (0x67aa bytes) 000843 1788971239.826507 trunc from 0x040000 to 0x01d412 (0x22bed bytes) 000844 1788971239.851932 trunc from 0x01d413 to 0x02bfe1 (0xebcf bytes) 000846 1788971239.874587 read 0x021185 thru 0x02af41 (0x9dbd bytes) 000847 1788971239.889533 punch from 0x0299fd to 0x02bfe1 (0x25e5 bytes) 000848 1788971239.903234 mapread 0x024399 thru 0x026132 (0x1d9a bytes) 000850 1788971239.905610 read 0x01dfd2 thru 0x029174 (0xb1a3 bytes) 000853 1788971239.906861 mapread 0x02b4a6 thru 0x02bfe1 (0xb3c bytes) 000854 1788971239.907848 trunc from 0x02bfe2 to 0x024670 (0x7971 bytes) 000857 1788971239.934797 mapread 0x01fa43 thru 0x023764 (0x3d22 bytes) 000858 1788971239.935649 punch from 0x00dd8d to 0x012a50 (0x4cc4 bytes) 000859 1788971239.941650 trunc from 0x024671 to 0x036841 (0x121d1 bytes) 000860 1788971239.966539 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x36842): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36842): Mirror 19: 0xccdc959b (stale) Mirror 21: 0xccdc959b (stale) Mirror 23: 0xccdc959b (stale) Mirror 25: 0xccdc959b (stale) Mirror 26: 0xccdc959b (stale) Mirror 27: 0xccdc959b (stale) Mirror 28: 0xccdc959b (stale) Mirror 29: 0xccdc959b (stale) Mirror 30: 0xccdc959b (stale) Mirror 31: 0x184695e5 (stale) Mirror 32: 0x5424ff93 (stale) CRC-32 checksum value for chunk [0, 0x36842): Mirror 1: 0x9c4279d0 Mirror 33: 0x9c4279d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36842: skipped 000862 1788971241.227551 punch from 0x02ae42 to 0x036841 (0xba00 bytes) 000863 1788971241.369790 punch from 0x0044e5 to 0x00afa6 (0x6ac2 bytes) 000864 1788971241.378146 trunc from 0x036842 to 0x008138 (0x2e709 bytes) 000865 1788971241.414701 mapwrite 0x021fda thru 0x030f04 (0xef2b bytes) 000866 1788971241.608281 trunc from 0x030f05 to 0x039f8a (0x9086 bytes) 000867 1788971241.658485 mapwrite 0x029b5b thru 0x031eb6 (0x835c bytes) 000869 1788971241.839810 write 0x00e907 thru 0x012386 (0x3a80 bytes) 000870 1788971241.868570 mapread 0x0203a9 thru 0x024b6a (0x47c2 bytes) 000872 1788971241.872044 mapwrite 0x03ab82 thru 0x03ffff (0x547e bytes) 000873 1788971242.002919 trunc from 0x040000 to 0x01cea0 (0x2315f bytes) 000874 1788971242.059812 mapread 0x00b117 thru 0x013267 (0x8151 bytes) 000875 1788971242.092507 mapread 0x016bca thru 0x01767a (0xab1 bytes) 000876 1788971242.105939 mapread 0x01bbba thru 0x01cea0 (0x12e7 bytes) 000877 1788971242.107011 mapwrite 0x010fe9 thru 0x01e380 (0xd398 bytes) 000878 1788971242.296636 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1e381): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e381): Mirror 19: 0x3a6d0546 (stale) Mirror 21: 0x3a6d0546 (stale) Mirror 23: 0x3a6d0546 (stale) Mirror 25: 0x3a6d0546 (stale) Mirror 26: 0x3a6d0546 (stale) Mirror 27: 0x3a6d0546 (stale) Mirror 28: 0x3a6d0546 (stale) Mirror 29: 0x3a6d0546 (stale) Mirror 30: 0x3a6d0546 (stale) Mirror 31: 0x3a6d0546 (stale) Mirror 32: 0xef456718 (stale) Mirror 33: 0xe340791b (stale) CRC-32 checksum value for chunk [0, 0x1e381): Mirror 1: 0xbc78da20 Mirror 34: 0xbc78da20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e381: skipped 000879 1788971243.815326 mapread 0x0199c8 thru 0x01ccf0 (0x3329 bytes) 000880 1788971243.881802 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e381): Mirror 19: 0x3a6d0546 (stale) Mirror 21: 0x3a6d0546 (stale) Mirror 23: 0x3a6d0546 (stale) Mirror 25: 0x3a6d0546 (stale) Mirror 26: 0x3a6d0546 (stale) Mirror 27: 0x3a6d0546 (stale) Mirror 28: 0x3a6d0546 (stale) Mirror 29: 0x3a6d0546 (stale) Mirror 31: 0x3a6d0546 (stale) Mirror 32: 0xef456718 (stale) Mirror 33: 0xe340791b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e381): Mirror 19: 0x3a6d0546 (stale) Mirror 21: 0x3a6d0546 (stale) Mirror 23: 0x3a6d0546 (stale) Mirror 25: 0x3a6d0546 (stale) Mirror 26: 0x3a6d0546 (stale) Mirror 27: 0x3a6d0546 (stale) Mirror 28: 0x3a6d0546 (stale) Mirror 29: 0x3a6d0546 (stale) Mirror 31: 0x3a6d0546 (stale) Mirror 32: 0xef456718 (stale) Mirror 33: 0xe340791b (stale) CRC-32 checksum value for chunk [0, 0x1e381): Mirror 1: 0xbc78da20 Mirror 34: 0xbc78da20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e381: skipped 000881 1788971244.889295 MIRROR_RESYNC 13 mirrors 000882 1788971249.768391 read 0x019edb thru 0x01e380 (0x44a6 bytes) 000883 1788971249.776197 mapwrite 0x00d74d thru 0x00ee7a (0x172e bytes) 000884 1788971249.925793 mapread 0x015712 thru 0x0171e6 (0x1ad5 bytes) 000885 1788971249.940250 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e381): Mirror 21: 0xbc78da20 (stale) Mirror 23: 0xbc78da20 (stale) Mirror 25: 0xbc78da20 (stale) Mirror 26: 0xbc78da20 (stale) Mirror 27: 0xbc78da20 (stale) Mirror 28: 0xbc78da20 (stale) Mirror 29: 0xbc78da20 (stale) Mirror 31: 0xbc78da20 (stale) Mirror 32: 0xbc78da20 (stale) Mirror 33: 0xbc78da20 (stale) Mirror 34: 0xbc78da20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e381): Mirror 21: 0xbc78da20 (stale) Mirror 23: 0xbc78da20 (stale) Mirror 25: 0xbc78da20 (stale) Mirror 26: 0xbc78da20 (stale) Mirror 27: 0xbc78da20 (stale) Mirror 28: 0xbc78da20 (stale) Mirror 29: 0xbc78da20 (stale) Mirror 31: 0xbc78da20 (stale) Mirror 32: 0xbc78da20 (stale) Mirror 33: 0xbc78da20 (stale) Mirror 34: 0xbc78da20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e381: skipped 000886 1788971250.436231 punch from 0x00f303 to 0x01d624 (0xe322 bytes) 000887 1788971250.455379 MIRROR_RESYNC 12 mirrors 000888 1788971253.510773 mapwrite 0x007ecb thru 0x013cf2 (0xbe28 bytes) 000891 1788971253.799093 MIRROR_RESYNC 12 mirrors 000893 1788971257.533347 mapwrite 0x01813f thru 0x0203ea (0x82ac bytes) 000894 1788971257.790679 punch from 0x01ba7b to 0x0203ea (0x4970 bytes) 000895 1788971257.796513 mapread 0x00cb14 thru 0x017772 (0xac5f bytes) 000896 1788971257.806855 mapread 0x00bc3f thru 0x00be9a (0x25c bytes) 000897 1788971257.811800 mapread 0x00f14c thru 0x0137cb (0x4680 bytes) 000898 1788971257.813321 write 0x017a05 thru 0x01f3ee (0x79ea bytes) 000899 1788971257.819810 trunc from 0x0203eb to 0x026ce2 (0x68f8 bytes) 000900 1788971257.848583 punch from 0x02239f to 0x026ce2 (0x4944 bytes) 000901 1788971257.864673 mapread 0x020571 thru 0x026ce2 (0x6772 bytes) truncating to largest ever: 0x3fa91 000902 1788971257.872628 trunc from 0x026ce3 to 0x03fa90 (0x18dae bytes) 000904 1788971257.888195 mapread 0x01dcc8 thru 0x01e27c (0x5b5 bytes) 000905 1788971257.893470 MIRROR_RESYNC 12 mirrors 000906 1788971261.862818 write 0x0229cd thru 0x029158 (0x678c bytes) 000908 1788971261.947594 mapread 0x004f9f thru 0x005c9b (0xcfd bytes) 000910 1788971261.959662 mapwrite 0x03f332 thru 0x03ffff (0xcce bytes) 000912 1788971262.081883 read 0x02bc37 thru 0x032392 (0x675c bytes) 000913 1788971262.094713 punch from 0x00fd69 to 0x01c781 (0xca19 bytes) 000914 1788971262.099498 read 0x02e842 thru 0x0375ce (0x8d8d bytes) 000915 1788971262.111252 mapread 0x03eea8 thru 0x03ffff (0x1158 bytes) 000916 1788971262.112619 punch from 0x025df1 to 0x035018 (0xf228 bytes) 000918 1788971262.121900 MIRROR_RESYNC 12 mirrors 000921 1788971267.555456 mapread 0x022f5c thru 0x02f16a (0xc20f bytes) 000922 1788971267.584419 write 0x008542 thru 0x015e30 (0xd8ef bytes) 000923 1788971267.667783 trunc from 0x040000 to 0x01f6bf (0x20940 bytes) 000925 1788971267.695071 mapread 0x00cc14 thru 0x00dd40 (0x112d bytes) 000927 1788971267.696455 punch from 0x006425 to 0x010894 (0xa470 bytes) 000929 1788971267.734387 trunc from 0x01f6c0 to 0x02230c (0x2c4d bytes) 000930 1788971267.756686 write 0x00b77b thru 0x01655e (0xade4 bytes) 000932 1788971267.774368 punch from 0x00b2ba to 0x011875 (0x65bc bytes) 000936 1788971267.798453 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2230d): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2230d): Mirror 21: 0x14451467 (stale) Mirror 23: 0x14451467 (stale) Mirror 25: 0x14451467 (stale) Mirror 26: 0x14451467 (stale) Mirror 27: 0x14451467 (stale) Mirror 28: 0x14451467 (stale) Mirror 29: 0x14451467 (stale) Mirror 31: 0x14451467 (stale) Mirror 32: 0x14451467 (stale) Mirror 33: 0x14451467 (stale) Mirror 34: 0x14451467 (stale) CRC-32 checksum value for chunk [0, 0x2230d): Mirror 1: 0xfcbceb99 Mirror 35: 0xfcbceb99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2230d: skipped 000937 1788971269.049586 mapread 0x00fe74 thru 0x0126bf (0x284c bytes) 000939 1788971269.083879 mapread 0x00f9a1 thru 0x01ef87 (0xf5e7 bytes) 000940 1788971269.104834 mapread 0x003922 thru 0x006e7d (0x355c bytes) 000942 1788971269.133229 write 0x01e02f thru 0x02c4c6 (0xe498 bytes) 000943 1788971269.217234 write 0x035e25 thru 0x03d6b1 (0x788d bytes) 000944 1788971269.218730 mapwrite 0x03e7cb thru 0x03ffff (0x1835 bytes) 000945 1788971269.339496 MIRROR_RESYNC 13 mirrors 000948 1788971276.101860 write 0x013856 thru 0x0190ab (0x5856 bytes) 000949 1788971276.258544 MIRROR_RESYNC 13 mirrors 000950 1788971283.537340 trunc from 0x040000 to 0x008495 (0x37b6a bytes) 000952 1788971283.675050 write 0x024a7d thru 0x02cf76 (0x84fa bytes) 000953 1788971283.678187 trunc from 0x02cf77 to 0x011c3b (0x1b33b bytes) 000954 1788971283.713766 write 0x016b84 thru 0x025afe (0xef7b bytes) 000955 1788971283.722359 write 0x02be26 thru 0x02f883 (0x3a5e bytes) 000958 1788971283.742374 MIRROR_RESYNC 13 mirrors 000963 1788971288.397199 punch from 0x020823 to 0x02313d (0x291b bytes) 000964 1788971288.496729 trunc from 0x02f884 to 0x019ea6 (0x159dd bytes) 000966 1788971288.532729 punch from 0x00d647 to 0x00dc4a (0x604 bytes) 000967 1788971288.538280 mapread 0x00e3d0 thru 0x01101b (0x2c4c bytes) 000968 1788971288.556253 trunc from 0x019ea7 to 0x02630f (0xc469 bytes) 000970 1788971288.578472 MIRROR_RESYNC 13 mirrors 000972 1788971293.276483 mapread 0x011504 thru 0x01bca2 (0xa79f bytes) 000973 1788971293.290768 mapread 0x014fed thru 0x01c816 (0x782a bytes) 000974 1788971293.296060 punch from 0x01a151 to 0x02376b (0x961b bytes) 000975 1788971293.360429 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x26310): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26310): Mirror 21: 0x79af149c (stale) Mirror 23: 0x79af149c (stale) Mirror 25: 0x79af149c (stale) Mirror 26: 0x79af149c (stale) Mirror 27: 0x79af149c (stale) Mirror 28: 0x79af149c (stale) Mirror 29: 0x79af149c (stale) Mirror 31: 0x79af149c (stale) Mirror 32: 0x79af149c (stale) Mirror 33: 0x79af149c (stale) Mirror 34: 0x79af149c (stale) Mirror 35: 0x79af149c (stale) CRC-32 checksum value for chunk [0, 0x26310): Mirror 1: 0x79af149c Mirror 36: 0x79af149c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26310: skipped 000976 1788971294.759297 punch from 0x01d95c to 0x02630f (0x89b4 bytes) 000981 1788971294.949128 mapwrite 0x02249b thru 0x026e4b (0x49b1 bytes) 000982 1788971295.252262 write 0x031300 thru 0x03f0f6 (0xddf7 bytes) 000984 1788971295.264464 mapwrite 0x000fd1 thru 0x00a0ef (0x911f bytes) 000989 1788971295.510153 punch from 0x033a1a to 0x037fda (0x45c1 bytes) 000990 1788971295.540652 punch from 0x03d6cb to 0x03f0f6 (0x1a2c bytes) 000992 1788971295.572154 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3f0f7): Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0f7): Mirror 21: 0x1fdac02b (stale) Mirror 23: 0x1fdac02b (stale) Mirror 25: 0x1fdac02b (stale) Mirror 26: 0x1fdac02b (stale) Mirror 27: 0x1fdac02b (stale) Mirror 28: 0x1fdac02b (stale) Mirror 29: 0x1fdac02b (stale) Mirror 31: 0x1fdac02b (stale) Mirror 32: 0x1fdac02b (stale) Mirror 33: 0x1fdac02b (stale) Mirror 34: 0x1fdac02b (stale) Mirror 35: 0x1fdac02b (stale) Mirror 36: 0x1fdac02b (stale) CRC-32 checksum value for chunk [0, 0x3f0f7): Mirror 1: 0x1ca796d0 Mirror 37: 0x1ca796d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0f7: skipped 000994 1788971297.224224 punch from 0x015337 to 0x019a1a (0x46e4 bytes) 000995 1788971297.384702 trunc from 0x03f0f7 to 0x01df07 (0x211ef bytes) 000996 1788971297.397311 write 0x036e64 thru 0x03ffff (0x919c bytes) 000997 1788971297.398637 write 0x00dd29 thru 0x0126a3 (0x497b bytes) 000999 1788971297.404239 write 0x03cf0a thru 0x03ffff (0x30f6 bytes) All operations completed A-OK! PASS 70a (210s) == sanity-flr test 71: check mirror extend parent fid ==== 12:28:25 (1788971305) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 12:28:31 (1788971311) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19796 s, 10.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235822 s, 4.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 12:28:40 (1788971320) Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre2 Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6081728 .. Extending file size to 6321472 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 6904160 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8448544 .. Extending file size to 8623584 .. Extending file size to 7119232 .. Extending file size to 7557568 .. Extending file size to 8289984 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8617344 .. Extending file size to 8823680 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9078304 .. Extending file size to 8935776 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57419 57420 57422 57425 57429 done 192.168.202.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg239-client.virtnet /mnt/lustre2 (opts:) 192.168.202.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg239-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (87s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:30:07 (1788971407) Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre2 Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7714848 .. Extending file size to 7700384 .. Extending file size to 9286432 .. Extending file size to 8256608 .. Extending file size to 8534848 .. Extending file size to 8920224 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59093 59094 59095 59096 59100 done 192.168.202.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg239-client.virtnet /mnt/lustre2 (opts:) 192.168.202.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg239-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:31:28 (1788971488) Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.098326 s, 21.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.543031 s, 3.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x650cbd4b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg239-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:31:49 (1788971509) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0824763 s, 25.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 12:31:56 (1788971516) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:32:06 (1788971526) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225093 s, 9.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:32:17 (1788971537) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.458305 s, 4.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:32:26 (1788971546) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209702 s, 10.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:33:00 (1788971580) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg239-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633473 s, 16.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:33:59 (1788971639) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg239-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247761 s, 8.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:34:57 (1788971697) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg239-server mds-ost sync done. Creating new pool test_204f oleg239-server: Pool lustre.test_204f created Adding targets to pool oleg239-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg239-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203472 s, 10.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg239-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg239-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg239-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:36:17 (1788971777) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:36:24 (1788971784) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:36:33 (1788971793) oleg239-server: Pool lustre.test_206 created Adding targets to pool oleg239-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg239-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg239-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg239-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg239-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:36:57 (1788971817) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471701 s, 22.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364015 s, 28.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5168aad Mirror 3: 0x5168aad PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:37:07 (1788971827) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:37:11 (1788971831) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:37:15 (1788971835) Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.123932 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2390000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x7e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x7e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x7e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x24e0000 Read this (4 bytes): BBBB 192.168.202.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg239-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:37:49 (1788971869) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0396122 s, 207 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:37:56 (1788971876) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061705 s, 17.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f7 lmm_fid: [0x200000404:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f7 lmm_fid: [0x200000404:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834423 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-09 12:37:57.000000000 -0400 Modify: 2026-09-09 12:37:57.000000000 -0400 Change: 2026-09-09 12:37:59.000000000 -0400 Birth: 2026-09-09 12:37:58.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:38:03 (1788971883) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:38:06 (1788971886) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.447218 s, 91.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0470626 s, 87.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 9 12:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f9:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 9 12:38 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:38:14 (1788971894) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:38:20 (1788971900) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1941 sec =========== 12:38:28 (1788971908) === sanity-flr: start cleanup 12:38:29 (1788971909) === === sanity-flr: finish cleanup 12:38:35 (1788971915) ===