-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 18:46:41 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg352-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:47:01 (1781218021) === oleg352-client.virtnet: executing check_config_client /mnt/lustre oleg352-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg352-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ee910037800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ee910037800.idle_timeout=debug disable quota as required oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:47:37 (1781218057) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7570] == sanity-flr test 0a: lfs mirror create with -N option == 18:47:40 (1781218060) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:47:58 (1781218078) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:48:02 (1781218082) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:48:05 (1781218085) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:48:29 (1781218109) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:48:33 (1781218113) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 18:48:36 (1781218116) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ee910037800.stats=clear osc.lustre-OST0001-osc-ffff9ee910037800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:48:44 (1781218124) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:48:56 (1781218136) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:49:06 (1781218146) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:49:10 (1781218150) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:49:19 (1781218159) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 18:49:30 (1781218170) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:49:39 (1781218179) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:49:43 (1781218183) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:49:49 (1781218189) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:49:59 (1781218199) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:50:06 (1781218206) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.399201 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227063 s, 4.6 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:50:17 (1781218217) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.262054 s, 28.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:50:26 (1781218226) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:50:39 (1781218239) Stopping /mnt/lustre-ost1 (opts:) on oleg352-server oleg352-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ee910037800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ee910037800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg352-server: oleg352-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg352-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg352-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg352-server oleg352-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ee910037800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ee910037800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg352-server: oleg352-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg352-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (130s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:52:49 (1781218369) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.438306 s, 19.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg352-server oleg352-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ee910037800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ee910037800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg352-server: oleg352-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg352-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:53:32 (1781218412) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg352-server oleg352-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ee910037800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ee910037800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg352-server: oleg352-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg352-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg352-server oleg352-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ee910037800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ee910037800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg352-server: oleg352-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg352-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (70s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:54:42 (1781218482) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.40731 s, 11.9 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 18:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.918s user 0m0.001s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg352-server oleg352-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ee910037800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ee910037800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.964s user 0m0.000s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg352-server: oleg352-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg352-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg352-server oleg352-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ee910037800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ee910037800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.159s user 0m0.001s sys 0m0.159s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg352-server: oleg352-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg352-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg352-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ee910037800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ee910037800.ost_server_uuid in FULL state after 0 sec PASS 33b (75s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:55:58 (1781218558) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:56:01 (1781218561) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:56:04 (1781218564) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 18:56:07 (1781218567) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 18:56:14 (1781218574) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.473852 s, 26.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:56:25 (1781218585) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076047 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400013 s, 26.2 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:56:32 (1781218592) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:56:39 (1781218599) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000728588 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000955816 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000834991 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00169242 s, 605 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 18:56:49 (1781218609) create test files with size 15753989 .. 0d52552df70f9b62e4041e83e46735e4 - 2b2b24750fe08b024f2deaf01f371906 - 22c1f254b5f3f9400e8141ee3dccf97f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg352-client.virtnet /mnt/lustre (opts:) Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 18:57:10 (1781218630) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:57:27 (1781218647) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:57:35 (1781218655) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898846 s, 11.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646533 s, 16.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 18:57:43 (1781218663) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15442 s, 13.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.51134 s, 8.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24488 s, 8.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 18:57:57 (1781218677) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 18:58:01 (1781218681) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:58:04 (1781218684) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404366 s, 25.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 18:58:12 (1781218692) llite.lustre-ffff9ee907f9e000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 44a: lfs mirror split check =========== 18:58:19 (1781218699) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:58:22 (1781218702) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:58:31 (1781218711) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.271361 s, 38.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1a:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg352-client.virtnet /mnt/lustre (opts:) Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:58:41 (1781218721) llite.lustre-ffff9ee910c90000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.259924 s, 40.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg352-client.virtnet /mnt/lustre (opts:) Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9ee921f2b800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:58:53 (1781218733) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:59:27 (1781218767) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:00:04 (1781218804) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 19:00:08 (1781218808) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.229237 s, 13.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.177198 s, 17.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbb20fec2 (nosync,stale) Mirror 2: 0xbb20fec2 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 45b3fa7e08ef35ac4d397bf0fea2421c - mirror 1 checksum: f2315711d9a994a1dd73950613589373 - mirror 2 checksum: 45b3fa7e08ef35ac4d397bf0fea2421c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 19:00:10 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 19:00:10 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 45b3fa7e08ef35ac4d397bf0fea2421c - mirror 1 checksum: f2315711d9a994a1dd73950613589373 - mirror 2 checksum: f2315711d9a994a1dd73950613589373 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 19:00:10 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 19:00:10 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:00:22 (1781218822) Failing mds1 on oleg352-server Stopping /mnt/lustre-mds1 (opts:) on oleg352-server 19:00:29 (1781218829) shut down facet: mds1 facet_host: oleg352-server facet_failover_host: oleg352-server Failover mds1 to oleg352-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 19:01:00 (1781218860) targets are mounted 19:01:00 (1781218860) facet_failover done oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435063 s, 24.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:01:16 (1781218876) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:01:19 (1781218879) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:01:23 (1781218883) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0116624 s, 1.4 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:01:31 (1781218891) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:01:35 (1781218895) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:01:38 (1781218898) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:02:03 (1781218923) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:02:31 (1781218951) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:02:55 (1781218975) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:03:20 (1781219000) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 19:03:28 (1781219008) mirror create pid 54144 mirror split pid 54145 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54144 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54145 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 19:04:36 (1781219076) Seed set to 9078 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000009 1781219078.452035 MIRROR_EXTEND to 2 mirrors 000010 1781219078.868177 MIRROR_EXTEND to 3 mirrors 000011 1781219079.216682 mapwrite 0x0106e2 thru 0x01baa9 (0xb3c8 bytes) 000013 1781219079.426176 mapread 0x00e87a thru 0x016089 (0x7810 bytes) 000016 1781219079.438475 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1baaa): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1baaa): Mirror 2: 0xf83a37ca (stale) Mirror 3: 0xf83a37ca (stale) CRC-32 checksum value for chunk [0, 0x1baaa): Mirror 1: 0x71c80049 Mirror 4: 0x71c80049 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1baaa: skipped truncating to largest ever: 0x6c53 000017 1781219080.406244 trunc from 0x01baaa to 0x006c52 (0x14e57 bytes) 000018 1781219080.610766 mapread 0x0042ea thru 0x006c52 (0x2969 bytes) 000019 1781219080.663426 read 0x0000a0 thru 0x00668f (0x65f0 bytes) 000021 1781219080.681825 write 0x008578 thru 0x011221 (0x8caa bytes) 000022 1781219080.683337 mapwrite 0x017055 thru 0x01f26c (0x8218 bytes) truncating to largest ever: 0x38f75 000024 1781219080.800984 trunc from 0x01f26d to 0x038f74 (0x19d08 bytes) 000026 1781219080.823861 mapread 0x02c88d thru 0x033248 (0x69bc bytes) 000027 1781219080.837320 mapread 0x01e4ed thru 0x0233f2 (0x4f06 bytes) 000028 1781219080.860809 MIRROR_RESYNC 4 mirrors 000029 1781219081.784680 write 0x024e41 thru 0x034743 (0xf903 bytes) 000032 1781219081.902277 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f75): Mirror 3: 0xb76a2988 (stale) Mirror 4: 0xb76a2988 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f75): Mirror 3: 0xb76a2988 (stale) Mirror 4: 0xb76a2988 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f75: skipped 000033 1781219082.444138 mapread 0x0376f0 thru 0x038f74 (0x1885 bytes) 000034 1781219082.456836 mapread 0x00c517 thru 0x00f5be (0x30a8 bytes) 000035 1781219082.467058 read 0x00d1ac thru 0x014d18 (0x7b6d bytes) 000036 1781219082.468030 trunc from 0x038f75 to 0x036a8f (0x24e5 bytes) 000037 1781219082.497388 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36a90): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a90): Mirror 3: 0xbfc352b8 (stale) Mirror 4: 0xbfc352b8 (stale) CRC-32 checksum value for chunk [0, 0x36a90): Mirror 1: 0xda7a6260 Mirror 5: 0xda7a6260 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a90: skipped 000038 1781219083.398306 trunc from 0x036a90 to 0x0195a6 (0x1d4e9 bytes) 000042 1781219083.508926 mapwrite 0x01c13a thru 0x01e321 (0x21e8 bytes) 000043 1781219083.608727 trunc from 0x01e322 to 0x01aa3a (0x38e7 bytes) 000045 1781219083.633119 MIRROR_RESYNC 4 mirrors 000047 1781219084.408778 trunc from 0x01aa3b to 0x009a29 (0x11011 bytes) 000049 1781219084.458790 mapwrite 0x03406f thru 0x03ffff (0xbf91 bytes) 000052 1781219084.568804 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4303b90f (stale) Mirror 4: 0x4303b90f (stale) Mirror 5: 0x4303b90f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4a9aae33 Mirror 6: 0x4a9aae33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000053 1781219085.203448 write 0x014542 thru 0x01525c (0xd1b bytes) 000055 1781219085.322711 read 0x025e58 thru 0x02e812 (0x89bb bytes) 000056 1781219085.331952 mapread 0x007804 thru 0x00d6f7 (0x5ef4 bytes) 000057 1781219085.344971 mapwrite 0x018489 thru 0x026ea8 (0xea20 bytes) 000059 1781219085.426839 read 0x03e5bb thru 0x03ffff (0x1a45 bytes) 000060 1781219085.442167 mapread 0x01470f thru 0x019bd8 (0x54ca bytes) 000061 1781219085.443630 trunc from 0x040000 to 0x019787 (0x26878 bytes) 000062 1781219085.500599 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19788): Mirror 4: 0xb1a518c5 (stale) Mirror 5: 0xb1a518c5 (stale) Mirror 6: 0xbe68b797 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19788): Mirror 4: 0xb1a518c5 (stale) Mirror 5: 0xb1a518c5 (stale) Mirror 6: 0xbe68b797 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19788: skipped truncating to largest ever: 0x3e98e 000064 1781219085.867362 trunc from 0x019788 to 0x03e98d (0x25206 bytes) 000065 1781219085.885026 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e98e): Mirror 4: 0xc151853 (stale) Mirror 6: 0x9e2561f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e98e): Mirror 4: 0xc151853 (stale) Mirror 6: 0x9e2561f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e98e: skipped 000066 1781219086.464914 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e98e): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e98e): Mirror 4: 0xc151853 (stale) Mirror 6: 0x9e2561f1 (stale) CRC-32 checksum value for chunk [0, 0x3e98e): Mirror 1: 0xd01918da Mirror 7: 0xd01918da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e98e: skipped 000067 1781219087.664684 mapread 0x0345c8 thru 0x03cda4 (0x87dd bytes) 000068 1781219087.715668 read 0x00f6ac thru 0x012d13 (0x3668 bytes) 000074 1781219087.744002 read 0x017a04 thru 0x017a33 (0x30 bytes) 000075 1781219087.770423 read 0x012faa thru 0x01c524 (0x957b bytes) 000077 1781219087.771380 mapread 0x010394 thru 0x01eefe (0xeb6b bytes) 000079 1781219087.773822 mapwrite 0x009223 thru 0x014a32 (0xb810 bytes) 000082 1781219087.981849 mapwrite 0x02c033 thru 0x039ced (0xdcbb bytes) 000083 1781219088.165815 mapread 0x012ca3 thru 0x015b8f (0x2eed bytes) 000084 1781219088.176637 MIRROR_RESYNC 4 mirrors 000085 1781219089.670184 trunc from 0x03e98e to 0x012528 (0x2c465 bytes) 000086 1781219089.787269 mapread 0x01117a thru 0x011b78 (0x9ff bytes) 000087 1781219089.809543 trunc from 0x012529 to 0x01be57 (0x992f bytes) 000088 1781219089.838829 read 0x01b73b thru 0x01be57 (0x71d bytes) 000090 1781219089.884331 MIRROR_RESYNC 4 mirrors 000092 1781219091.103208 trunc from 0x01be58 to 0x02340e (0x75b7 bytes) 000096 1781219091.186524 trunc from 0x02340f to 0x008a2d (0x1a9e1 bytes) 000097 1781219091.216014 read 0x0088b5 thru 0x008a2d (0x179 bytes) 000100 1781219091.226398 mapwrite 0x024ff6 thru 0x031eb5 (0xcec0 bytes) 000103 1781219091.362584 read 0x02359d thru 0x031b61 (0xe5c5 bytes) 000104 1781219091.364358 read 0x0017c4 thru 0x006edf (0x571c bytes) 000105 1781219091.383715 mapread 0x0142e9 thru 0x01a49e (0x61b6 bytes) 000106 1781219091.389194 trunc from 0x031eb6 to 0x00dd54 (0x24161 bytes) 000107 1781219091.449056 read 0x009d05 thru 0x00db9c (0x3e98 bytes) 000108 1781219091.450123 mapread 0x009c94 thru 0x00dd54 (0x40c1 bytes) 000109 1781219091.460926 read 0x005207 thru 0x00c513 (0x730d bytes) 000110 1781219091.461684 mapread 0x00d6f4 thru 0x00dd54 (0x661 bytes) 000114 1781219091.488398 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdd55): Mirror 6: 0xf129a888 (stale) Mirror 7: 0xf129a888 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdd55): Mirror 6: 0xf129a888 (stale) Mirror 7: 0xf129a888 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdd55: skipped 000115 1781219091.832498 mapread 0x005be0 thru 0x00c272 (0x6693 bytes) 000116 1781219091.867851 write 0x016485 thru 0x01e37a (0x7ef6 bytes) 000117 1781219091.879309 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e37b): Mirror 7: 0x99c79914 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e37b): Mirror 7: 0x99c79914 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e37b: skipped 000118 1781219092.239487 read 0x00fbdf thru 0x014f5d (0x537f bytes) 000121 1781219092.257218 MIRROR_RESYNC 2 mirrors 000122 1781219093.002039 trunc from 0x01e37b to 0x00add5 (0x135a5 bytes) 000124 1781219093.106963 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xadd6): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xadd6): Mirror 7: 0x7a64317 (stale) CRC-32 checksum value for chunk [0, 0xadd6): Mirror 1: 0x7a64317 Mirror 8: 0x7a64317 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xadd6: skipped 000125 1781219094.166459 mapwrite 0x011e66 thru 0x01a901 (0x8a9c bytes) 000127 1781219094.386521 write 0x015bd3 thru 0x01bf3f (0x636d bytes) 000129 1781219094.390326 mapwrite 0x0294a6 thru 0x02c6ea (0x3245 bytes) 000130 1781219094.501730 trunc from 0x02c6eb to 0x0084b0 (0x2423a bytes) 000132 1781219094.522297 mapwrite 0x01b826 thru 0x028711 (0xceec bytes) 000133 1781219094.641144 mapwrite 0x038006 thru 0x03ffff (0x7ffa bytes) 000135 1781219094.772241 mapread 0x03f70f thru 0x03ffff (0x8f1 bytes) 000136 1781219094.777912 MIRROR_RESYNC 3 mirrors 000138 1781219096.120026 mapread 0x00deff thru 0x0157d1 (0x78d3 bytes) 000139 1781219096.188245 mapread 0x03816b thru 0x03ffff (0x7e95 bytes) 000140 1781219096.202780 MIRROR_RESYNC 3 mirrors 000144 1781219096.577846 MIRROR_EXTEND to 4 mirrors 000145 1781219097.194009 MIRROR_EXTEND to 5 mirrors 000149 1781219098.227342 read 0x018468 thru 0x02440d (0xbfa6 bytes) 000151 1781219098.250704 MIRROR_RESYNC 5 mirrors 000152 1781219098.895362 MIRROR_RESYNC 5 mirrors 000153 1781219099.607358 mapwrite 0x01b3f8 thru 0x028e6a (0xda73 bytes) 000154 1781219099.951682 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xff48f468 (stale) Mirror 8: 0xff48f468 (stale) Mirror 9: 0xff48f468 (stale) Mirror 10: 0xff48f468 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbfe6681d Mirror 11: 0xbfe6681d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000155 1781219100.907158 mapwrite 0x00b174 thru 0x01a479 (0xf306 bytes) 000160 1781219101.151049 trunc from 0x040000 to 0x01d516 (0x22ae9 bytes) 000161 1781219101.185927 trunc from 0x01d517 to 0x0147c7 (0x8d4f bytes) 000163 1781219101.217006 mapwrite 0x0334be thru 0x03ffff (0xcb42 bytes) 000164 1781219101.348655 trunc from 0x040000 to 0x0046eb (0x3b914 bytes) 000165 1781219101.417273 mapwrite 0x000e01 thru 0x007724 (0x6924 bytes) 000166 1781219101.587663 trunc from 0x007725 to 0x005381 (0x23a3 bytes) 000167 1781219101.614182 write 0x0370de thru 0x03e871 (0x7794 bytes) 000168 1781219101.621794 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e872): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e872): Mirror 7: 0xcc743fdd (stale) Mirror 8: 0xcc743fdd (stale) Mirror 9: 0xcc743fdd (stale) Mirror 10: 0xcc743fdd (stale) Mirror 11: 0x22777a90 (stale) CRC-32 checksum value for chunk [0, 0x3e872): Mirror 1: 0xd8f6581b Mirror 12: 0xd8f6581b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e872: skipped 000171 1781219102.669456 trunc from 0x03e872 to 0x028072 (0x167ff bytes) 000172 1781219102.914220 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x28073): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28073): Mirror 7: 0x694aa635 (stale) Mirror 8: 0x694aa635 (stale) Mirror 9: 0x694aa635 (stale) Mirror 10: 0x694aa635 (stale) Mirror 11: 0x230f9fbc (stale) Mirror 12: 0xde2588b7 (stale) CRC-32 checksum value for chunk [0, 0x28073): Mirror 1: 0xde2588b7 Mirror 13: 0xde2588b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28073: skipped 000173 1781219104.057748 mapread 0x0158cc thru 0x023520 (0xdc55 bytes) 000174 1781219104.102535 write 0x011322 thru 0x013e00 (0x2adf bytes) 000176 1781219104.205293 trunc from 0x028073 to 0x02f695 (0x7623 bytes) 000178 1781219104.230280 mapwrite 0x012442 thru 0x019bb2 (0x7771 bytes) 000180 1781219104.285332 mapread 0x019772 thru 0x026b5d (0xd3ec bytes) 000181 1781219104.307998 trunc from 0x02f696 to 0x02259b (0xd0fa bytes) 000182 1781219104.325814 read 0x01b6eb thru 0x02259b (0x6eb1 bytes) 000183 1781219104.326735 write 0x02d667 thru 0x03c167 (0xeb01 bytes) 000187 1781219104.333004 trunc from 0x03c168 to 0x02778e (0x149d9 bytes) 000188 1781219104.352939 mapwrite 0x0103a5 thru 0x019765 (0x93c1 bytes) 000189 1781219104.402320 trunc from 0x02778f to 0x03b2bd (0x13b2f bytes) 000190 1781219104.421277 read 0x0389ed thru 0x03b2bd (0x28d1 bytes) 000191 1781219104.430660 read 0x02c8c2 thru 0x038421 (0xbb60 bytes) 000192 1781219104.431521 trunc from 0x03b2be to 0x01ca74 (0x1e849 bytes) 000193 1781219104.445963 mapread 0x00233f thru 0x007964 (0x5626 bytes) 000194 1781219104.447261 mapwrite 0x02033b thru 0x021611 (0x12d7 bytes) 000195 1781219104.524860 trunc from 0x021612 to 0x02a723 (0x9112 bytes) 000196 1781219104.545221 mapread 0x0064db thru 0x00d1d6 (0x6cfc bytes) 000199 1781219104.556049 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2a724): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a724): Mirror 7: 0xc72f6c44 (stale) Mirror 8: 0xc72f6c44 (stale) Mirror 9: 0xc72f6c44 (stale) Mirror 10: 0xc72f6c44 (stale) Mirror 11: 0xf726d27a (stale) Mirror 12: 0x4cb8535f (stale) Mirror 13: 0x4cb8535f (stale) CRC-32 checksum value for chunk [0, 0x2a724): Mirror 1: 0x11b367d5 Mirror 14: 0x11b367d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a724: skipped 000200 1781219105.615851 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a724): Mirror 8: 0xc72f6c44 (stale) Mirror 9: 0xc72f6c44 (stale) Mirror 10: 0xc72f6c44 (stale) Mirror 11: 0xf726d27a (stale) Mirror 12: 0x4cb8535f (stale) Mirror 13: 0x4cb8535f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a724): Mirror 8: 0xc72f6c44 (stale) Mirror 9: 0xc72f6c44 (stale) Mirror 10: 0xc72f6c44 (stale) Mirror 11: 0xf726d27a (stale) Mirror 12: 0x4cb8535f (stale) Mirror 13: 0x4cb8535f (stale) CRC-32 checksum value for chunk [0, 0x2a724): Mirror 1: 0x11b367d5 Mirror 14: 0x11b367d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a724: skipped 000201 1781219106.632527 mapread 0x003e78 thru 0x009c17 (0x5da0 bytes) truncating to largest ever: 0x3eea9 000204 1781219106.732947 trunc from 0x02a724 to 0x03eea8 (0x14785 bytes) 000205 1781219106.952146 read 0x0384a4 thru 0x03de3e (0x599b bytes) 000206 1781219106.974997 trunc from 0x03eea9 to 0x0321b6 (0xccf2 bytes) 000207 1781219107.006621 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x321b7): Mirror 9: 0xfbe60b25 (stale) Mirror 10: 0xfbe60b25 (stale) Mirror 11: 0xb05b17af (stale) Mirror 12: 0xcec2c5ea (stale) Mirror 13: 0xcec2c5ea (stale) Mirror 14: 0xc98c8066 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x321b7): Mirror 9: 0xfbe60b25 (stale) Mirror 10: 0xfbe60b25 (stale) Mirror 11: 0xb05b17af (stale) Mirror 12: 0xcec2c5ea (stale) Mirror 13: 0xcec2c5ea (stale) Mirror 14: 0xc98c8066 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x321b7: skipped 000209 1781219107.612202 read 0x00dc87 thru 0x00dc90 (0xa bytes) 000212 1781219107.623051 MIRROR_RESYNC 7 mirrors 000213 1781219108.935729 mapread 0x02ca0b thru 0x0321b6 (0x57ac bytes) 000215 1781219109.002656 write 0x005a2e thru 0x012ab5 (0xd088 bytes) 000216 1781219109.124004 mapread 0x002aff thru 0x007959 (0x4e5b bytes) 000221 1781219109.129250 read 0x02f557 thru 0x0321b6 (0x2c60 bytes) 000222 1781219109.135976 mapwrite 0x026037 thru 0x02bfa6 (0x5f70 bytes) 000223 1781219109.192793 mapread 0x02cd60 thru 0x0321b6 (0x5457 bytes) 000224 1781219109.199454 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x321b7): Mirror 9: 0xc98c8066 (stale) Mirror 10: 0xc98c8066 (stale) Mirror 11: 0xc98c8066 (stale) Mirror 13: 0xc98c8066 (stale) Mirror 14: 0xc98c8066 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x321b7): Mirror 9: 0xc98c8066 (stale) Mirror 10: 0xc98c8066 (stale) Mirror 11: 0xc98c8066 (stale) Mirror 13: 0xc98c8066 (stale) Mirror 14: 0xc98c8066 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x321b7: skipped 000225 1781219109.763860 write 0x014c74 thru 0x01b132 (0x64bf bytes) 000226 1781219109.774256 mapread 0x0209c2 thru 0x02833a (0x7979 bytes) 000228 1781219109.786443 mapwrite 0x003e2a thru 0x00c99d (0x8b74 bytes) 000230 1781219109.845441 read 0x00fe43 thru 0x013f05 (0x40c3 bytes) 000231 1781219109.852195 trunc from 0x0321b7 to 0x0317a6 (0xa10 bytes) 000232 1781219109.877375 write 0x00494b thru 0x006e16 (0x24cc bytes) 000233 1781219109.883752 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x317a7): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317a7): Mirror 9: 0x4b819d5d (stale) Mirror 10: 0x4b819d5d (stale) Mirror 11: 0x4b819d5d (stale) Mirror 13: 0x4b819d5d (stale) Mirror 14: 0x4b819d5d (stale) CRC-32 checksum value for chunk [0, 0x317a7): Mirror 1: 0x25271c39 Mirror 15: 0x25271c39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317a7: skipped 000234 1781219110.977742 mapread 0x02198b thru 0x02b243 (0x98b9 bytes) 000236 1781219111.022109 read 0x00adaf thru 0x013f15 (0x9167 bytes) 000238 1781219111.035009 write 0x00684a thru 0x00cbf0 (0x63a7 bytes) 000240 1781219111.214063 trunc from 0x0317a7 to 0x03c46c (0xacc6 bytes) 000241 1781219111.254960 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c46d): Mirror 9: 0xfc61a2f8 (stale) Mirror 10: 0xfc61a2f8 (stale) Mirror 13: 0xfc61a2f8 (stale) Mirror 14: 0xfc61a2f8 (stale) Mirror 15: 0x73ee3a70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c46d): Mirror 9: 0xfc61a2f8 (stale) Mirror 10: 0xfc61a2f8 (stale) Mirror 13: 0xfc61a2f8 (stale) Mirror 14: 0xfc61a2f8 (stale) Mirror 15: 0x73ee3a70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c46d: skipped 000242 1781219111.747233 write 0x026a49 thru 0x02c776 (0x5d2e bytes) 000243 1781219111.777014 write 0x01aff2 thru 0x02489f (0x98ae bytes) 000245 1781219111.807785 trunc from 0x03c46d to 0x0083e6 (0x34086 bytes) 000249 1781219111.846519 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x83e7): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x83e7): Mirror 9: 0xf84ac376 (stale) Mirror 10: 0xf84ac376 (stale) Mirror 13: 0xf84ac376 (stale) Mirror 14: 0xf84ac376 (stale) Mirror 15: 0xe01fbe2 (stale) CRC-32 checksum value for chunk [0, 0x83e7): Mirror 1: 0x11fd5d3f Mirror 16: 0x11fd5d3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x83e7: skipped 000250 1781219113.074444 mapread 0x004559 thru 0x0083e6 (0x3e8e bytes) 000253 1781219113.119648 read 0x006074 thru 0x0083e6 (0x2373 bytes) 000254 1781219113.127756 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x83e7): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x83e7): Mirror 9: 0xf84ac376 (stale) Mirror 10: 0xf84ac376 (stale) Mirror 13: 0xf84ac376 (stale) Mirror 14: 0xf84ac376 (stale) Mirror 15: 0xe01fbe2 (stale) CRC-32 checksum value for chunk [0, 0x83e7): Mirror 1: 0x11fd5d3f Mirror 16: 0x11fd5d3f Mirror 17: 0x11fd5d3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x83e7: skipped 000255 1781219114.176776 trunc from 0x0083e7 to 0x00dc7a (0x5894 bytes) 000256 1781219114.423368 read 0x005e81 thru 0x00ad6c (0x4eec bytes) 000258 1781219114.488449 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdc7b): Mirror 9: 0x718daf8b (stale) Mirror 13: 0x718daf8b (stale) Mirror 14: 0x718daf8b (stale) Mirror 15: 0xa47c0de6 (stale) Mirror 16: 0x163c429 (stale) Mirror 17: 0x163c429 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdc7b): Mirror 9: 0x718daf8b (stale) Mirror 13: 0x718daf8b (stale) Mirror 14: 0x718daf8b (stale) Mirror 15: 0xa47c0de6 (stale) Mirror 16: 0x163c429 (stale) Mirror 17: 0x163c429 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdc7b: skipped 000259 1781219115.214860 write 0x03a45a thru 0x03ffff (0x5ba6 bytes) 000260 1781219115.221651 read 0x02fa48 thru 0x03392d (0x3ee6 bytes) 000262 1781219115.231634 write 0x015253 thru 0x0175fa (0x23a8 bytes) 000266 1781219115.244081 trunc from 0x040000 to 0x02e1df (0x11e20 bytes) 000267 1781219115.265616 trunc from 0x02e1e0 to 0x02771f (0x6ac0 bytes) 000268 1781219115.294026 read 0x014cc9 thru 0x01ab9f (0x5ed7 bytes) 000279 1781219115.345540 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x27720): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27720): Mirror 9: 0x25624b99 (stale) Mirror 13: 0x25624b99 (stale) Mirror 14: 0x25624b99 (stale) Mirror 15: 0x8a6b594c (stale) Mirror 16: 0x36b051fa (stale) Mirror 17: 0x36b051fa (stale) CRC-32 checksum value for chunk [0, 0x27720): Mirror 1: 0x9606a143 Mirror 18: 0x9606a143 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27720: skipped 000280 1781219116.306482 read 0x01a304 thru 0x021ca3 (0x79a0 bytes) 000282 1781219116.323993 MIRROR_RESYNC 8 mirrors truncating to largest ever: 0x3f422 000283 1781219117.723931 trunc from 0x027720 to 0x03f421 (0x17d02 bytes) 000284 1781219117.804338 write 0x02aae0 thru 0x02dcbb (0x31dc bytes) 000285 1781219117.818217 mapwrite 0x020015 thru 0x0280ba (0x80a6 bytes) 000287 1781219117.883935 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f422): Mirror 9: 0x894469c1 (stale) Mirror 13: 0x894469c1 (stale) Mirror 15: 0x894469c1 (stale) Mirror 16: 0x894469c1 (stale) Mirror 17: 0x894469c1 (stale) Mirror 18: 0x894469c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f422): Mirror 9: 0x894469c1 (stale) Mirror 13: 0x894469c1 (stale) Mirror 15: 0x894469c1 (stale) Mirror 16: 0x894469c1 (stale) Mirror 17: 0x894469c1 (stale) Mirror 18: 0x894469c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f422: skipped 000289 1781219118.575040 trunc from 0x03f422 to 0x01f221 (0x20200 bytes) 000291 1781219118.598764 read 0x015dca thru 0x01acd2 (0x4f09 bytes) 000292 1781219118.612032 read 0x002a1a thru 0x007437 (0x4a1e bytes) 000293 1781219118.627448 write 0x01bc68 thru 0x022bfd (0x6f96 bytes) 000295 1781219118.631022 trunc from 0x022bfe to 0x0022dc (0x20921 bytes) 000297 1781219118.655009 read 0x000e99 thru 0x0022dc (0x1444 bytes) 000299 1781219118.657653 trunc from 0x0022dd to 0x03bfbc (0x39ce0 bytes) 000300 1781219118.676632 mapread 0x01e354 thru 0x02998a (0xb637 bytes) 000303 1781219118.697196 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfbd): Mirror 13: 0x79b23df3 (stale) Mirror 15: 0x79b23df3 (stale) Mirror 16: 0x79b23df3 (stale) Mirror 17: 0x79b23df3 (stale) Mirror 18: 0x79b23df3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfbd): Mirror 13: 0x79b23df3 (stale) Mirror 15: 0x79b23df3 (stale) Mirror 16: 0x79b23df3 (stale) Mirror 17: 0x79b23df3 (stale) Mirror 18: 0x79b23df3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfbd: skipped 000305 1781219119.217599 write 0x012957 thru 0x0227a6 (0xfe50 bytes) 000307 1781219119.234015 read 0x00e07e thru 0x01dfe9 (0xff6c bytes) 000310 1781219119.244657 mapwrite 0x038b3f thru 0x03df9b (0x545d bytes) 000311 1781219119.330201 write 0x0398d2 thru 0x03ffff (0x672e bytes) 000313 1781219119.337062 trunc from 0x040000 to 0x007213 (0x38dec bytes) 000314 1781219119.362636 MIRROR_RESYNC 6 mirrors 000317 1781219121.103522 mapwrite 0x023a18 thru 0x031266 (0xd84f bytes) 000319 1781219121.274559 write 0x0098f8 thru 0x014382 (0xaa8b bytes) 000320 1781219121.290513 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31267): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31267): Mirror 13: 0xbd93c290 (stale) Mirror 15: 0xbd93c290 (stale) Mirror 16: 0xbd93c290 (stale) Mirror 17: 0xbd93c290 (stale) Mirror 18: 0xbd93c290 (stale) CRC-32 checksum value for chunk [0, 0x31267): Mirror 1: 0xa57f8b88 Mirror 19: 0xa57f8b88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31267: skipped 000322 1781219122.286258 read 0x026193 thru 0x0269c7 (0x835 bytes) 000324 1781219122.292679 trunc from 0x031267 to 0x0084ea (0x28d7c bytes) 000326 1781219122.454336 trunc from 0x0084eb to 0x03a535 (0x3204b bytes) 000327 1781219122.474366 trunc from 0x03a536 to 0x01ff0b (0x1a62a bytes) 000329 1781219122.495666 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1ff0c): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff0c): Mirror 13: 0x334a986a (stale) Mirror 15: 0x334a986a (stale) Mirror 16: 0x334a986a (stale) Mirror 17: 0x334a986a (stale) Mirror 18: 0x334a986a (stale) Mirror 19: 0xfc7849cd (stale) CRC-32 checksum value for chunk [0, 0x1ff0c): Mirror 1: 0x334a986a Mirror 20: 0x334a986a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff0c: skipped 000330 1781219123.241432 write 0x012e2e thru 0x01625e (0x3431 bytes) 000331 1781219123.353519 write 0x011771 thru 0x014135 (0x29c5 bytes) 000332 1781219123.364476 write 0x018aa0 thru 0x02142a (0x898b bytes) 000333 1781219123.366911 read 0x016abd thru 0x02142a (0xa96e bytes) 000334 1781219123.376480 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2142b): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2142b): Mirror 13: 0x60af1998 (stale) Mirror 15: 0x60af1998 (stale) Mirror 16: 0x60af1998 (stale) Mirror 17: 0x60af1998 (stale) Mirror 18: 0x60af1998 (stale) Mirror 19: 0xa202d3da (stale) Mirror 20: 0x60af1998 (stale) CRC-32 checksum value for chunk [0, 0x2142b): Mirror 1: 0x4545468c Mirror 21: 0x4545468c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2142b: skipped 000335 1781219124.305832 mapwrite 0x01b810 thru 0x028b11 (0xd302 bytes) 000336 1781219124.502561 mapwrite 0x0023fb thru 0x007050 (0x4c56 bytes) 000337 1781219124.559587 mapread 0x0169bd thru 0x017189 (0x7cd bytes) 000338 1781219124.565725 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28b12): Mirror 13: 0xace47d54 (stale) Mirror 15: 0xace47d54 (stale) Mirror 16: 0xace47d54 (stale) Mirror 18: 0xace47d54 (stale) Mirror 19: 0xc33129eb (stale) Mirror 20: 0xace47d54 (stale) Mirror 21: 0x1f82d3b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28b12): Mirror 13: 0xace47d54 (stale) Mirror 15: 0xace47d54 (stale) Mirror 16: 0xace47d54 (stale) Mirror 18: 0xace47d54 (stale) Mirror 19: 0xc33129eb (stale) Mirror 20: 0xace47d54 (stale) Mirror 21: 0x1f82d3b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b12: skipped 000340 1781219125.153583 mapread 0x022031 thru 0x028b11 (0x6ae1 bytes) 000341 1781219125.179236 trunc from 0x028b12 to 0x02501d (0x3af4 bytes) 000342 1781219125.200604 mapwrite 0x009373 thru 0x00ccb6 (0x3944 bytes) 000343 1781219125.291999 mapwrite 0x00e38a thru 0x01389b (0x5512 bytes) 000344 1781219125.340169 write 0x02d3bc thru 0x02ff8b (0x2bd0 bytes) 000346 1781219125.342897 read 0x003268 thru 0x004f37 (0x1cd0 bytes) 000348 1781219125.344906 write 0x01d0a6 thru 0x02c558 (0xf4b3 bytes) 000349 1781219125.351841 trunc from 0x02ff8c to 0x03ebca (0xec3f bytes) 000353 1781219125.369111 mapwrite 0x017a69 thru 0x022c0b (0xb1a3 bytes) 000355 1781219125.413639 mapread 0x020784 thru 0x026726 (0x5fa3 bytes) 000356 1781219125.415516 trunc from 0x03ebcb to 0x023045 (0x1bb85 bytes) 000358 1781219125.433595 trunc from 0x023046 to 0x023d03 (0xcbe bytes) 000359 1781219125.448997 mapwrite 0x01ccfb thru 0x026d97 (0xa09d bytes) 000360 1781219125.515618 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26d98): Mirror 15: 0xb2b11840 (stale) Mirror 16: 0xb2b11840 (stale) Mirror 18: 0xb2b11840 (stale) Mirror 19: 0x8ddac3ea (stale) Mirror 20: 0xb2b11840 (stale) Mirror 21: 0xe5ae5484 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26d98): Mirror 15: 0xb2b11840 (stale) Mirror 16: 0xb2b11840 (stale) Mirror 18: 0xb2b11840 (stale) Mirror 19: 0x8ddac3ea (stale) Mirror 20: 0xb2b11840 (stale) Mirror 21: 0xe5ae5484 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26d98: skipped 000361 1781219125.887146 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x26d98): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26d98): Mirror 15: 0xb2b11840 (stale) Mirror 16: 0xb2b11840 (stale) Mirror 18: 0xb2b11840 (stale) Mirror 19: 0x8ddac3ea (stale) Mirror 20: 0xb2b11840 (stale) Mirror 21: 0xe5ae5484 (stale) CRC-32 checksum value for chunk [0, 0x26d98): Mirror 1: 0x30d244fa Mirror 22: 0x30d244fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26d98: skipped 000362 1781219126.827902 mapwrite 0x00a649 thru 0x00c5b2 (0x1f6a bytes) 000363 1781219127.076791 MIRROR_RESYNC 8 mirrors 000364 1781219128.176219 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000366 1781219128.793077 mapwrite 0x03765a thru 0x03fb02 (0x84a9 bytes) 000367 1781219129.045693 write 0x000843 thru 0x0082ec (0x7aaa bytes) 000368 1781219129.073808 mapwrite 0x034048 thru 0x03ffff (0xbfb8 bytes) 000370 1781219129.266185 mapwrite 0x029c4f thru 0x031a87 (0x7e39 bytes) 000374 1781219129.410913 read 0x01df1e thru 0x024f2e (0x7011 bytes) 000376 1781219129.437634 write 0x02d559 thru 0x035af5 (0x859d bytes) 000377 1781219129.442826 read 0x001bd0 thru 0x001f5d (0x38e bytes) 000379 1781219129.449733 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe24b2d17 (stale) Mirror 19: 0xe24b2d17 (stale) Mirror 20: 0xe24b2d17 (stale) Mirror 21: 0xe24b2d17 (stale) Mirror 22: 0xe24b2d17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe24b2d17 (stale) Mirror 19: 0xe24b2d17 (stale) Mirror 20: 0xe24b2d17 (stale) Mirror 21: 0xe24b2d17 (stale) Mirror 22: 0xe24b2d17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000381 1781219129.999138 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe24b2d17 (stale) Mirror 19: 0xe24b2d17 (stale) Mirror 20: 0xe24b2d17 (stale) Mirror 21: 0xe24b2d17 (stale) Mirror 22: 0xe24b2d17 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a9516d5 Mirror 23: 0x1a9516d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1781219131.590987 read 0x008999 thru 0x00d5b3 (0x4c1b bytes) 000384 1781219131.646928 MIRROR_RESYNC 7 mirrors 000385 1781219133.600270 mapread 0x03a950 thru 0x03cbc6 (0x2277 bytes) 000386 1781219133.641118 trunc from 0x040000 to 0x035182 (0xae7d bytes) 000388 1781219133.729316 mapread 0x024272 thru 0x02d268 (0x8ff7 bytes) 000391 1781219133.748846 write 0x0284e3 thru 0x02c0b9 (0x3bd7 bytes) 000392 1781219133.750553 mapread 0x012ecc thru 0x01f8b4 (0xc9e9 bytes) 000393 1781219133.765701 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35183): Mirror 18: 0x9e5b55c4 (stale) Mirror 19: 0x9e5b55c4 (stale) Mirror 20: 0x9e5b55c4 (stale) Mirror 21: 0x9e5b55c4 (stale) Mirror 23: 0x9e5b55c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35183): Mirror 18: 0x9e5b55c4 (stale) Mirror 19: 0x9e5b55c4 (stale) Mirror 20: 0x9e5b55c4 (stale) Mirror 21: 0x9e5b55c4 (stale) Mirror 23: 0x9e5b55c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35183: skipped 000394 1781219134.270898 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35183): Mirror 18: 0x9e5b55c4 (stale) Mirror 19: 0x9e5b55c4 (stale) Mirror 20: 0x9e5b55c4 (stale) Mirror 21: 0x9e5b55c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35183): Mirror 18: 0x9e5b55c4 (stale) Mirror 19: 0x9e5b55c4 (stale) Mirror 20: 0x9e5b55c4 (stale) Mirror 21: 0x9e5b55c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35183: skipped 000395 1781219134.643953 MIRROR_RESYNC 5 mirrors 000396 1781219136.025440 MIRROR_EXTEND to 6 mirrors 000398 1781219136.807611 trunc from 0x035183 to 0x00544b (0x2fd37 bytes) 000399 1781219136.934028 mapwrite 0x015e66 thru 0x01b115 (0x52b0 bytes) 000400 1781219137.148142 mapwrite 0x00db59 thru 0x010763 (0x2c0b bytes) 000402 1781219137.207817 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b116): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b116): Mirror 18: 0xac5fd876 (stale) Mirror 19: 0xac5fd876 (stale) Mirror 20: 0xac5fd876 (stale) Mirror 21: 0xac5fd876 (stale) Mirror 22: 0xac5fd876 (stale) CRC-32 checksum value for chunk [0, 0x1b116): Mirror 1: 0x5df13ae Mirror 23: 0x5df13ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b116: skipped 000403 1781219138.395095 read 0x00f877 thru 0x015f2a (0x66b4 bytes) 000404 1781219138.436044 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1b116): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b116): Mirror 18: 0xac5fd876 (stale) Mirror 19: 0xac5fd876 (stale) Mirror 20: 0xac5fd876 (stale) Mirror 21: 0xac5fd876 (stale) Mirror 22: 0xac5fd876 (stale) CRC-32 checksum value for chunk [0, 0x1b116): Mirror 1: 0x5df13ae Mirror 23: 0x5df13ae Mirror 24: 0x5df13ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b116: skipped 000409 1781219139.546324 write 0x03d7c8 thru 0x03ffff (0x2838 bytes) 000410 1781219139.655420 trunc from 0x040000 to 0x03d4c3 (0x2b3c bytes) 000411 1781219139.680715 trunc from 0x03d4c4 to 0x001c2c (0x3b897 bytes) 000412 1781219139.696356 mapread 0x00079b thru 0x001c2c (0x1492 bytes) 000414 1781219139.708953 mapwrite 0x03a3a2 thru 0x03ffff (0x5c5e bytes) 000415 1781219139.795340 mapwrite 0x021192 thru 0x02ccf3 (0xbb62 bytes) 000416 1781219139.884681 trunc from 0x040000 to 0x039c8d (0x6372 bytes) 000417 1781219139.903050 read 0x00c29e thru 0x0162bd (0xa020 bytes) 000419 1781219139.917741 read 0x00d19e thru 0x01695a (0x97bd bytes) 000420 1781219139.918732 read 0x023df2 thru 0x0324d1 (0xe6e0 bytes) 000422 1781219139.925812 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39c8e): Mirror 18: 0x48c7bf5e (stale) Mirror 19: 0x48c7bf5e (stale) Mirror 20: 0x48c7bf5e (stale) Mirror 21: 0x48c7bf5e (stale) Mirror 22: 0x48c7bf5e (stale) Mirror 24: 0x77e54453 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c8e): Mirror 18: 0x48c7bf5e (stale) Mirror 19: 0x48c7bf5e (stale) Mirror 20: 0x48c7bf5e (stale) Mirror 21: 0x48c7bf5e (stale) Mirror 22: 0x48c7bf5e (stale) Mirror 24: 0x77e54453 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c8e: skipped 000423 1781219140.383466 mapwrite 0x018431 thru 0x01e59c (0x616c bytes) 000424 1781219140.466965 mapread 0x00b143 thru 0x01761a (0xc4d8 bytes) 000425 1781219140.476530 read 0x029f1a thru 0x0389d4 (0xeabb bytes) 000429 1781219140.486847 read 0x012c84 thru 0x01f8ff (0xcc7c bytes) 000430 1781219140.488077 mapwrite 0x019be9 thru 0x02352e (0x9946 bytes) 000432 1781219140.538169 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39c8e): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c8e): Mirror 18: 0x48c7bf5e (stale) Mirror 19: 0x48c7bf5e (stale) Mirror 20: 0x48c7bf5e (stale) Mirror 21: 0x48c7bf5e (stale) Mirror 22: 0x48c7bf5e (stale) Mirror 24: 0x77e54453 (stale) CRC-32 checksum value for chunk [0, 0x39c8e): Mirror 1: 0xd5df306 Mirror 25: 0xd5df306 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c8e: skipped 000433 1781219141.606267 write 0x026eff thru 0x029ae2 (0x2be4 bytes) 000435 1781219141.814220 read 0x012695 thru 0x013f13 (0x187f bytes) 000436 1781219141.825326 mapwrite 0x021e7a thru 0x028558 (0x66df bytes) 000437 1781219141.995706 mapwrite 0x02f033 thru 0x03b2e2 (0xc2b0 bytes) 000438 1781219142.148031 mapwrite 0x01e550 thru 0x02cebf (0xe970 bytes) 000439 1781219142.198712 mapwrite 0x018571 thru 0x02501d (0xcaad bytes) 000440 1781219142.272300 read 0x024b54 thru 0x027fbc (0x3469 bytes) 000442 1781219142.274044 mapwrite 0x015849 thru 0x02102e (0xb7e6 bytes) 000443 1781219142.355986 write 0x03e704 thru 0x03ffff (0x18fc bytes) 000444 1781219142.357604 write 0x03bcd9 thru 0x03ffff (0x4327 bytes) 000446 1781219142.359689 trunc from 0x040000 to 0x01e220 (0x21ddf bytes) 000448 1781219142.376621 read 0x000bd8 thru 0x00ce46 (0xc26f bytes) 000449 1781219142.386099 mapread 0x0132d8 thru 0x01e220 (0xaf49 bytes) 000450 1781219142.391450 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e221): Mirror 18: 0x7a359e0 (stale) Mirror 20: 0x7a359e0 (stale) Mirror 21: 0x7a359e0 (stale) Mirror 22: 0x7a359e0 (stale) Mirror 24: 0x83cb39aa (stale) Mirror 25: 0xa1001e48 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e221): Mirror 18: 0x7a359e0 (stale) Mirror 20: 0x7a359e0 (stale) Mirror 21: 0x7a359e0 (stale) Mirror 22: 0x7a359e0 (stale) Mirror 24: 0x83cb39aa (stale) Mirror 25: 0xa1001e48 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e221: skipped 000452 1781219142.812181 write 0x023d43 thru 0x033c41 (0xfeff bytes) 000453 1781219142.820717 mapread 0x02e52b thru 0x033c41 (0x5717 bytes) 000455 1781219142.823520 trunc from 0x033c42 to 0x0121b7 (0x21a8a bytes) 000456 1781219142.843015 read 0x010753 thru 0x0121b7 (0x1a65 bytes) 000457 1781219142.848054 mapwrite 0x01b0f7 thru 0x02223e (0x7148 bytes) 000458 1781219142.925648 mapread 0x0123d0 thru 0x012414 (0x45 bytes) 000462 1781219142.929573 write 0x02754d thru 0x0305c6 (0x907a bytes) 000463 1781219142.932560 mapread 0x02e141 thru 0x0305c6 (0x2486 bytes) 000464 1781219142.934303 mapread 0x02d1e2 thru 0x0305c6 (0x33e5 bytes) 000465 1781219142.935534 read 0x005058 thru 0x0069f1 (0x199a bytes) 000469 1781219142.947628 trunc from 0x0305c7 to 0x035d1e (0x5758 bytes) 000470 1781219142.961564 mapread 0x0094d3 thru 0x01053a (0x7068 bytes) 000471 1781219142.962814 mapread 0x00a9cf thru 0x00edf5 (0x4427 bytes) 000473 1781219142.971561 write 0x01e57b thru 0x02de9b (0xf921 bytes) 000474 1781219142.974053 mapread 0x0270e5 thru 0x027b1a (0xa36 bytes) 000475 1781219142.976695 mapread 0x003ddf thru 0x006d71 (0x2f93 bytes) 000476 1781219142.978581 mapwrite 0x03bec9 thru 0x03ffff (0x4137 bytes) 000478 1781219143.074302 read 0x0131f9 thru 0x01f3ba (0xc1c2 bytes) 000482 1781219143.081792 write 0x01e9ab thru 0x026c01 (0x8257 bytes) 000484 1781219143.088914 mapread 0x002905 thru 0x0058d8 (0x2fd4 bytes) 000486 1781219143.093888 read 0x0355dd thru 0x03ffff (0xaa23 bytes) 000491 1781219143.118392 MIRROR_RESYNC 7 mirrors 000492 1781219144.288166 read 0x022271 thru 0x0233af (0x113f bytes) 000493 1781219144.296852 MIRROR_EXTEND to 8 mirrors 000494 1781219145.018858 trunc from 0x040000 to 0x03b6e2 (0x491d bytes) 000495 1781219145.098643 write 0x01a76f thru 0x02515b (0xa9ed bytes) 000496 1781219145.107518 trunc from 0x03b6e3 to 0x007012 (0x346d0 bytes) 000497 1781219145.129102 mapread 0x005aa5 thru 0x007012 (0x156e bytes) 000498 1781219145.147697 MIRROR_RESYNC 8 mirrors 000499 1781219146.658555 read 0x00058d thru 0x007012 (0x6a86 bytes) 000502 1781219146.667999 read 0x00086e thru 0x007012 (0x67a5 bytes) 000504 1781219146.669362 mapread 0x005c74 thru 0x007012 (0x139f bytes) 000505 1781219146.674362 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000506 1781219147.020770 write 0x0125cd thru 0x01bb94 (0x95c8 bytes) 000507 1781219147.206998 read 0x00e942 thru 0x00ed13 (0x3d2 bytes) 000508 1781219147.213311 mapwrite 0x03c644 thru 0x03ffff (0x39bc bytes) 000509 1781219147.321627 write 0x03958a thru 0x03ffff (0x6a76 bytes) 000510 1781219147.322762 mapread 0x00c76e thru 0x00de94 (0x1727 bytes) 000511 1781219147.343291 MIRROR_RESYNC 7 mirrors 000514 1781219148.379475 mapwrite 0x018a8a thru 0x019f58 (0x14cf bytes) 000518 1781219148.524380 mapwrite 0x03446e thru 0x03ffff (0xbb92 bytes) 000520 1781219148.579711 trunc from 0x040000 to 0x02edc3 (0x1123c bytes) 000521 1781219148.628036 mapwrite 0x02fdf9 thru 0x036476 (0x667e bytes) 000522 1781219148.704717 trunc from 0x036477 to 0x035e60 (0x616 bytes) 000523 1781219148.736792 write 0x015f01 thru 0x01f9fa (0x9afa bytes) 000524 1781219148.747451 mapread 0x0110b2 thru 0x018f06 (0x7e55 bytes) 000526 1781219148.817323 MIRROR_RESYNC 7 mirrors 000528 1781219150.114679 trunc from 0x035e61 to 0x0318ad (0x45b3 bytes) 000530 1781219150.272518 write 0x00aaef thru 0x01979e (0xecb0 bytes) 000533 1781219150.293488 mapread 0x02dafa thru 0x0318ad (0x3db4 bytes) 000534 1781219150.321496 read 0x0137c4 thru 0x01f3e6 (0xbc23 bytes) 000538 1781219150.333666 mapwrite 0x0198f9 thru 0x01a6bd (0xdc5 bytes) 000540 1781219150.401919 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x318ae): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x318ae): Mirror 18: 0x559b9a1f (stale) Mirror 20: 0x559b9a1f (stale) Mirror 21: 0x559b9a1f (stale) Mirror 22: 0x559b9a1f (stale) Mirror 24: 0x559b9a1f (stale) Mirror 25: 0x559b9a1f (stale) CRC-32 checksum value for chunk [0, 0x318ae): Mirror 1: 0xd4af5a23 Mirror 26: 0xd4af5a23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x318ae: skipped 000541 1781219151.455566 mapwrite 0x036c38 thru 0x03cf02 (0x62cb bytes) 000545 1781219151.699981 mapread 0x028f12 thru 0x02f496 (0x6585 bytes) 000547 1781219151.721744 MIRROR_RESYNC 8 mirrors 000549 1781219153.160743 mapread 0x01b02b thru 0x022611 (0x75e7 bytes) 000551 1781219153.179386 trunc from 0x03cf03 to 0x0105c1 (0x2c941 bytes) 000552 1781219153.263741 read 0x008e3c thru 0x0105c1 (0x7786 bytes) 000553 1781219153.273631 mapwrite 0x0165d9 thru 0x024098 (0xdac0 bytes) 000554 1781219153.360466 mapread 0x014a25 thru 0x01c705 (0x7ce1 bytes) 000557 1781219153.363555 trunc from 0x024099 to 0x03806a (0x13fd2 bytes) 000558 1781219153.383349 trunc from 0x03806b to 0x000d3f (0x3732b bytes) 000560 1781219153.400089 write 0x00f589 thru 0x01a961 (0xb3d9 bytes) 000561 1781219153.402243 mapwrite 0x03cec4 thru 0x03ffff (0x313c bytes) 000564 1781219153.461989 read 0x02dba5 thru 0x0367c1 (0x8c1d bytes) 000565 1781219153.486636 MIRROR_RESYNC 8 mirrors 000566 1781219154.880334 mapwrite 0x0325cf thru 0x035340 (0x2d72 bytes) 000568 1781219155.089693 read 0x038d86 thru 0x03ad1c (0x1f97 bytes) 000569 1781219155.090450 read 0x015720 thru 0x0237eb (0xe0cc bytes) 000571 1781219155.109337 read 0x01a4f9 thru 0x0233b2 (0x8eba bytes) 000572 1781219155.110294 read 0x022361 thru 0x02ac61 (0x8901 bytes) 000575 1781219155.112500 mapwrite 0x022129 thru 0x024a07 (0x28df bytes) 000579 1781219155.159801 mapwrite 0x028de4 thru 0x0365e6 (0xd803 bytes) 000580 1781219155.215510 mapwrite 0x03bf66 thru 0x03ffff (0x409a bytes) 000581 1781219155.262848 mapwrite 0x019dfc thru 0x025eff (0xc104 bytes) 000582 1781219155.321405 mapwrite 0x02c8f6 thru 0x02e198 (0x18a3 bytes) 000583 1781219155.378533 read 0x036f2f thru 0x03ffff (0x90d1 bytes) 000584 1781219155.384436 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x5d74a8df (stale) Mirror 20: 0x5d74a8df (stale) Mirror 21: 0x5d74a8df (stale) Mirror 22: 0x5d74a8df (stale) Mirror 24: 0x5d74a8df (stale) Mirror 25: 0x5d74a8df (stale) Mirror 26: 0x5d74a8df (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x514be6c Mirror 27: 0x514be6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000585 1781219156.207206 mapread 0x017cce thru 0x01989b (0x1bce bytes) 000586 1781219156.253474 trunc from 0x040000 to 0x03a341 (0x5cbe bytes) 000587 1781219156.426615 trunc from 0x03a342 to 0x0124f5 (0x27e4c bytes) 000588 1781219156.472606 read 0x00f698 thru 0x0124f5 (0x2e5e bytes) 000591 1781219156.485169 trunc from 0x0124f6 to 0x00ea11 (0x3ae4 bytes) 000592 1781219156.512008 write 0x00f495 thru 0x018e43 (0x99af bytes) 000594 1781219156.518520 mapwrite 0x02d6f4 thru 0x0373a2 (0x9caf bytes) 000595 1781219156.601011 mapwrite 0x02847f thru 0x0374d8 (0xf05a bytes) 000596 1781219156.667524 trunc from 0x0374d9 to 0x02a60b (0xcecd bytes) 000597 1781219156.685208 mapread 0x029c27 thru 0x02a60b (0x9e5 bytes) 000598 1781219156.686296 write 0x02b4cc thru 0x03978f (0xe2c4 bytes) 000599 1781219156.689210 write 0x031e30 thru 0x038a80 (0x6c51 bytes) 000601 1781219156.691813 mapwrite 0x00a52e thru 0x010fa7 (0x6a7a bytes) 000604 1781219156.753808 MIRROR_RESYNC 9 mirrors 000605 1781219158.660565 read 0x018b02 thru 0x021c9d (0x919c bytes) 000606 1781219158.674599 trunc from 0x039790 to 0x031b9e (0x7bf1 bytes) 000608 1781219158.772065 mapwrite 0x022d95 thru 0x024664 (0x18d0 bytes) 000610 1781219158.838921 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x31b9f): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b9f): Mirror 18: 0x30fea11b (stale) Mirror 20: 0x30fea11b (stale) Mirror 21: 0x30fea11b (stale) Mirror 22: 0x30fea11b (stale) Mirror 24: 0x30fea11b (stale) Mirror 25: 0x30fea11b (stale) Mirror 26: 0x30fea11b (stale) Mirror 27: 0x30fea11b (stale) CRC-32 checksum value for chunk [0, 0x31b9f): Mirror 1: 0xe4ac6b56 Mirror 28: 0xe4ac6b56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b9f: skipped 000614 1781219160.060074 MIRROR_RESYNC 10 mirrors 000615 1781219161.596609 mapread 0x024edf thru 0x031b9e (0xccc0 bytes) 000616 1781219161.623792 write 0x03a4dd thru 0x03c138 (0x1c5c bytes) 000617 1781219161.722901 write 0x00baa4 thru 0x01a8d0 (0xee2d bytes) 000619 1781219161.746305 mapread 0x038221 thru 0x03c138 (0x3f18 bytes) 000620 1781219161.769003 write 0x02b2bb thru 0x0300b1 (0x4df7 bytes) 000621 1781219161.780362 trunc from 0x03c139 to 0x01c633 (0x1fb05 bytes) 000622 1781219161.803378 trunc from 0x01c634 to 0x03f0ef (0x22abc bytes) 000623 1781219161.836311 MIRROR_RESYNC 10 mirrors 000626 1781219163.580289 trunc from 0x03f0f0 to 0x00acfc (0x343f3 bytes) 000627 1781219163.699715 trunc from 0x00acfd to 0x02ad4d (0x20051 bytes) 000628 1781219163.725432 trunc from 0x02ad4e to 0x024e78 (0x5ed5 bytes) 000629 1781219163.749492 mapwrite 0x00077b thru 0x00f29b (0xeb21 bytes) 000633 1781219163.837985 mapread 0x01ae24 thru 0x024e78 (0xa055 bytes) 000634 1781219163.856843 trunc from 0x024e79 to 0x03f179 (0x1a301 bytes) 000635 1781219163.874660 read 0x02e2b7 thru 0x038293 (0x9fdd bytes) 000636 1781219163.895833 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f17a): Mirror 18: 0x90b24d92 (stale) Mirror 20: 0x90b24d92 (stale) Mirror 21: 0x90b24d92 (stale) Mirror 22: 0x90b24d92 (stale) Mirror 25: 0x90b24d92 (stale) Mirror 26: 0x90b24d92 (stale) Mirror 27: 0x90b24d92 (stale) Mirror 28: 0x90b24d92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f17a): Mirror 18: 0x90b24d92 (stale) Mirror 20: 0x90b24d92 (stale) Mirror 21: 0x90b24d92 (stale) Mirror 22: 0x90b24d92 (stale) Mirror 25: 0x90b24d92 (stale) Mirror 26: 0x90b24d92 (stale) Mirror 27: 0x90b24d92 (stale) Mirror 28: 0x90b24d92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f17a: skipped 000638 1781219164.385333 read 0x02d005 thru 0x03785e (0xa85a bytes) 000642 1781219164.394192 write 0x01ae98 thru 0x020b8d (0x5cf6 bytes) 000643 1781219164.404108 mapwrite 0x01e4d8 thru 0x021f2e (0x3a57 bytes) 000644 1781219164.456452 write 0x02e757 thru 0x03051e (0x1dc8 bytes) 000645 1781219164.457731 mapwrite 0x01614e thru 0x0161ff (0xb2 bytes) 000646 1781219164.506696 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f17a): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f17a): Mirror 18: 0x90b24d92 (stale) Mirror 20: 0x90b24d92 (stale) Mirror 21: 0x90b24d92 (stale) Mirror 22: 0x90b24d92 (stale) Mirror 25: 0x90b24d92 (stale) Mirror 26: 0x90b24d92 (stale) Mirror 27: 0x90b24d92 (stale) Mirror 28: 0x90b24d92 (stale) CRC-32 checksum value for chunk [0, 0x3f17a): Mirror 1: 0xde2b74f4 Mirror 29: 0xde2b74f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f17a: skipped 000647 1781219165.495624 mapwrite 0x02be35 thru 0x034cf1 (0x8ebd bytes) 000648 1781219165.765221 mapread 0x0045c5 thru 0x00ab65 (0x65a1 bytes) 000649 1781219165.780758 mapread 0x0365ec thru 0x03f179 (0x8b8e bytes) 000650 1781219165.818769 read 0x000ecf thru 0x007d4d (0x6e7f bytes) 000651 1781219165.831091 MIRROR_RESYNC 10 mirrors 000653 1781219167.700492 write 0x02d207 thru 0x037472 (0xa26c bytes) 000655 1781219167.802365 trunc from 0x03f17a to 0x0186a3 (0x26ad6 bytes) 000657 1781219167.833626 read 0x0154bb thru 0x0168e8 (0x142e bytes) 000658 1781219167.839821 mapread 0x00579e thru 0x00fb3b (0xa39e bytes) 000659 1781219167.853450 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x186a4): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x186a4): Mirror 18: 0xe6c0f8a1 (stale) Mirror 20: 0xe6c0f8a1 (stale) Mirror 21: 0xe6c0f8a1 (stale) Mirror 22: 0xe6c0f8a1 (stale) Mirror 25: 0xe6c0f8a1 (stale) Mirror 26: 0xe6c0f8a1 (stale) Mirror 27: 0xe6c0f8a1 (stale) Mirror 28: 0xe6c0f8a1 (stale) Mirror 29: 0xe6c0f8a1 (stale) CRC-32 checksum value for chunk [0, 0x186a4): Mirror 1: 0xe6c0f8a1 Mirror 30: 0xe6c0f8a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x186a4: skipped 000660 1781219168.800844 trunc from 0x0186a4 to 0x033216 (0x1ab73 bytes) 000661 1781219168.944129 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x33217): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33217): Mirror 18: 0xa834e0f9 (stale) Mirror 20: 0xa834e0f9 (stale) Mirror 21: 0xa834e0f9 (stale) Mirror 22: 0xa834e0f9 (stale) Mirror 25: 0xa834e0f9 (stale) Mirror 26: 0xa834e0f9 (stale) Mirror 27: 0xa834e0f9 (stale) Mirror 28: 0xa834e0f9 (stale) Mirror 29: 0xa834e0f9 (stale) Mirror 30: 0xbf2418fd (stale) CRC-32 checksum value for chunk [0, 0x33217): Mirror 1: 0xbf2418fd Mirror 31: 0xbf2418fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33217: skipped 000662 1781219170.013572 mapread 0x025ccd thru 0x02adf8 (0x512c bytes) 000665 1781219170.033543 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33217): Mirror 18: 0xa834e0f9 (stale) Mirror 20: 0xa834e0f9 (stale) Mirror 21: 0xa834e0f9 (stale) Mirror 22: 0xa834e0f9 (stale) Mirror 25: 0xa834e0f9 (stale) Mirror 27: 0xa834e0f9 (stale) Mirror 28: 0xa834e0f9 (stale) Mirror 29: 0xa834e0f9 (stale) Mirror 30: 0xbf2418fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33217): Mirror 18: 0xa834e0f9 (stale) Mirror 20: 0xa834e0f9 (stale) Mirror 21: 0xa834e0f9 (stale) Mirror 22: 0xa834e0f9 (stale) Mirror 25: 0xa834e0f9 (stale) Mirror 27: 0xa834e0f9 (stale) Mirror 28: 0xa834e0f9 (stale) Mirror 29: 0xa834e0f9 (stale) Mirror 30: 0xbf2418fd (stale) CRC-32 checksum value for chunk [0, 0x33217): Mirror 1: 0xbf2418fd Mirror 31: 0xbf2418fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33217: skipped 000667 1781219170.890683 MIRROR_RESYNC 11 mirrors 000668 1781219172.571094 mapread 0x032dba thru 0x033216 (0x45d bytes) 000670 1781219172.582109 mapread 0x0063af thru 0x014b28 (0xe77a bytes) 000673 1781219172.603943 mapwrite 0x013573 thru 0x014639 (0x10c7 bytes) 000674 1781219172.781958 read 0x02ef92 thru 0x033216 (0x4285 bytes) 000675 1781219172.794302 read 0x01f485 thru 0x02559b (0x6117 bytes) 000678 1781219172.833242 trunc from 0x033217 to 0x025769 (0xdaad bytes) 000681 1781219172.860287 trunc from 0x02576a to 0x0265f6 (0xe8d bytes) 000682 1781219172.881655 mapwrite 0x03da76 thru 0x03ffff (0x258a bytes) 000683 1781219172.957486 mapwrite 0x00c73e thru 0x01a490 (0xdd53 bytes) 000684 1781219173.007121 read 0x03cf45 thru 0x03ffff (0x30bb bytes) 000686 1781219173.009235 mapwrite 0x036526 thru 0x03ffff (0x9ada bytes) 000691 1781219173.056126 mapread 0x01b927 thru 0x01d178 (0x1852 bytes) 000692 1781219173.057602 write 0x0306ec thru 0x033fb2 (0x38c7 bytes) 000693 1781219173.076946 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x45faff20 (stale) Mirror 20: 0x45faff20 (stale) Mirror 21: 0x45faff20 (stale) Mirror 22: 0x45faff20 (stale) Mirror 25: 0x45faff20 (stale) Mirror 27: 0x45faff20 (stale) Mirror 28: 0x45faff20 (stale) Mirror 30: 0x45faff20 (stale) Mirror 31: 0x45faff20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x45faff20 (stale) Mirror 20: 0x45faff20 (stale) Mirror 21: 0x45faff20 (stale) Mirror 22: 0x45faff20 (stale) Mirror 25: 0x45faff20 (stale) Mirror 27: 0x45faff20 (stale) Mirror 28: 0x45faff20 (stale) Mirror 30: 0x45faff20 (stale) Mirror 31: 0x45faff20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1781219173.719363 mapwrite 0x009490 thru 0x01896e (0xf4df bytes) 000695 1781219173.809446 mapread 0x005032 thru 0x00abc3 (0x5b92 bytes) 000696 1781219173.817181 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x45faff20 (stale) Mirror 21: 0x45faff20 (stale) Mirror 22: 0x45faff20 (stale) Mirror 25: 0x45faff20 (stale) Mirror 27: 0x45faff20 (stale) Mirror 28: 0x45faff20 (stale) Mirror 30: 0x45faff20 (stale) Mirror 31: 0x45faff20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x45faff20 (stale) Mirror 21: 0x45faff20 (stale) Mirror 22: 0x45faff20 (stale) Mirror 25: 0x45faff20 (stale) Mirror 27: 0x45faff20 (stale) Mirror 28: 0x45faff20 (stale) Mirror 30: 0x45faff20 (stale) Mirror 31: 0x45faff20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000697 1781219174.349183 trunc from 0x040000 to 0x02e2ad (0x11d52 bytes) 000699 1781219174.366806 write 0x034c3e thru 0x03fed1 (0xb294 bytes) 000702 1781219174.376533 mapread 0x01f6d2 thru 0x022426 (0x2d55 bytes) 000704 1781219174.402440 read 0x034b57 thru 0x03f951 (0xadfb bytes) 000706 1781219174.408815 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 20: 0x493dd437 (stale) Mirror 21: 0x493dd437 (stale) Mirror 22: 0x493dd437 (stale) Mirror 25: 0x493dd437 (stale) Mirror 27: 0x493dd437 (stale) Mirror 28: 0x493dd437 (stale) Mirror 30: 0x493dd437 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fed2): Mirror 20: 0x493dd437 (stale) Mirror 21: 0x493dd437 (stale) Mirror 22: 0x493dd437 (stale) Mirror 25: 0x493dd437 (stale) Mirror 27: 0x493dd437 (stale) Mirror 28: 0x493dd437 (stale) Mirror 30: 0x493dd437 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fed2: skipped 000707 1781219175.096823 write 0x028ba0 thru 0x028f91 (0x3f2 bytes) 000710 1781219175.106955 trunc from 0x03fed2 to 0x037d57 (0x817a bytes) 000711 1781219175.136417 write 0x00a424 thru 0x019c2f (0xf80c bytes) 000713 1781219175.164501 mapwrite 0x010a41 thru 0x0143db (0x399b bytes) 000716 1781219175.251738 write 0x01ec25 thru 0x0269ec (0x7dc8 bytes) 000717 1781219175.275552 trunc from 0x037d58 to 0x020e79 (0x16ede bytes) 000718 1781219175.310530 trunc from 0x020e7a to 0x012d45 (0xe134 bytes) 000720 1781219175.333537 mapread 0x00ba66 thru 0x012d45 (0x72e0 bytes) 000722 1781219175.335356 read 0x0120c8 thru 0x012d45 (0xc7e bytes) 000723 1781219175.346824 write 0x007785 thru 0x0102a4 (0x8b20 bytes) 000724 1781219175.354289 mapwrite 0x02527b thru 0x034e5e (0xfbe4 bytes) 000725 1781219175.554082 mapwrite 0x006927 thru 0x00f5e2 (0x8cbc bytes) 000727 1781219175.623533 mapread 0x00fbfe thru 0x01d99e (0xdda1 bytes) 000732 1781219175.654761 write 0x03a1d6 thru 0x03ffff (0x5e2a bytes) 000738 1781219175.669309 MIRROR_RESYNC 8 mirrors 000739 1781219177.175243 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000741 1781219177.581596 trunc from 0x040000 to 0x002eae (0x3d151 bytes) 000742 1781219177.687814 MIRROR_RESYNC 7 mirrors 000744 1781219178.885200 write 0x0367b9 thru 0x03eec9 (0x8711 bytes) 000745 1781219178.971713 mapwrite 0x016b8a thru 0x01b155 (0x45cc bytes) 000746 1781219179.071366 mapwrite 0x03db52 thru 0x03ffff (0x24ae bytes) 000747 1781219179.173654 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x413cac29 (stale) Mirror 21: 0x413cac29 (stale) Mirror 27: 0x413cac29 (stale) Mirror 28: 0x413cac29 (stale) Mirror 30: 0x413cac29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x413cac29 (stale) Mirror 21: 0x413cac29 (stale) Mirror 27: 0x413cac29 (stale) Mirror 28: 0x413cac29 (stale) Mirror 30: 0x413cac29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000748 1781219179.630660 write 0x00ebe5 thru 0x01240f (0x382b bytes) 000751 1781219179.658236 read 0x038f23 thru 0x03ffff (0x70dd bytes) 000757 1781219179.691336 write 0x0214fd thru 0x021770 (0x274 bytes) 000758 1781219179.712403 mapwrite 0x0232c1 thru 0x02c45d (0x919d bytes) 000761 1781219179.792916 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x413cac29 (stale) Mirror 21: 0x413cac29 (stale) Mirror 27: 0x413cac29 (stale) Mirror 28: 0x413cac29 (stale) Mirror 30: 0x413cac29 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb434dc15 Mirror 31: 0xb434dc15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000763 1781219180.798606 mapwrite 0x02d293 thru 0x03936f (0xc0dd bytes) 000766 1781219181.053906 write 0x0106a9 thru 0x015f59 (0x58b1 bytes) 000767 1781219181.072596 mapread 0x01a775 thru 0x028338 (0xdbc4 bytes) 000769 1781219181.113203 MIRROR_RESYNC 7 mirrors 000770 1781219182.760334 mapread 0x010409 thru 0x017b31 (0x7729 bytes) 000772 1781219182.785151 read 0x02fc69 thru 0x03b08c (0xb424 bytes) 000776 1781219182.797303 mapwrite 0x01dbfa thru 0x02350e (0x5915 bytes) 000777 1781219183.037856 trunc from 0x040000 to 0x03046d (0xfb92 bytes) 000778 1781219183.068683 mapread 0x01f61b thru 0x0270e9 (0x7acf bytes) 000779 1781219183.081245 mapread 0x00f2e0 thru 0x01c4af (0xd1d0 bytes) 000781 1781219183.090794 mapread 0x028d1a thru 0x03046d (0x7754 bytes) 000784 1781219183.097775 mapread 0x0026c3 thru 0x00ea45 (0xc383 bytes) 000785 1781219183.107318 read 0x000fc1 thru 0x010707 (0xf747 bytes) 000786 1781219183.115048 mapread 0x027eb2 thru 0x03046d (0x85bc bytes) 000787 1781219183.116575 read 0x01eaca thru 0x028e75 (0xa3ac bytes) 000789 1781219183.118612 mapwrite 0x00d55c thru 0x015b48 (0x85ed bytes) 000792 1781219183.168435 trunc from 0x03046e to 0x02a7c0 (0x5cad bytes) 000794 1781219183.204231 mapwrite 0x0339d1 thru 0x03458c (0xbbc bytes) 000795 1781219183.304306 mapwrite 0x038b78 thru 0x03c436 (0x38bf bytes) 000796 1781219183.408684 write 0x02c27c thru 0x034049 (0x7dce bytes) 000801 1781219183.415827 write 0x022f17 thru 0x031713 (0xe7fd bytes) 000802 1781219183.418145 trunc from 0x03c437 to 0x01c11d (0x20319 bytes) 000803 1781219183.433767 mapread 0x00dbe5 thru 0x0173f3 (0x980f bytes) 000807 1781219183.439134 trunc from 0x01c11e to 0x02879a (0xc67d bytes) 000811 1781219183.467565 trunc from 0x02879b to 0x036a51 (0xe2b7 bytes) 000812 1781219183.504188 mapwrite 0x00b05d thru 0x01a2c5 (0xf269 bytes) 000813 1781219183.654131 mapread 0x0065b5 thru 0x0145c4 (0xe010 bytes) 000814 1781219183.656156 mapwrite 0x03ba2e thru 0x03ffff (0x45d2 bytes) 000815 1781219183.795398 mapwrite 0x03e59b thru 0x03f561 (0xfc7 bytes) 000816 1781219183.919722 mapwrite 0x011ac5 thru 0x019266 (0x77a2 bytes) 000819 1781219183.986463 mapread 0x021186 thru 0x0224c6 (0x1341 bytes) 000820 1781219184.022999 mapread 0x03cd95 thru 0x03ffff (0x326b bytes) 000821 1781219184.027582 mapread 0x034c19 thru 0x035533 (0x91b bytes) 000824 1781219184.082576 mapread 0x00806f thru 0x00ac4f (0x2be1 bytes) 000825 1781219184.083725 mapread 0x007952 thru 0x00bf86 (0x4635 bytes) 000826 1781219184.084792 mapread 0x020bc8 thru 0x0228ef (0x1d28 bytes) 000827 1781219184.108257 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xaea78d6f (stale) Mirror 21: 0xaea78d6f (stale) Mirror 27: 0xaea78d6f (stale) Mirror 28: 0xaea78d6f (stale) Mirror 31: 0xaea78d6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xaea78d6f (stale) Mirror 21: 0xaea78d6f (stale) Mirror 27: 0xaea78d6f (stale) Mirror 28: 0xaea78d6f (stale) Mirror 31: 0xaea78d6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000829 1781219184.783403 read 0x008ff2 thru 0x014c88 (0xbc97 bytes) 000831 1781219184.794270 write 0x03f6d0 thru 0x03ffff (0x930 bytes) 000832 1781219184.800088 trunc from 0x040000 to 0x004c4b (0x3b3b4 bytes) 000834 1781219184.817577 read 0x002f18 thru 0x004c4b (0x1d34 bytes) 000836 1781219184.828282 mapread 0x00331d thru 0x004c4b (0x192f bytes) 000837 1781219184.833252 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x4c4c): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4c4c): Mirror 20: 0x62bf864d (stale) Mirror 21: 0x62bf864d (stale) Mirror 27: 0x62bf864d (stale) Mirror 28: 0x62bf864d (stale) Mirror 31: 0x62bf864d (stale) CRC-32 checksum value for chunk [0, 0x4c4c): Mirror 1: 0x62bf864d Mirror 32: 0x62bf864d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4c4c: skipped 000838 1781219186.099346 write 0x025da1 thru 0x0342a1 (0xe501 bytes) 000839 1781219186.195798 trunc from 0x0342a2 to 0x01d170 (0x17131 bytes) 000840 1781219186.223544 mapread 0x0082db thru 0x015ca5 (0xd9cb bytes) 000842 1781219186.281795 mapwrite 0x02fe15 thru 0x0307a6 (0x992 bytes) 000846 1781219186.397007 trunc from 0x0307a7 to 0x013bc2 (0x1cbe4 bytes) 000847 1781219186.413834 trunc from 0x013bc3 to 0x03e6b4 (0x2aaf2 bytes) 000848 1781219186.435181 mapwrite 0x008649 thru 0x015780 (0xd138 bytes) 000850 1781219186.526178 trunc from 0x03e6b5 to 0x01af1f (0x23795 bytes) 000851 1781219186.540342 trunc from 0x01af20 to 0x027231 (0xc312 bytes) 000852 1781219186.558213 mapread 0x00049e thru 0x0024c9 (0x202c bytes) 000855 1781219186.579017 write 0x018a3a thru 0x022ec4 (0xa48b bytes) 000857 1781219186.622155 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27232): Mirror 20: 0x3c85900e (stale) Mirror 21: 0x3c85900e (stale) Mirror 27: 0x3c85900e (stale) Mirror 28: 0x3c85900e (stale) Mirror 31: 0x3c85900e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27232): Mirror 20: 0x3c85900e (stale) Mirror 21: 0x3c85900e (stale) Mirror 27: 0x3c85900e (stale) Mirror 28: 0x3c85900e (stale) Mirror 31: 0x3c85900e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27232: skipped 000859 1781219187.244660 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27232): Mirror 21: 0x3c85900e (stale) Mirror 27: 0x3c85900e (stale) Mirror 28: 0x3c85900e (stale) Mirror 31: 0x3c85900e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27232): Mirror 21: 0x3c85900e (stale) Mirror 27: 0x3c85900e (stale) Mirror 28: 0x3c85900e (stale) Mirror 31: 0x3c85900e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27232: skipped 000861 1781219187.783414 read 0x01d385 thru 0x024483 (0x70ff bytes) 000863 1781219187.797298 mapread 0x0187a8 thru 0x021c41 (0x949a bytes) 000865 1781219187.855511 write 0x01e2da thru 0x021ff7 (0x3d1e bytes) 000866 1781219187.861080 read 0x022c10 thru 0x0238b0 (0xca1 bytes) 000867 1781219187.877182 MIRROR_RESYNC 5 mirrors 000869 1781219189.395529 write 0x03eac1 thru 0x03ffff (0x153f bytes) 000870 1781219189.530930 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x72b0de36 (stale) Mirror 27: 0x72b0de36 (stale) Mirror 28: 0x72b0de36 (stale) Mirror 31: 0x72b0de36 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x458d9e5c Mirror 32: 0x458d9e5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000871 1781219190.731419 read 0x03fad0 thru 0x03ffff (0x530 bytes) 000873 1781219190.743317 trunc from 0x040000 to 0x00f035 (0x30fca bytes) 000875 1781219190.865623 write 0x02bcf6 thru 0x02f234 (0x353f bytes) 000876 1781219190.867650 read 0x0243e3 thru 0x025863 (0x1481 bytes) 000877 1781219190.883341 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f235): Mirror 21: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f235): Mirror 21: 0x3e393264 (stale) Mirror 27: 0x3e393264 (stale) Mirror 28: 0x3e393264 (stale) Mirror 31: 0x3e393264 (stale) Mirror 32: 0x3e393264 (stale) CRC-32 checksum value for chunk [0, 0x2f235): Mirror 1: 0xa28bf7bc Mirror 33: 0xa28bf7bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f235: skipped 000878 1781219191.825643 mapread 0x00cb46 thru 0x01b65f (0xeb1a bytes) 000880 1781219191.858950 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2f235): Mirror 21: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f235): Mirror 21: 0x3e393264 (stale) Mirror 27: 0x3e393264 (stale) Mirror 28: 0x3e393264 (stale) Mirror 31: 0x3e393264 (stale) Mirror 32: 0x3e393264 (stale) CRC-32 checksum value for chunk [0, 0x2f235): Mirror 1: 0xa28bf7bc Mirror 33: 0xa28bf7bc Mirror 34: 0xa28bf7bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f235: skipped 000882 1781219193.051169 write 0x015d99 thru 0x023daf (0xe017 bytes) 000884 1781219193.196734 write 0x004222 thru 0x011c58 (0xda37 bytes) 000885 1781219193.217780 trunc from 0x02f235 to 0x00f024 (0x20210 bytes) 000886 1781219193.245294 read 0x0023c9 thru 0x0093bd (0x6ff5 bytes) 000888 1781219193.248479 read 0x004112 thru 0x00f024 (0xaf13 bytes) 000889 1781219193.250576 mapwrite 0x023b90 thru 0x030dab (0xd21c bytes) 000891 1781219193.362246 write 0x02a6ba thru 0x03690b (0xc252 bytes) 000893 1781219193.372323 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3690c): Mirror 27: 0x2062cc50 (stale) Mirror 28: 0x2062cc50 (stale) Mirror 31: 0x2062cc50 (stale) Mirror 32: 0x2062cc50 (stale) Mirror 33: 0xac027d38 (stale) Mirror 34: 0xac027d38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3690c): Mirror 27: 0x2062cc50 (stale) Mirror 28: 0x2062cc50 (stale) Mirror 31: 0x2062cc50 (stale) Mirror 32: 0x2062cc50 (stale) Mirror 33: 0xac027d38 (stale) Mirror 34: 0xac027d38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3690c: skipped 000895 1781219193.944132 trunc from 0x03690c to 0x001e1d (0x34aee bytes) 000897 1781219193.979079 MIRROR_RESYNC 7 mirrors 000898 1781219195.328242 trunc from 0x001e1e to 0x00de1b (0xbffe bytes) 000899 1781219195.455995 trunc from 0x00de1c to 0x03c5f2 (0x2e7d7 bytes) 000900 1781219195.494777 trunc from 0x03c5f3 to 0x012601 (0x29ff1 bytes) 000903 1781219195.523949 mapwrite 0x01ac8c thru 0x029b22 (0xee97 bytes) 000904 1781219195.698954 write 0x02d41c thru 0x02d7dd (0x3c2 bytes) 000905 1781219195.701642 mapread 0x007cc7 thru 0x015e33 (0xe16d bytes) 000906 1781219195.739172 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d7de): Mirror 27: 0xe909e66c (stale) Mirror 28: 0xe909e66c (stale) Mirror 31: 0xe909e66c (stale) Mirror 32: 0xe909e66c (stale) Mirror 33: 0xe909e66c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d7de): Mirror 27: 0xe909e66c (stale) Mirror 28: 0xe909e66c (stale) Mirror 31: 0xe909e66c (stale) Mirror 32: 0xe909e66c (stale) Mirror 33: 0xe909e66c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d7de: skipped 000907 1781219196.178938 trunc from 0x02d7de to 0x039912 (0xc135 bytes) 000908 1781219196.194240 write 0x02d0e4 thru 0x0390b6 (0xbfd3 bytes) 000909 1781219196.205789 trunc from 0x039913 to 0x00f84d (0x2a0c5 bytes) 000910 1781219196.220679 mapread 0x004515 thru 0x00f84d (0xb339 bytes) 000913 1781219196.234103 write 0x021a2a thru 0x0296ae (0x7c85 bytes) 000914 1781219196.237144 trunc from 0x0296af to 0x03ae0b (0x1175d bytes) 000916 1781219196.268021 trunc from 0x03ae0c to 0x001482 (0x39989 bytes) 000917 1781219196.296722 read 0x000c5b thru 0x001482 (0x828 bytes) 000919 1781219196.321031 MIRROR_RESYNC 6 mirrors 000920 1781219197.396529 read 0x000c49 thru 0x001482 (0x83a bytes) 000922 1781219197.402338 trunc from 0x001483 to 0x010620 (0xf19e bytes) 000923 1781219197.484553 trunc from 0x010621 to 0x01457c (0x3f5c bytes) 000925 1781219197.504180 mapread 0x00aa21 thru 0x01457c (0x9b5c bytes) 000926 1781219197.530030 MIRROR_RESYNC 6 mirrors 000928 1781219198.697377 mapwrite 0x02d2de thru 0x034402 (0x7125 bytes) 000931 1781219198.869258 mapread 0x0322fb thru 0x034402 (0x2108 bytes) 000932 1781219198.883798 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x34403): Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34403): Mirror 27: 0xdc439b20 (stale) Mirror 28: 0xdc439b20 (stale) Mirror 31: 0xdc439b20 (stale) Mirror 32: 0xdc439b20 (stale) Mirror 33: 0xdc439b20 (stale) CRC-32 checksum value for chunk [0, 0x34403): Mirror 1: 0xe8d8078 Mirror 34: 0xe8d8078 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34403: skipped 000933 1781219200.009055 mapwrite 0x025f20 thru 0x0307ea (0xa8cb bytes) 000935 1781219200.308994 MIRROR_RESYNC 7 mirrors 000937 1781219201.485183 write 0x036e8e thru 0x03ffff (0x9172 bytes) 000938 1781219201.585261 trunc from 0x040000 to 0x000c6b (0x3f394 bytes) 000939 1781219201.605066 mapwrite 0x01dfba thru 0x022953 (0x499a bytes) 000940 1781219201.738529 trunc from 0x022954 to 0x01b71b (0x7238 bytes) 000945 1781219201.761255 trunc from 0x01b71c to 0x0278d4 (0xc1b9 bytes) 000946 1781219201.792209 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x278d5): Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278d5): Mirror 27: 0xa7812112 (stale) Mirror 28: 0xa7812112 (stale) Mirror 31: 0xa7812112 (stale) Mirror 32: 0xa7812112 (stale) Mirror 33: 0xa7812112 (stale) Mirror 34: 0xa7812112 (stale) CRC-32 checksum value for chunk [0, 0x278d5): Mirror 1: 0x3ac2961d Mirror 35: 0x3ac2961d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278d5: skipped 000949 1781219202.860358 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x278d5): Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278d5): Mirror 27: 0xa7812112 (stale) Mirror 28: 0xa7812112 (stale) Mirror 31: 0xa7812112 (stale) Mirror 32: 0xa7812112 (stale) Mirror 33: 0xa7812112 (stale) Mirror 34: 0xa7812112 (stale) CRC-32 checksum value for chunk [0, 0x278d5): Mirror 1: 0x3ac2961d Mirror 35: 0x3ac2961d Mirror 36: 0x3ac2961d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278d5: skipped 000950 1781219203.918296 trunc from 0x0278d5 to 0x031d08 (0xa434 bytes) 000951 1781219204.082264 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d09): Mirror 28: 0xad7a7587 (stale) Mirror 31: 0xad7a7587 (stale) Mirror 32: 0xad7a7587 (stale) Mirror 33: 0xad7a7587 (stale) Mirror 34: 0xad7a7587 (stale) Mirror 35: 0x6f80893d (stale) Mirror 36: 0x6f80893d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d09): Mirror 28: 0xad7a7587 (stale) Mirror 31: 0xad7a7587 (stale) Mirror 32: 0xad7a7587 (stale) Mirror 33: 0xad7a7587 (stale) Mirror 34: 0xad7a7587 (stale) Mirror 35: 0x6f80893d (stale) Mirror 36: 0x6f80893d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d09: skipped 000952 1781219204.672936 write 0x00490a thru 0x012192 (0xd889 bytes) 000954 1781219204.702161 mapwrite 0x007bc4 thru 0x00e772 (0x6baf bytes) 000955 1781219204.790748 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d09): Mirror 28: 0xad7a7587 (stale) Mirror 31: 0xad7a7587 (stale) Mirror 32: 0xad7a7587 (stale) Mirror 33: 0xad7a7587 (stale) Mirror 34: 0xad7a7587 (stale) Mirror 35: 0x6f80893d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d09): Mirror 28: 0xad7a7587 (stale) Mirror 31: 0xad7a7587 (stale) Mirror 32: 0xad7a7587 (stale) Mirror 33: 0xad7a7587 (stale) Mirror 34: 0xad7a7587 (stale) Mirror 35: 0x6f80893d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d09: skipped 000957 1781219205.288126 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31d09): Mirror 28: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d09): Mirror 28: 0xad7a7587 (stale) Mirror 31: 0xad7a7587 (stale) Mirror 32: 0xad7a7587 (stale) Mirror 33: 0xad7a7587 (stale) Mirror 34: 0xad7a7587 (stale) Mirror 35: 0x6f80893d (stale) CRC-32 checksum value for chunk [0, 0x31d09): Mirror 1: 0xea28581e Mirror 36: 0xea28581e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d09: skipped 000958 1781219206.505464 mapwrite 0x029fb7 thru 0x02a605 (0x64f bytes) 000961 1781219206.839959 MIRROR_RESYNC 8 mirrors 000963 1781219208.622244 write 0x02d616 thru 0x02ea57 (0x1442 bytes) 000964 1781219208.782785 trunc from 0x031d09 to 0x023ae5 (0xe223 bytes) 000965 1781219208.804566 mapwrite 0x00b886 thru 0x019db8 (0xe533 bytes) 000966 1781219208.882145 trunc from 0x023ae6 to 0x017bd0 (0xbf15 bytes) 000967 1781219208.910143 write 0x01a822 thru 0x0271e1 (0xc9c0 bytes) 000974 1781219208.926132 MIRROR_RESYNC 8 mirrors 000975 1781219210.544283 mapwrite 0x01245a thru 0x012a12 (0x5b9 bytes) 000976 1781219210.744299 mapwrite 0x01c442 thru 0x02b58e (0xf14d bytes) 000978 1781219211.041537 trunc from 0x02b58f to 0x0391cd (0xdc3f bytes) 000979 1781219211.079485 trunc from 0x0391ce to 0x000653 (0x38b7a bytes) 000980 1781219211.117189 mapread 0x000195 thru 0x000653 (0x4bf bytes) 000981 1781219211.129123 mapread 0x000603 thru 0x000653 (0x51 bytes) 000982 1781219211.132960 trunc from 0x000654 to 0x02f8e8 (0x2f295 bytes) 000986 1781219211.175338 mapwrite 0x00a891 thru 0x017757 (0xcec7 bytes) 000987 1781219211.342150 mapread 0x00ab6f thru 0x00e5c6 (0x3a58 bytes) 000992 1781219211.348168 write 0x0158f1 thru 0x0241cb (0xe8db bytes) 000993 1781219211.359014 write 0x02a96f thru 0x032984 (0x8016 bytes) 000994 1781219211.371346 write 0x01d2c0 thru 0x02ac4e (0xd98f bytes) 000995 1781219211.377431 write 0x036782 thru 0x03d01d (0x689c bytes) 001000 1781219211.386898 read 0x027b2f thru 0x028063 (0x535 bytes) All operations completed A-OK! PASS 70a (141s) == sanity-flr test 71: check mirror extend parent fid ==== 19:06:57 (1781219217) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 19:07:00 (1781219220) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309354 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310057 s, 3.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 19:07:09 (1781219229) Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre2 Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6880800 .. Extending file size to 5617088 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7278496 .. Extending file size to 7839712 .. Extending file size to 7590368 .. Extending file size to 8375584 .. Extending file size to 8624672 .. Extending file size to 8660992 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9256256 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9341120 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57324 57325 57327 57328 57329 done 192.168.203.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg352-client.virtnet /mnt/lustre2 (opts:) 192.168.203.152@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg352-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:08:34 (1781219314) Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre2 Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5701440 .. Extending file size to 5999680 .. failed Extending file size to 7062176 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8034112 .. Extending file size to 9070272 .. Extending file size to 8335392 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58968 58969 58970 58973 58974 done 192.168.203.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg352-client.virtnet /mnt/lustre2 (opts:) 192.168.203.152@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg352-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:09:55 (1781219395) Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159366 s, 13.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.395528 s, 5.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x70658f81 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.203.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg352-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:10:17 (1781219417) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0655278 s, 32.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 19:10:24 (1781219424) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:10:32 (1781219432) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22064 s, 9.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:10:40 (1781219440) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206477 s, 10.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:10:47 (1781219447) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183185 s, 11.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:11:21 (1781219481) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg352-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110769 s, 9.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:12:25 (1781219545) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg352-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299988 s, 7.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:13:28 (1781219608) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg352-server mds-ost sync done. Creating new pool test_204f oleg352-server: Pool lustre.test_204f created Adding targets to pool oleg352-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg352-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.429415 s, 4.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg352-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg352-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg352-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:14:49 (1781219689) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:14:56 (1781219696) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:15:04 (1781219704) oleg352-server: Pool lustre.test_206 created Adding targets to pool oleg352-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg352-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg352-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg352-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg352-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:15:30 (1781219730) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865513 s, 12.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962734 s, 10.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x23ea247d Mirror 3: 0x23ea247d PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:15:42 (1781219742) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:15:46 (1781219746) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:15:49 (1781219749) Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.205036 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1bf0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1bf0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1bf0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1740000 Read this (4 bytes): BBBB 192.168.203.152@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg352-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:16:25 (1781219785) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0571525 s, 143 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:16:32 (1781219792) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612149 s, 17.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834453 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 19:16:34.000000000 -0400 Modify: 2026-06-11 19:16:34.000000000 -0400 Change: 2026-06-11 19:16:35.000000000 -0400 Birth: 2026-06-11 19:16:34.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:16:41 (1781219801) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:16:44 (1781219804) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.507348 s, 80.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0384081 s, 107 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 19:16 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x217:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 19:16 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:16:52 (1781219812) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:16:59 (1781219819) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1824 sec =========== 19:17:07 (1781219827) === sanity-flr: start cleanup 19:17:09 (1781219829) === === sanity-flr: finish cleanup 19:17:15 (1781219835) ===