-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 21:08:21 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg127-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:08:43 (1781226523) === oleg127-client.virtnet: executing check_config_client /mnt/lustre oleg127-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg127-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff946f4606e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff946f4606e800.idle_timeout=debug disable quota as required oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:09:25 (1781226565) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8229] == sanity-flr test 0a: lfs mirror create with -N option == 21:09:28 (1781226568) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:09:47 (1781226587) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:09:51 (1781226591) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:09:55 (1781226595) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:10:19 (1781226619) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:10:24 (1781226624) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 21:10:29 (1781226629) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff946f4606e800.stats=clear osc.lustre-OST0001-osc-ffff946f4606e800.stats=clear PASS 0g (12s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:10:42 (1781226642) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:10:55 (1781226655) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:11:04 (1781226664) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:11:07 (1781226667) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:11:16 (1781226676) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 21:11:26 (1781226686) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:11:34 (1781226694) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:11:42 (1781226702) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:11:51 (1781226711) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:12:03 (1781226723) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:12:11 (1781226731) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.926779 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116388 s, 9.0 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:12:22 (1781226742) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.546881 s, 28.8 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:12:33 (1781226753) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:12:46 (1781226766) Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff946f4606e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff946f4606e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff946f4606e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff946f4606e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (86s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:14:12 (1781226852) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.645975 s, 14.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff946f4606e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff946f4606e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:14:54 (1781226894) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff946f4606e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff946f4606e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff946f4606e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff946f4606e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:16:20 (1781226980) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.20992 s, 10.1 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 21:16 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.820s user 0m0.002s sys 0m0.160s Stopping /mnt/lustre-ost1 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff946f4606e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff946f4606e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.644s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg127-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg127-server oleg127-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff946f4606e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff946f4606e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.682s user 0m0.001s sys 0m0.200s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg127-server: oleg127-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg127-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff946f4606e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff946f4606e800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:18:02 (1781227082) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:18:05 (1781227085) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:18:10 (1781227090) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 21:18:14 (1781227094) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 21:18:24 (1781227104) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.768088 s, 16.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:18:38 (1781227118) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750273 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659299 s, 15.9 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:18:47 (1781227127) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:18:58 (1781227138) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00456363 s, 224 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00176048 s, 582 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00277751 s, 369 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00183062 s, 559 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 21:19:12 (1781227152) create test files with size 15744925 .. d2c4398babc42e1e440dd40a22a97cdf - 958c527b6372031bc7c1601e373cb6fb - 9fbdfd36009220349c2213f274e675f3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 21:19:37 (1781227177) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (20s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:19:57 (1781227197) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:20:04 (1781227204) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744631 s, 14.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318644 s, 32.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 21:20:13 (1781227213) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0946184 s, 22.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.459023 s, 9.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189321 s, 11.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 21:20:34 (1781227234) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 21:20:37 (1781227237) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:20:42 (1781227242) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991032 s, 10.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 21:20:51 (1781227251) llite.lustre-ffff946f54e0a000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 44a: lfs mirror split check =========== 21:21:01 (1781227261) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.297742 s, 10.6 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:21:12 (1781227272) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:21:24 (1781227284) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.396887 s, 26.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:21:37 (1781227297) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff946f43437800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.460866 s, 22.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg127-client.virtnet /mnt/lustre (opts:) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff946f4797a000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:21:56 (1781227316) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:22:40 (1781227360) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (51s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:23:31 (1781227411) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 21:23:35 (1781227415) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.455853 s, 6.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.130431 s, 24.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd77288f7 (nosync,stale) Mirror 2: 0xd77288f7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 77d183c02c17204e1e2e47c9c07dcdc0 - mirror 1 checksum: 80c298e18d531651162fe3b38217c291 - mirror 2 checksum: 77d183c02c17204e1e2e47c9c07dcdc0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 21:23:39 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 21:23:39 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 77d183c02c17204e1e2e47c9c07dcdc0 - mirror 1 checksum: 80c298e18d531651162fe3b38217c291 - mirror 2 checksum: 80c298e18d531651162fe3b38217c291 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 21:23:39 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 21:23:39 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:23:47 (1781227427) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593659 s, 17.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:23:55 (1781227435) Failing mds1 on oleg127-server Stopping /mnt/lustre-mds1 (opts:) on oleg127-server 21:24:08 (1781227448) shut down facet: mds1 facet_host: oleg127-server facet_failover_host: oleg127-server Failover mds1 to oleg127-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 21:24:39 (1781227479) targets are mounted 21:24:39 (1781227479) facet_failover done oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364359 s, 28.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:24:59 (1781227499) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0561399 s, 365 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0279982 s, 731 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:25:11 (1781227511) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0682665 s, 300 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00906828 s, 2.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:25:23 (1781227523) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00880354 s, 1.9 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:25:32 (1781227532) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0754566 s, 271 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:25:40 (1781227540) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0187084 s, 219 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.028072 s, 146 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:25:51 (1781227551) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:26:17 (1781227577) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:26:48 (1781227608) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:27:14 (1781227634) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:27:41 (1781227661) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 21:27:48 (1781227668) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57825 mirror split pid 57827 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57825 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57827 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 21:28:58 (1781227738) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7741 fallocating to largest ever: 0x40000 000001 1781227741.776109 fallocate 0x03f826 thru 0x03ffff (0x7da bytes) skipping zero length zero range skipping zero size read 000004 1781227741.790825 mapwrite 0x03decc thru 0x03ffff (0x2134 bytes) 000005 1781227741.851092 write 0x0203a2 thru 0x02678d (0x63ec bytes) 000007 1781227741.862544 trunc from 0x026de8 to 0x031c54 (0xae6d bytes) 000008 1781227741.886223 read 0x028ac1 thru 0x029035 (0x575 bytes) 000009 1781227741.900111 punch from 0x02efc2 to 0x0368d8 (0x7917 bytes) 000010 1781227741.909803 punch from 0x001c17 to 0x0118b1 (0xfc9b bytes) 000012 1781227741.920239 mapread 0x01095a thru 0x0196af (0x8d56 bytes) 000013 1781227741.955872 MIRROR_EXTEND to 2 mirrors 000014 1781227743.005528 read 0x039167 thru 0x03ffff (0x6e99 bytes) 000015 1781227743.017782 mapwrite 0x03ef64 thru 0x03ffff (0x109c bytes) 000016 1781227743.097925 fallocate 0x0109c2 thru 0x018ffe (0x863d bytes) 000017 1781227743.106288 fallocate 0x027b0f thru 0x036ba3 (0xf095 bytes) 000018 1781227743.116293 mapwrite 0x025e91 thru 0x030a9c (0xac0c bytes) 000019 1781227743.199347 punch from 0x005efb to 0x010298 (0xa39e bytes) 000020 1781227743.214019 fallocate 0x000e5b thru 0x00f3e3 (0xe589 bytes) 000021 1781227743.226290 punch from 0x0347f4 to 0x03a6c4 (0x5ed1 bytes) 000022 1781227743.235047 trunc from 0x005755 to 0x0093f5 (0x3ca1 bytes) 000023 1781227743.265724 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6327032c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaa6b6560 Mirror 3: 0xaa6b6560 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000024 1781227745.217228 trunc from 0x0002e3 to 0x00c86a (0xc588 bytes) 000025 1781227745.323490 write 0x01f01a thru 0x02855e (0x9545 bytes) 000026 1781227745.363057 fallocate 0x027ab7 thru 0x030d4f (0x9299 bytes) 000027 1781227745.386478 punch from 0x019082 to 0x023f6a (0xaee9 bytes) 000028 1781227745.441034 read 0x036773 thru 0x03f140 (0x89ce bytes) 000029 1781227745.461043 write 0x032b27 thru 0x03fcc3 (0xd19d bytes) 000030 1781227745.493891 MIRROR_RESYNC 3 mirrors 000031 1781227746.841035 read 0x00138b thru 0x005b31 (0x47a7 bytes) 000032 1781227746.865210 fallocate 0x0176a7 thru 0x023a28 (0xc382 bytes) 000033 1781227747.011160 mapread 0x01e97c thru 0x0247fe (0x5e83 bytes) 000034 1781227747.068466 fallocate 0x009968 thru 0x012ae7 (0x9180 bytes) 000035 1781227747.075291 mapwrite 0x02fcbd thru 0x03f4d7 (0xf81b bytes) 000037 1781227747.200390 trunc from 0x015b11 to 0x01f7cf (0x9cbf bytes) 000038 1781227747.212332 mapwrite 0x02f5be thru 0x03a4f6 (0xaf39 bytes) 000039 1781227747.279533 read 0x031cef thru 0x032367 (0x679 bytes) 000040 1781227747.280260 trunc from 0x033f96 to 0x039d91 (0x5dfc bytes) 000042 1781227747.289405 fallocate 0x028758 thru 0x0354f2 (0xcd9b bytes) 000043 1781227747.308125 punch from 0x007986 to 0x014ea6 (0xd521 bytes) 000044 1781227747.324181 trunc from 0x040000 to 0x0349af (0xb650 bytes) 000045 1781227747.394270 trunc from 0x0107e4 to 0x01c144 (0xb961 bytes) 000046 1781227747.408749 write 0x030ac9 thru 0x03ffff (0xf537 bytes) 000047 1781227747.410754 punch from 0x003417 to 0x0124de (0xf0c8 bytes) 000048 1781227747.427729 mapwrite 0x011afa thru 0x01aa50 (0x8f57 bytes) 000049 1781227747.514776 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc1df1313 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc1df1313 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000051 1781227747.979417 write 0x035985 thru 0x036031 (0x6ad bytes) 000052 1781227748.009845 trunc from 0x040000 to 0x02c3be (0x13c41 bytes) 000053 1781227748.055034 trunc from 0x017bb3 to 0x022609 (0xaa57 bytes) 000054 1781227748.075106 MIRROR_RESYNC 2 mirrors 000055 1781227748.883617 trunc from 0x00f0a9 to 0x014b87 (0x5adf bytes) 000056 1781227748.952981 mapread 0x020dd7 thru 0x02358f (0x27b9 bytes) 000057 1781227748.978902 trunc from 0x02577a to 0x02c3be (0x6c45 bytes) 000058 1781227748.985651 mapwrite 0x000657 thru 0x00584a (0x51f4 bytes) 000059 1781227749.018342 trunc from 0x024a5c to 0x02c3be (0x7963 bytes) 000062 1781227749.027432 fallocate 0x008ede thru 0x014b4d (0xbc70 bytes) 000063 1781227749.034246 read 0x015649 thru 0x023f4a (0xe902 bytes) 000064 1781227749.050480 fallocate 0x01fcbe thru 0x02f580 (0xf8c3 bytes) 000065 1781227749.057378 write 0x0351d9 thru 0x03ffff (0xae27 bytes) 000066 1781227749.062574 punch from 0x00a618 to 0x013b4c (0x9535 bytes) 000067 1781227749.074718 trunc from 0x040000 to 0x00ca30 (0x335cf bytes) 000068 1781227749.103613 trunc from 0x0074bd to 0x00ca30 (0x5574 bytes) 000071 1781227749.130629 MIRROR_RESYNC 2 mirrors 000072 1781227749.729702 punch from 0x0050ab to 0x00ca30 (0x7986 bytes) 000073 1781227749.779287 punch from 0x00778b to 0x00ca30 (0x52a6 bytes) 000075 1781227749.787960 read 0x004921 thru 0x004d9a (0x47a bytes) 000076 1781227749.794360 mapread 0x005bd9 thru 0x00ca30 (0x6e58 bytes) 000077 1781227749.835254 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xca31): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xca31): Mirror 2: 0xd9f8ca14 (stale) CRC-32 checksum value for chunk [0, 0xca31): Mirror 1: 0x6fd43ae2 Mirror 3: 0x6fd43ae2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xca31: skipped 000078 1781227750.874864 mapread 0x00b806 thru 0x00ca30 (0x122b bytes) 000080 1781227750.892104 punch from 0x006a30 to 0x00ca30 (0x6001 bytes) 000081 1781227750.970666 punch from 0x009bd2 to 0x00ca30 (0x2e5f bytes) 000082 1781227750.978862 trunc from 0x00ca31 to 0x022b32 (0x16102 bytes) 000083 1781227751.018051 punch from 0x00a2d6 to 0x00d492 (0x31bd bytes) 000084 1781227751.028086 punch from 0x005614 to 0x006931 (0x131e bytes) 000085 1781227751.036305 read 0x009404 thru 0x015073 (0xbc70 bytes) 000086 1781227751.104275 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22b33): Mirror 3: 0xbe6bb3e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b33): Mirror 3: 0xbe6bb3e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b33: skipped 000087 1781227751.713057 fallocate 0x00cee1 thru 0x010977 (0x3a97 bytes) 000088 1781227751.751170 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) 000089 1781227751.988726 punch from 0x012e2a to 0x0135f4 (0x7cb bytes) 000091 1781227751.996211 fallocate 0x015a27 thru 0x01a91f (0x4ef9 bytes) 000092 1781227752.013527 punch from 0x020fa8 to 0x021e44 (0xe9d bytes) 000094 1781227752.023244 read 0x006475 thru 0x013f50 (0xdadc bytes) 000095 1781227752.038766 mapread 0x010a7a thru 0x019757 (0x8cde bytes) 000097 1781227752.072574 trunc from 0x022b33 to 0x01b348 (0x77ea bytes) 000098 1781227752.098873 read 0x0065c2 thru 0x011903 (0xb342 bytes) 000099 1781227752.100676 write 0x02ebcf thru 0x0375d5 (0x8a07 bytes) 000100 1781227752.106159 trunc from 0x000269 to 0x004707 (0x449f bytes) 000101 1781227752.121135 MIRROR_EXTEND to 2 mirrors 000102 1781227753.270087 punch from 0x00bb82 to 0x019283 (0xd702 bytes) 000104 1781227753.364103 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x375d6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x375d6): Mirror 2: 0xf3161692 (stale) CRC-32 checksum value for chunk [0, 0x375d6): Mirror 1: 0xf3161692 Mirror 3: 0xf3161692 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x375d6: skipped 000106 1781227755.021496 mapread 0x02dc91 thru 0x02e958 (0xcc8 bytes) 000107 1781227755.072327 MIRROR_RESYNC 3 mirrors 000109 1781227756.410342 punch from 0x02b56e to 0x033a67 (0x84fa bytes) 000110 1781227756.495459 punch from 0x004c8c to 0x014938 (0xfcad bytes) 000111 1781227756.507236 trunc from 0x032f1b to 0x036b73 (0x3c59 bytes) 000112 1781227756.517079 mapwrite 0x00125d thru 0x001e84 (0xc28 bytes) 000113 1781227756.552111 write 0x00042f thru 0x00e075 (0xdc47 bytes) 000114 1781227756.570649 trunc from 0x0375d6 to 0x0145ae (0x23027 bytes) 000115 1781227756.606020 read 0x010aaf thru 0x0145ae (0x3b00 bytes) 000116 1781227756.618737 trunc from 0x0145af to 0x032746 (0x1e198 bytes) 000117 1781227756.646030 trunc from 0x0087c9 to 0x00bde9 (0x3621 bytes) 000118 1781227756.675062 punch from 0x00f2f8 to 0x01eea6 (0xfbaf bytes) 000119 1781227756.692726 MIRROR_RESYNC 3 mirrors 000120 1781227757.662507 read 0x02fde3 thru 0x032746 (0x2964 bytes) 000121 1781227757.672963 mapwrite 0x028824 thru 0x035e1e (0xd5fb bytes) 000122 1781227757.785715 punch from 0x005fde to 0x011486 (0xb4a9 bytes) 000123 1781227757.793579 punch from 0x023d85 to 0x02d110 (0x938c bytes) 000124 1781227757.801963 write 0x039be0 thru 0x03ffff (0x6420 bytes) 000125 1781227757.804368 fallocate 0x004328 thru 0x00518f (0xe68 bytes) 000126 1781227757.810465 punch from 0x02685e to 0x0278af (0x1052 bytes) 000127 1781227757.817967 read 0x0060d2 thru 0x00dc65 (0x7b94 bytes) 000128 1781227757.855355 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4e2e5dfd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4e2e5dfd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000129 1781227758.272210 MIRROR_RESYNC 2 mirrors 000130 1781227759.187314 punch from 0x00650d to 0x006f4e (0xa42 bytes) 000131 1781227759.300357 punch from 0x037e03 to 0x0384cd (0x6cb bytes) 000133 1781227759.309373 mapread 0x002a03 thru 0x00a97b (0x7f79 bytes) 000134 1781227759.339396 read 0x01f052 thru 0x02e09a (0xf049 bytes) 000136 1781227759.361313 punch from 0x008e64 to 0x0091f8 (0x395 bytes) 000137 1781227759.371322 mapwrite 0x006a37 thru 0x00b8f2 (0x4ebc bytes) 000138 1781227759.411042 mapread 0x01c5c0 thru 0x020dee (0x482f bytes) 000139 1781227759.431141 trunc from 0x003743 to 0x00817b (0x4a39 bytes) 000142 1781227759.443715 trunc from 0x008154 to 0x00d537 (0x53e4 bytes) 000143 1781227759.457457 mapread 0x014c57 thru 0x01bf5f (0x7309 bytes) 000144 1781227759.472237 fallocate 0x01587d thru 0x01d8b5 (0x8039 bytes) 000145 1781227759.480549 mapwrite 0x01fb28 thru 0x02e0d8 (0xe5b1 bytes) 000146 1781227759.535616 trunc from 0x0357cb to 0x03ffff (0xa835 bytes) 000147 1781227759.544840 read 0x01ccf4 thru 0x01f189 (0x2496 bytes) 000148 1781227759.546069 punch from 0x01683d to 0x01c2c3 (0x5a87 bytes) 000149 1781227759.568000 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) 000150 1781227759.803419 trunc from 0x040000 to 0x01d1d5 (0x22e2a bytes) 000151 1781227759.829374 trunc from 0x01b971 to 0x01d1d5 (0x1865 bytes) 000153 1781227759.837270 trunc from 0x01d1d6 to 0x01a51b (0x2cba bytes) 000154 1781227759.854515 trunc from 0x01a51c to 0x02dc85 (0x1376a bytes) 000155 1781227759.872509 fallocate 0x020ecb thru 0x02ca11 (0xbb47 bytes) 000157 1781227759.879573 fallocate 0x00b5b7 thru 0x013f9a (0x89e4 bytes) 000158 1781227759.889420 fallocate 0x03df63 thru 0x03ffff (0x209d bytes) 000159 1781227759.896537 trunc from 0x040000 to 0x039b12 (0x64ed bytes) 000160 1781227759.918714 trunc from 0x039b13 to 0x012b29 (0x26fe9 bytes) 000161 1781227759.940967 fallocate 0x01ee8b thru 0x020ea1 (0x2017 bytes) 000164 1781227759.948147 fallocate 0x02b8e3 thru 0x03ae04 (0xf522 bytes) 000165 1781227759.959535 read 0x029404 thru 0x03183b (0x8438 bytes) 000166 1781227759.971411 mapwrite 0x0199e2 thru 0x01d0b5 (0x36d4 bytes) 000167 1781227760.008529 punch from 0x00ed74 to 0x00f9ad (0xc3a bytes) 000168 1781227760.014366 punch from 0x016cff to 0x01f76e (0x8a70 bytes) 000170 1781227760.020676 trunc from 0x03ae05 to 0x00be43 (0x2efc1 bytes) 000171 1781227760.044003 mapread 0x003078 thru 0x00be43 (0x8dcc bytes) 000173 1781227760.060988 mapread 0x00bdc3 thru 0x00be43 (0x81 bytes) 000174 1781227760.062059 read 0x0048a5 thru 0x00be43 (0x759f bytes) 000175 1781227760.063426 mapread 0x00a778 thru 0x00be43 (0x16cc bytes) 000176 1781227760.064813 read 0x001f70 thru 0x002582 (0x613 bytes) 000177 1781227760.065792 trunc from 0x00be44 to 0x025be8 (0x19da5 bytes) 000178 1781227760.081285 write 0x02c474 thru 0x036c5c (0xa7e9 bytes) 000179 1781227760.083006 punch from 0x00ab0d to 0x018a3b (0xdf2f bytes) 000180 1781227760.090488 fallocate 0x00ecbd thru 0x018de7 (0xa12b bytes) 000183 1781227760.103586 trunc from 0x036c5d to 0x010a98 (0x261c4 bytes) 000184 1781227760.121002 read 0x0006f5 thru 0x00348e (0x2d9a bytes) 000185 1781227760.131774 trunc from 0x00db9c to 0x010a98 (0x2efd bytes) 000186 1781227760.137478 write 0x0287e4 thru 0x02db6f (0x538c bytes) 000187 1781227760.139442 trunc from 0x02db70 to 0x01020c (0x1d963 bytes) 000188 1781227760.157382 write 0x01f1f3 thru 0x02bdba (0xcbc8 bytes) 000189 1781227760.161399 fallocate 0x026e29 thru 0x02eba5 (0x7d7d bytes) 000190 1781227760.167732 punch from 0x001875 to 0x00a1a0 (0x892c bytes) 000191 1781227760.173512 mapread 0x02d331 thru 0x02eba5 (0x1875 bytes) 000192 1781227760.189270 punch from 0x0061f5 to 0x00e429 (0x8235 bytes) 000193 1781227760.197763 mapwrite 0x00c1ab thru 0x00ff0c (0x3d62 bytes) 000194 1781227760.240742 trunc from 0x00e32b to 0x01ba4a (0xd720 bytes) 000195 1781227760.248280 mapread 0x00d721 thru 0x0188bd (0xb19d bytes) 000196 1781227760.310336 mapwrite 0x01e2d7 thru 0x01f726 (0x1450 bytes) 000197 1781227760.324285 read 0x00bc54 thru 0x014bc0 (0x8f6d bytes) 000198 1781227760.339502 write 0x006e55 thru 0x0080bf (0x126b bytes) 000199 1781227760.341101 punch from 0x00633a to 0x01251f (0xc1e6 bytes) 000200 1781227760.352966 trunc from 0x023f69 to 0x0263fe (0x2496 bytes) 000201 1781227760.359377 read 0x0167d5 thru 0x01a9e8 (0x4214 bytes) 000202 1781227760.360436 fallocate 0x0116fe thru 0x0211ff (0xfb02 bytes) 000203 1781227760.366098 trunc from 0x017231 to 0x01f225 (0x7ff5 bytes) 000204 1781227760.371643 write 0x0319bb thru 0x03323b (0x1881 bytes) 000205 1781227760.372836 read 0x0104a4 thru 0x01221e (0x1d7b bytes) 000206 1781227760.391381 punch from 0x029424 to 0x03323b (0x9e18 bytes) 000208 1781227760.412528 mapread 0x00cd45 thru 0x00fd2d (0x2fe9 bytes) 000210 1781227760.415937 read 0x007b16 thru 0x010d72 (0x925d bytes) 000211 1781227760.417431 write 0x004a4b thru 0x010d80 (0xc336 bytes) 000212 1781227760.422375 write 0x010967 thru 0x01af97 (0xa631 bytes) 000213 1781227760.426939 fallocate 0x037372 thru 0x03ffff (0x8c8e bytes) 000214 1781227760.433545 fallocate 0x039e13 thru 0x03b2e8 (0x14d6 bytes) 000215 1781227760.442681 write 0x005e37 thru 0x00efd9 (0x91a3 bytes) 000216 1781227760.445536 mapwrite 0x025fa8 thru 0x029e85 (0x3ede bytes) 000217 1781227760.499249 trunc from 0x00b873 to 0x00c039 (0x7c7 bytes) 000218 1781227760.507063 punch from 0x0125df to 0x019cd4 (0x76f6 bytes) 000219 1781227760.515266 punch from 0x0251d3 to 0x0291af (0x3fdd bytes) 000220 1781227760.523382 mapread 0x029a56 thru 0x039958 (0xff03 bytes) 000221 1781227760.530522 trunc from 0x0333c6 to 0x03b2e8 (0x7f23 bytes) 000222 1781227760.540395 trunc from 0x037caa to 0x0395ca (0x1921 bytes) 000223 1781227760.546984 read 0x009258 thru 0x0139b8 (0xa761 bytes) 000224 1781227760.572518 punch from 0x01ff8f to 0x024382 (0x43f4 bytes) 000225 1781227760.580309 punch from 0x025f23 to 0x02b6f6 (0x57d4 bytes) 000226 1781227760.595202 read 0x0231d6 thru 0x023d4d (0xb78 bytes) 000229 1781227760.619454 mapread 0x0046de thru 0x01142c (0xcd4f bytes) 000230 1781227760.622465 trunc from 0x028751 to 0x03267a (0x9f2a bytes) 000231 1781227760.628382 read 0x025ba6 thru 0x0335f8 (0xda53 bytes) 000232 1781227760.643739 punch from 0x01eab4 to 0x02105d (0x25aa bytes) 000233 1781227760.649280 trunc from 0x03b2e9 to 0x03c611 (0x1329 bytes) 000236 1781227760.671671 write 0x022b0c thru 0x02ebd6 (0xc0cb bytes) 000237 1781227760.673940 fallocate 0x01fc8b thru 0x021758 (0x1ace bytes) 000238 1781227760.687705 punch from 0x0149b8 to 0x021412 (0xca5b bytes) 000239 1781227760.694433 mapread 0x028154 thru 0x02c271 (0x411e bytes) 000240 1781227760.697233 punch from 0x01da20 to 0x02c04c (0xe62d bytes) 000242 1781227760.728983 write 0x02e15c thru 0x03d660 (0xf505 bytes) 000243 1781227760.731445 trunc from 0x03d661 to 0x01e2e7 (0x1f379 bytes) 000244 1781227760.746520 punch from 0x019bdc to 0x01e2e7 (0x470c bytes) 000246 1781227760.753871 fallocate 0x03e32e thru 0x03ffff (0x1cd2 bytes) 000248 1781227760.760225 mapread 0x00216e thru 0x00ca24 (0xa8b7 bytes) 000250 1781227760.764556 trunc from 0x0055cf to 0x015398 (0xfdca bytes) 000251 1781227760.776762 MIRROR_EXTEND to 2 mirrors 000252 1781227761.708723 read 0x009f93 thru 0x015026 (0xb094 bytes) 000253 1781227761.723197 read 0x0128be thru 0x01e2e7 (0xba2a bytes) 000254 1781227761.772195 mapread 0x019afc thru 0x01e2e7 (0x47ec bytes) 000255 1781227761.774196 fallocate 0x02b77e thru 0x03216f (0x69f2 bytes) 000256 1781227761.847278 trunc from 0x023c65 to 0x02bdea (0x8186 bytes) 000257 1781227761.861271 punch from 0x00c82b to 0x00e38e (0x1b64 bytes) 000258 1781227761.867305 fallocate 0x00c4e9 thru 0x0109d6 (0x44ee bytes) 000259 1781227761.873993 trunc from 0x0018f2 to 0x009daf (0x84be bytes) 000260 1781227761.885387 trunc from 0x032170 to 0x02fa10 (0x275f bytes) 000261 1781227761.925528 mapwrite 0x0392bd thru 0x03ffff (0x6d43 bytes) 000262 1781227762.013450 MIRROR_RESYNC 2 mirrors 000263 1781227762.884924 trunc from 0x040000 to 0x03b900 (0x46ff bytes) 000264 1781227762.960018 read 0x0034d8 thru 0x011671 (0xe19a bytes) 000265 1781227763.003994 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b901): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b901): Mirror 2: 0x75b2ba6a (stale) CRC-32 checksum value for chunk [0, 0x3b901): Mirror 1: 0x75b2ba6a Mirror 3: 0x75b2ba6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b901: skipped 000266 1781227764.306252 mapread 0x005075 thru 0x00cddc (0x7d68 bytes) 000267 1781227764.373039 write 0x00161f thru 0x00b431 (0x9e13 bytes) 000269 1781227764.628972 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b901): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b901): Mirror 2: 0x75b2ba6a (stale) Mirror 3: 0x75b2ba6a (stale) CRC-32 checksum value for chunk [0, 0x3b901): Mirror 1: 0xee47e629 Mirror 4: 0xee47e629 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b901: skipped 000270 1781227766.047955 fallocate 0x01e6be thru 0x02c481 (0xddc4 bytes) 000271 1781227766.124905 punch from 0x01ce40 to 0x01edb8 (0x1f79 bytes) 000272 1781227766.132159 write 0x027a83 thru 0x030d27 (0x92a5 bytes) 000273 1781227766.149937 fallocate 0x001a71 thru 0x002464 (0x9f4 bytes) 000274 1781227766.156182 trunc from 0x03b901 to 0x032a07 (0x8ef9 bytes) 000275 1781227766.188562 mapwrite 0x03171a thru 0x0358b8 (0x419f bytes) 000276 1781227766.283702 mapwrite 0x01ec3a thru 0x025a8f (0x6e56 bytes) 000278 1781227766.314134 fallocate 0x0155d5 thru 0x016bd3 (0x15ff bytes) 000279 1781227766.319895 read 0x008fbe thru 0x0114f2 (0x8535 bytes) 000280 1781227766.330357 mapwrite 0x01c450 thru 0x02bd1a (0xf8cb bytes) 000281 1781227766.354864 write 0x01c3ca thru 0x02bc51 (0xf888 bytes) 000282 1781227766.360199 mapread 0x00d599 thru 0x0190be (0xbb26 bytes) 000283 1781227766.377997 MIRROR_RESYNC 4 mirrors 000285 1781227767.860135 trunc from 0x0358b9 to 0x023e5e (0x11a5a bytes) 000286 1781227767.915943 trunc from 0x015694 to 0x01a911 (0x527e bytes) 000287 1781227767.920944 write 0x02b1d2 thru 0x03957d (0xe3ac bytes) 000288 1781227767.923173 mapwrite 0x00bcbe thru 0x01a8b7 (0xebfa bytes) 000289 1781227768.002301 trunc from 0x03957e to 0x02cd94 (0xc7e9 bytes) 000290 1781227768.032313 trunc from 0x02cd95 to 0x028d1e (0x4076 bytes) 000291 1781227768.057674 mapread 0x00452b thru 0x008e25 (0x48fb bytes) 000292 1781227768.079562 mapwrite 0x0372b0 thru 0x03b773 (0x44c4 bytes) 000293 1781227768.156429 mapread 0x035d44 thru 0x037b4b (0x1e08 bytes) 000294 1781227768.168068 punch from 0x0310ef to 0x032168 (0x107a bytes) 000295 1781227768.195364 mapread 0x007b67 thru 0x007fe1 (0x47b bytes) 000296 1781227768.203726 trunc from 0x018654 to 0x025b76 (0xd523 bytes) 000297 1781227768.221422 trunc from 0x01c948 to 0x029f50 (0xd609 bytes) 000298 1781227768.239375 write 0x006bbc thru 0x00dca2 (0x70e7 bytes) 000299 1781227768.243148 punch from 0x031590 to 0x03b773 (0xa1e4 bytes) 000300 1781227768.260506 fallocate 0x032ee9 thru 0x039de3 (0x6efb bytes) 000301 1781227768.270478 trunc from 0x0390c0 to 0x03b773 (0x26b4 bytes) 000302 1781227768.283477 trunc from 0x0249bf to 0x02ca8e (0x80d0 bytes) 000303 1781227768.291484 punch from 0x019c30 to 0x02862c (0xe9fd bytes) 000304 1781227768.301466 fallocate 0x02e2bc thru 0x03b5a8 (0xd2ed bytes) 000305 1781227768.312583 trunc from 0x03b774 to 0x00d537 (0x2e23c bytes) 000306 1781227768.354938 write 0x008faa thru 0x0163a4 (0xd3fb bytes) 000307 1781227768.360346 read 0x0124ee thru 0x0163a4 (0x3eb7 bytes) 000308 1781227768.361662 fallocate 0x03e605 thru 0x03ffff (0x19fb bytes) 000309 1781227768.370092 punch from 0x0265d2 to 0x02f207 (0x8c36 bytes) 000310 1781227768.377337 mapread 0x03d20b thru 0x03f1fa (0x1ff0 bytes) 000311 1781227768.396673 fallocate 0x03a568 thru 0x03caee (0x2587 bytes) 000312 1781227768.404699 read 0x03afd5 thru 0x03bb77 (0xba3 bytes) 000313 1781227768.406321 trunc from 0x040000 to 0x00fc93 (0x3036c bytes) 000314 1781227768.432987 mapread 0x008541 thru 0x00fc93 (0x7753 bytes) 000315 1781227768.443874 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xfc94): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfc94): Mirror 2: 0x9182d6c (stale) Mirror 3: 0x9182d6c (stale) Mirror 4: 0x9182d6c (stale) CRC-32 checksum value for chunk [0, 0xfc94): Mirror 1: 0xabbb2768 Mirror 5: 0xabbb2768 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfc94: skipped 000316 1781227769.508881 mapread 0x000fa0 thru 0x00b409 (0xa46a bytes) 000317 1781227769.571939 trunc from 0x0009ff to 0x00b988 (0xaf8a bytes) 000318 1781227769.634019 mapwrite 0x00d706 thru 0x00e80a (0x1105 bytes) 000320 1781227769.663089 trunc from 0x00df76 to 0x00e91c (0x9a7 bytes) 000321 1781227769.669583 read 0x007fec thru 0x00fc93 (0x7ca8 bytes) 000322 1781227769.677420 trunc from 0x00fc94 to 0x00de09 (0x1e8a bytes) 000323 1781227769.738014 write 0x03e75f thru 0x03ffff (0x18a1 bytes) 000324 1781227769.739447 write 0x02f3b7 thru 0x038c0e (0x9858 bytes) 000325 1781227769.764452 read 0x0149e8 thru 0x018b5e (0x4177 bytes) 000326 1781227769.803441 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xae41955f (stale) Mirror 3: 0xae41955f (stale) Mirror 4: 0xae41955f (stale) Mirror 5: 0x6e581bbd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa83d05cf Mirror 6: 0xa83d05cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000327 1781227771.134520 trunc from 0x00e8bd to 0x0186f4 (0x9e38 bytes) 000330 1781227771.229743 punch from 0x00635e to 0x006581 (0x224 bytes) 000331 1781227771.241726 fallocate 0x00d9cd thru 0x011f38 (0x456c bytes) 000333 1781227771.253316 trunc from 0x040000 to 0x03fb77 (0x488 bytes) 000334 1781227771.294669 read 0x00e2f1 thru 0x01a120 (0xbe30 bytes) 000335 1781227771.315292 fallocate 0x02faca thru 0x03ddbd (0xe2f4 bytes) 000337 1781227771.328415 mapwrite 0x002ca4 thru 0x0107f1 (0xdb4e bytes) 000339 1781227771.372908 write 0x03b86a thru 0x03ffff (0x4796 bytes) 000340 1781227771.382835 trunc from 0x010bcb to 0x01ab98 (0x9fce bytes) 000341 1781227771.390300 punch from 0x03727c to 0x03971e (0x24a3 bytes) 000342 1781227771.396033 mapread 0x01b78f thru 0x01e688 (0x2efa bytes) 000343 1781227771.411721 mapread 0x0073cb thru 0x0097fc (0x2432 bytes) 000344 1781227771.413002 trunc from 0x040000 to 0x016d14 (0x292eb bytes) 000345 1781227771.429843 read 0x003434 thru 0x00e861 (0xb42e bytes) 000346 1781227771.430736 trunc from 0x001819 to 0x005cfc (0x44e4 bytes) 000347 1781227771.437011 mapread 0x000242 thru 0x0058e3 (0x56a2 bytes) 000348 1781227771.445236 fallocate 0x03766d thru 0x03beba (0x484e bytes) 000349 1781227771.449638 mapwrite 0x01985f thru 0x01c839 (0x2fdb bytes) 000350 1781227771.489310 write 0x03bd7b thru 0x03d3bf (0x1645 bytes) 000351 1781227771.493198 trunc from 0x03d3c0 to 0x008abf (0x34900 bytes) 000352 1781227771.512860 punch from 0x0065e3 to 0x0079c1 (0x13df bytes) 000353 1781227771.520256 write 0x038f5e thru 0x03ffff (0x70a2 bytes) 000354 1781227771.523345 punch from 0x0371a3 to 0x03bc11 (0x4a6f bytes) 000357 1781227771.547552 MIRROR_RESYNC 6 mirrors 000358 1781227773.678324 punch from 0x0250e8 to 0x02f337 (0xa250 bytes) 000359 1781227773.754540 punch from 0x03c7e1 to 0x03ffff (0x381f bytes) 000360 1781227773.761798 mapwrite 0x039cea thru 0x03ffff (0x6316 bytes) 000361 1781227773.881208 mapwrite 0x01301d thru 0x01c202 (0x91e6 bytes) 000362 1781227773.937473 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc1130c0e (stale) Mirror 3: 0xc1130c0e (stale) Mirror 4: 0xc1130c0e (stale) Mirror 5: 0xc1130c0e (stale) Mirror 6: 0xc1130c0e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc3831757 Mirror 7: 0xc3831757 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000363 1781227775.447308 trunc from 0x040000 to 0x01b2d6 (0x24d29 bytes) 000364 1781227775.576040 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1b2d7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b2d7): Mirror 2: 0x79f928ab (stale) Mirror 3: 0x79f928ab (stale) Mirror 4: 0x79f928ab (stale) Mirror 5: 0x79f928ab (stale) Mirror 6: 0x79f928ab (stale) Mirror 7: 0x5790545b (stale) CRC-32 checksum value for chunk [0, 0x1b2d7): Mirror 1: 0x5790545b Mirror 8: 0x5790545b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b2d7: skipped 000365 1781227777.462377 trunc from 0x01b2d7 to 0x00d66a (0xdc6c bytes) 000366 1781227777.539242 mapread 0x004444 thru 0x00d66a (0x9227 bytes) 000367 1781227777.564434 punch from 0x004776 to 0x008b80 (0x440b bytes) 000369 1781227777.572962 punch from 0x006977 to 0x008d9b (0x2425 bytes) 000370 1781227777.580137 fallocate 0x0086b7 thru 0x00f9f8 (0x7342 bytes) 000371 1781227777.589044 fallocate 0x01effc thru 0x02a9c9 (0xb9ce bytes) 000372 1781227777.602579 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a9ca): Mirror 2: 0xf161d5e9 (stale) Mirror 3: 0xf161d5e9 (stale) Mirror 5: 0xf161d5e9 (stale) Mirror 6: 0xf161d5e9 (stale) Mirror 7: 0xec4df6c1 (stale) Mirror 8: 0x85c03daa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a9ca): Mirror 2: 0xf161d5e9 (stale) Mirror 3: 0xf161d5e9 (stale) Mirror 5: 0xf161d5e9 (stale) Mirror 6: 0xf161d5e9 (stale) Mirror 7: 0xec4df6c1 (stale) Mirror 8: 0x85c03daa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a9ca: skipped 000373 1781227778.236983 fallocate 0x0133f2 thru 0x01d1e8 (0x9df7 bytes) 000374 1781227778.244565 mapread 0x025a3d thru 0x02a605 (0x4bc9 bytes) 000375 1781227778.266716 mapread 0x01170d thru 0x016e77 (0x576b bytes) 000376 1781227778.277370 read 0x027416 thru 0x02a9c9 (0x35b4 bytes) 000377 1781227778.280026 read 0x005c6c thru 0x00d2e3 (0x7678 bytes) 000378 1781227778.291288 trunc from 0x02a9ca to 0x019983 (0x11046 bytes) 000379 1781227778.317343 mapwrite 0x01b14e thru 0x0247a9 (0x965c bytes) 000380 1781227778.427502 mapwrite 0x000105 thru 0x0026b8 (0x25b4 bytes) 000381 1781227778.452433 mapwrite 0x0363c6 thru 0x03ffff (0x9c3a bytes) 000382 1781227778.566904 mapwrite 0x02889e thru 0x031d10 (0x9473 bytes) 000383 1781227778.625242 mapread 0x019fea thru 0x02776e (0xd785 bytes) 000384 1781227778.669266 MIRROR_RESYNC 7 mirrors 000385 1781227782.523667 fallocate 0x03bd73 thru 0x03dfca (0x2258 bytes) 000386 1781227782.573309 punch from 0x00dbf6 to 0x01c746 (0xeb51 bytes) 000387 1781227782.586485 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xde6516f6 (stale) Mirror 3: 0xde6516f6 (stale) Mirror 5: 0xde6516f6 (stale) Mirror 6: 0xde6516f6 (stale) Mirror 7: 0xde6516f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xde6516f6 (stale) Mirror 3: 0xde6516f6 (stale) Mirror 5: 0xde6516f6 (stale) Mirror 6: 0xde6516f6 (stale) Mirror 7: 0xde6516f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1781227783.205898 trunc from 0x014f10 to 0x01d0e3 (0x81d4 bytes) 000389 1781227783.214083 read 0x01685f thru 0x0265e9 (0xfd8b bytes) 000390 1781227783.226922 fallocate 0x0011ed thru 0x0031ce (0x1fe2 bytes) 000391 1781227783.234428 fallocate 0x02fee3 thru 0x03ca8c (0xcbaa bytes) 000392 1781227783.244964 trunc from 0x00505f to 0x00c38b (0x732d bytes) 000394 1781227783.260583 fallocate 0x01a5d2 thru 0x025dda (0xb809 bytes) 000396 1781227783.268095 mapwrite 0x024639 thru 0x030322 (0xbcea bytes) 000397 1781227783.337306 write 0x005aad thru 0x014705 (0xec59 bytes) 000398 1781227783.362853 trunc from 0x001a27 to 0x003835 (0x1e0f bytes) 000399 1781227783.370197 mapread 0x0020e4 thru 0x007341 (0x525e bytes) 000400 1781227783.378691 trunc from 0x031027 to 0x035408 (0x43e2 bytes) 000401 1781227783.386290 trunc from 0x00cc6e to 0x018e74 (0xc207 bytes) 000402 1781227783.408377 trunc from 0x040000 to 0x00c8e3 (0x3371c bytes) 000403 1781227783.441982 mapwrite 0x0062e3 thru 0x01552c (0xf24a bytes) 000404 1781227783.573082 mapread 0x005d48 thru 0x00cee5 (0x719e bytes) 000405 1781227783.574475 punch from 0x0068d6 to 0x00eb75 (0x82a0 bytes) 000406 1781227783.580878 write 0x00dd99 thru 0x014b09 (0x6d71 bytes) 000407 1781227783.596714 write 0x0129a2 thru 0x014908 (0x1f67 bytes) 000408 1781227783.597891 mapwrite 0x031f84 thru 0x0328d2 (0x94f bytes) 000409 1781227783.671933 trunc from 0x0328d3 to 0x0013b8 (0x3151a bytes) 000410 1781227783.707384 read 0x001237 thru 0x0013b8 (0x182 bytes) 000413 1781227783.712108 read 0x000038 thru 0x0013b8 (0x1381 bytes) 000414 1781227783.727150 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13b9): Mirror 2: 0x42ff476d (stale) Mirror 3: 0x42ff476d (stale) Mirror 5: 0x42ff476d (stale) Mirror 7: 0x42ff476d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13b9): Mirror 2: 0x42ff476d (stale) Mirror 3: 0x42ff476d (stale) Mirror 5: 0x42ff476d (stale) Mirror 7: 0x42ff476d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13b9: skipped 000415 1781227784.263358 trunc from 0x0013b9 to 0x007d12 (0x695a bytes) 000416 1781227784.294395 read 0x001b2b thru 0x007d12 (0x61e8 bytes) 000417 1781227784.307323 mapwrite 0x01abe9 thru 0x0250c0 (0xa4d8 bytes) 000418 1781227784.373461 fallocate 0x03c16c thru 0x03ffff (0x3e94 bytes) 000419 1781227784.379859 punch from 0x03de5e to 0x03ffff (0x21a2 bytes) 000420 1781227784.385684 write 0x00c25a thru 0x00e6f8 (0x249f bytes) 000421 1781227784.396004 read 0x0239e2 thru 0x02b081 (0x76a0 bytes) 000422 1781227784.407860 trunc from 0x01c66b to 0x029b7b (0xd511 bytes) 000423 1781227784.419989 write 0x03bc6c thru 0x03f2c2 (0x3657 bytes) 000424 1781227784.423170 mapwrite 0x017505 thru 0x01d47f (0x5f7b bytes) 000425 1781227784.456937 write 0x01e3c3 thru 0x02d9c9 (0xf607 bytes) 000426 1781227784.467452 fallocate 0x01620b thru 0x022494 (0xc28a bytes) 000427 1781227784.473236 write 0x02f931 thru 0x032b83 (0x3253 bytes) 000428 1781227784.475390 read 0x028e13 thru 0x02f084 (0x6272 bytes) 000429 1781227784.476912 write 0x014626 thru 0x01ff44 (0xb91f bytes) 000430 1781227784.479102 punch from 0x003547 to 0x012108 (0xebc2 bytes) 000431 1781227784.485076 mapwrite 0x00af5a thru 0x013744 (0x87eb bytes) 000434 1781227784.534109 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xde6516f6 (stale) Mirror 5: 0xde6516f6 (stale) Mirror 7: 0xde6516f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xde6516f6 (stale) Mirror 5: 0xde6516f6 (stale) Mirror 7: 0xde6516f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000435 1781227785.219841 write 0x00a1b3 thru 0x01165e (0x74ac bytes) 000436 1781227785.265542 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xde6516f6 (stale) Mirror 5: 0xde6516f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xde6516f6 (stale) Mirror 5: 0xde6516f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000437 1781227786.152132 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xde6516f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xde6516f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000438 1781227786.615197 punch from 0x010958 to 0x018c34 (0x82dd bytes) 000439 1781227786.627862 write 0x00ec27 thru 0x00f21c (0x5f6 bytes) 000441 1781227786.651180 MIRROR_RESYNC 2 mirrors 000442 1781227788.014770 punch from 0x0045ec to 0x00d37b (0x8d90 bytes) 000443 1781227788.090006 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) 000444 1781227788.378966 trunc from 0x020950 to 0x0273f7 (0x6aa8 bytes) 000446 1781227788.394999 write 0x007596 thru 0x00a853 (0x32be bytes) 000447 1781227788.420030 mapwrite 0x01d61f thru 0x0201a2 (0x2b84 bytes) 000448 1781227788.484902 trunc from 0x040000 to 0x003ca4 (0x3c35b bytes) 000449 1781227788.506743 punch from 0x000c54 to 0x003ca4 (0x3051 bytes) 000450 1781227788.513987 punch from 0x0029bc to 0x003ca4 (0x12e9 bytes) 000451 1781227788.525867 mapwrite 0x005475 thru 0x00c5ed (0x7179 bytes) 000452 1781227788.580112 write 0x0158d8 thru 0x01f727 (0x9e50 bytes) 000453 1781227788.583856 punch from 0x00d557 to 0x01516c (0x7c16 bytes) 000454 1781227788.608334 trunc from 0x01f728 to 0x003869 (0x1bebe bytes) 000455 1781227788.644338 trunc from 0x00386a to 0x02dc5b (0x2a3f2 bytes) 000456 1781227788.682784 write 0x030dcd thru 0x03ffff (0xf233 bytes) 000457 1781227788.685338 fallocate 0x02053f thru 0x021e8f (0x1951 bytes) 000458 1781227788.701862 mapread 0x012044 thru 0x0140fb (0x20b8 bytes) 000459 1781227788.729369 trunc from 0x001ac1 to 0x0072cd (0x580d bytes) 000460 1781227788.739992 trunc from 0x00ed82 to 0x01bff7 (0xd276 bytes) 000461 1781227788.753814 punch from 0x0259ab to 0x02c353 (0x69a9 bytes) 000463 1781227788.766109 read 0x02ed96 thru 0x0321f1 (0x345c bytes) 000464 1781227788.777879 punch from 0x017241 to 0x017477 (0x237 bytes) 000465 1781227788.789173 read 0x031991 thru 0x03a34e (0x89be bytes) 000466 1781227788.796048 read 0x00d82f thru 0x0106bf (0x2e91 bytes) 000467 1781227788.819384 fallocate 0x01a9f9 thru 0x026095 (0xb69d bytes) 000468 1781227788.834833 trunc from 0x00b532 to 0x00bffc (0xacb bytes) 000469 1781227788.850206 mapread 0x01ff09 thru 0x02eb09 (0xec01 bytes) 000470 1781227788.854244 mapwrite 0x00ee7a thru 0x0125de (0x3765 bytes) 000471 1781227788.882520 write 0x036d07 thru 0x03ae41 (0x413b bytes) 000473 1781227788.884938 mapread 0x03c368 thru 0x03ffff (0x3c98 bytes) 000474 1781227788.888157 punch from 0x005430 to 0x008e9e (0x3a6f bytes) 000475 1781227788.894509 trunc from 0x019818 to 0x020d4c (0x7535 bytes) 000476 1781227788.903263 read 0x016b47 thru 0x0222de (0xb798 bytes) 000477 1781227788.933851 write 0x0328e6 thru 0x03e9bb (0xc0d6 bytes) 000478 1781227788.936279 punch from 0x02e151 to 0x034f3d (0x6ded bytes) 000479 1781227788.976231 mapwrite 0x00e423 thru 0x01befe (0xdadc bytes) 000480 1781227789.016288 mapread 0x0326e0 thru 0x0385de (0x5eff bytes) 000481 1781227789.093265 write 0x02e7ad thru 0x033a64 (0x52b8 bytes) 000482 1781227789.111671 read 0x020810 thru 0x027d15 (0x7506 bytes) 000483 1781227789.112497 trunc from 0x040000 to 0x0168cd (0x29732 bytes) 000486 1781227789.171928 trunc from 0x00740b to 0x00f15a (0x7d50 bytes) 000487 1781227789.193007 trunc from 0x0168ce to 0x0097d8 (0xd0f5 bytes) 000488 1781227789.253414 trunc from 0x0086d1 to 0x0097d8 (0x1108 bytes) 000489 1781227789.275320 fallocate 0x015ad6 thru 0x018eb7 (0x33e2 bytes) 000490 1781227789.291778 mapread 0x00939f thru 0x0097d8 (0x43a bytes) 000491 1781227789.302795 write 0x02cf45 thru 0x02e10b (0x11c7 bytes) 000492 1781227789.306854 trunc from 0x02e10c to 0x00e556 (0x1fbb5 bytes) 000493 1781227789.335208 fallocate 0x02a750 thru 0x02abe8 (0x499 bytes) 000494 1781227789.342433 punch from 0x00465e to 0x00808c (0x3a2f bytes) 000495 1781227789.352784 trunc from 0x004a9f to 0x0082a9 (0x380b bytes) 000497 1781227789.376075 mapread 0x008c72 thru 0x00e556 (0x58e5 bytes) 000498 1781227789.399421 read 0x0058c7 thru 0x00a305 (0x4a3f bytes) 000499 1781227789.408467 punch from 0x0033d3 to 0x00d45e (0xa08c bytes) 000500 1781227789.417394 mapwrite 0x020fa9 thru 0x027722 (0x677a bytes) 000501 1781227789.512376 punch from 0x004259 to 0x008e26 (0x4bce bytes) 000502 1781227789.520234 trunc from 0x0177e4 to 0x01b05f (0x387c bytes) 000503 1781227789.533223 trunc from 0x00eff2 to 0x010c9a (0x1ca9 bytes) 000504 1781227789.542887 trunc from 0x027723 to 0x0385a0 (0x10e7e bytes) 000505 1781227789.565210 write 0x000e67 thru 0x001c7a (0xe14 bytes) 000506 1781227789.579041 punch from 0x01ed17 to 0x027db0 (0x909a bytes) 000507 1781227789.590964 punch from 0x01ebe3 to 0x0237f1 (0x4c0f bytes) 000508 1781227789.629501 MIRROR_EXTEND to 2 mirrors 000509 1781227790.837931 trunc from 0x02a72a to 0x02ae98 (0x76f bytes) 000510 1781227790.888540 mapwrite 0x0006d8 thru 0x0014a6 (0xdcf bytes) 000511 1781227790.921139 fallocate 0x0096a9 thru 0x0131f2 (0x9b4a bytes) 000512 1781227790.929756 trunc from 0x0385a1 to 0x01c8c9 (0x1bcd7 bytes) 000513 1781227790.971862 punch from 0x00fed9 to 0x01896e (0x8a96 bytes) 000514 1781227790.978487 read 0x0015e0 thru 0x00da1b (0xc43c bytes) 000515 1781227790.994873 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c8ca): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c8ca): Mirror 2: 0xa27c69f4 (stale) CRC-32 checksum value for chunk [0, 0x1c8ca): Mirror 1: 0x7e93a665 Mirror 3: 0x7e93a665 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c8ca: skipped 000516 1781227792.013921 fallocate 0x012129 thru 0x01be65 (0x9d3d bytes) 000517 1781227792.054194 mapread 0x019f67 thru 0x01c8c9 (0x2963 bytes) 000518 1781227792.074236 read 0x01278e thru 0x013ded (0x1660 bytes) 000520 1781227792.081165 mapread 0x011c51 thru 0x0190d5 (0x7485 bytes) 000521 1781227792.094760 trunc from 0x01c8ca to 0x00debf (0xea0a bytes) 000522 1781227792.122689 mapread 0x0082a8 thru 0x00ca41 (0x479a bytes) 000523 1781227792.133936 punch from 0x000e1b to 0x007235 (0x641b bytes) 000524 1781227792.139510 trunc from 0x004fe6 to 0x00a46d (0x5488 bytes) 000525 1781227792.144228 trunc from 0x00dec0 to 0x017126 (0x9267 bytes) 000526 1781227792.158687 trunc from 0x017127 to 0x017c0c (0xae6 bytes) 000527 1781227792.176770 mapread 0x016489 thru 0x017c0c (0x1784 bytes) 000528 1781227792.191093 punch from 0x00ab34 to 0x017a76 (0xcf43 bytes) 000530 1781227792.196909 trunc from 0x00bc42 to 0x01066f (0x4a2e bytes) 000531 1781227792.202386 trunc from 0x017c0d to 0x01b30c (0x3700 bytes) 000532 1781227792.217712 fallocate 0x02ceaf thru 0x0326e4 (0x5836 bytes) 000533 1781227792.223946 mapread 0x018253 thru 0x01b30c (0x30ba bytes) 000534 1781227792.234463 trunc from 0x01b30d to 0x0375fe (0x1c2f2 bytes) 000535 1781227792.258707 read 0x004d07 thru 0x007884 (0x2b7e bytes) 000536 1781227792.294725 fallocate 0x03741d thru 0x03ffff (0x8be3 bytes) 000537 1781227792.302185 read 0x037437 thru 0x0375fe (0x1c8 bytes) 000538 1781227792.302942 write 0x0286cc thru 0x0373ca (0xecff bytes) 000539 1781227792.305919 mapwrite 0x016926 thru 0x01bd7d (0x5458 bytes) 000540 1781227792.331212 trunc from 0x0375ff to 0x0162c0 (0x2133e bytes) 000541 1781227792.374777 MIRROR_RESYNC 3 mirrors 000542 1781227793.209240 mapwrite 0x00ca67 thru 0x018d29 (0xc2c3 bytes) 000543 1781227793.330445 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18d2a): Mirror 3: 0xf0fb4151 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18d2a): Mirror 3: 0xf0fb4151 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18d2a: skipped 000544 1781227793.633978 mapwrite 0x035b65 thru 0x03f62e (0x9aca bytes) 000545 1781227793.725457 punch from 0x02624b to 0x026716 (0x4cc bytes) 000546 1781227793.732189 trunc from 0x014a01 to 0x0244d8 (0xfad8 bytes) 000547 1781227793.738895 trunc from 0x03f62f to 0x03ffaa (0x97c bytes) 000548 1781227793.767051 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ffab): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffab): Mirror 3: 0x1b5b19a6 (stale) CRC-32 checksum value for chunk [0, 0x3ffab): Mirror 1: 0xfb057364 Mirror 4: 0xfb057364 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffab: skipped 000549 1781227794.940719 fallocate 0x018fa8 thru 0x019a92 (0xaeb bytes) 000550 1781227795.015211 trunc from 0x03db75 to 0x03ffaa (0x2436 bytes) 000551 1781227795.021073 mapread 0x02b686 thru 0x035bde (0xa559 bytes) 000552 1781227795.044238 mapwrite 0x020ab0 thru 0x027e89 (0x73da bytes) 000553 1781227795.069898 fallocate 0x0273b1 thru 0x028a8e (0x16de bytes) 000554 1781227795.078412 fallocate 0x023ec7 thru 0x03057e (0xc6b8 bytes) 000555 1781227795.086699 trunc from 0x03ffab to 0x024852 (0x1b758 bytes) 000556 1781227795.156739 mapwrite 0x019155 thru 0x01e2ec (0x5198 bytes) 000557 1781227795.214503 trunc from 0x024853 to 0x016689 (0xe1c9 bytes) 000558 1781227795.245210 trunc from 0x00f6cb to 0x00fe0d (0x743 bytes) 000559 1781227795.263029 read 0x005d61 thru 0x00fe6c (0xa10c bytes) 000560 1781227795.280056 fallocate 0x015622 thru 0x024e19 (0xf7f8 bytes) 000561 1781227795.311895 read 0x01fba9 thru 0x02071f (0xb77 bytes) 000562 1781227795.336505 fallocate 0x0229cc thru 0x029d35 (0x736a bytes) 000563 1781227795.350944 read 0x010e8b thru 0x01b24a (0xa3c0 bytes) 000564 1781227795.351928 read 0x0149f0 thru 0x020885 (0xbe96 bytes) 000565 1781227795.352659 read 0x006a4d thru 0x00d899 (0x6e4d bytes) 000566 1781227795.357492 trunc from 0x00edd2 to 0x01de3d (0xf06c bytes) 000567 1781227795.364666 punch from 0x000b74 to 0x004099 (0x3526 bytes) 000568 1781227795.380672 mapread 0x0263fc thru 0x029d35 (0x393a bytes) 000569 1781227795.425445 mapread 0x0090e0 thru 0x00a4bc (0x13dd bytes) 000570 1781227795.446539 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29d36): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d36): Mirror 3: 0xbe801289 (stale) Mirror 4: 0x41ee2970 (stale) CRC-32 checksum value for chunk [0, 0x29d36): Mirror 1: 0xd84744ca Mirror 5: 0xd84744ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d36: skipped 000571 1781227797.208526 trunc from 0x029d36 to 0x019c83 (0x100b2 bytes) 000572 1781227797.281576 fallocate 0x02bafe thru 0x02ffb4 (0x44b7 bytes) 000573 1781227797.288447 mapread 0x02f4bd thru 0x02ffb4 (0xaf8 bytes) 000574 1781227797.302787 mapwrite 0x00274a thru 0x00622c (0x3ae3 bytes) 000575 1781227797.394839 read 0x01e1ea thru 0x02a3e6 (0xc1fd bytes) 000576 1781227797.410475 write 0x017913 thru 0x01f980 (0x806e bytes) 000577 1781227797.430887 trunc from 0x020764 to 0x02771f (0x6fbc bytes) 000578 1781227797.443652 punch from 0x01d9d3 to 0x0243e0 (0x6a0e bytes) 000579 1781227797.496144 trunc from 0x02ffb5 to 0x035fbe (0x600a bytes) 000580 1781227797.549430 punch from 0x026577 to 0x035fbe (0xfa48 bytes) 000581 1781227797.573697 trunc from 0x011921 to 0x01dcb8 (0xc398 bytes) 000582 1781227797.588101 mapwrite 0x01df45 thru 0x01e880 (0x93c bytes) 000583 1781227797.615866 write 0x032cdd thru 0x03ffff (0xd323 bytes) 000584 1781227797.634945 fallocate 0x01d544 thru 0x01d9a5 (0x462 bytes) 000585 1781227797.642727 read 0x01bbac thru 0x01dd60 (0x21b5 bytes) 000586 1781227797.644285 trunc from 0x02794a to 0x0312a4 (0x995b bytes) 000587 1781227797.651754 read 0x012ffa thru 0x018b3b (0x5b42 bytes) 000588 1781227797.687827 mapwrite 0x01bb09 thru 0x022618 (0x6b10 bytes) 000589 1781227797.722411 mapwrite 0x02d4e0 thru 0x0367ce (0x92ef bytes) 000590 1781227797.763929 mapwrite 0x03725f thru 0x03ffff (0x8da1 bytes) 000591 1781227797.801864 mapwrite 0x004619 thru 0x0103c1 (0xbda9 bytes) 000592 1781227797.857112 write 0x0102e6 thru 0x01b10a (0xae25 bytes) 000593 1781227797.864385 read 0x017c37 thru 0x01eb57 (0x6f21 bytes) 000594 1781227797.869266 mapread 0x016c6e thru 0x0188be (0x1c51 bytes) 000595 1781227797.872752 punch from 0x039ac5 to 0x03ffff (0x653b bytes) 000596 1781227797.888852 trunc from 0x040000 to 0x0232b7 (0x1cd48 bytes) 000598 1781227797.957876 fallocate 0x03314f thru 0x03eac9 (0xb97b bytes) 000600 1781227798.005047 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x232b8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232b8): Mirror 3: 0x4052eec0 (stale) Mirror 4: 0x152682d2 (stale) Mirror 5: 0xc34b3fba (stale) CRC-32 checksum value for chunk [0, 0x232b8): Mirror 1: 0xf52ce43b Mirror 6: 0xf52ce43b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232b8: skipped 000601 1781227799.943833 fallocate 0x003eaf thru 0x0115bd (0xd70f bytes) 000602 1781227800.004926 punch from 0x018625 to 0x0232b7 (0xac93 bytes) 000603 1781227800.012157 trunc from 0x0232b8 to 0x003f30 (0x1f387 bytes) 000604 1781227800.054995 punch from 0x0010da to 0x003f30 (0x2e57 bytes) 000605 1781227800.061482 punch from 0x0031d2 to 0x003f30 (0xd5f bytes) 000606 1781227800.066816 mapread 0x00155a thru 0x003f30 (0x29d7 bytes) 000609 1781227800.086908 read 0x0017e9 thru 0x003f30 (0x2748 bytes) 000610 1781227800.087977 punch from 0x003bfc to 0x003f30 (0x335 bytes) 000611 1781227800.094855 mapread 0x003575 thru 0x003f30 (0x9bc bytes) 000612 1781227800.106995 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f31): Mirror 4: 0x2b72b042 (stale) Mirror 5: 0x2c5686c7 (stale) Mirror 6: 0x298ac059 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f31): Mirror 4: 0x2b72b042 (stale) Mirror 5: 0x2c5686c7 (stale) Mirror 6: 0x298ac059 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f31: skipped 000613 1781227800.499895 mapwrite 0x021b03 thru 0x02bde5 (0xa2e3 bytes) 000614 1781227800.589942 trunc from 0x01e052 to 0x02bde5 (0xdd94 bytes) 000615 1781227800.601772 fallocate 0x0136f7 thru 0x01488d (0x1197 bytes) 000616 1781227800.609694 mapwrite 0x03d314 thru 0x03ffff (0x2cec bytes) 000618 1781227800.659013 write 0x00cc26 thru 0x013ab0 (0x6e8b bytes) 000620 1781227800.672958 trunc from 0x00e234 to 0x01da91 (0xf85e bytes) 000621 1781227800.699515 write 0x035e80 thru 0x0382bf (0x2440 bytes) 000622 1781227800.713387 mapwrite 0x006d9f thru 0x00f78b (0x89ed bytes) 000624 1781227800.741019 punch from 0x03f345 to 0x03ffff (0xcbb bytes) 000625 1781227800.748059 punch from 0x03a082 to 0x03bca9 (0x1c28 bytes) 000626 1781227800.753027 mapread 0x007f16 thru 0x0171a1 (0xf28c bytes) 000627 1781227800.800214 punch from 0x023f62 to 0x0249a3 (0xa42 bytes) 000628 1781227800.808096 mapread 0x00ad72 thru 0x00eed6 (0x4165 bytes) 000629 1781227800.818033 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3f69b63 (stale) Mirror 5: 0xade23824 (stale) Mirror 6: 0x19c2b2fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x26c0e168 Mirror 7: 0x26c0e168 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000630 1781227802.437287 punch from 0x006dbb to 0x0105ae (0x97f4 bytes) 000631 1781227802.492821 trunc from 0x040000 to 0x03e753 (0x18ac bytes) 000632 1781227802.524173 mapwrite 0x03bdf4 thru 0x03ffff (0x420c bytes) 000633 1781227802.602023 write 0x00ba3d thru 0x019081 (0xd645 bytes) 000634 1781227802.621731 trunc from 0x00bd5b to 0x014af6 (0x8d9c bytes) 000635 1781227802.645540 read 0x03b56b thru 0x03ffff (0x4a95 bytes) 000636 1781227802.646970 fallocate 0x02f167 thru 0x03a8aa (0xb744 bytes) 000638 1781227802.674632 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3f69b63 (stale) Mirror 5: 0xade23824 (stale) Mirror 6: 0x19c2b2fe (stale) Mirror 7: 0x26c0e168 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x97829261 Mirror 8: 0x97829261 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000639 1781227804.603672 trunc from 0x040000 to 0x003e5a (0x3c1a5 bytes) 000640 1781227804.661168 trunc from 0x003e5b to 0x00ab40 (0x6ce6 bytes) 000641 1781227804.681248 trunc from 0x0052d4 to 0x00ab40 (0x586d bytes) 000642 1781227804.687264 punch from 0x00959d to 0x00ab40 (0x15a4 bytes) 000643 1781227804.692779 trunc from 0x00534c to 0x00ab40 (0x57f5 bytes) 000644 1781227804.697990 trunc from 0x00ab41 to 0x02e133 (0x235f3 bytes) 000645 1781227804.713457 punch from 0x006b25 to 0x015721 (0xebfd bytes) 000646 1781227804.720894 read 0x029e0a thru 0x02e133 (0x432a bytes) 000647 1781227804.732013 trunc from 0x02e134 to 0x009f0a (0x24229 bytes) 000648 1781227804.753493 fallocate 0x02f050 thru 0x03993c (0xa8ed bytes) 000649 1781227804.773826 mapwrite 0x01de93 thru 0x025da8 (0x7f16 bytes) 000651 1781227804.852929 trunc from 0x03993d to 0x01b6ff (0x1e23d bytes) 000652 1781227804.883539 fallocate 0x02712e thru 0x030155 (0x9028 bytes) 000653 1781227804.897476 mapread 0x006267 thru 0x00a98f (0x4729 bytes) 000654 1781227804.923309 mapwrite 0x003a97 thru 0x0043db (0x945 bytes) 000656 1781227804.942524 punch from 0x00be0f to 0x010f65 (0x5157 bytes) 000657 1781227804.948761 mapread 0x01a352 thru 0x01b6ff (0x13ae bytes) 000658 1781227804.949702 trunc from 0x014a23 to 0x0171dc (0x27ba bytes) 000659 1781227804.960758 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b700): Mirror 5: 0x9ae50e5c (stale) Mirror 6: 0xfd27c743 (stale) Mirror 7: 0xc03a28e7 (stale) Mirror 8: 0x12c64d6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b700): Mirror 5: 0x9ae50e5c (stale) Mirror 6: 0xfd27c743 (stale) Mirror 7: 0xc03a28e7 (stale) Mirror 8: 0x12c64d6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b700: skipped 000661 1781227805.394822 write 0x03db9c thru 0x03ffff (0x2464 bytes) 000662 1781227805.404412 punch from 0x035ce0 to 0x03ffff (0xa320 bytes) 000663 1781227805.414953 trunc from 0x03e589 to 0x03ffff (0x1a77 bytes) 000664 1781227805.420537 mapread 0x01d50e thru 0x02ccc5 (0xf7b8 bytes) 000665 1781227805.450298 mapwrite 0x02eb6b thru 0x03e31b (0xf7b1 bytes) 000666 1781227805.498766 punch from 0x029ef4 to 0x034093 (0xa1a0 bytes) 000667 1781227805.504585 mapwrite 0x00ac05 thru 0x00ef90 (0x438c bytes) 000668 1781227805.533858 write 0x006abb thru 0x00ce8a (0x63d0 bytes) 000671 1781227805.542549 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xade23824 (stale) Mirror 6: 0x19c2b2fe (stale) Mirror 7: 0x26c0e168 (stale) Mirror 8: 0x97829261 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7c596839 Mirror 9: 0x7c596839 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000672 1781227806.969970 read 0x01c6ee thru 0x01d27e (0xb91 bytes) 000673 1781227806.977396 trunc from 0x013f7b to 0x0147e7 (0x86d bytes) 000674 1781227807.032825 mapwrite 0x02559b thru 0x02ec13 (0x9679 bytes) 000676 1781227807.103772 mapwrite 0x00ac89 thru 0x00d938 (0x2cb0 bytes) 000677 1781227807.125961 trunc from 0x03421d to 0x03e098 (0x9e7c bytes) 000678 1781227807.132635 mapread 0x00f1ef thru 0x0189bf (0x97d1 bytes) 000679 1781227807.145279 mapread 0x03c5a2 thru 0x03ffff (0x3a5e bytes) 000680 1781227807.162776 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xade23824 (stale) Mirror 6: 0x19c2b2fe (stale) Mirror 7: 0x26c0e168 (stale) Mirror 8: 0x97829261 (stale) Mirror 9: 0x7c596839 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x60625eef Mirror 10: 0x60625eef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000681 1781227808.335555 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xade23824 (stale) Mirror 6: 0x19c2b2fe (stale) Mirror 7: 0x26c0e168 (stale) Mirror 8: 0x97829261 (stale) Mirror 9: 0x7c596839 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xade23824 (stale) Mirror 6: 0x19c2b2fe (stale) Mirror 7: 0x26c0e168 (stale) Mirror 8: 0x97829261 (stale) Mirror 9: 0x7c596839 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000682 1781227808.789941 write 0x0173fc thru 0x027201 (0xfe06 bytes) 000683 1781227808.831532 fallocate 0x03db5b thru 0x03ffff (0x24a5 bytes) 000684 1781227808.837537 trunc from 0x02ee02 to 0x03dbe4 (0xede3 bytes) 000685 1781227808.848184 read 0x01d4a5 thru 0x02273c (0x5298 bytes) 000686 1781227808.854440 read 0x01a1ba thru 0x024729 (0xa570 bytes) 000687 1781227808.856361 mapread 0x0390e0 thru 0x03ffff (0x6f20 bytes) 000689 1781227808.911233 punch from 0x024479 to 0x02f390 (0xaf18 bytes) 000690 1781227808.927872 punch from 0x007975 to 0x00ecc4 (0x7350 bytes) 000691 1781227808.935341 trunc from 0x040000 to 0x00b3e1 (0x34c1e bytes) 000693 1781227808.982947 mapread 0x003b14 thru 0x00b3e1 (0x78ce bytes) 000694 1781227809.007876 fallocate 0x03fa0f thru 0x03ffff (0x5f1 bytes) 000695 1781227809.011918 punch from 0x0251dd to 0x02e459 (0x927d bytes) 000696 1781227809.015737 trunc from 0x03927c to 0x03c09c (0x2e21 bytes) 000697 1781227809.019451 fallocate 0x0119b0 thru 0x017d51 (0x63a2 bytes) 000698 1781227809.024309 read 0x02ab05 thru 0x032a0b (0x7f07 bytes) 000699 1781227809.058038 trunc from 0x03b94c to 0x03ffff (0x46b4 bytes) 000700 1781227809.067641 MIRROR_RESYNC 6 mirrors 000701 1781227811.345354 trunc from 0x036926 to 0x038a23 (0x20fe bytes) 000702 1781227811.405500 punch from 0x006c00 to 0x00a82e (0x3c2f bytes) 000703 1781227811.418087 trunc from 0x03d4cc to 0x03ffff (0x2b34 bytes) 000704 1781227811.436175 read 0x027848 thru 0x036c52 (0xf40b bytes) 000705 1781227811.447368 mapwrite 0x0398a8 thru 0x03ffff (0x6758 bytes) 000706 1781227811.509018 fallocate 0x026b41 thru 0x0307a3 (0x9c63 bytes) 000707 1781227811.515579 mapread 0x015245 thru 0x015fcc (0xd88 bytes) 000708 1781227811.536876 write 0x035470 thru 0x03ffff (0xab90 bytes) 000709 1781227811.540504 fallocate 0x0230a1 thru 0x02916f (0x60cf bytes) 000710 1781227811.548531 punch from 0x0299c3 to 0x035040 (0xb67e bytes) 000711 1781227811.570797 mapwrite 0x034006 thru 0x037c94 (0x3c8f bytes) 000713 1781227811.604001 read 0x0292b8 thru 0x035a68 (0xc7b1 bytes) 000714 1781227811.625294 mapread 0x03f189 thru 0x03ffff (0xe77 bytes) 000715 1781227811.626769 fallocate 0x018d7f thru 0x026d6b (0xdfed bytes) 000716 1781227811.633164 fallocate 0x02a264 thru 0x02e560 (0x42fd bytes) 000717 1781227811.639392 punch from 0x01d5b9 to 0x0247dc (0x7224 bytes) 000718 1781227811.644135 trunc from 0x00bfc1 to 0x011a61 (0x5aa1 bytes) 000719 1781227811.651003 write 0x001480 thru 0x00478b (0x330c bytes) 000721 1781227811.653167 punch from 0x039d78 to 0x03ffff (0x6288 bytes) 000722 1781227811.664482 mapread 0x031435 thru 0x03fe8e (0xea5a bytes) 000723 1781227811.717470 trunc from 0x02665d to 0x02ebd9 (0x857d bytes) 000724 1781227811.725001 fallocate 0x02b7c7 thru 0x03698d (0xb1c7 bytes) 000725 1781227811.732337 trunc from 0x040000 to 0x028d93 (0x1726c bytes) 000726 1781227811.772782 MIRROR_RESYNC 6 mirrors 000727 1781227813.514389 write 0x00a1ac thru 0x016602 (0xc457 bytes) 000728 1781227813.574510 MIRROR_RESYNC 6 mirrors 000729 1781227815.425433 trunc from 0x028d94 to 0x03027f (0x74ec bytes) 000731 1781227815.497106 read 0x02acfc thru 0x03027f (0x5584 bytes) 000732 1781227815.522402 mapwrite 0x037d2b thru 0x03ffff (0x82d5 bytes) 000733 1781227815.649800 trunc from 0x040000 to 0x01e6df (0x21920 bytes) 000734 1781227815.720551 mapwrite 0x0099a1 thru 0x015666 (0xbcc6 bytes) 000735 1781227815.826468 trunc from 0x01e6e0 to 0x00d907 (0x10dd8 bytes) 000736 1781227815.855154 punch from 0x000027 to 0x008544 (0x851e bytes) 000737 1781227815.866033 mapwrite 0x0069b7 thru 0x00c509 (0x5b53 bytes) 000738 1781227815.920811 punch from 0x0038a8 to 0x00d907 (0xa060 bytes) 000740 1781227815.938401 punch from 0x0048a8 to 0x00512d (0x886 bytes) 000741 1781227815.960043 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd908): Mirror 5: 0x41b4f228 (stale) Mirror 6: 0x41b4f228 (stale) Mirror 7: 0x41b4f228 (stale) Mirror 9: 0x41b4f228 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd908): Mirror 5: 0x41b4f228 (stale) Mirror 6: 0x41b4f228 (stale) Mirror 7: 0x41b4f228 (stale) Mirror 9: 0x41b4f228 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd908: skipped 000742 1781227816.368106 punch from 0x009091 to 0x00d907 (0x4877 bytes) 000743 1781227816.373957 trunc from 0x00d908 to 0x01b5ee (0xdce7 bytes) 000744 1781227816.393927 trunc from 0x01006e to 0x01a894 (0xa827 bytes) 000745 1781227816.399492 mapread 0x017342 thru 0x01b5ee (0x42ad bytes) 000746 1781227816.427798 fallocate 0x0237be thru 0x02e96f (0xb1b2 bytes) 000747 1781227816.437411 trunc from 0x01b5ef to 0x004ec0 (0x1672e bytes) 000748 1781227816.478486 MIRROR_RESYNC 5 mirrors 000749 1781227817.626460 punch from 0x0037f1 to 0x003cdb (0x4eb bytes) 000750 1781227817.678670 mapwrite 0x013c6a thru 0x015230 (0x15c7 bytes) 000751 1781227817.760745 write 0x03ed12 thru 0x03ffff (0x12ee bytes) 000752 1781227817.761932 mapwrite 0x028104 thru 0x0336ea (0xb5e7 bytes) 000754 1781227817.826719 mapwrite 0x0347aa thru 0x03af8c (0x67e3 bytes) 000755 1781227817.873392 trunc from 0x032743 to 0x03a42d (0x7ceb bytes) 000756 1781227817.905649 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000757 1781227818.471411 read 0x016949 thru 0x01e88b (0x7f43 bytes) 000758 1781227818.494549 mapread 0x024827 thru 0x02c0e3 (0x78bd bytes) 000760 1781227818.521196 punch from 0x03f744 to 0x03ffff (0x8bc bytes) 000761 1781227818.531805 write 0x02fa66 thru 0x03013b (0x6d6 bytes) 000762 1781227818.562802 read 0x007271 thru 0x010784 (0x9514 bytes) 000763 1781227818.590054 trunc from 0x040000 to 0x030d81 (0xf27e bytes) 000764 1781227818.630580 write 0x00480c thru 0x00e3a7 (0x9b9c bytes) 000765 1781227818.633476 read 0x004386 thru 0x007549 (0x31c4 bytes) 000766 1781227818.645138 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30d82): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d82): Mirror 5: 0x81a8b539 (stale) Mirror 7: 0x81a8b539 (stale) Mirror 9: 0x81a8b539 (stale) CRC-32 checksum value for chunk [0, 0x30d82): Mirror 1: 0x9b0baf56 Mirror 10: 0x9b0baf56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d82: skipped 000767 1781227820.125328 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30d82): Mirror 7: 0x81a8b539 (stale) Mirror 9: 0x81a8b539 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d82): Mirror 7: 0x81a8b539 (stale) Mirror 9: 0x81a8b539 (stale) CRC-32 checksum value for chunk [0, 0x30d82): Mirror 1: 0x9b0baf56 Mirror 10: 0x9b0baf56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d82: skipped 000768 1781227821.022782 fallocate 0x02f95d thru 0x038040 (0x86e4 bytes) 000769 1781227821.142397 fallocate 0x018138 thru 0x01c1c4 (0x408d bytes) 000770 1781227821.176876 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38041): Mirror 7: 0xd61627f3 (stale) Mirror 10: 0x43fd9a22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38041): Mirror 7: 0xd61627f3 (stale) Mirror 10: 0x43fd9a22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38041: skipped 000772 1781227821.846039 trunc from 0x038041 to 0x0205f5 (0x17a4b bytes) 000773 1781227821.893978 punch from 0x01f481 to 0x0205f5 (0x1175 bytes) 000775 1781227821.920377 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x205f6): Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x205f6): Mirror 7: 0x78b6b7ed (stale) Mirror 10: 0x21131d54 (stale) CRC-32 checksum value for chunk [0, 0x205f6): Mirror 1: 0x21131d54 Mirror 11: 0x21131d54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x205f6: skipped 000776 1781227823.442581 MIRROR_RESYNC 4 mirrors 000777 1781227824.858788 trunc from 0x0205f6 to 0x02f06c (0xea77 bytes) 000779 1781227824.965524 write 0x00c628 thru 0x01126a (0x4c43 bytes) 000780 1781227824.984426 read 0x02e22a thru 0x02e55a (0x331 bytes) 000781 1781227824.996585 read 0x01e058 thru 0x02c1ab (0xe154 bytes) 000782 1781227825.029892 mapwrite 0x023319 thru 0x025890 (0x2578 bytes) 000783 1781227825.071134 trunc from 0x02f06d to 0x03001f (0xfb3 bytes) 000785 1781227825.098107 write 0x014e6c thru 0x01dc23 (0x8db8 bytes) 000786 1781227825.100476 mapwrite 0x00b2bc thru 0x00ea4e (0x3793 bytes) 000787 1781227825.123794 punch from 0x0196ff to 0x02312f (0x9a31 bytes) 000788 1781227825.136222 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30020): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30020): Mirror 7: 0x54fc9ebb (stale) Mirror 10: 0x54fc9ebb (stale) Mirror 11: 0x54fc9ebb (stale) CRC-32 checksum value for chunk [0, 0x30020): Mirror 1: 0x12be56d3 Mirror 12: 0x12be56d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30020: skipped 000789 1781227826.664823 read 0x02168e thru 0x03001f (0xe992 bytes) 000790 1781227826.687513 punch from 0x02fcce to 0x03001f (0x352 bytes) 000791 1781227826.771750 read 0x01eb26 thru 0x02e15c (0xf637 bytes) 000792 1781227826.815345 mapwrite 0x0072ff thru 0x00a8fd (0x35ff bytes) 000793 1781227826.846532 read 0x02f45f thru 0x03001f (0xbc1 bytes) 000794 1781227826.847271 mapread 0x0017cd thru 0x0036bf (0x1ef3 bytes) 000795 1781227826.848055 write 0x03ff1d thru 0x03ffff (0xe3 bytes) 000796 1781227826.853489 fallocate 0x0081db thru 0x011cb4 (0x9ada bytes) 000797 1781227826.861828 punch from 0x029ca5 to 0x02f095 (0x53f1 bytes) 000798 1781227826.866853 trunc from 0x040000 to 0x029f4c (0x160b3 bytes) 000799 1781227826.963479 mapwrite 0x00e191 thru 0x01cecf (0xed3f bytes) 000800 1781227827.033392 mapwrite 0x00bf7b thru 0x016df9 (0xae7f bytes) 000801 1781227827.054070 trunc from 0x029f4d to 0x02babd (0x1b71 bytes) 000802 1781227827.073827 trunc from 0x025475 to 0x025e63 (0x9ef bytes) 000803 1781227827.079911 fallocate 0x01c1a6 thru 0x01e129 (0x1f84 bytes) 000804 1781227827.085492 mapread 0x024769 thru 0x02babd (0x7355 bytes) 000805 1781227827.096491 read 0x004be9 thru 0x00670b (0x1b23 bytes) 000807 1781227827.098386 mapread 0x025e4d thru 0x02babd (0x5c71 bytes) 000808 1781227827.105320 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2babe): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2babe): Mirror 7: 0x435116e5 (stale) Mirror 10: 0x435116e5 (stale) Mirror 11: 0x435116e5 (stale) Mirror 12: 0x19777a92 (stale) CRC-32 checksum value for chunk [0, 0x2babe): Mirror 1: 0x5dec11d2 Mirror 13: 0x5dec11d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2babe: skipped 000809 1781227828.204727 trunc from 0x02babe to 0x024c68 (0x6e55 bytes) 000810 1781227828.291076 trunc from 0x000bd8 to 0x00ab36 (0x9f5f bytes) 000811 1781227828.298835 read 0x0231ae thru 0x024c68 (0x1abb bytes) 000812 1781227828.310312 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24c69): Mirror 7: 0xb68a06a0 (stale) Mirror 10: 0xb68a06a0 (stale) Mirror 12: 0x7c01bef2 (stale) Mirror 13: 0x15e6ff0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c69): Mirror 7: 0xb68a06a0 (stale) Mirror 10: 0xb68a06a0 (stale) Mirror 12: 0x7c01bef2 (stale) Mirror 13: 0x15e6ff0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c69: skipped 000813 1781227828.898553 trunc from 0x024706 to 0x024c68 (0x563 bytes) 000814 1781227828.907006 read 0x001b30 thru 0x007e02 (0x62d3 bytes) 000816 1781227828.918022 mapread 0x01dafd thru 0x02132b (0x382f bytes) 000817 1781227828.942281 mapread 0x0134b5 thru 0x01dbf7 (0xa743 bytes) 000818 1781227828.959830 read 0x01fad9 thru 0x01fed6 (0x3fe bytes) 000819 1781227828.961815 fallocate 0x024923 thru 0x02d95c (0x903a bytes) 000820 1781227828.976221 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d95d): Mirror 10: 0xaf7864b0 (stale) Mirror 12: 0x8510a240 (stale) Mirror 13: 0x6dde0969 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d95d): Mirror 10: 0xaf7864b0 (stale) Mirror 12: 0x8510a240 (stale) Mirror 13: 0x6dde0969 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d95d: skipped 000821 1781227829.358155 write 0x03e44d thru 0x03ffff (0x1bb3 bytes) 000822 1781227829.371396 fallocate 0x00c274 thru 0x01a1d6 (0xdf63 bytes) 000823 1781227829.384907 trunc from 0x040000 to 0x02a45a (0x15ba5 bytes) 000824 1781227829.417163 read 0x0179a0 thru 0x02092b (0x8f8c bytes) 000825 1781227829.430280 read 0x022d69 thru 0x025835 (0x2acd bytes) 000826 1781227829.447329 punch from 0x01b37a to 0x025c6c (0xa8f3 bytes) 000827 1781227829.461069 mapread 0x010357 thru 0x0168a5 (0x654f bytes) 000829 1781227829.462968 mapread 0x00ffae thru 0x01243d (0x2490 bytes) 000830 1781227829.465232 mapread 0x01be0f thru 0x01f22e (0x3420 bytes) 000831 1781227829.511809 fallocate 0x0210c7 thru 0x025b7b (0x4ab5 bytes) 000832 1781227829.540004 mapread 0x0112a8 thru 0x0179e3 (0x673c bytes) 000833 1781227829.554247 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a45b): Mirror 12: 0x542036cf (stale) Mirror 13: 0xce7b85a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a45b): Mirror 12: 0x542036cf (stale) Mirror 13: 0xce7b85a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a45b: skipped 000834 1781227830.090340 punch from 0x007b36 to 0x010f90 (0x945b bytes) 000835 1781227830.102945 mapwrite 0x039685 thru 0x03ffff (0x697b bytes) 000836 1781227830.199726 trunc from 0x011e6f to 0x01f180 (0xd312 bytes) 000837 1781227830.205166 trunc from 0x02282e to 0x02c56c (0x9d3f bytes) 000838 1781227830.210003 trunc from 0x01603e to 0x01e397 (0x835a bytes) 000839 1781227830.219285 fallocate 0x01c7fe thru 0x0219a3 (0x51a6 bytes) 000840 1781227830.225159 trunc from 0x040000 to 0x035b3a (0xa4c5 bytes) 000841 1781227830.264343 write 0x033543 thru 0x03883f (0x52fd bytes) 000842 1781227830.275714 fallocate 0x02368d thru 0x028d3f (0x56b3 bytes) 000843 1781227830.285119 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38840): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38840): Mirror 12: 0x5473dc0c (stale) Mirror 13: 0x4d298544 (stale) CRC-32 checksum value for chunk [0, 0x38840): Mirror 1: 0x43573565 Mirror 14: 0x43573565 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38840: skipped 000844 1781227831.685541 trunc from 0x038840 to 0x03aadd (0x229e bytes) 000845 1781227831.745077 write 0x03657b thru 0x03db1d (0x75a3 bytes) 000846 1781227831.757220 trunc from 0x0038f8 to 0x00a57e (0x6c87 bytes) 000847 1781227831.765094 trunc from 0x01b71e to 0x025d26 (0xa609 bytes) 000848 1781227831.772550 punch from 0x02dc0b to 0x02fcb3 (0x20a9 bytes) 000849 1781227831.782481 write 0x038fd4 thru 0x03ffff (0x702c bytes) 000850 1781227831.786209 write 0x026d34 thru 0x0311a9 (0xa476 bytes) 000851 1781227831.804835 punch from 0x03504d to 0x03e831 (0x97e5 bytes) 000852 1781227831.839015 mapread 0x03cc09 thru 0x03ffff (0x33f7 bytes) 000855 1781227831.873101 fallocate 0x01bdd4 thru 0x020c18 (0x4e45 bytes) 000856 1781227831.881504 write 0x00e193 thru 0x019057 (0xaec5 bytes) 000857 1781227831.923327 trunc from 0x040000 to 0x03855d (0x7aa2 bytes) 000858 1781227831.966078 trunc from 0x031582 to 0x03855d (0x6fdc bytes) 000859 1781227831.975391 mapwrite 0x02ec54 thru 0x03974e (0xaafb bytes) 000860 1781227832.063636 mapwrite 0x0242c6 thru 0x02692d (0x2668 bytes) 000861 1781227832.116559 mapread 0x007a5b thru 0x00b800 (0x3da6 bytes) 000862 1781227832.126170 trunc from 0x03974f to 0x013d64 (0x259ea bytes) 000863 1781227832.156517 mapwrite 0x02b836 thru 0x02bdfc (0x5c7 bytes) 000864 1781227832.207378 trunc from 0x02bdfd to 0x033715 (0x7919 bytes) 000865 1781227832.225389 trunc from 0x033716 to 0x033bea (0x4d5 bytes) 000866 1781227832.246722 mapwrite 0x01bd33 thru 0x01db09 (0x1dd7 bytes) 000867 1781227832.273320 read 0x0067d5 thru 0x013056 (0xc882 bytes) 000868 1781227832.286270 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33beb): Mirror 13: 0x4171ee0f (stale) Mirror 14: 0x9560d9e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33beb): Mirror 13: 0x4171ee0f (stale) Mirror 14: 0x9560d9e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33beb: skipped 000869 1781227832.755514 trunc from 0x033beb to 0x005137 (0x2eab3 bytes) 000870 1781227832.786180 trunc from 0x005138 to 0x0348f1 (0x2f7ba bytes) 000872 1781227832.813129 fallocate 0x01b4f8 thru 0x01cbdf (0x16e8 bytes) 000873 1781227832.825258 punch from 0x00a6d1 to 0x00b320 (0xc50 bytes) 000874 1781227832.835484 trunc from 0x0348f2 to 0x0335d0 (0x1321 bytes) 000875 1781227832.859467 trunc from 0x0335d1 to 0x000f9e (0x32632 bytes) 000877 1781227832.891545 trunc from 0x000d45 to 0x000f9e (0x25a bytes) 000878 1781227832.906527 write 0x03222c thru 0x03ae3d (0x8c12 bytes) 000879 1781227832.912387 mapread 0x00e4d4 thru 0x019395 (0xaec2 bytes) 000880 1781227832.967811 write 0x000c5c thru 0x006894 (0x5c39 bytes) 000882 1781227832.986870 read 0x0045ee thru 0x0123ea (0xddfd bytes) 000883 1781227832.993699 trunc from 0x03ae3e to 0x007a79 (0x333c4 bytes) 000884 1781227833.033402 punch from 0x0039a3 to 0x007a79 (0x40d7 bytes) 000885 1781227833.068585 trunc from 0x0029a9 to 0x007a79 (0x50d1 bytes) 000886 1781227833.103968 MIRROR_RESYNC 3 mirrors 000887 1781227834.287064 trunc from 0x007a7a to 0x01e383 (0x1690a bytes) 000888 1781227834.337057 mapwrite 0x02885a thru 0x02a00b (0x17b2 bytes) 000889 1781227834.410155 mapread 0x01d44f thru 0x020743 (0x32f5 bytes) 000890 1781227834.422215 trunc from 0x004f33 to 0x00a3b1 (0x547f bytes) 000893 1781227834.429682 punch from 0x019fe1 to 0x027ae6 (0xdb06 bytes) 000894 1781227834.435167 trunc from 0x02552c to 0x02a00b (0x4ae0 bytes) 000895 1781227834.443071 punch from 0x018e69 to 0x027225 (0xe3bd bytes) 000896 1781227834.448432 trunc from 0x02a00c to 0x02f4c3 (0x54b8 bytes) 000898 1781227834.467511 punch from 0x00d5dd to 0x0153e7 (0x7e0b bytes) 000899 1781227834.473921 trunc from 0x00b462 to 0x015eee (0xaa8d bytes) 000900 1781227834.479028 read 0x0168de thru 0x024f4b (0xe66e bytes) 000901 1781227834.499826 trunc from 0x017fc8 to 0x021d8f (0x9dc8 bytes) 000902 1781227834.510711 write 0x028fa1 thru 0x02aebb (0x1f1b bytes) 000903 1781227834.543923 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f4c4): Mirror 13: 0xad1304d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f4c4): Mirror 13: 0xad1304d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f4c4: skipped 000904 1781227835.010549 fallocate 0x00aec4 thru 0x00d614 (0x2751 bytes) 000905 1781227835.021780 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x190:0x0]: Invalid argument (22) 000906 1781227835.259436 mapread 0x004019 thru 0x013da4 (0xfd8c bytes) 000907 1781227835.298416 trunc from 0x02f4c4 to 0x0367c9 (0x7306 bytes) 000908 1781227835.323288 write 0x020dad thru 0x02a8d9 (0x9b2d bytes) 000909 1781227835.346074 mapread 0x01f883 thru 0x02ab72 (0xb2f0 bytes) 000910 1781227835.347864 write 0x00127b thru 0x009856 (0x85dc bytes) 000911 1781227835.350996 trunc from 0x0367ca to 0x01d250 (0x19579 bytes) 000912 1781227835.374267 fallocate 0x03d4a5 thru 0x03ffff (0x2b5b bytes) 000913 1781227835.381697 punch from 0x02d4d4 to 0x0329f8 (0x5525 bytes) 000914 1781227835.387400 punch from 0x0166ba to 0x01d7f4 (0x713b bytes) 000915 1781227835.394011 mapwrite 0x01ce27 thru 0x0281d9 (0xb3b3 bytes) 000916 1781227835.452502 mapread 0x03b0cf thru 0x03ffff (0x4f31 bytes) 000917 1781227835.477261 read 0x01a611 thru 0x02857c (0xdf6c bytes) 000918 1781227835.480444 write 0x0308a5 thru 0x03a778 (0x9ed4 bytes) 000920 1781227835.503288 mapread 0x00b98c thru 0x00cf85 (0x15fa bytes) 000921 1781227835.506186 read 0x027951 thru 0x033e89 (0xc539 bytes) 000922 1781227835.508191 write 0x014708 thru 0x022dd0 (0xe6c9 bytes) 000923 1781227835.512399 fallocate 0x01ce19 thru 0x01dcf0 (0xed8 bytes) 000924 1781227835.522389 punch from 0x020b9c to 0x024686 (0x3aeb bytes) 000925 1781227835.547509 fallocate 0x0065ef thru 0x00ad4e (0x4760 bytes) 000926 1781227835.555343 punch from 0x008be7 to 0x017cfb (0xf115 bytes) 000927 1781227835.568497 write 0x03d7a1 thru 0x03ffff (0x285f bytes) 000928 1781227835.573982 read 0x01091b thru 0x01cca5 (0xc38b bytes) 000929 1781227835.600387 write 0x021f3a thru 0x02334e (0x1415 bytes) 000930 1781227835.608868 write 0x017b69 thru 0x01cd19 (0x51b1 bytes) 000931 1781227835.617514 write 0x03be7b thru 0x03ffff (0x4185 bytes) 000932 1781227835.627918 mapwrite 0x01220a thru 0x016205 (0x3ffc bytes) 000934 1781227835.669174 mapread 0x037d48 thru 0x03ffff (0x82b8 bytes) 000935 1781227835.671097 fallocate 0x023d14 thru 0x024d4f (0x103c bytes) 000936 1781227835.683073 mapwrite 0x003263 thru 0x005f1f (0x2cbd bytes) 000937 1781227835.703987 trunc from 0x040000 to 0x0181c6 (0x27e39 bytes) 000938 1781227835.724334 read 0x00d342 thru 0x0181c6 (0xae85 bytes) 000939 1781227835.725353 write 0x03d0c0 thru 0x03ffff (0x2f40 bytes) 000940 1781227835.727529 trunc from 0x008a43 to 0x0151f1 (0xc7af bytes) 000942 1781227835.732478 mapread 0x034d3f thru 0x03a9c6 (0x5c88 bytes) 000943 1781227835.766776 mapread 0x009230 thru 0x00b1c1 (0x1f92 bytes) 000944 1781227835.767980 trunc from 0x00dbe8 to 0x01ce3f (0xf258 bytes) 000945 1781227835.785878 fallocate 0x00a135 thru 0x00a297 (0x163 bytes) 000946 1781227835.800193 punch from 0x01754e to 0x025a69 (0xe51c bytes) 000947 1781227835.819912 mapread 0x006634 thru 0x01315a (0xcb27 bytes) 000948 1781227835.879252 trunc from 0x040000 to 0x02b389 (0x14c76 bytes) 000950 1781227835.903345 read 0x0035a3 thru 0x006e5a (0x38b8 bytes) 000951 1781227835.904674 mapwrite 0x02cff3 thru 0x039535 (0xc543 bytes) 000953 1781227835.971463 trunc from 0x016dc3 to 0x026057 (0xf295 bytes) 000954 1781227835.978363 trunc from 0x00ea84 to 0x00ef9c (0x519 bytes) 000955 1781227835.984345 mapwrite 0x00f0e0 thru 0x010e4e (0x1d6f bytes) 000956 1781227835.999545 fallocate 0x017d7f thru 0x023e97 (0xc119 bytes) 000958 1781227836.015011 mapread 0x030691 thru 0x031ec6 (0x1836 bytes) 000959 1781227836.017249 fallocate 0x017b22 thru 0x01d627 (0x5b06 bytes) 000960 1781227836.024071 punch from 0x035f72 to 0x039535 (0x35c4 bytes) 000961 1781227836.031929 fallocate 0x00f2a3 thru 0x01c6b2 (0xd410 bytes) 000962 1781227836.037519 fallocate 0x02f4f3 thru 0x03d14f (0xdc5d bytes) 000963 1781227836.044151 mapread 0x03578f thru 0x039535 (0x3da7 bytes) 000964 1781227836.072969 fallocate 0x0368ae thru 0x03ffff (0x9752 bytes) 000965 1781227836.078877 punch from 0x000760 to 0x005e27 (0x56c8 bytes) 000966 1781227836.085390 mapread 0x03a6f7 thru 0x03ffff (0x5909 bytes) 000967 1781227836.096224 trunc from 0x0113bf to 0x01524c (0x3e8e bytes) 000968 1781227836.103195 read 0x028638 thru 0x02ccfd (0x46c6 bytes) 000969 1781227836.105001 fallocate 0x02666e thru 0x03234a (0xbcdd bytes) 000971 1781227836.111381 punch from 0x01c902 to 0x029127 (0xc826 bytes) 000972 1781227836.117969 read 0x017466 thru 0x01bf37 (0x4ad2 bytes) 000975 1781227836.134035 read 0x02e75f thru 0x0345f8 (0x5e9a bytes) 000976 1781227836.135017 fallocate 0x028928 thru 0x0361b3 (0xd88c bytes) 000977 1781227836.141888 write 0x021c52 thru 0x02538e (0x373d bytes) 000978 1781227836.143992 fallocate 0x00cc80 thru 0x012ecc (0x624d bytes) 000979 1781227836.150282 mapwrite 0x02edea thru 0x039a0b (0xac22 bytes) 000980 1781227836.174854 punch from 0x02cc23 to 0x035539 (0x8917 bytes) 000981 1781227836.181478 write 0x02dddd thru 0x0324e3 (0x4707 bytes) 000982 1781227836.196551 trunc from 0x040000 to 0x02c66f (0x13990 bytes) 000984 1781227836.221060 mapwrite 0x0291d8 thru 0x038745 (0xf56e bytes) 000985 1781227836.352010 mapread 0x0246e1 thru 0x030007 (0xb927 bytes) 000986 1781227836.355247 write 0x006bb4 thru 0x0107bd (0x9c0a bytes) 000987 1781227836.358134 mapread 0x036d9d thru 0x038745 (0x19a9 bytes) 000988 1781227836.375706 MIRROR_EXTEND to 2 mirrors 000989 1781227837.648431 mapwrite 0x02857d thru 0x036aad (0xe531 bytes) 000990 1781227837.833840 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38746): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38746): Mirror 2: 0x6b9329db (stale) CRC-32 checksum value for chunk [0, 0x38746): Mirror 1: 0x5c413d92 Mirror 3: 0x5c413d92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38746: skipped 000991 1781227839.678940 punch from 0x0328d8 to 0x038745 (0x5e6e bytes) 000992 1781227839.784778 write 0x03b1fd thru 0x03ffff (0x4e03 bytes) 000993 1781227839.803582 trunc from 0x040000 to 0x014f52 (0x2b0ad bytes) 000994 1781227839.838396 trunc from 0x011a15 to 0x014f52 (0x353e bytes) 000995 1781227839.848553 trunc from 0x004e34 to 0x00a1d9 (0x53a6 bytes) 000996 1781227839.858240 read 0x0023c8 thru 0x0039c1 (0x15fa bytes) 000997 1781227839.872546 mapread 0x00ea09 thru 0x010370 (0x1968 bytes) 000998 1781227839.914965 punch from 0x0067a6 to 0x01384d (0xd0a8 bytes) 000999 1781227839.931864 punch from 0x011ee2 to 0x014448 (0x2567 bytes) 001000 1781227839.947968 mapread 0x007a50 thru 0x0143ed (0xc99e bytes) All operations completed A-OK! PASS 70a (108s) == sanity-flr test 71: check mirror extend parent fid ==== 21:30:46 (1781227846) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 21:30:59 (1781227859) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28395 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201283 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2aa:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 21:31:10 (1781227870) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8852480 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4088064 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8113664 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8205120 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9023936 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9029696 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9217664 .. Waiting 61470 61471 61472 61476 61480 done 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:32:37 (1781227957) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3878592 .. Extending file size to 655744 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1547552 .. Extending file size to 6258176 .. Extending file size to 7624128 .. Extending file size to 7105408 .. Extending file size to 7954496 .. Extending file size to 8624768 .. Extending file size to 8703392 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8838144 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9079712 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9027200 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63114 63115 63116 63120 63123 done 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) 192.168.201.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:33:58 (1781228038) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2608 1268696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17772 3589248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25524 3581496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43296 7170744 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2608 1268696 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17772 3589248 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25524 3581496 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43296 7170744 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227998 s, 9.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187834 s, 11.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x7cfc0f2d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:34:20 (1781228060) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114048 s, 18.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 21:34:28 (1781228068) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c5:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:34:36 (1781228076) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140878 s, 14.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:34:47 (1781228087) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182654 s, 11.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:34:56 (1781228096) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116024 s, 18.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c9:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:35:31 (1781228131) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12032 s, 8.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:36:21 (1781228181) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14085 s, 14.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:37:21 (1781228241) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg127-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg127-server: Pool lustre.test_204f created Adding targets to pool oleg127-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg127-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13729 s, 15.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg127-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg127-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg127-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:38:41 (1781228321) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:38:52 (1781228332) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:39:03 (1781228343) oleg127-server: Pool lustre.test_206 created Adding targets to pool oleg127-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg127-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg127-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg127-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg127-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:39:33 (1781228373) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771764 s, 13.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570416 s, 18.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa637840a Mirror 3: 0xa637840a PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:39:45 (1781228385) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:39:49 (1781228389) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:39:53 (1781228393) Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0807757 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x11f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1470000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1470000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1470000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xcc0000 Read this (4 bytes): BBBB 192.168.201.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg127-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:40:34 (1781228434) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.030168 s, 272 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:40:43 (1781228443) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046479 s, 22.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f7 lmm_fid: [0x200000405:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f7 lmm_fid: [0x200000405:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d5:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611639 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 21:40:45.000000000 -0400 Modify: 2026-06-11 21:40:45.000000000 -0400 Change: 2026-06-11 21:40:47.000000000 -0400 Birth: 2026-06-11 21:40:46.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:40:53 (1781228453) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440112 s, 23.8 MB/s oleg127-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg127-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg127-server: O/280000401/d19/2f3: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:41:17 (1781228477) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.45164 s, 90.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.023842 s, 172 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 21:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1fc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 21:41 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:41:27 (1781228487) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:41:36 (1781228496) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2006 sec =========== 21:41:48 (1781228508) === sanity-flr: start cleanup 21:41:50 (1781228510) === === sanity-flr: finish cleanup 21:41:59 (1781228519) ===