-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 05:04:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg259-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 05:04:36 (1789117476) === oleg259-client.virtnet: executing check_config_client /mnt/lustre oleg259-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg259-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93ea11e02000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93ea11e02000.idle_timeout=debug disable quota as required oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' Setting lod.lustre-MDT0001-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanity-flr: finish setup 05:05:39 (1789117539) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8233] == sanity-flr test 0a: lfs mirror create with -N option == 05:05:44 (1789117544) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:06:05 (1789117565) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:06:09 (1789117569) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:06:13 (1789117573) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:06:40 (1789117600) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:06:44 (1789117604) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 05:06:49 (1789117609) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93ea11e02000.stats=clear osc.lustre-OST0001-osc-ffff93ea11e02000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:06:58 (1789117618) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:07:11 (1789117631) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:07:22 (1789117642) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 05:07:26 (1789117646) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:07:36 (1789117656) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 05:07:49 (1789117669) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:07:57 (1789117677) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:08:06 (1789117686) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:08:15 (1789117695) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:08:27 (1789117707) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:08:34 (1789117714) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.619919 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107464 s, 9.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:08:44 (1789117724) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.286482 s, 29.3 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 05:08:53 (1789117733) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:09:04 (1789117744) Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93ea11e02000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93ea11e02000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93ea11e02000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93ea11e02000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:10:21 (1789117821) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.578055 s, 16.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93ea11e02000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93ea11e02000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:11:07 (1789117867) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93ea11e02000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93ea11e02000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93ea11e02000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93ea11e02000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:12:34 (1789117954) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.55292 s, 14.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 05:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.927s user 0m0.000s sys 0m0.171s Stopping /mnt/lustre-ost1 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93ea11e02000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93ea11e02000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.770s user 0m0.000s sys 0m0.204s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg259-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg259-server oleg259-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93ea11e02000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93ea11e02000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.181s user 0m0.002s sys 0m0.170s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg259-server: oleg259-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg259-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93ea11e02000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93ea11e02000.ost_server_uuid in FULL state after 0 sec PASS 33b (114s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:14:28 (1789118068) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:14:33 (1789118073) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:14:39 (1789118079) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 05:14:44 (1789118084) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 05:14:52 (1789118092) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.707344 s, 17.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:15:06 (1789118106) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541527 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542532 s, 19.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:15:15 (1789118115) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:15:22 (1789118122) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00172342 s, 594 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00326746 s, 313 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0019922 s, 514 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122158 s, 838 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 05:15:32 (1789118132) create test files with size 15730508 .. 1d913d156d8cab2fc87937140370b160 - 2e358581bf3cfc364e9a8c812af7ab61 - 40060fabfda5b7b14b430a9959c513e2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 05:15:59 (1789118159) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:16:16 (1789118176) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:16:25 (1789118185) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101756 s, 10.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb2:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb3:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499263 s, 21.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 05:16:35 (1789118195) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123489 s, 17.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.371815 s, 11.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246069 s, 8.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 05:16:54 (1789118214) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 05:16:57 (1789118217) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:17:01 (1789118221) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236338 s, 44.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 05:17:10 (1789118230) llite.lustre-ffff93ea14c93800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 05:17:19 (1789118239) llite.lustre-ffff93ea14c93800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416359 s, 25.2 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255319 s, 41.1 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 05:17:27 (1789118247) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.344336 s, 9.1 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:17:38 (1789118258) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:17:49 (1789118269) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.387314 s, 27.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x8:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:18:02 (1789118282) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff93ea029b4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.288548 s, 36.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg259-client.virtnet /mnt/lustre (opts:) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff93ea08e9d800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:18:19 (1789118299) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (51s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:19:10 (1789118350) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:19:51 (1789118391) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 05:19:55 (1789118395) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.227992 s, 13.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.138373 s, 22.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x3f4a5a6a (nosync,stale) Mirror 2: 0x3f4a5a6a (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0ddbc9e1ee3ae28146be62c9d545bd5c - mirror 1 checksum: a46fc45c4fd4827b20741744095138ad - mirror 2 checksum: 0ddbc9e1ee3ae28146be62c9d545bd5c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T05:19:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T05:19:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0ddbc9e1ee3ae28146be62c9d545bd5c - mirror 1 checksum: a46fc45c4fd4827b20741744095138ad - mirror 2 checksum: a46fc45c4fd4827b20741744095138ad - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T05:19:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T05:19:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:20:06 (1789118406) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098346 s, 10.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (11s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:20:17 (1789118417) Failing mds1 on oleg259-server Stopping /mnt/lustre-mds1 (opts:) on oleg259-server 05:20:30 (1789118430) shut down facet: mds1 facet_host: oleg259-server facet_failover_host: oleg259-server Failover mds1 to oleg259-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 05:21:04 (1789118464) targets are mounted 05:21:04 (1789118464) facet_failover done oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463966 s, 22.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:21:23 (1789118483) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0422917 s, 484 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0289876 s, 707 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:21:34 (1789118494) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0964037 s, 212 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.012751 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:21:45 (1789118505) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0197339 s, 830 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:21:56 (1789118516) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.104898 s, 195 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (12s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:22:07 (1789118527) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0245827 s, 167 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0272765 s, 150 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:22:16 (1789118536) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:22:42 (1789118562) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:23:12 (1789118592) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:23:38 (1789118618) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:24:04 (1789118644) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 05:24:14 (1789118654) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 59125 mirror split pid 59126 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 59125 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 59126 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 05:25:23 (1789118723) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 8725 truncating to largest ever: 0x387ae 000001 1789118725.587757 trunc from 00000000 to 0x0387ad (0x387ae bytes) 000002 1789118725.608342 mapread 0x01938f thru 0x020690 (0x7302 bytes) 000003 1789118725.644214 punch from 0x02b39a to 0x02f3bb (0x4022 bytes) 000004 1789118725.650392 mapread 0x031b24 thru 0x0387ad (0x6c8a bytes) 000005 1789118725.674210 read 0x028b97 thru 0x037f57 (0xf3c1 bytes) 000006 1789118725.680393 fallocate 0x01a875 thru 0x027872 (0xcffe bytes) 000007 1789118725.703062 MIRROR_EXTEND to 2 mirrors 000008 1789118726.598181 fallocate 0x00199b thru 0x00ce6f (0xb4d5 bytes) 000009 1789118726.676257 MIRROR_RESYNC 2 mirrors 000011 1789118727.574411 trunc from 0x02bf70 to 0x02e9f7 (0x2a88 bytes) 000012 1789118727.687274 mapread 0x011861 thru 0x020075 (0xe815 bytes) 000013 1789118727.732261 mapwrite 0x00607e thru 0x01543d (0xf3c0 bytes) 000014 1789118727.784986 trunc from 0x02cd42 to 0x02d196 (0x455 bytes) 000015 1789118727.792898 write 0x039c85 thru 0x039f82 (0x2fe bytes) 000016 1789118727.795414 punch from 0x031b0c to 0x039f82 (0x8477 bytes) 000017 1789118727.832080 MIRROR_RESYNC 2 mirrors 000018 1789118728.504626 trunc from 0x014614 to 0x01f18e (0xab7b bytes) 000019 1789118728.560825 write 0x03a49c thru 0x03ffff (0x5b64 bytes) 000020 1789118728.562968 mapread 0x025bef thru 0x027407 (0x1819 bytes) 000021 1789118728.584728 fallocate 0x0265b8 thru 0x03400e (0xda57 bytes) 000022 1789118728.600005 trunc from 0x040000 to 0x013b5c (0x2c4a3 bytes) 000023 1789118728.624599 read 0x01087e thru 0x013b5c (0x32df bytes) 000025 1789118728.633372 mapread 0x00ce8d thru 0x013b5c (0x6cd0 bytes) 000026 1789118728.649502 trunc from 0x013b5d to 0x01781d (0x3cc1 bytes) 000027 1789118728.670398 write 0x029a4e thru 0x02eb90 (0x5143 bytes) 000028 1789118728.673328 write 0x00e912 thru 0x01b11b (0xc80a bytes) 000029 1789118728.679783 write 0x013175 thru 0x01ced3 (0x9d5f bytes) 000030 1789118728.687500 read 0x00609e thru 0x00d10f (0x7072 bytes) 000031 1789118728.688517 punch from 0x01f862 to 0x0233d0 (0x3b6f bytes) 000033 1789118728.698348 MIRROR_RESYNC 2 mirrors 000034 1789118729.204572 mapwrite 0x0025e2 thru 0x005f6e (0x398d bytes) 000035 1789118729.303992 trunc from 0x0048e4 to 0x00a6a6 (0x5dc3 bytes) 000036 1789118729.310363 mapread 0x019101 thru 0x01f8bc (0x67bc bytes) 000037 1789118729.322365 trunc from 0x006626 to 0x00f7d5 (0x91b0 bytes) 000038 1789118729.332916 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) fallocating to largest ever: 0x40000 000040 1789118729.472848 fallocate 0x0372c3 thru 0x03ffff (0x8d3d bytes) 000041 1789118729.486789 fallocate 0x03768a thru 0x03ffff (0x8976 bytes) 000043 1789118729.493847 mapread 0x02fccb thru 0x03a8e8 (0xac1e bytes) 000044 1789118729.524466 read 0x005f59 thru 0x00eb9e (0x8c46 bytes) 000045 1789118729.537793 punch from 0x011deb to 0x019304 (0x751a bytes) 000046 1789118729.544907 read 0x02d7c1 thru 0x035e85 (0x86c5 bytes) 000047 1789118729.545954 mapwrite 0x01ea8c thru 0x02cb3e (0xe0b3 bytes) 000049 1789118729.580723 mapread 0x018919 thru 0x027df8 (0xf4e0 bytes) 000051 1789118729.583030 read 0x00995c thru 0x012103 (0x87a8 bytes) 000053 1789118729.584285 fallocate 0x02d0a0 thru 0x0372d9 (0xa23a bytes) 000054 1789118729.590772 trunc from 0x040000 to 0x0261a6 (0x19e59 bytes) 000056 1789118729.646246 write 0x039b7e thru 0x03a164 (0x5e7 bytes) 000057 1789118729.648793 mapwrite 0x01e731 thru 0x0286e4 (0x9fb4 bytes) 000058 1789118729.687253 fallocate 0x024dce thru 0x03120b (0xc43e bytes) 000059 1789118729.695788 trunc from 0x028c56 to 0x02fd24 (0x70cf bytes) 000060 1789118729.707181 trunc from 0x03a165 to 0x02fb7b (0xa5e9 bytes) 000061 1789118729.731929 trunc from 0x02fb7c to 0x029366 (0x6815 bytes) 000062 1789118729.753392 read 0x02472d thru 0x029366 (0x4c3a bytes) 000063 1789118729.759818 write 0x03aacc thru 0x03ffff (0x5534 bytes) 000064 1789118729.761803 trunc from 0x03ad67 to 0x03ffff (0x5299 bytes) 000065 1789118729.775530 mapread 0x03824f thru 0x039695 (0x1447 bytes) 000067 1789118729.786171 write 0x01afbb thru 0x026b10 (0xbb56 bytes) 000068 1789118729.789266 mapread 0x02c90e thru 0x03c434 (0xfb27 bytes) 000070 1789118729.791321 read 0x005896 thru 0x00d505 (0x7c70 bytes) 000072 1789118729.792646 write 0x008612 thru 0x00e8e1 (0x62d0 bytes) 000073 1789118729.794829 trunc from 0x040000 to 0x0122b4 (0x2dd4b bytes) 000074 1789118729.812422 write 0x0109ef thru 0x016f77 (0x6589 bytes) 000075 1789118729.819932 MIRROR_EXTEND to 2 mirrors 000076 1789118730.447350 read 0x00026e thru 0x007e3b (0x7bce bytes) 000077 1789118730.459573 read 0x015a5e thru 0x016f77 (0x151a bytes) 000079 1789118730.480050 fallocate 0x03704d thru 0x03944a (0x23fe bytes) 000080 1789118730.588390 write 0x009e1d thru 0x00b841 (0x1a25 bytes) 000081 1789118730.661516 punch from 0x00b0dd to 0x01968e (0xe5b2 bytes) 000082 1789118730.675968 mapwrite 0x03a48e thru 0x03d400 (0x2f73 bytes) 000083 1789118730.792357 fallocate 0x021bca thru 0x0274ad (0x58e4 bytes) 000084 1789118730.809015 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d401): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d401): Mirror 2: 0x48df0036 (stale) CRC-32 checksum value for chunk [0, 0x3d401): Mirror 1: 0x70d8640e Mirror 3: 0x70d8640e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d401: skipped 000085 1789118731.980631 read 0x020b98 thru 0x02e631 (0xda9a bytes) 000086 1789118731.992642 fallocate 0x039642 thru 0x03cdb5 (0x3774 bytes) 000088 1789118732.035827 trunc from 0x036268 to 0x03c97d (0x6716 bytes) 000089 1789118732.043015 punch from 0x026da9 to 0x032ccc (0xbf24 bytes) 000090 1789118732.052706 trunc from 0x01ca76 to 0x02bf2d (0xf4b8 bytes) 000091 1789118732.059762 mapwrite 0x0078e5 thru 0x008f7e (0x169a bytes) 000092 1789118732.089054 mapwrite 0x039161 thru 0x039f05 (0xda5 bytes) 000093 1789118732.118325 write 0x03e1bb thru 0x03ffff (0x1e45 bytes) 000094 1789118732.126328 mapwrite 0x0350d0 thru 0x03ffff (0xaf30 bytes) 000095 1789118732.153886 fallocate 0x0031a1 thru 0x012934 (0xf794 bytes) 000096 1789118732.161558 punch from 0x002bfa to 0x00f500 (0xc907 bytes) 000097 1789118732.172836 MIRROR_RESYNC 3 mirrors 000098 1789118733.326255 MIRROR_EXTEND to 4 mirrors 000099 1789118734.169992 trunc from 0x040000 to 0x010857 (0x2f7a8 bytes) 000100 1789118734.250128 write 0x025aff thru 0x03047b (0xa97d bytes) 000101 1789118734.259830 write 0x01ecc1 thru 0x025e4c (0x718c bytes) 000102 1789118734.276013 fallocate 0x014d6e thru 0x01831c (0x35af bytes) 000103 1789118734.288017 mapread 0x008ac8 thru 0x0187da (0xfd13 bytes) 000104 1789118734.334355 trunc from 0x03047c to 0x017a97 (0x189e4 bytes) 000105 1789118734.360041 trunc from 0x002693 to 0x0042e0 (0x1c4e bytes) 000106 1789118734.369400 fallocate 0x014a91 thru 0x015d09 (0x1279 bytes) 000107 1789118734.375646 mapwrite 0x035ffd thru 0x03ffff (0xa003 bytes) 000108 1789118734.437278 punch from 0x01ddb6 to 0x0212c1 (0x350c bytes) 000109 1789118734.448055 trunc from 0x040000 to 0x029fc5 (0x1603a bytes) 000111 1789118734.471271 mapread 0x007a67 thru 0x015ef0 (0xe48a bytes) 000112 1789118734.473788 mapwrite 0x010761 thru 0x01ba7c (0xb31c bytes) 000114 1789118734.507765 write 0x0244e2 thru 0x0344b8 (0xffd7 bytes) 000115 1789118734.515645 fallocate 0x02d9a7 thru 0x02f0d5 (0x172f bytes) 000116 1789118734.521465 write 0x030eee thru 0x03b307 (0xa41a bytes) 000117 1789118734.524920 trunc from 0x03b308 to 0x007ee8 (0x3341f bytes) 000118 1789118734.551333 read 0x006d9f thru 0x00718b (0x3ed bytes) 000119 1789118734.552238 mapread 0x00308d thru 0x007ee8 (0x4e5c bytes) 000120 1789118734.560206 trunc from 0x007ee9 to 0x0283c2 (0x204da bytes) 000121 1789118734.580920 write 0x03d58e thru 0x03ffff (0x2a72 bytes) 000122 1789118734.587288 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb5155e83 (stale) Mirror 3: 0xb5155e83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb5155e83 (stale) Mirror 3: 0xb5155e83 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000123 1789118735.048952 mapwrite 0x0174c0 thru 0x01870f (0x1250 bytes) 000124 1789118735.072681 trunc from 0x040000 to 0x00ef5e (0x310a1 bytes) 000125 1789118735.098279 write 0x01708d thru 0x019da9 (0x2d1d bytes) 000128 1789118735.101726 write 0x00285f thru 0x00a1b6 (0x7958 bytes) 000129 1789118735.117109 read 0x006229 thru 0x014983 (0xe75b bytes) 000130 1789118735.150317 trunc from 0x019daa to 0x018ad5 (0x12d4 bytes) 000132 1789118735.180290 mapread 0x018273 thru 0x018ad5 (0x863 bytes) 000133 1789118735.181369 write 0x01620d thru 0x01e7cd (0x85c1 bytes) 000134 1789118735.190553 mapread 0x0142eb thru 0x01e7cd (0xa4e3 bytes) 000135 1789118735.199034 punch from 0x007354 to 0x00e36a (0x7017 bytes) 000136 1789118735.214025 mapwrite 0x01d115 thru 0x026b06 (0x99f2 bytes) 000137 1789118735.293062 mapread 0x013169 thru 0x016ad0 (0x3968 bytes) 000138 1789118735.294534 punch from 0x00f210 to 0x01d4cf (0xe2c0 bytes) 000139 1789118735.315299 trunc from 0x0038f4 to 0x012be7 (0xf2f4 bytes) 000140 1789118735.327518 write 0x01e989 thru 0x027872 (0x8eea bytes) 000141 1789118735.332324 mapread 0x02340e thru 0x027872 (0x4465 bytes) 000142 1789118735.334547 trunc from 0x015bc0 to 0x0202b5 (0xa6f6 bytes) 000143 1789118735.380236 fallocate 0x001ce3 thru 0x006fcf (0x52ed bytes) 000144 1789118735.403289 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27873): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27873): Mirror 2: 0x69f5e961 (stale) Mirror 3: 0x69f5e961 (stale) CRC-32 checksum value for chunk [0, 0x27873): Mirror 1: 0x5e7086f4 Mirror 4: 0x5e7086f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27873: skipped 000145 1789118736.473682 read 0x011c27 thru 0x017f9a (0x6374 bytes) 000146 1789118736.483510 write 0x0056f4 thru 0x01439b (0xeca8 bytes) 000147 1789118736.552772 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27873): Mirror 3: 0x69f5e961 (stale) Mirror 4: 0x5e7086f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27873): Mirror 3: 0x69f5e961 (stale) Mirror 4: 0x5e7086f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27873: skipped 000148 1789118736.984121 write 0x024d1f thru 0x0315db (0xc8bd bytes) 000150 1789118737.013232 read 0x016661 thru 0x02551f (0xeebf bytes) 000151 1789118737.047797 mapwrite 0x00d325 thru 0x00ef9b (0x1c77 bytes) 000152 1789118737.081318 trunc from 0x00b61a to 0x00d242 (0x1c29 bytes) 000153 1789118737.088260 fallocate 0x013aa7 thru 0x016fb3 (0x350d bytes) 000154 1789118737.098247 trunc from 0x0315dc to 0x0366d8 (0x50fd bytes) 000155 1789118737.168898 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x366d9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366d9): Mirror 3: 0x5819aa0a (stale) Mirror 4: 0xd61d3bc8 (stale) CRC-32 checksum value for chunk [0, 0x366d9): Mirror 1: 0x511ad485 Mirror 5: 0x511ad485 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366d9: skipped 000156 1789118738.387674 fallocate 0x0158b7 thru 0x0207bd (0xaf07 bytes) 000157 1789118738.432662 trunc from 0x024801 to 0x02c20b (0x7a0b bytes) 000158 1789118738.439830 mapwrite 0x000c87 thru 0x00305e (0x23d8 bytes) 000159 1789118738.488266 punch from 0x009abf to 0x0182bd (0xe7ff bytes) 000160 1789118738.496625 punch from 0x0161a0 to 0x017672 (0x14d3 bytes) 000161 1789118738.502214 mapread 0x0185d8 thru 0x01d301 (0x4d2a bytes) 000162 1789118738.514726 punch from 0x031b76 to 0x0366d8 (0x4b63 bytes) 000163 1789118738.520494 fallocate 0x01f715 thru 0x02278f (0x307b bytes) 000164 1789118738.526236 punch from 0x005362 to 0x012eb3 (0xdb52 bytes) 000165 1789118738.535480 trunc from 0x0319dc to 0x0366d8 (0x4cfd bytes) 000167 1789118738.547833 fallocate 0x0374b1 thru 0x03ffff (0x8b4f bytes) 000168 1789118738.563452 trunc from 0x040000 to 0x019ac9 (0x26536 bytes) 000169 1789118738.626644 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19aca): Mirror 3: 0x3407b039 (stale) Mirror 5: 0xbd82621a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19aca): Mirror 3: 0x3407b039 (stale) Mirror 5: 0xbd82621a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19aca: skipped 000171 1789118738.935643 punch from 0x001c6e to 0x004ef9 (0x328c bytes) 000174 1789118738.943166 write 0x00e594 thru 0x00f7a1 (0x120e bytes) 000175 1789118738.957468 punch from 0x011086 to 0x019ac9 (0x8a44 bytes) 000176 1789118738.963253 punch from 0x012f3d to 0x019ac9 (0x6b8d bytes) 000177 1789118738.968655 fallocate 0x015204 thru 0x0187a6 (0x35a3 bytes) 000178 1789118738.977362 fallocate 0x0348a5 thru 0x036361 (0x1abd bytes) 000179 1789118738.986168 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36362): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36362): Mirror 3: 0xeed172a0 (stale) Mirror 5: 0x5740cf7e (stale) CRC-32 checksum value for chunk [0, 0x36362): Mirror 1: 0x6a42df47 Mirror 6: 0x6a42df47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36362: skipped 000180 1789118739.995498 mapwrite 0x01ea4c thru 0x02c375 (0xd92a bytes) 000182 1789118740.174231 fallocate 0x025c55 thru 0x027059 (0x1405 bytes) 000183 1789118740.186772 trunc from 0x00f699 to 0x010083 (0x9eb bytes) 000184 1789118740.196668 mapwrite 0x012f7a thru 0x021fdc (0xf063 bytes) 000185 1789118740.267712 fallocate 0x038fb4 thru 0x039fd1 (0x101e bytes) 000186 1789118740.273176 punch from 0x0034c0 to 0x005b63 (0x26a4 bytes) 000187 1789118740.281570 mapwrite 0x015957 thru 0x022b5c (0xd206 bytes) 000188 1789118740.312107 mapwrite 0x03d57e thru 0x03f945 (0x23c8 bytes) 000189 1789118740.400582 punch from 0x02bda7 to 0x034150 (0x83aa bytes) 000190 1789118740.413871 punch from 0x028318 to 0x033b16 (0xb7ff bytes) 000191 1789118740.424250 read 0x01903b thru 0x01de73 (0x4e39 bytes) 000193 1789118740.426694 read 0x03325b thru 0x03480f (0x15b5 bytes) 000194 1789118740.451938 mapwrite 0x0369f6 thru 0x038fd5 (0x25e0 bytes) 000195 1789118740.472199 trunc from 0x03f946 to 0x02828e (0x176b7 bytes) 000196 1789118740.489850 mapwrite 0x0156d4 thru 0x01e635 (0x8f62 bytes) 000197 1789118740.523592 punch from 0x003d62 to 0x009717 (0x59b6 bytes) 000198 1789118740.536177 MIRROR_RESYNC 4 mirrors 000199 1789118741.501130 fallocate 0x0238c6 thru 0x02de34 (0xa56f bytes) 000200 1789118741.551154 punch from 0x015366 to 0x023b1b (0xe7b6 bytes) 000201 1789118741.561688 MIRROR_RESYNC 4 mirrors 000202 1789118742.549879 trunc from 0x00d735 to 0x019d2b (0xc5f7 bytes) 000203 1789118742.621908 write 0x028e15 thru 0x02edfd (0x5fe9 bytes) 000204 1789118742.642418 read 0x01657f thru 0x01ff4d (0x99cf bytes) 000205 1789118742.657649 trunc from 0x006d61 to 0x01476e (0xda0e bytes) 000206 1789118742.667737 mapread 0x00ad41 thru 0x01443d (0x96fd bytes) 000207 1789118742.707786 read 0x013616 thru 0x023416 (0xfe01 bytes) 000208 1789118742.736379 write 0x032761 thru 0x03cba0 (0xa440 bytes) 000209 1789118742.741898 write 0x027695 thru 0x029c4d (0x25b9 bytes) 000210 1789118742.748042 trunc from 0x03cba1 to 0x01b7ce (0x213d2 bytes) 000211 1789118742.792319 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1b7cf): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b7cf): Mirror 3: 0x7d01f693 (stale) Mirror 5: 0x7d01f693 (stale) Mirror 6: 0x7d01f693 (stale) CRC-32 checksum value for chunk [0, 0x1b7cf): Mirror 1: 0x6285b150 Mirror 7: 0x6285b150 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b7cf: skipped 000212 1789118744.167292 MIRROR_RESYNC 5 mirrors 000213 1789118745.610785 read 0x00bf40 thru 0x01667a (0xa73b bytes) 000214 1789118745.626789 read 0x0049b4 thru 0x00c538 (0x7b85 bytes) 000215 1789118745.636188 trunc from 0x01b7cf to 0x00f625 (0xc1a9 bytes) 000216 1789118745.690506 read 0x0016aa thru 0x002387 (0xcde bytes) 000217 1789118745.711000 mapread 0x006777 thru 0x00ce25 (0x66af bytes) 000218 1789118745.716692 MIRROR_RESYNC 5 mirrors 000219 1789118747.193117 mapread 0x00ebdd thru 0x00f625 (0xa49 bytes) 000220 1789118747.199673 punch from 0x00bb92 to 0x00f625 (0x3a94 bytes) 000221 1789118747.229112 punch from 0x007aba to 0x00f625 (0x7b6c bytes) 000222 1789118747.233578 read 0x00cb95 thru 0x00f625 (0x2a91 bytes) 000223 1789118747.239015 write 0x03d692 thru 0x03ffff (0x296e bytes) 000224 1789118747.240611 read 0x001af4 thru 0x005dbc (0x42c9 bytes) 000225 1789118747.250569 punch from 0x024c52 to 0x031bc9 (0xcf78 bytes) 000226 1789118747.254893 mapread 0x03e359 thru 0x03ffff (0x1ca7 bytes) 000227 1789118747.255639 mapread 0x0248f2 thru 0x02670b (0x1e1a bytes) 000228 1789118747.266097 trunc from 0x040000 to 0x0338fe (0xc701 bytes) 000229 1789118747.290086 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x338ff): Mirror 3: 0xd6bf8553 (stale) Mirror 5: 0xd6bf8553 (stale) Mirror 6: 0xd6bf8553 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338ff): Mirror 3: 0xd6bf8553 (stale) Mirror 5: 0xd6bf8553 (stale) Mirror 6: 0xd6bf8553 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338ff: skipped 000230 1789118747.835569 read 0x026cf4 thru 0x02faec (0x8df9 bytes) 000231 1789118747.850265 punch from 0x0075ff to 0x014d3a (0xd73c bytes) 000233 1789118747.864898 write 0x00f26d thru 0x012247 (0x2fdb bytes) 000234 1789118747.927922 trunc from 0x0338ff to 0x01fedb (0x13a23 bytes) 000235 1789118747.963219 trunc from 0x01fedc to 0x00d45c (0x12a7f bytes) 000236 1789118748.019755 MIRROR_RESYNC 4 mirrors 000237 1789118749.037641 mapread 0x004f8c thru 0x00d45c (0x84d1 bytes) 000238 1789118749.087224 write 0x01b158 thru 0x02540f (0xa2b8 bytes) 000239 1789118749.198013 fallocate 0x012791 thru 0x01320d (0xa7d bytes) 000240 1789118749.207821 read 0x01525e thru 0x01ae2e (0x5bd1 bytes) 000241 1789118749.260390 punch from 0x00d8e5 to 0x00ef47 (0x1663 bytes) 000242 1789118749.273788 trunc from 0x01b874 to 0x02540f (0x9b9c bytes) 000243 1789118749.354249 punch from 0x011bd6 to 0x014d97 (0x31c2 bytes) 000244 1789118749.376051 trunc from 0x008596 to 0x00ea4f (0x64ba bytes) 000245 1789118749.410218 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25410): Mirror 3: 0x3d3d39ee (stale) Mirror 5: 0x3d3d39ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25410): Mirror 3: 0x3d3d39ee (stale) Mirror 5: 0x3d3d39ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25410: skipped 000246 1789118749.905604 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25410): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25410): Mirror 3: 0x3d3d39ee (stale) Mirror 5: 0x3d3d39ee (stale) CRC-32 checksum value for chunk [0, 0x25410): Mirror 1: 0x5db5564d Mirror 6: 0x5db5564d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25410: skipped 000247 1789118751.226922 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25410): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25410): Mirror 3: 0x3d3d39ee (stale) Mirror 5: 0x3d3d39ee (stale) CRC-32 checksum value for chunk [0, 0x25410): Mirror 1: 0x5db5564d Mirror 6: 0x5db5564d Mirror 7: 0x5db5564d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25410: skipped 000248 1789118752.484532 fallocate 0x027bca thru 0x030c3c (0x9073 bytes) 000249 1789118752.575797 fallocate 0x02498d thru 0x02d512 (0x8b86 bytes) 000250 1789118752.582653 trunc from 0x025410 to 0x00acc6 (0x1a749 bytes) 000251 1789118752.634957 mapread 0x006e81 thru 0x00acc6 (0x3e46 bytes) 000252 1789118752.719584 mapwrite 0x0375d5 thru 0x03ffff (0x8a2b bytes) 000253 1789118752.847997 mapwrite 0x010ca6 thru 0x01bb86 (0xaee1 bytes) 000254 1789118752.913243 trunc from 0x01c73d to 0x02ab92 (0xe456 bytes) 000255 1789118752.932109 punch from 0x026753 to 0x031502 (0xadb0 bytes) 000256 1789118752.940218 trunc from 0x040000 to 0x037f9f (0x8060 bytes) 000257 1789118752.976359 read 0x010a5e thru 0x0199cb (0x8f6e bytes) 000258 1789118752.977509 write 0x022297 thru 0x025a7c (0x37e6 bytes) 000259 1789118753.015878 read 0x00e067 thru 0x013817 (0x57b1 bytes) 000260 1789118753.017876 mapwrite 0x030492 thru 0x03194c (0x14bb bytes) 000261 1789118753.065080 mapwrite 0x038f33 thru 0x03ffff (0x70cd bytes) 000262 1789118753.203671 MIRROR_RESYNC 5 mirrors 000263 1789118755.339729 read 0x0389b8 thru 0x03ffff (0x7648 bytes) 000264 1789118755.356813 trunc from 0x0061f2 to 0x0151ce (0xefdd bytes) 000265 1789118755.408545 punch from 0x035525 to 0x03f3f0 (0x9ecc bytes) 000266 1789118755.414654 read 0x00e5d5 thru 0x0146cc (0x60f8 bytes) 000267 1789118755.445799 mapwrite 0x0004b9 thru 0x0056dd (0x5225 bytes) 000268 1789118755.468105 mapwrite 0x000966 thru 0x00419e (0x3839 bytes) 000269 1789118755.488551 punch from 0x018aa4 to 0x01e5e7 (0x5b44 bytes) 000270 1789118755.500117 MIRROR_RESYNC 5 mirrors 000271 1789118757.636535 read 0x011139 thru 0x0124f6 (0x13be bytes) 000272 1789118757.643407 write 0x01c680 thru 0x0284ec (0xbe6d bytes) 000273 1789118757.757978 mapwrite 0x03c61b thru 0x03ffff (0x39e5 bytes) 000274 1789118757.855662 mapread 0x013c0d thru 0x022d1c (0xf110 bytes) 000275 1789118757.879948 punch from 0x02b501 to 0x02ea81 (0x3581 bytes) 000276 1789118757.887594 punch from 0x02585d to 0x030280 (0xaa24 bytes) 000277 1789118757.894801 fallocate 0x0130d7 thru 0x01a81a (0x7744 bytes) 000278 1789118757.908840 trunc from 0x00ce39 to 0x016502 (0x96ca bytes) 000279 1789118757.933246 fallocate 0x028fb2 thru 0x02ba04 (0x2a53 bytes) 000280 1789118757.941423 read 0x039187 thru 0x03e94b (0x57c5 bytes) 000281 1789118757.960475 mapwrite 0x02321b thru 0x031342 (0xe128 bytes) 000282 1789118758.068355 write 0x03fb90 thru 0x03ffff (0x470 bytes) 000283 1789118758.070151 write 0x0043c0 thru 0x0089aa (0x45eb bytes) 000284 1789118758.072888 punch from 0x00b219 to 0x00bb9b (0x983 bytes) 000285 1789118758.082472 trunc from 0x040000 to 0x01181e (0x2e7e1 bytes) 000287 1789118758.149157 trunc from 0x01181f to 0x02bbb9 (0x1a39b bytes) 000288 1789118758.240007 mapread 0x018579 thru 0x01a38d (0x1e15 bytes) 000289 1789118758.305730 trunc from 0x0144ec to 0x01794d (0x3462 bytes) 000290 1789118758.325805 read 0x0124d5 thru 0x018465 (0x5f91 bytes) 000291 1789118758.346316 fallocate 0x003639 thru 0x00d746 (0xa10e bytes) 000292 1789118758.359561 MIRROR_RESYNC 5 mirrors 000293 1789118760.318763 trunc from 0x01e4fb to 0x022629 (0x412f bytes) 000294 1789118760.346541 mapwrite 0x0074f6 thru 0x014873 (0xd37e bytes) 000295 1789118760.395629 read 0x01aaf3 thru 0x021a3e (0x6f4c bytes) 000296 1789118760.404970 MIRROR_RESYNC 5 mirrors 000297 1789118761.748905 write 0x0242d5 thru 0x02f90f (0xb63b bytes) 000298 1789118761.819715 trunc from 0x02f910 to 0x00c1ff (0x23710 bytes) 000299 1789118761.855108 write 0x03717e thru 0x03e4ea (0x736d bytes) 000300 1789118761.857067 punch from 0x01dc49 to 0x021bd3 (0x3f8b bytes) 000301 1789118761.871282 trunc from 0x03e4eb to 0x006a31 (0x37ab9 bytes) 000302 1789118761.895975 trunc from 0x006a32 to 0x02bb7a (0x25149 bytes) 000303 1789118761.930609 read 0x018d41 thru 0x01c8d8 (0x3b98 bytes) 000304 1789118761.951365 write 0x033ea4 thru 0x03ecee (0xae4b bytes) 000305 1789118761.957332 punch from 0x007489 to 0x00aa10 (0x3588 bytes) 000306 1789118761.979680 write 0x018b7b thru 0x019b89 (0x100f bytes) 000307 1789118761.983626 read 0x01f11d thru 0x02cb55 (0xda39 bytes) 000308 1789118762.009933 read 0x011769 thru 0x0135e6 (0x1e7e bytes) 000309 1789118762.010896 punch from 0x02d0db to 0x034e51 (0x7d77 bytes) 000310 1789118762.043318 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ecef): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecef): Mirror 3: 0x42d11ada (stale) Mirror 5: 0x42d11ada (stale) Mirror 6: 0x42d11ada (stale) Mirror 7: 0x42d11ada (stale) CRC-32 checksum value for chunk [0, 0x3ecef): Mirror 1: 0x20d1107c Mirror 8: 0x20d1107c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecef: skipped 000311 1789118763.276893 punch from 0x0007f7 to 0x007e9d (0x76a7 bytes) 000312 1789118763.315963 read 0x02aef4 thru 0x02e784 (0x3891 bytes) 000313 1789118763.348835 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ecef): Mirror 3: 0x42d11ada (stale) Mirror 5: 0x42d11ada (stale) Mirror 7: 0x42d11ada (stale) Mirror 8: 0x20d1107c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecef): Mirror 3: 0x42d11ada (stale) Mirror 5: 0x42d11ada (stale) Mirror 7: 0x42d11ada (stale) Mirror 8: 0x20d1107c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecef: skipped 000315 1789118763.998298 trunc from 0x03ecef to 0x02ef21 (0xfdcd bytes) 000316 1789118764.036734 fallocate 0x02047e thru 0x02533f (0x4ec2 bytes) 000318 1789118764.063748 punch from 0x02ac04 to 0x02ef21 (0x431e bytes) 000319 1789118764.067944 punch from 0x01de3f to 0x029a7d (0xbc3f bytes) 000320 1789118764.072271 trunc from 0x02ef22 to 0x02b986 (0x359b bytes) 000323 1789118764.115255 mapread 0x010d46 thru 0x015046 (0x4301 bytes) 000324 1789118764.150759 punch from 0x02a8dd to 0x02b986 (0x10aa bytes) 000325 1789118764.162394 write 0x004c32 thru 0x007153 (0x2522 bytes) 000326 1789118764.183259 mapwrite 0x0107b0 thru 0x01e95b (0xe1ac bytes) 000327 1789118764.218168 fallocate 0x03c99b thru 0x03ffff (0x3665 bytes) 000328 1789118764.223704 trunc from 0x00a29c to 0x01771d (0xd482 bytes) 000329 1789118764.229515 mapwrite 0x02bdaf thru 0x03bdaa (0xfffc bytes) 000330 1789118764.282054 trunc from 0x0028d0 to 0x00fdcf (0xd500 bytes) 000331 1789118764.288363 mapread 0x035e29 thru 0x03bdaa (0x5f82 bytes) 000332 1789118764.289377 write 0x014a60 thru 0x01fbc7 (0xb168 bytes) 000333 1789118764.296668 read 0x027b95 thru 0x032412 (0xa87e bytes) 000334 1789118764.297382 mapread 0x01f16f thru 0x02e2ae (0xf140 bytes) 000335 1789118764.299644 trunc from 0x02be38 to 0x02d0c3 (0x128c bytes) 000336 1789118764.305538 mapread 0x01b658 thru 0x029b83 (0xe52c bytes) 000338 1789118764.307053 write 0x010aa1 thru 0x0195d4 (0x8b34 bytes) 000340 1789118764.314413 fallocate 0x000272 thru 0x006289 (0x6018 bytes) 000341 1789118764.319710 read 0x018998 thru 0x01b132 (0x279b bytes) 000342 1789118764.320354 punch from 0x00e812 to 0x01e673 (0xfe62 bytes) 000344 1789118764.335311 read 0x003096 thru 0x01064c (0xd5b7 bytes) 000345 1789118764.352100 trunc from 0x03bdab to 0x02f09d (0xcd0d bytes) 000346 1789118764.369347 mapwrite 0x036113 thru 0x036f97 (0xe85 bytes) 000347 1789118764.414581 trunc from 0x036f98 to 0x025e04 (0x11193 bytes) 000348 1789118764.444985 trunc from 0x025e05 to 0x0025af (0x23855 bytes) 000349 1789118764.474927 trunc from 0x0025b0 to 0x01fef9 (0x1d94a bytes) 000350 1789118764.518539 read 0x00f2fe thru 0x01ec89 (0xf98c bytes) 000352 1789118764.552323 mapwrite 0x035d94 thru 0x035e36 (0xa3 bytes) 000353 1789118764.655065 mapread 0x01fa72 thru 0x021971 (0x1f00 bytes) 000354 1789118764.660975 punch from 0x019f6a to 0x01e1ab (0x4242 bytes) 000356 1789118764.686560 mapwrite 0x005ea1 thru 0x009ac8 (0x3c28 bytes) 000357 1789118764.755075 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e37): Mirror 3: 0x34727e1b (stale) Mirror 5: 0x34727e1b (stale) Mirror 8: 0x1e30797c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e37): Mirror 3: 0x34727e1b (stale) Mirror 5: 0x34727e1b (stale) Mirror 8: 0x1e30797c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e37: skipped 000359 1789118765.158476 write 0x024886 thru 0x03249d (0xdc18 bytes) 000360 1789118765.174014 trunc from 0x035e37 to 0x022167 (0x13ccf bytes) 000361 1789118765.196960 trunc from 0x003cdf to 0x00a9ab (0x6ccd bytes) 000362 1789118765.212620 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22168): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22168): Mirror 3: 0xc53f7cee (stale) Mirror 5: 0xc53f7cee (stale) Mirror 8: 0x636234d5 (stale) CRC-32 checksum value for chunk [0, 0x22168): Mirror 1: 0x1108645 Mirror 9: 0x1108645 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22168: skipped 000363 1789118766.653876 mapread 0x01af22 thru 0x01b3db (0x4ba bytes) 000364 1789118766.675907 punch from 0x01cccb to 0x01fc03 (0x2f39 bytes) 000365 1789118766.774089 read 0x00d304 thru 0x017ee1 (0xabde bytes) 000366 1789118766.790700 trunc from 0x020016 to 0x022167 (0x2152 bytes) 000367 1789118766.803186 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22168): Mirror 3: 0xc53f7cee (stale) Mirror 5: 0xc53f7cee (stale) Mirror 8: 0x636234d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22168): Mirror 3: 0xc53f7cee (stale) Mirror 5: 0xc53f7cee (stale) Mirror 8: 0x636234d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22168: skipped 000368 1789118767.201816 trunc from 0x020894 to 0x022167 (0x18d4 bytes) 000369 1789118767.208092 write 0x02b85e thru 0x02d797 (0x1f3a bytes) 000370 1789118767.214199 trunc from 0x01fdba to 0x01fe0d (0x54 bytes) 000372 1789118767.221543 trunc from 0x003d2a to 0x010f57 (0xd22e bytes) 000373 1789118767.227318 punch from 0x02c573 to 0x02d797 (0x1225 bytes) 000374 1789118767.259041 MIRROR_RESYNC 4 mirrors 000375 1789118768.565800 MIRROR_RESYNC 4 mirrors 000376 1789118769.097204 read 0x0125fb thru 0x015bf4 (0x35fa bytes) 000377 1789118769.103806 read 0x00504d thru 0x00b771 (0x6725 bytes) 000378 1789118769.113310 read 0x01c01e thru 0x028360 (0xc343 bytes) 000379 1789118769.113803 mapwrite 0x03f8d8 thru 0x03ffff (0x728 bytes) 000380 1789118769.189266 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6d3d8072 (stale) Mirror 8: 0x6d3d8072 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6d3d8072 (stale) Mirror 8: 0x6d3d8072 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000381 1789118769.712754 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6d3d8072 (stale) Mirror 8: 0x6d3d8072 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3828ab5f Mirror 9: 0x3828ab5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1789118770.793783 trunc from 0x0099af to 0x014a96 (0xb0e8 bytes) 000383 1789118770.856355 mapwrite 0x02487c thru 0x02e5ad (0x9d32 bytes) 000384 1789118770.909583 write 0x03abfd thru 0x03dcd3 (0x30d7 bytes) 000385 1789118770.924063 fallocate 0x003f20 thru 0x005047 (0x1128 bytes) 000386 1789118770.929189 fallocate 0x028ed5 thru 0x02a65e (0x178a bytes) 000387 1789118770.933926 mapread 0x02a269 thru 0x033283 (0x901b bytes) 000388 1789118770.943685 punch from 0x00f290 to 0x016736 (0x74a7 bytes) 000389 1789118770.949785 write 0x03f265 thru 0x03ffff (0xd9b bytes) 000390 1789118770.956622 mapread 0x024994 thru 0x032b9b (0xe208 bytes) 000391 1789118770.984219 MIRROR_RESYNC 4 mirrors 000393 1789118772.594764 fallocate 0x01b0fd thru 0x02661c (0xb520 bytes) 000394 1789118772.646173 trunc from 0x040000 to 0x02c9d1 (0x1362e bytes) 000395 1789118772.672182 trunc from 0x02c9d2 to 0x00cc24 (0x1fdad bytes) 000396 1789118772.696621 read 0x007560 thru 0x00778b (0x22c bytes) 000397 1789118772.711250 MIRROR_RESYNC 4 mirrors 000398 1789118773.649719 trunc from 0x00700c to 0x00cc24 (0x5c19 bytes) 000399 1789118773.739364 read 0x003123 thru 0x00b7b0 (0x868e bytes) 000401 1789118773.762013 mapwrite 0x01f382 thru 0x02e68c (0xf30b bytes) 000402 1789118773.899594 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e68d): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e68d): Mirror 5: 0x2b43987d (stale) Mirror 8: 0x2b43987d (stale) Mirror 9: 0x2b43987d (stale) CRC-32 checksum value for chunk [0, 0x2e68d): Mirror 1: 0x77b49877 Mirror 10: 0x77b49877 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e68d: skipped 000403 1789118775.740131 trunc from 0x02e68d to 0x014fe1 (0x196ab bytes) 000404 1789118775.791532 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x14fe2): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14fe2): Mirror 5: 0x375417ed (stale) Mirror 8: 0x375417ed (stale) Mirror 9: 0x375417ed (stale) Mirror 10: 0x375417ed (stale) CRC-32 checksum value for chunk [0, 0x14fe2): Mirror 1: 0x375417ed Mirror 11: 0x375417ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14fe2: skipped 000405 1789118777.027680 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14fe2): Mirror 8: 0x375417ed (stale) Mirror 9: 0x375417ed (stale) Mirror 10: 0x375417ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14fe2): Mirror 8: 0x375417ed (stale) Mirror 9: 0x375417ed (stale) Mirror 10: 0x375417ed (stale) CRC-32 checksum value for chunk [0, 0x14fe2): Mirror 1: 0x375417ed Mirror 11: 0x375417ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14fe2: skipped 000406 1789118777.814279 read 0x014ca5 thru 0x014fe1 (0x33d bytes) 000407 1789118777.819995 trunc from 0x006632 to 0x00eb52 (0x8521 bytes) 000408 1789118777.891015 read 0x00e20d thru 0x014fe1 (0x6dd5 bytes) 000409 1789118777.905094 fallocate 0x0304cb thru 0x034ec9 (0x49ff bytes) 000410 1789118777.911937 read 0x00fdc3 thru 0x010b03 (0xd41 bytes) 000411 1789118777.912914 trunc from 0x00761d to 0x014fe1 (0xd9c5 bytes) 000413 1789118777.920076 mapwrite 0x00188f thru 0x003473 (0x1be5 bytes) 000414 1789118777.940167 fallocate 0x03bfcc thru 0x03ffff (0x4034 bytes) 000415 1789118777.957254 trunc from 0x0038df to 0x00ff92 (0xc6b4 bytes) 000416 1789118777.975832 MIRROR_RESYNC 5 mirrors 000417 1789118779.185224 trunc from 0x00f1dd to 0x014fe1 (0x5e05 bytes) 000418 1789118779.260749 trunc from 0x008d21 to 0x014fe1 (0xc2c1 bytes) 000419 1789118779.272153 trunc from 0x014fe2 to 0x034ad8 (0x1faf7 bytes) 000420 1789118779.330586 mapwrite 0x00d625 thru 0x019d36 (0xc712 bytes) 000421 1789118779.504960 fallocate 0x03528c thru 0x0388cd (0x3642 bytes) 000422 1789118779.530114 fallocate 0x007189 thru 0x008ac4 (0x193c bytes) 000423 1789118779.560245 read 0x024ebe thru 0x02f57f (0xa6c2 bytes) 000424 1789118779.604206 fallocate 0x01df83 thru 0x022ebe (0x4f3c bytes) 000425 1789118779.654277 mapread 0x0367ee thru 0x037963 (0x1176 bytes) 000426 1789118779.681613 write 0x015884 thru 0x0160e2 (0x85f bytes) 000427 1789118779.684670 fallocate 0x037c06 thru 0x038927 (0xd22 bytes) 000428 1789118779.696148 trunc from 0x038928 to 0x031afd (0x6e2a bytes) 000429 1789118779.717486 punch from 0x00031e to 0x009632 (0x9315 bytes) 000430 1789118779.728012 mapwrite 0x000302 thru 0x001dd6 (0x1ad5 bytes) 000431 1789118779.755226 mapread 0x020a55 thru 0x022203 (0x17af bytes) 000432 1789118779.759997 trunc from 0x031afe to 0x03c8df (0xade2 bytes) 000433 1789118779.787313 read 0x03b93c thru 0x03c8df (0xfa4 bytes) 000434 1789118779.802123 trunc from 0x03c8e0 to 0x01e7f2 (0x1e0ed bytes) 000437 1789118779.815897 trunc from 0x00e6f2 to 0x01bbc1 (0xd4d0 bytes) 000438 1789118779.820574 write 0x03399d thru 0x03d877 (0x9edb bytes) 000439 1789118779.822322 fallocate 0x03aaa8 thru 0x03ffff (0x5558 bytes) 000440 1789118779.828934 fallocate 0x01bb68 thru 0x029ba4 (0xe03d bytes) 000441 1789118779.834234 mapwrite 0x01c3ff thru 0x01d18d (0xd8f bytes) 000442 1789118779.847348 write 0x01910d thru 0x01e53d (0x5431 bytes) 000443 1789118779.857544 MIRROR_RESYNC 5 mirrors 000444 1789118781.066893 mapwrite 0x039a1c thru 0x03ffff (0x65e4 bytes) 000445 1789118781.189228 trunc from 0x015825 to 0x01af0b (0x56e7 bytes) 000446 1789118781.202359 write 0x0315c5 thru 0x03ffff (0xea3b bytes) 000447 1789118781.210882 mapread 0x002ae3 thru 0x010114 (0xd632 bytes) 000448 1789118781.233772 read 0x00a333 thru 0x00b2a5 (0xf73 bytes) 000449 1789118781.235845 trunc from 0x040000 to 0x02814d (0x17eb2 bytes) 000450 1789118781.289352 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2814e): Mirror 8: 0xfc3e53e9 (stale) Mirror 10: 0xfc3e53e9 (stale) Mirror 11: 0xfc3e53e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2814e): Mirror 8: 0xfc3e53e9 (stale) Mirror 10: 0xfc3e53e9 (stale) Mirror 11: 0xfc3e53e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2814e: skipped 000451 1789118781.906558 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2814e): Mirror 8: 0xfc3e53e9 (stale) Mirror 10: 0xfc3e53e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2814e): Mirror 8: 0xfc3e53e9 (stale) Mirror 10: 0xfc3e53e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2814e: skipped 000452 1789118782.195207 mapwrite 0x002336 thru 0x010913 (0xe5de bytes) 000453 1789118782.243876 trunc from 0x02814e to 0x030809 (0x86bc bytes) 000454 1789118782.269304 trunc from 0x03080a to 0x01af8f (0x1587a bytes) 000455 1789118782.290033 mapread 0x00bf59 thru 0x01250f (0x65b7 bytes) 000456 1789118782.298999 read 0x000e22 thru 0x006c5a (0x5e39 bytes) 000457 1789118782.304601 mapwrite 0x01fcb3 thru 0x02a526 (0xa874 bytes) 000458 1789118782.346878 write 0x02fb25 thru 0x030fba (0x1496 bytes) 000459 1789118782.348593 trunc from 0x030fbb to 0x013df0 (0x1d1ca bytes) 000461 1789118782.363820 mapwrite 0x0056d8 thru 0x01034b (0xac74 bytes) 000462 1789118782.391889 trunc from 0x002d89 to 0x0080bd (0x5335 bytes) 000463 1789118782.401002 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x13df1): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13df1): Mirror 8: 0x1c2d1511 (stale) Mirror 10: 0x1c2d1511 (stale) CRC-32 checksum value for chunk [0, 0x13df1): Mirror 1: 0x3cc3929b Mirror 11: 0x3cc3929b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13df1: skipped 000464 1789118783.045187 write 0x000551 thru 0x00ff7e (0xfa2e bytes) 000465 1789118783.125821 mapwrite 0x0355b8 thru 0x03ffff (0xaa48 bytes) 000466 1789118783.321874 MIRROR_RESYNC 4 mirrors 000467 1789118784.948891 read 0x000a3c thru 0x010263 (0xf828 bytes) 000468 1789118784.961882 mapread 0x00cc00 thru 0x017380 (0xa781 bytes) 000470 1789118784.984996 mapread 0x03f3dd thru 0x03ffff (0xc23 bytes) 000471 1789118784.996308 mapwrite 0x00efa9 thru 0x01cada (0xdb32 bytes) 000472 1789118785.094950 write 0x011e1b thru 0x01210c (0x2f2 bytes) 000473 1789118785.106196 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xa59d73e4 (stale) Mirror 10: 0xa59d73e4 (stale) Mirror 11: 0xa59d73e4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x85e3691f Mirror 12: 0x85e3691f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000474 1789118786.772245 punch from 0x02a05c to 0x0357d2 (0xb777 bytes) 000476 1789118786.855179 mapwrite 0x0155f0 thru 0x02511c (0xfb2d bytes) 000477 1789118786.947261 mapread 0x0333a1 thru 0x03ffff (0xcc5f bytes) 000478 1789118786.972269 write 0x005bed thru 0x00bce6 (0x60fa bytes) 000479 1789118787.008967 trunc from 0x040000 to 0x0313ee (0xec11 bytes) 000482 1789118787.067717 trunc from 0x005f67 to 0x015611 (0xf6ab bytes) 000483 1789118787.108060 trunc from 0x00da9a to 0x0128ba (0x4e21 bytes) 000484 1789118787.117914 fallocate 0x02acda thru 0x038d86 (0xe0ad bytes) 000485 1789118787.129754 punch from 0x012bcf to 0x01a893 (0x7cc5 bytes) 000486 1789118787.152205 punch from 0x0278a8 to 0x0313ee (0x9b47 bytes) 000487 1789118787.169353 write 0x039d75 thru 0x03ffff (0x628b bytes) 000489 1789118787.175727 mapwrite 0x031f81 thru 0x03ffff (0xe07f bytes) 000490 1789118787.234945 read 0x027101 thru 0x0296db (0x25db bytes) 000491 1789118787.250170 trunc from 0x040000 to 0x01a85e (0x257a1 bytes) 000492 1789118787.296413 trunc from 0x01a85f to 0x00fc65 (0xabf9 bytes) 000493 1789118787.318756 trunc from 0x00fc66 to 0x0218d6 (0x11c71 bytes) 000494 1789118787.335848 read 0x020eef thru 0x020f7d (0x8f bytes) 000495 1789118787.364036 write 0x0118c7 thru 0x018ad0 (0x720a bytes) 000496 1789118787.366008 mapread 0x011c47 thru 0x0174a7 (0x5861 bytes) 000497 1789118787.369369 read 0x011b0c thru 0x0123e7 (0x8dc bytes) 000498 1789118787.376533 fallocate 0x035998 thru 0x03ffff (0xa668 bytes) 000499 1789118787.393835 MIRROR_RESYNC 5 mirrors 000500 1789118789.117261 read 0x002710 thru 0x0083c5 (0x5cb6 bytes) 000501 1789118789.129791 write 0x0169c7 thru 0x017438 (0xa72 bytes) 000502 1789118789.175974 fallocate 0x001498 thru 0x0100cf (0xec38 bytes) 000505 1789118789.183677 trunc from 0x0218d7 to 0x00d510 (0x143c6 bytes) 000506 1789118789.206323 trunc from 0x00d511 to 0x03ce58 (0x2f948 bytes) 000507 1789118789.228115 mapread 0x007d1e thru 0x00f3d5 (0x76b8 bytes) 000508 1789118789.247910 fallocate 0x01725d thru 0x023b14 (0xc8b8 bytes) 000509 1789118789.253766 trunc from 0x0008f2 to 0x007486 (0x6b95 bytes) 000510 1789118789.258521 write 0x006045 thru 0x00d8de (0x789a bytes) 000511 1789118789.265275 write 0x010507 thru 0x01cb3a (0xc634 bytes) 000512 1789118789.275742 mapread 0x000a43 thru 0x008d6b (0x8329 bytes) 000513 1789118789.299462 punch from 0x008c39 to 0x00f948 (0x6d10 bytes) 000514 1789118789.320836 read 0x00da45 thru 0x015853 (0x7e0f bytes) 000515 1789118789.334085 trunc from 0x01dc0b to 0x0265be (0x89b4 bytes) 000516 1789118789.342219 fallocate 0x01af61 thru 0x029cd1 (0xed71 bytes) 000517 1789118789.346907 mapread 0x018afe thru 0x01f8cd (0x6dd0 bytes) 000518 1789118789.362351 fallocate 0x024a3d thru 0x034091 (0xf655 bytes) 000519 1789118789.368179 mapwrite 0x02967e thru 0x038490 (0xee13 bytes) 000520 1789118789.389319 write 0x00fe00 thru 0x01b503 (0xb704 bytes) 000521 1789118789.392009 trunc from 0x03ce59 to 0x0265af (0x168a9 bytes) 000522 1789118789.409398 mapread 0x00eefe thru 0x010a51 (0x1b54 bytes) 000523 1789118789.410542 mapwrite 0x0326e6 thru 0x03e1ca (0xbae5 bytes) 000524 1789118789.505552 mapwrite 0x026aa6 thru 0x02da23 (0x6f7e bytes) 000525 1789118789.551346 mapwrite 0x01c73c thru 0x0214df (0x4da4 bytes) 000526 1789118789.580291 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e1cb): Mirror 8: 0xb2953ada (stale) Mirror 11: 0xb2953ada (stale) Mirror 12: 0xb2953ada (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1cb): Mirror 8: 0xb2953ada (stale) Mirror 11: 0xb2953ada (stale) Mirror 12: 0xb2953ada (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1cb: skipped 000527 1789118789.928145 trunc from 0x03e1cb to 0x01e48b (0x1fd3f bytes) 000528 1789118789.953198 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1e48c): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e48c): Mirror 8: 0x9521a4fd (stale) Mirror 11: 0x9521a4fd (stale) Mirror 12: 0x9521a4fd (stale) CRC-32 checksum value for chunk [0, 0x1e48c): Mirror 1: 0xdf7d5c32 Mirror 13: 0xdf7d5c32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e48c: skipped 000529 1789118790.767632 punch from 0x001d5b to 0x009144 (0x73ea bytes) 000530 1789118790.813226 fallocate 0x0067eb thru 0x012c76 (0xc48c bytes) 000531 1789118790.824384 trunc from 0x01e48c to 0x02e909 (0x1047e bytes) 000532 1789118790.886322 fallocate 0x01b0c3 thru 0x01c7e1 (0x171f bytes) 000533 1789118790.910712 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2e90a): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e90a): Mirror 8: 0x34d4e2e4 (stale) Mirror 11: 0x34d4e2e4 (stale) Mirror 12: 0x34d4e2e4 (stale) Mirror 13: 0x5cd3c910 (stale) CRC-32 checksum value for chunk [0, 0x2e90a): Mirror 1: 0x5020db40 Mirror 14: 0x5020db40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e90a: skipped 000534 1789118791.873340 trunc from 0x02e90a to 0x0300aa (0x17a1 bytes) 000536 1789118791.922193 mapread 0x01e11f thru 0x02569c (0x757e bytes) 000537 1789118791.958110 mapread 0x029d22 thru 0x02a600 (0x8df bytes) 000538 1789118791.975346 write 0x0270f4 thru 0x0344c4 (0xd3d1 bytes) 000539 1789118791.981769 trunc from 0x0344c5 to 0x016a96 (0x1da2e bytes) 000540 1789118792.011879 trunc from 0x016a97 to 0x023fe5 (0xd54f bytes) 000541 1789118792.036962 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 8: 0x60573d10 (stale) Mirror 11: 0x60573d10 (stale) Mirror 12: 0x60573d10 (stale) Mirror 13: 0x3ba303d5 (stale) Mirror 14: 0x6abd266b (stale) CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 1: 0x9a950dca Mirror 15: 0x9a950dca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fe6: skipped 000542 1789118793.486894 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 8: 0x60573d10 (stale) Mirror 11: 0x60573d10 (stale) Mirror 12: 0x60573d10 (stale) Mirror 13: 0x3ba303d5 (stale) Mirror 14: 0x6abd266b (stale) CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 1: 0x9a950dca Mirror 15: 0x9a950dca Mirror 16: 0x9a950dca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fe6: skipped 000543 1789118794.879087 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 8: 0x60573d10 (stale) Mirror 11: 0x60573d10 (stale) Mirror 12: 0x60573d10 (stale) Mirror 14: 0x6abd266b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 8: 0x60573d10 (stale) Mirror 11: 0x60573d10 (stale) Mirror 12: 0x60573d10 (stale) Mirror 14: 0x6abd266b (stale) CRC-32 checksum value for chunk [0, 0x23fe6): Mirror 1: 0x9a950dca Mirror 15: 0x9a950dca Mirror 16: 0x9a950dca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fe6: skipped 000544 1789118795.878502 mapread 0x00494f thru 0x004953 (0x5 bytes) 000546 1789118795.892410 read 0x0149a4 thru 0x015f5c (0x15b9 bytes) 000547 1789118795.904312 write 0x03b372 thru 0x03c272 (0xf01 bytes) 000548 1789118795.999953 mapwrite 0x0053f0 thru 0x00d026 (0x7c37 bytes) 000549 1789118796.089946 mapread 0x021718 thru 0x029944 (0x822d bytes) 000550 1789118796.107211 trunc from 0x035aa4 to 0x0393ab (0x3908 bytes) 000551 1789118796.112861 write 0x0074d6 thru 0x00ae0d (0x3938 bytes) 000552 1789118796.114373 write 0x036775 thru 0x03d68e (0x6f1a bytes) 000553 1789118796.121930 fallocate 0x0144e4 thru 0x016e96 (0x29b3 bytes) 000554 1789118796.126994 mapwrite 0x012439 thru 0x01a5b6 (0x817e bytes) 000555 1789118796.153742 trunc from 0x03d68f to 0x022d78 (0x1a916 bytes) 000556 1789118796.174645 punch from 0x01168f to 0x01c705 (0xb077 bytes) 000557 1789118796.181950 trunc from 0x0113b2 to 0x01c3ac (0xaffb bytes) 000558 1789118796.189155 fallocate 0x013352 thru 0x015780 (0x242f bytes) 000559 1789118796.196351 write 0x024df5 thru 0x028b3f (0x3d4b bytes) 000560 1789118796.197646 punch from 0x01d660 to 0x01e338 (0xcd9 bytes) 000562 1789118796.210630 trunc from 0x000ece to 0x00564d (0x4780 bytes) 000563 1789118796.220670 mapwrite 0x012d14 thru 0x01516f (0x245c bytes) 000564 1789118796.249355 mapread 0x006017 thru 0x00fea5 (0x9e8f bytes) 000567 1789118796.253389 read 0x018847 thru 0x0195e4 (0xd9e bytes) 000568 1789118796.254676 trunc from 0x028b40 to 0x00f7ae (0x19391 bytes) 000569 1789118796.279413 mapread 0x0069ed thru 0x006e43 (0x457 bytes) 000570 1789118796.281281 punch from 0x00e72a to 0x00f7ae (0x1085 bytes) 000571 1789118796.287573 mapwrite 0x008f75 thru 0x018e43 (0xfecf bytes) 000572 1789118796.346370 trunc from 0x00bb6f to 0x0181da (0xc66c bytes) 000573 1789118796.357216 write 0x02d882 thru 0x0368f3 (0x9072 bytes) 000574 1789118796.373574 MIRROR_RESYNC 7 mirrors 000575 1789118799.125844 MIRROR_RESYNC 7 mirrors 000576 1789118799.787879 trunc from 0x0368f4 to 0x00bed6 (0x2aa1d bytes) 000577 1789118799.852979 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xbed7): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbed7): Mirror 8: 0x8db2f964 (stale) Mirror 11: 0x8db2f964 (stale) Mirror 12: 0x8db2f964 (stale) Mirror 14: 0x8db2f964 (stale) Mirror 15: 0x8db2f964 (stale) Mirror 16: 0x8db2f964 (stale) CRC-32 checksum value for chunk [0, 0xbed7): Mirror 1: 0x8db2f964 Mirror 17: 0x8db2f964 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbed7: skipped 000578 1789118800.828908 read 0x007659 thru 0x00aea6 (0x384e bytes) 000579 1789118800.835831 trunc from 0x009c47 to 0x00bed6 (0x2290 bytes) 000580 1789118800.883344 fallocate 0x0350ce thru 0x03b2e0 (0x6213 bytes) 000581 1789118800.892900 MIRROR_RESYNC 8 mirrors 000583 1789118802.336503 trunc from 0x00943b to 0x00bed6 (0x2a9c bytes) 000584 1789118802.396594 mapread 0x002253 thru 0x00bed6 (0x9c84 bytes) 000585 1789118802.421719 trunc from 0x005e85 to 0x00bed6 (0x6052 bytes) 000586 1789118802.426849 write 0x007134 thru 0x00d7b2 (0x667f bytes) 000587 1789118802.439096 write 0x00efec thru 0x0124dc (0x34f1 bytes) 000588 1789118802.445753 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x124dd): Mirror 8: 0x8b249457 (stale) Mirror 12: 0x8b249457 (stale) Mirror 14: 0x8b249457 (stale) Mirror 15: 0x8b249457 (stale) Mirror 16: 0x8b249457 (stale) Mirror 17: 0x8b249457 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x124dd): Mirror 8: 0x8b249457 (stale) Mirror 12: 0x8b249457 (stale) Mirror 14: 0x8b249457 (stale) Mirror 15: 0x8b249457 (stale) Mirror 16: 0x8b249457 (stale) Mirror 17: 0x8b249457 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x124dd: skipped 000589 1789118803.135529 fallocate 0x03ab65 thru 0x03ffff (0x549b bytes) 000590 1789118803.170421 trunc from 0x0124dd to 0x029fdc (0x17b00 bytes) 000591 1789118803.232145 fallocate 0x006293 thru 0x007124 (0xe92 bytes) 000592 1789118803.250921 mapread 0x01faf7 thru 0x0257b5 (0x5cbf bytes) 000593 1789118803.304399 read 0x011ea1 thru 0x01498a (0x2aea bytes) 000594 1789118803.321244 read 0x0208c4 thru 0x023ff5 (0x3732 bytes) 000595 1789118803.323272 mapread 0x012671 thru 0x013688 (0x1018 bytes) 000596 1789118803.324329 punch from 0x00559f to 0x007516 (0x1f78 bytes) 000597 1789118803.341884 trunc from 0x0082d9 to 0x00b5da (0x3302 bytes) 000598 1789118803.350205 trunc from 0x029fdd to 0x02c742 (0x2766 bytes) 000599 1789118803.395293 read 0x01e9d3 thru 0x02c742 (0xdd70 bytes) 000600 1789118803.419657 mapread 0x0169e9 thru 0x02593d (0xef55 bytes) 000601 1789118803.422774 mapread 0x0137ab thru 0x02290c (0xf162 bytes) 000602 1789118803.440744 MIRROR_RESYNC 7 mirrors 000603 1789118805.799216 trunc from 0x02848e to 0x02c742 (0x42b5 bytes) 000604 1789118805.847139 trunc from 0x00052e to 0x001083 (0xb56 bytes) 000605 1789118805.854099 trunc from 0x02c743 to 0x03fe12 (0x136d0 bytes) 000606 1789118805.878556 read 0x03e7dd thru 0x03fe12 (0x1636 bytes) 000607 1789118805.889297 fallocate 0x01e1d4 thru 0x020c82 (0x2aaf bytes) 000609 1789118805.894782 trunc from 0x035b22 to 0x038a5f (0x2f3e bytes) 000610 1789118805.904941 trunc from 0x03fe13 to 0x02452c (0x1b8e6 bytes) 000611 1789118805.918505 fallocate 0x01cc03 thru 0x027856 (0xac54 bytes) 000612 1789118805.922591 mapwrite 0x03882d thru 0x03b21a (0x29ee bytes) 000613 1789118805.964378 fallocate 0x005165 thru 0x00e8e4 (0x9780 bytes) 000614 1789118805.969266 read 0x023a4e thru 0x02e62c (0xabdf bytes) 000615 1789118805.979636 mapread 0x02474d thru 0x032ccb (0xe57f bytes) 000616 1789118805.980648 punch from 0x0276e2 to 0x035495 (0xddb4 bytes) 000617 1789118805.985370 trunc from 0x03b21b to 0x0046f9 (0x36b21 bytes) 000618 1789118806.004127 mapread 0x002495 thru 0x0046f9 (0x2265 bytes) 000619 1789118806.005351 trunc from 0x0002fc to 0x0046f9 (0x43fe bytes) 000620 1789118806.011651 mapread 0x00449a thru 0x0046f9 (0x260 bytes) 000622 1789118806.017525 mapwrite 0x026983 thru 0x03639b (0xfa19 bytes) 000623 1789118806.119412 read 0x0177b2 thru 0x01b3d2 (0x3c21 bytes) 000624 1789118806.137869 write 0x024e84 thru 0x02a3ef (0x556c bytes) 000625 1789118806.143717 MIRROR_RESYNC 7 mirrors 000626 1789118807.122896 MIRROR_RESYNC 7 mirrors 000627 1789118807.796040 write 0x02f4fe thru 0x03d290 (0xdd93 bytes) 000628 1789118807.855584 fallocate 0x02bc45 thru 0x02fc25 (0x3fe1 bytes) 000629 1789118807.866056 mapwrite 0x00d6c6 thru 0x0100ad (0x29e8 bytes) 000630 1789118807.924314 trunc from 0x03d291 to 0x00dc90 (0x2f600 bytes) 000632 1789118807.989543 MIRROR_RESYNC 7 mirrors 000633 1789118809.166628 trunc from 0x00dc91 to 0x023ef2 (0x16262 bytes) 000634 1789118809.217575 trunc from 0x023ef3 to 0x00257d (0x21975 bytes) 000635 1789118809.235557 trunc from 0x00257e to 0x03deed (0x3b970 bytes) 000637 1789118809.269256 mapread 0x03b592 thru 0x03deed (0x295c bytes) 000639 1789118809.291212 fallocate 0x002d57 thru 0x00d492 (0xa73c bytes) 000640 1789118809.296114 trunc from 0x03deee to 0x00cfa1 (0x30f4c bytes) 000641 1789118809.322035 punch from 0x006b2c to 0x00cfa1 (0x6476 bytes) 000642 1789118809.327053 fallocate 0x02b69a thru 0x033ad2 (0x8439 bytes) 000644 1789118809.332333 mapread 0x021c1c thru 0x02d807 (0xbbec bytes) 000645 1789118809.351317 mapread 0x011fa7 thru 0x01634b (0x43a5 bytes) 000646 1789118809.358686 fallocate 0x03e3ea thru 0x03ffff (0x1c16 bytes) 000647 1789118809.362740 read 0x0007b7 thru 0x00af31 (0xa77b bytes) 000648 1789118809.370751 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33ad3): Mirror 8: 0x683d319b (stale) Mirror 14: 0x683d319b (stale) Mirror 15: 0x683d319b (stale) Mirror 16: 0x683d319b (stale) Mirror 17: 0x683d319b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ad3): Mirror 8: 0x683d319b (stale) Mirror 14: 0x683d319b (stale) Mirror 15: 0x683d319b (stale) Mirror 16: 0x683d319b (stale) Mirror 17: 0x683d319b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ad3: skipped 000649 1789118809.659221 punch from 0x0184f8 to 0x0245cd (0xc0d6 bytes) 000650 1789118809.663002 mapwrite 0x008a44 thru 0x00e497 (0x5a54 bytes) 000651 1789118809.706691 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33ad3): Mirror 8: 0x683d319b (stale) Mirror 14: 0x683d319b (stale) Mirror 15: 0x683d319b (stale) Mirror 16: 0x683d319b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ad3): Mirror 8: 0x683d319b (stale) Mirror 14: 0x683d319b (stale) Mirror 15: 0x683d319b (stale) Mirror 16: 0x683d319b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ad3: skipped 000652 1789118810.096770 mapwrite 0x01036d thru 0x010f27 (0xbbb bytes) 000654 1789118810.136970 mapread 0x02da39 thru 0x0337b2 (0x5d7a bytes) 000655 1789118810.159952 mapwrite 0x02ce1e thru 0x03c228 (0xf40b bytes) 000657 1789118810.248590 punch from 0x01f6c7 to 0x02446a (0x4da4 bytes) 000658 1789118810.264128 mapread 0x0126fe thru 0x01963e (0x6f41 bytes) 000659 1789118810.290408 fallocate 0x01236b thru 0x01b912 (0x95a8 bytes) 000660 1789118810.306832 read 0x02578c thru 0x0325ce (0xce43 bytes) 000661 1789118810.315347 write 0x0286d9 thru 0x035d25 (0xd64d bytes) 000662 1789118810.319037 mapread 0x018e15 thru 0x019a34 (0xc20 bytes) 000663 1789118810.320153 read 0x00cb2b thru 0x016a99 (0x9f6f bytes) 000664 1789118810.331835 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c229): Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c229): Mirror 8: 0xc95430b3 (stale) Mirror 14: 0xc95430b3 (stale) Mirror 15: 0xc95430b3 (stale) Mirror 16: 0xc95430b3 (stale) CRC-32 checksum value for chunk [0, 0x3c229): Mirror 1: 0x6662626d Mirror 17: 0x6662626d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c229: skipped 000665 1789118811.675129 read 0x0205f0 thru 0x020897 (0x2a8 bytes) 000666 1789118811.684960 punch from 0x013827 to 0x014434 (0xc0e bytes) 000667 1789118811.734870 trunc from 0x03c229 to 0x033ee2 (0x8346 bytes) 000668 1789118811.768555 trunc from 0x029b0a to 0x03066f (0x6b66 bytes) 000669 1789118811.784481 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33ee3): Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ee3): Mirror 8: 0x48e29d51 (stale) Mirror 14: 0x48e29d51 (stale) Mirror 15: 0x48e29d51 (stale) Mirror 16: 0x48e29d51 (stale) Mirror 17: 0xc3ed3b25 (stale) CRC-32 checksum value for chunk [0, 0x33ee3): Mirror 1: 0x7ffba82a Mirror 18: 0x7ffba82a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ee3: skipped 000670 1789118812.886946 trunc from 0x023928 to 0x02aa0c (0x70e5 bytes) 000671 1789118812.958659 punch from 0x009d12 to 0x012cad (0x8f9c bytes) 000672 1789118812.966754 trunc from 0x033ee3 to 0x02b3c7 (0x8b1b bytes) 000673 1789118813.000310 mapwrite 0x03f606 thru 0x03ffff (0x9fa bytes) 000674 1789118813.084753 fallocate 0x002746 thru 0x0116b5 (0xef70 bytes) 000675 1789118813.096614 mapread 0x025a18 thru 0x02ec79 (0x9262 bytes) 000677 1789118813.126248 trunc from 0x018b48 to 0x0227d5 (0x9c8e bytes) 000678 1789118813.134755 write 0x03f858 thru 0x03ffff (0x7a8 bytes) 000679 1789118813.136261 mapwrite 0x027065 thru 0x030abe (0x9a5a bytes) 000680 1789118813.195809 trunc from 0x00b915 to 0x016560 (0xac4c bytes) 000681 1789118813.201331 mapread 0x014ac6 thru 0x01e885 (0x9dc0 bytes) 000682 1789118813.221248 MIRROR_RESYNC 7 mirrors 000683 1789118815.298840 MIRROR_EXTEND to 8 mirrors 000684 1789118816.104204 mapread 0x037818 thru 0x03feda (0x86c3 bytes) 000685 1789118816.128572 write 0x011da9 thru 0x017519 (0x5771 bytes) 000688 1789118816.195164 punch from 0x0120fb to 0x017758 (0x565e bytes) 000689 1789118816.211103 mapwrite 0x03f071 thru 0x03fa91 (0xa21 bytes) 000690 1789118816.237701 punch from 0x03e8ae to 0x03ffff (0x1752 bytes) 000691 1789118816.244024 trunc from 0x012063 to 0x021c48 (0xfbe6 bytes) 000692 1789118816.250037 write 0x0077f5 thru 0x010376 (0x8b82 bytes) 000694 1789118816.267670 trunc from 0x040000 to 0x00e717 (0x318e8 bytes) 000695 1789118816.301067 read 0x007afb thru 0x00a098 (0x259e bytes) 000696 1789118816.308192 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe718): Mirror 14: 0x40a2f909 (stale) Mirror 15: 0x40a2f909 (stale) Mirror 16: 0x40a2f909 (stale) Mirror 17: 0x40a2f909 (stale) Mirror 18: 0x40a2f909 (stale) Mirror 19: 0x40a2f909 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe718): Mirror 14: 0x40a2f909 (stale) Mirror 15: 0x40a2f909 (stale) Mirror 16: 0x40a2f909 (stale) Mirror 17: 0x40a2f909 (stale) Mirror 18: 0x40a2f909 (stale) Mirror 19: 0x40a2f909 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe718: skipped 000697 1789118816.775066 trunc from 0x0038bd to 0x007bc5 (0x4309 bytes) 000698 1789118816.782483 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xe718): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe718): Mirror 14: 0x40a2f909 (stale) Mirror 15: 0x40a2f909 (stale) Mirror 16: 0x40a2f909 (stale) Mirror 17: 0x40a2f909 (stale) Mirror 18: 0x40a2f909 (stale) Mirror 19: 0x40a2f909 (stale) CRC-32 checksum value for chunk [0, 0xe718): Mirror 1: 0x108512db Mirror 20: 0x108512db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe718: skipped 000699 1789118817.904468 mapread 0x00dd51 thru 0x00e717 (0x9c7 bytes) 000700 1789118817.953199 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xe718): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe718): Mirror 14: 0x40a2f909 (stale) Mirror 15: 0x40a2f909 (stale) Mirror 16: 0x40a2f909 (stale) Mirror 17: 0x40a2f909 (stale) Mirror 18: 0x40a2f909 (stale) Mirror 19: 0x40a2f909 (stale) CRC-32 checksum value for chunk [0, 0xe718): Mirror 1: 0x108512db Mirror 20: 0x108512db Mirror 21: 0x108512db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe718: skipped 000701 1789118819.337839 write 0x02bbb4 thru 0x0307e2 (0x4c2f bytes) 000702 1789118819.441509 read 0x01cb71 thru 0x02bb06 (0xef96 bytes) 000703 1789118819.452581 punch from 0x028648 to 0x02b494 (0x2e4d bytes) 000704 1789118819.470608 fallocate 0x01e7e8 thru 0x02cfa2 (0xe7bb bytes) 000705 1789118819.476577 write 0x004ddd thru 0x005409 (0x62d bytes) 000706 1789118819.499369 mapwrite 0x00477d thru 0x009bdf (0x5463 bytes) 000707 1789118819.590125 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x307e3): Mirror 14: 0xce382566 (stale) Mirror 15: 0xce382566 (stale) Mirror 16: 0xce382566 (stale) Mirror 17: 0xce382566 (stale) Mirror 18: 0xce382566 (stale) Mirror 19: 0xce382566 (stale) Mirror 21: 0x99d9726 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307e3): Mirror 14: 0xce382566 (stale) Mirror 15: 0xce382566 (stale) Mirror 16: 0xce382566 (stale) Mirror 17: 0xce382566 (stale) Mirror 18: 0xce382566 (stale) Mirror 19: 0xce382566 (stale) Mirror 21: 0x99d9726 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307e3: skipped 000708 1789118820.672314 trunc from 0x016b4a to 0x01adb8 (0x426f bytes) 000709 1789118820.702598 fallocate 0x02468b thru 0x02e066 (0x99dc bytes) 000710 1789118820.714188 trunc from 0x02ebff to 0x0307e2 (0x1be4 bytes) 000711 1789118820.722572 trunc from 0x02e0d7 to 0x0307e2 (0x270c bytes) 000712 1789118820.746047 punch from 0x02d4e8 to 0x02d62a (0x143 bytes) 000713 1789118820.754683 write 0x0136dd thru 0x019c12 (0x6536 bytes) 000714 1789118820.775593 write 0x028f20 thru 0x02a80a (0x18eb bytes) 000715 1789118820.787412 read 0x00f65b thru 0x01eb1b (0xf4c1 bytes) 000716 1789118820.799951 mapwrite 0x01aa60 thru 0x01ef61 (0x4502 bytes) 000717 1789118820.831177 trunc from 0x00f2fb to 0x01a2a3 (0xafa9 bytes) 000718 1789118820.855406 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x307e3): Mirror 14: 0xce382566 (stale) Mirror 15: 0xce382566 (stale) Mirror 16: 0xce382566 (stale) Mirror 17: 0xce382566 (stale) Mirror 18: 0xce382566 (stale) Mirror 19: 0xce382566 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307e3): Mirror 14: 0xce382566 (stale) Mirror 15: 0xce382566 (stale) Mirror 16: 0xce382566 (stale) Mirror 17: 0xce382566 (stale) Mirror 18: 0xce382566 (stale) Mirror 19: 0xce382566 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307e3: skipped 000720 1789118821.913754 read 0x0070a2 thru 0x008fc5 (0x1f24 bytes) 000721 1789118821.931173 mapread 0x023637 thru 0x02f219 (0xbbe3 bytes) 000722 1789118821.994330 fallocate 0x02c34d thru 0x038b73 (0xc827 bytes) 000723 1789118822.003493 punch from 0x029183 to 0x02c449 (0x32c7 bytes) 000724 1789118822.014832 mapread 0x00162a thru 0x00f035 (0xda0c bytes) 000725 1789118822.069677 punch from 0x02f8f5 to 0x02fd83 (0x48f bytes) 000726 1789118822.085077 write 0x004d8a thru 0x0099a2 (0x4c19 bytes) 000727 1789118822.094474 write 0x010b90 thru 0x01d905 (0xcd76 bytes) 000728 1789118822.123091 read 0x019e56 thru 0x01c5dc (0x2787 bytes) 000729 1789118822.123728 read 0x030540 thru 0x0307e2 (0x2a3 bytes) 000731 1789118822.124774 read 0x014fa8 thru 0x01a8e7 (0x5940 bytes) 000732 1789118822.129064 trunc from 0x0307e3 to 0x001dfe (0x2e9e4 bytes) 000733 1789118822.150560 trunc from 0x0016c7 to 0x001dfe (0x738 bytes) 000734 1789118822.160841 write 0x00e3a5 thru 0x01c3fd (0xe059 bytes) 000735 1789118822.166525 MIRROR_RESYNC 7 mirrors 000736 1789118823.692016 read 0x00f4b1 thru 0x01b32b (0xbe7b bytes) 000737 1789118823.711321 write 0x035842 thru 0x038b33 (0x32f2 bytes) 000738 1789118823.785910 trunc from 0x038b34 to 0x0264c3 (0x12670 bytes) 000739 1789118823.813025 trunc from 0x0264c4 to 0x010fbf (0x15504 bytes) 000740 1789118823.847062 mapwrite 0x0359ff thru 0x03ffff (0xa601 bytes) 000741 1789118823.978199 mapwrite 0x01d3da thru 0x023d28 (0x694f bytes) 000742 1789118824.045966 mapwrite 0x03a3e9 thru 0x03ffff (0x5c17 bytes) 000743 1789118824.076924 mapread 0x018fe4 thru 0x0261cd (0xd1ea bytes) 000744 1789118824.094156 read 0x023564 thru 0x031edc (0xe979 bytes) 000745 1789118824.136574 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x21a55315 (stale) Mirror 15: 0x21a55315 (stale) Mirror 16: 0x21a55315 (stale) Mirror 17: 0x21a55315 (stale) Mirror 18: 0x21a55315 (stale) Mirror 19: 0x21a55315 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9d9a95a9 Mirror 20: 0x9d9a95a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000746 1789118825.504395 fallocate 0x035cc9 thru 0x037f60 (0x2298 bytes) 000747 1789118825.597504 trunc from 0x040000 to 0x03fa50 (0x5af bytes) 000748 1789118825.713055 read 0x01ee0c thru 0x01f376 (0x56b bytes) 000749 1789118825.727717 mapread 0x00dab2 thru 0x01b6a9 (0xdbf8 bytes) 000750 1789118825.772359 punch from 0x012916 to 0x01c957 (0xa042 bytes) 000751 1789118825.790413 MIRROR_RESYNC 8 mirrors 000752 1789118827.997501 trunc from 0x036f1e to 0x03d8b0 (0x6993 bytes) 000754 1789118828.061541 fallocate 0x008a29 thru 0x0168bc (0xde94 bytes) 000755 1789118828.085073 read 0x01ea30 thru 0x022365 (0x3936 bytes) 000756 1789118828.094189 mapwrite 0x006331 thru 0x00ff84 (0x9c54 bytes) 000757 1789118828.184508 punch from 0x037d5e to 0x03fa50 (0x7cf3 bytes) 000758 1789118828.193832 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa51): Mirror 15: 0xf055db51 (stale) Mirror 16: 0xf055db51 (stale) Mirror 17: 0xf055db51 (stale) Mirror 18: 0xf055db51 (stale) Mirror 19: 0xf055db51 (stale) Mirror 20: 0xf055db51 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa51): Mirror 15: 0xf055db51 (stale) Mirror 16: 0xf055db51 (stale) Mirror 17: 0xf055db51 (stale) Mirror 18: 0xf055db51 (stale) Mirror 19: 0xf055db51 (stale) Mirror 20: 0xf055db51 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa51: skipped 000759 1789118828.781051 trunc from 0x03fa51 to 0x00d59e (0x324b2 bytes) 000761 1789118828.829798 mapwrite 0x023356 thru 0x02e04f (0xacfa bytes) 000762 1789118828.919138 write 0x006840 thru 0x008177 (0x1938 bytes) 000763 1789118828.978488 write 0x0333b0 thru 0x037854 (0x44a5 bytes) 000764 1789118828.985176 mapwrite 0x012864 thru 0x017dd0 (0x556d bytes) 000765 1789118829.060065 mapread 0x027a3d thru 0x02e60d (0x6bd1 bytes) 000766 1789118829.062216 mapread 0x00d1dc thru 0x01c601 (0xf426 bytes) 000767 1789118829.087404 trunc from 0x021048 to 0x02d9c2 (0xc97b bytes) 000768 1789118829.101843 mapwrite 0x003d5c thru 0x00f58a (0xb82f bytes) 000769 1789118829.178252 fallocate 0x00510e thru 0x012220 (0xd113 bytes) 000770 1789118829.190049 mapwrite 0x01fa3e thru 0x029513 (0x9ad6 bytes) 000771 1789118829.237100 trunc from 0x037855 to 0x0175e2 (0x20272 bytes) 000772 1789118829.261065 write 0x032cf4 thru 0x03860f (0x591c bytes) 000773 1789118829.270322 MIRROR_RESYNC 7 mirrors 000774 1789118831.559604 punch from 0x0010b5 to 0x00be34 (0xad80 bytes) 000775 1789118831.604264 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38610): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38610): Mirror 15: 0x7903ba3a (stale) Mirror 16: 0x7903ba3a (stale) Mirror 17: 0x7903ba3a (stale) Mirror 18: 0x7903ba3a (stale) Mirror 19: 0x7903ba3a (stale) Mirror 20: 0x7903ba3a (stale) CRC-32 checksum value for chunk [0, 0x38610): Mirror 1: 0x6293a63a Mirror 21: 0x6293a63a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38610: skipped 000776 1789118832.930944 trunc from 0x038610 to 0x02d24a (0xb3c5 bytes) 000777 1789118832.995100 punch from 0x028e3d to 0x02d24a (0x440e bytes) 000778 1789118833.006552 fallocate 0x00126e thru 0x009942 (0x86d5 bytes) 000779 1789118833.014818 mapread 0x00bcea thru 0x01859d (0xc8b4 bytes) 000780 1789118833.058008 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 15: 0xd031166c (stale) Mirror 16: 0xd031166c (stale) Mirror 17: 0xd031166c (stale) Mirror 18: 0xd031166c (stale) Mirror 19: 0xd031166c (stale) Mirror 20: 0xd031166c (stale) Mirror 21: 0x86a2b71a (stale) CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 1: 0x86a2b71a Mirror 22: 0x86a2b71a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d24b: skipped 000782 1789118834.453066 punch from 0x020bc0 to 0x021fef (0x1430 bytes) 000783 1789118834.508107 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 15: 0xd031166c (stale) Mirror 16: 0xd031166c (stale) Mirror 17: 0xd031166c (stale) Mirror 18: 0xd031166c (stale) Mirror 19: 0xd031166c (stale) Mirror 20: 0xd031166c (stale) Mirror 21: 0x86a2b71a (stale) Mirror 22: 0x86a2b71a (stale) CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 1: 0x86a2b71a Mirror 23: 0x86a2b71a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d24b: skipped 000784 1789118836.183327 mapread 0x016166 thru 0x01ab5e (0x49f9 bytes) 000785 1789118836.256536 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 15: 0xd031166c (stale) Mirror 16: 0xd031166c (stale) Mirror 17: 0xd031166c (stale) Mirror 18: 0xd031166c (stale) Mirror 19: 0xd031166c (stale) Mirror 20: 0xd031166c (stale) Mirror 21: 0x86a2b71a (stale) Mirror 22: 0x86a2b71a (stale) CRC-32 checksum value for chunk [0, 0x2d24b): Mirror 1: 0x86a2b71a Mirror 23: 0x86a2b71a Mirror 24: 0x86a2b71a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d24b: skipped 000786 1789118837.671973 read 0x02c0dd thru 0x02d24a (0x116e bytes) 000788 1789118837.685874 read 0x00ced4 thru 0x00d698 (0x7c5 bytes) 000789 1789118837.711957 read 0x011f33 thru 0x01b99f (0x9a6d bytes) 000790 1789118837.712757 trunc from 0x00d676 to 0x0158ea (0x8275 bytes) 000791 1789118837.799255 trunc from 0x027c35 to 0x028e27 (0x11f3 bytes) 000792 1789118837.806137 fallocate 0x017673 thru 0x022adc (0xb46a bytes) 000793 1789118837.813725 trunc from 0x02d24b to 0x006725 (0x26b25 bytes) 000794 1789118837.877917 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x6726): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6726): Mirror 15: 0x5c98a6d4 (stale) Mirror 16: 0x5c98a6d4 (stale) Mirror 17: 0x5c98a6d4 (stale) Mirror 18: 0x5c98a6d4 (stale) Mirror 19: 0x5c98a6d4 (stale) Mirror 20: 0x5c98a6d4 (stale) Mirror 21: 0x9e2cb53c (stale) Mirror 22: 0x9e2cb53c (stale) Mirror 23: 0x9e2cb53c (stale) Mirror 24: 0x9e2cb53c (stale) CRC-32 checksum value for chunk [0, 0x6726): Mirror 1: 0x9e2cb53c Mirror 25: 0x9e2cb53c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6726: skipped 000796 1789118839.465767 mapwrite 0x0343a3 thru 0x03ffff (0xbc5d bytes) 000797 1789118839.615605 write 0x00b2b8 thru 0x0166fd (0xb446 bytes) 000799 1789118839.638325 fallocate 0x0155ba thru 0x01bce3 (0x672a bytes) 000800 1789118839.647226 mapwrite 0x03b704 thru 0x03ffff (0x48fc bytes) 000801 1789118839.665241 fallocate 0x02d162 thru 0x03527e (0x811d bytes) 000802 1789118839.670473 write 0x020a6d thru 0x02756e (0x6b02 bytes) 000803 1789118839.681618 read 0x027bcb thru 0x02c451 (0x4887 bytes) 000804 1789118839.682253 mapwrite 0x02671c thru 0x02cf03 (0x67e8 bytes) 000805 1789118839.701200 punch from 0x02512f to 0x028e4c (0x3d1e bytes) 000806 1789118839.711686 MIRROR_RESYNC 12 mirrors 000807 1789118844.058498 trunc from 0x040000 to 0x030676 (0xf989 bytes) 000808 1789118844.128241 trunc from 0x020597 to 0x02c0db (0xbb45 bytes) 000809 1789118844.134307 write 0x023fb2 thru 0x0287f0 (0x483f bytes) 000810 1789118844.165939 MIRROR_RESYNC 12 mirrors 000811 1789118848.971894 trunc from 0x030677 to 0x0056cd (0x2afa9 bytes) 000813 1789118849.029653 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x56ce): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x56ce): Mirror 15: 0xacf28409 (stale) Mirror 16: 0xacf28409 (stale) Mirror 17: 0xacf28409 (stale) Mirror 18: 0xacf28409 (stale) Mirror 19: 0xacf28409 (stale) Mirror 20: 0xacf28409 (stale) Mirror 21: 0xacf28409 (stale) Mirror 22: 0xacf28409 (stale) Mirror 23: 0xacf28409 (stale) Mirror 24: 0xacf28409 (stale) Mirror 25: 0xacf28409 (stale) CRC-32 checksum value for chunk [0, 0x56ce): Mirror 1: 0xacf28409 Mirror 26: 0xacf28409 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x56ce: skipped 000814 1789118850.446328 fallocate 0x015dfb thru 0x01f258 (0x945e bytes) 000816 1789118850.493654 punch from 0x0026fc to 0x0056cd (0x2fd2 bytes) 000817 1789118850.498904 punch from 0x002bfb to 0x0056cd (0x2ad3 bytes) 000818 1789118850.508415 MIRROR_RESYNC 13 mirrors 000819 1789118852.095869 trunc from 0x001c2f to 0x0056cd (0x3a9f bytes) 000820 1789118852.136486 write 0x0170de thru 0x021c78 (0xab9b bytes) 000821 1789118852.138701 fallocate 0x02dba8 thru 0x03217b (0x45d4 bytes) 000822 1789118852.145267 read 0x00baea thru 0x017939 (0xbe50 bytes) 000823 1789118852.157043 punch from 0x0289db to 0x02c20d (0x3833 bytes) 000824 1789118852.164638 trunc from 0x031f88 to 0x03217b (0x1f4 bytes) 000825 1789118852.170065 trunc from 0x03217c to 0x02169e (0x10add bytes) 000826 1789118852.195728 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2169f): Mirror 16: 0x72e5c454 (stale) Mirror 17: 0x72e5c454 (stale) Mirror 18: 0x72e5c454 (stale) Mirror 19: 0x72e5c454 (stale) Mirror 20: 0x72e5c454 (stale) Mirror 21: 0x72e5c454 (stale) Mirror 22: 0x72e5c454 (stale) Mirror 23: 0x72e5c454 (stale) Mirror 24: 0x72e5c454 (stale) Mirror 25: 0x72e5c454 (stale) Mirror 26: 0x72e5c454 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2169f): Mirror 16: 0x72e5c454 (stale) Mirror 17: 0x72e5c454 (stale) Mirror 18: 0x72e5c454 (stale) Mirror 19: 0x72e5c454 (stale) Mirror 20: 0x72e5c454 (stale) Mirror 21: 0x72e5c454 (stale) Mirror 22: 0x72e5c454 (stale) Mirror 23: 0x72e5c454 (stale) Mirror 24: 0x72e5c454 (stale) Mirror 25: 0x72e5c454 (stale) Mirror 26: 0x72e5c454 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2169f: skipped 000827 1789118852.721406 read 0x0126e1 thru 0x01dde9 (0xb709 bytes) 000828 1789118852.729704 trunc from 0x02169f to 0x026f8c (0x58ee bytes) 000829 1789118852.752241 read 0x018338 thru 0x021896 (0x955f bytes) 000830 1789118852.775913 punch from 0x01e8e1 to 0x026f8c (0x86ac bytes) 000832 1789118852.782116 mapwrite 0x00dd36 thru 0x00fcbc (0x1f87 bytes) 000833 1789118852.795729 trunc from 0x014b31 to 0x0196f1 (0x4bc1 bytes) 000834 1789118852.801232 trunc from 0x022f0b to 0x026c08 (0x3cfe bytes) 000835 1789118852.805673 mapread 0x0145b3 thru 0x01803e (0x3a8c bytes) 000836 1789118852.828236 trunc from 0x026f8d to 0x00bd0a (0x1b282 bytes) 000838 1789118852.848186 punch from 0x00196b to 0x008f8c (0x7622 bytes) 000839 1789118852.854277 punch from 0x00b4e3 to 0x00bd0a (0x828 bytes) 000840 1789118852.859138 mapwrite 0x00c045 thru 0x00cb9b (0xb57 bytes) 000841 1789118852.903840 mapwrite 0x014903 thru 0x01baa7 (0x71a5 bytes) 000842 1789118853.022848 mapread 0x019c75 thru 0x01baa7 (0x1e33 bytes) 000843 1789118853.023897 mapwrite 0x002ae0 thru 0x003625 (0xb46 bytes) 000844 1789118853.057260 mapwrite 0x019a1c thru 0x01d66b (0x3c50 bytes) 000845 1789118853.142293 mapwrite 0x0273e0 thru 0x02a60b (0x322c bytes) 000846 1789118853.185096 mapwrite 0x03b19a thru 0x03ffff (0x4e66 bytes) 000847 1789118853.229931 trunc from 0x034cc8 to 0x03bc81 (0x6fba bytes) 000848 1789118853.237649 fallocate 0x01d24e thru 0x027b3b (0xa8ee bytes) 000850 1789118853.243181 mapwrite 0x02602f thru 0x0318b9 (0xb88b bytes) 000851 1789118853.268234 mapwrite 0x0295e8 thru 0x02a18e (0xba7 bytes) 000852 1789118853.282267 fallocate 0x033e16 thru 0x03ac0e (0x6df9 bytes) 000853 1789118853.295182 MIRROR_RESYNC 12 mirrors 000854 1789118857.472275 fallocate 0x0219d5 thru 0x025b28 (0x4154 bytes) 000855 1789118857.534695 trunc from 0x024550 to 0x02cfaf (0x8a60 bytes) 000856 1789118857.553353 trunc from 0x033443 to 0x03f5a3 (0xc161 bytes) 000857 1789118857.559702 trunc from 0x030882 to 0x03eb2f (0xe2ae bytes) 000858 1789118857.565717 read 0x02ed29 thru 0x037b9f (0x8e77 bytes) 000859 1789118857.574580 trunc from 0x03e7e2 to 0x03ffff (0x181e bytes) 000860 1789118857.579084 fallocate 0x03d32a thru 0x03ffff (0x2cd6 bytes) 000861 1789118857.584728 trunc from 0x040000 to 0x001a63 (0x3e59c bytes) 000862 1789118857.611957 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a64): Mirror 16: 0x56e875d6 (stale) Mirror 17: 0x56e875d6 (stale) Mirror 18: 0x56e875d6 (stale) Mirror 20: 0x56e875d6 (stale) Mirror 21: 0x56e875d6 (stale) Mirror 22: 0x56e875d6 (stale) Mirror 23: 0x56e875d6 (stale) Mirror 24: 0x56e875d6 (stale) Mirror 25: 0x56e875d6 (stale) Mirror 26: 0x56e875d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a64): Mirror 16: 0x56e875d6 (stale) Mirror 17: 0x56e875d6 (stale) Mirror 18: 0x56e875d6 (stale) Mirror 20: 0x56e875d6 (stale) Mirror 21: 0x56e875d6 (stale) Mirror 22: 0x56e875d6 (stale) Mirror 23: 0x56e875d6 (stale) Mirror 24: 0x56e875d6 (stale) Mirror 25: 0x56e875d6 (stale) Mirror 26: 0x56e875d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a64: skipped 000863 1789118858.055119 punch from 0x00100a to 0x001a63 (0xa5a bytes) 000864 1789118858.061683 fallocate 0x006463 thru 0x00f5a1 (0x913f bytes) 000865 1789118858.071508 trunc from 0x00f5a2 to 0x01f034 (0xfa93 bytes) 000866 1789118858.096612 mapread 0x017544 thru 0x01cbe1 (0x569e bytes) 000867 1789118858.117670 write 0x03df55 thru 0x03ffff (0x20ab bytes) 000868 1789118858.119785 write 0x00ff09 thru 0x01e31b (0xe413 bytes) 000869 1789118858.126668 trunc from 0x02f605 to 0x032194 (0x2b90 bytes) 000870 1789118858.133621 read 0x0317e0 thru 0x035af3 (0x4314 bytes) 000871 1789118858.143990 fallocate 0x01b8b2 thru 0x02929e (0xd9ed bytes) 000872 1789118858.149180 mapwrite 0x037ae2 thru 0x03ffff (0x851e bytes) 000873 1789118858.186015 write 0x00b904 thru 0x016fa7 (0xb6a4 bytes) 000874 1789118858.194929 trunc from 0x040000 to 0x0040f4 (0x3bf0b bytes) 000875 1789118858.215884 write 0x03016e thru 0x03d076 (0xcf09 bytes) 000876 1789118858.219834 trunc from 0x0288f4 to 0x02ee41 (0x654e bytes) 000877 1789118858.226821 mapwrite 0x01a150 thru 0x029ecb (0xfd7c bytes) 000878 1789118858.280209 trunc from 0x034b01 to 0x03d076 (0x8576 bytes) 000879 1789118858.285067 punch from 0x003bb2 to 0x0129bb (0xee0a bytes) 000880 1789118858.290269 write 0x02dc38 thru 0x03cfb0 (0xf379 bytes) 000881 1789118858.302660 trunc from 0x03d077 to 0x0009db (0x3c69b bytes) 000882 1789118858.321560 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9dc): Mirror 16: 0xa47bb47e (stale) Mirror 17: 0xa47bb47e (stale) Mirror 18: 0xa47bb47e (stale) Mirror 20: 0xa47bb47e (stale) Mirror 21: 0xa47bb47e (stale) Mirror 22: 0xa47bb47e (stale) Mirror 23: 0xa47bb47e (stale) Mirror 25: 0xa47bb47e (stale) Mirror 26: 0xa47bb47e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9dc): Mirror 16: 0xa47bb47e (stale) Mirror 17: 0xa47bb47e (stale) Mirror 18: 0xa47bb47e (stale) Mirror 20: 0xa47bb47e (stale) Mirror 21: 0xa47bb47e (stale) Mirror 22: 0xa47bb47e (stale) Mirror 23: 0xa47bb47e (stale) Mirror 25: 0xa47bb47e (stale) Mirror 26: 0xa47bb47e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9dc: skipped 000883 1789118858.842623 read 0x0005ab thru 0x0009db (0x431 bytes) 000884 1789118858.851748 mapread 0x00006a thru 0x00009b (0x32 bytes) 000886 1789118858.853387 mapread 0x000077 thru 0x0009db (0x965 bytes) 000887 1789118858.857026 fallocate 0x029ee0 thru 0x02e3b0 (0x44d1 bytes) 000888 1789118858.863340 punch from 0x01870e to 0x024754 (0xc047 bytes) 000889 1789118858.869129 trunc from 0x016b93 to 0x020072 (0x94e0 bytes) 000890 1789118858.873503 trunc from 0x02e3b1 to 0x039d6f (0xb9bf bytes) 000891 1789118858.898126 mapwrite 0x01c1a5 thru 0x024788 (0x85e4 bytes) 000892 1789118858.939307 fallocate 0x013e2c thru 0x01588f (0x1a64 bytes) 000893 1789118858.949745 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39d70): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d70): Mirror 16: 0x4ff52660 (stale) Mirror 17: 0x4ff52660 (stale) Mirror 18: 0x4ff52660 (stale) Mirror 20: 0x4ff52660 (stale) Mirror 21: 0x4ff52660 (stale) Mirror 22: 0x4ff52660 (stale) Mirror 23: 0x4ff52660 (stale) Mirror 25: 0x4ff52660 (stale) Mirror 26: 0x4ff52660 (stale) CRC-32 checksum value for chunk [0, 0x39d70): Mirror 1: 0x1f42d054 Mirror 27: 0x1f42d054 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d70: skipped 000894 1789118859.934290 mapwrite 0x03a813 thru 0x03f386 (0x4b74 bytes) 000895 1789118860.030648 write 0x000aeb thru 0x00fe99 (0xf3af bytes) 000897 1789118860.047292 fallocate 0x0132fd thru 0x018d81 (0x5a85 bytes) 000898 1789118860.053153 read 0x010cda thru 0x01a3e4 (0x970b bytes) 000899 1789118860.063076 mapwrite 0x001f19 thru 0x0117ea (0xf8d2 bytes) 000900 1789118860.081060 write 0x008e02 thru 0x010ffd (0x81fc bytes) 000901 1789118860.082417 trunc from 0x023e8f to 0x026ad5 (0x2c47 bytes) 000902 1789118860.088016 mapread 0x000f39 thru 0x00e6da (0xd7a2 bytes) 000903 1789118860.090650 trunc from 0x012ea6 to 0x017a1e (0x4b79 bytes) 000904 1789118860.097264 punch from 0x03956f to 0x03f386 (0x5e18 bytes) 000905 1789118860.103422 write 0x00d747 thru 0x013a4a (0x6304 bytes) 000906 1789118860.110267 fallocate 0x02d538 thru 0x0340b1 (0x6b7a bytes) 000907 1789118860.115716 punch from 0x00e5b1 to 0x012af9 (0x4549 bytes) 000908 1789118860.131759 mapwrite 0x0270f2 thru 0x027e7c (0xd8b bytes) 000909 1789118860.152258 punch from 0x010041 to 0x015d12 (0x5cd2 bytes) 000910 1789118860.159164 trunc from 0x0008ec to 0x00b659 (0xad6e bytes) 000911 1789118860.169246 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3f387): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f387): Mirror 16: 0xea749b2f (stale) Mirror 17: 0xea749b2f (stale) Mirror 18: 0xea749b2f (stale) Mirror 20: 0xea749b2f (stale) Mirror 21: 0xea749b2f (stale) Mirror 22: 0xea749b2f (stale) Mirror 23: 0xea749b2f (stale) Mirror 25: 0xea749b2f (stale) Mirror 26: 0xea749b2f (stale) Mirror 27: 0x9e119315 (stale) CRC-32 checksum value for chunk [0, 0x3f387): Mirror 1: 0x9e860256 Mirror 28: 0x9e860256 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f387: skipped 000912 1789118861.187602 fallocate 0x02d363 thru 0x034dc1 (0x7a5f bytes) 000913 1789118861.240771 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f387): Mirror 16: 0xea749b2f (stale) Mirror 17: 0xea749b2f (stale) Mirror 20: 0xea749b2f (stale) Mirror 21: 0xea749b2f (stale) Mirror 22: 0xea749b2f (stale) Mirror 23: 0xea749b2f (stale) Mirror 25: 0xea749b2f (stale) Mirror 26: 0xea749b2f (stale) Mirror 27: 0x9e119315 (stale) Mirror 28: 0x9e860256 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f387): Mirror 16: 0xea749b2f (stale) Mirror 17: 0xea749b2f (stale) Mirror 20: 0xea749b2f (stale) Mirror 21: 0xea749b2f (stale) Mirror 22: 0xea749b2f (stale) Mirror 23: 0xea749b2f (stale) Mirror 25: 0xea749b2f (stale) Mirror 26: 0xea749b2f (stale) Mirror 27: 0x9e119315 (stale) Mirror 28: 0x9e860256 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f387: skipped 000914 1789118861.873116 write 0x00ef83 thru 0x01c6d0 (0xd74e bytes) 000915 1789118861.892624 punch from 0x018f39 to 0x0278d7 (0xe99f bytes) 000916 1789118861.913354 fallocate 0x006491 thru 0x01617d (0xfced bytes) 000917 1789118861.918747 trunc from 0x038d2b to 0x03f386 (0x665c bytes) 000918 1789118861.924784 trunc from 0x03f387 to 0x003878 (0x3bb0e bytes) 000919 1789118861.979393 mapwrite 0x01eaae thru 0x0211d6 (0x2729 bytes) 000920 1789118862.038008 mapread 0x005515 thru 0x00b6d5 (0x61c1 bytes) 000921 1789118862.057999 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x211d7): Mirror 16: 0x32e2d15b (stale) Mirror 20: 0x32e2d15b (stale) Mirror 21: 0x32e2d15b (stale) Mirror 22: 0x32e2d15b (stale) Mirror 23: 0x32e2d15b (stale) Mirror 25: 0x32e2d15b (stale) Mirror 26: 0x32e2d15b (stale) Mirror 27: 0x5e8dc893 (stale) Mirror 28: 0xcab85671 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x211d7): Mirror 16: 0x32e2d15b (stale) Mirror 20: 0x32e2d15b (stale) Mirror 21: 0x32e2d15b (stale) Mirror 22: 0x32e2d15b (stale) Mirror 23: 0x32e2d15b (stale) Mirror 25: 0x32e2d15b (stale) Mirror 26: 0x32e2d15b (stale) Mirror 27: 0x5e8dc893 (stale) Mirror 28: 0xcab85671 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x211d7: skipped 000922 1789118862.537111 write 0x01fa4b thru 0x024c52 (0x5208 bytes) 000923 1789118862.548927 mapwrite 0x0352c7 thru 0x03ffff (0xad39 bytes) 000924 1789118862.600843 mapwrite 0x033141 thru 0x03dcdb (0xab9b bytes) 000925 1789118862.622637 mapwrite 0x0309b4 thru 0x031dc1 (0x140e bytes) 000927 1789118862.635725 trunc from 0x00fb0c to 0x0174fc (0x79f1 bytes) 000929 1789118862.641685 read 0x01ef51 thru 0x023153 (0x4203 bytes) 000930 1789118862.648788 trunc from 0x040000 to 0x01f140 (0x20ebf bytes) 000931 1789118862.667635 trunc from 0x01f141 to 0x017bae (0x7592 bytes) 000932 1789118862.688880 mapread 0x0014ab thru 0x0095ac (0x8102 bytes) 000933 1789118862.699845 punch from 0x00ab0f to 0x0100cc (0x55be bytes) 000934 1789118862.707502 mapwrite 0x0107d0 thru 0x0152b0 (0x4ae1 bytes) 000935 1789118862.734166 trunc from 0x017baf to 0x02357a (0xb9cc bytes) 000936 1789118862.748734 fallocate 0x025f72 thru 0x033af4 (0xdb83 bytes) 000937 1789118862.754625 trunc from 0x00faf3 to 0x0174a9 (0x79b7 bytes) 000938 1789118862.761678 fallocate 0x024ba2 thru 0x02cf0b (0x836a bytes) 000940 1789118862.773398 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2357b): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2357b): Mirror 16: 0x4b74ea79 (stale) Mirror 20: 0x4b74ea79 (stale) Mirror 21: 0x4b74ea79 (stale) Mirror 22: 0x4b74ea79 (stale) Mirror 23: 0x4b74ea79 (stale) Mirror 25: 0x4b74ea79 (stale) Mirror 26: 0x4b74ea79 (stale) Mirror 27: 0x683149bc (stale) Mirror 28: 0x5dc15ec (stale) CRC-32 checksum value for chunk [0, 0x2357b): Mirror 1: 0xfea4b918 Mirror 29: 0xfea4b918 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2357b: skipped 000944 1789118864.056645 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2357b): Mirror 16: 0x4b74ea79 (stale) Mirror 20: 0x4b74ea79 (stale) Mirror 22: 0x4b74ea79 (stale) Mirror 23: 0x4b74ea79 (stale) Mirror 25: 0x4b74ea79 (stale) Mirror 26: 0x4b74ea79 (stale) Mirror 27: 0x683149bc (stale) Mirror 28: 0x5dc15ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2357b): Mirror 16: 0x4b74ea79 (stale) Mirror 20: 0x4b74ea79 (stale) Mirror 22: 0x4b74ea79 (stale) Mirror 23: 0x4b74ea79 (stale) Mirror 25: 0x4b74ea79 (stale) Mirror 26: 0x4b74ea79 (stale) Mirror 27: 0x683149bc (stale) Mirror 28: 0x5dc15ec (stale) CRC-32 checksum value for chunk [0, 0x2357b): Mirror 1: 0xfea4b918 Mirror 29: 0xfea4b918 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2357b: skipped 000946 1789118865.034004 trunc from 0x015fc8 to 0x021967 (0xb9a0 bytes) 000947 1789118865.111265 punch from 0x0129c9 to 0x015cff (0x3337 bytes) 000948 1789118865.125153 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2357b): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2357b): Mirror 16: 0x4b74ea79 (stale) Mirror 20: 0x4b74ea79 (stale) Mirror 22: 0x4b74ea79 (stale) Mirror 23: 0x4b74ea79 (stale) Mirror 25: 0x4b74ea79 (stale) Mirror 26: 0x4b74ea79 (stale) Mirror 27: 0x683149bc (stale) Mirror 28: 0x5dc15ec (stale) Mirror 29: 0xfea4b918 (stale) CRC-32 checksum value for chunk [0, 0x2357b): Mirror 1: 0xfea4b918 Mirror 30: 0xfea4b918 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2357b: skipped 000949 1789118866.515624 punch from 0x00a457 to 0x014719 (0xa2c3 bytes) 000952 1789118866.573266 mapwrite 0x024a50 thru 0x02a2ea (0x589b bytes) 000954 1789118866.644733 trunc from 0x023e4e to 0x024b23 (0xcd6 bytes) 000955 1789118866.650185 read 0x0275a1 thru 0x02a2ea (0x2d4a bytes) 000956 1789118866.651123 punch from 0x003346 to 0x00d071 (0x9d2c bytes) 000957 1789118866.660923 trunc from 0x011d2d to 0x01390a (0x1bde bytes) 000958 1789118866.670719 MIRROR_RESYNC 11 mirrors 000959 1789118868.559610 fallocate 0x006ba1 thru 0x006fa6 (0x406 bytes) 000960 1789118868.612098 write 0x03e545 thru 0x03ffff (0x1abb bytes) 000961 1789118868.621205 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xf33c5d69 (stale) Mirror 20: 0xf33c5d69 (stale) Mirror 22: 0xf33c5d69 (stale) Mirror 23: 0xf33c5d69 (stale) Mirror 25: 0xf33c5d69 (stale) Mirror 26: 0xf33c5d69 (stale) Mirror 27: 0xf33c5d69 (stale) Mirror 28: 0xf33c5d69 (stale) Mirror 29: 0xf33c5d69 (stale) Mirror 30: 0xf33c5d69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa95ce355 Mirror 31: 0xa95ce355 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000962 1789118870.286275 trunc from 0x036098 to 0x036434 (0x39d bytes) 000963 1789118870.336854 punch from 0x037404 to 0x03ffff (0x8bfc bytes) 000964 1789118870.343254 fallocate 0x000128 thru 0x00e4bb (0xe394 bytes) 000966 1789118870.349356 fallocate 0x01337c thru 0x01ff86 (0xcc0b bytes) 000968 1789118870.358691 trunc from 0x01d51d to 0x02cb27 (0xf60b bytes) 000969 1789118870.363695 trunc from 0x040000 to 0x00bab9 (0x34546 bytes) 000970 1789118870.398617 punch from 0x00198a to 0x00bab9 (0xa130 bytes) 000971 1789118870.405602 punch from 0x00b052 to 0x00bab9 (0xa68 bytes) 000972 1789118870.415836 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbaba): Mirror 16: 0xef294c48 (stale) Mirror 20: 0xef294c48 (stale) Mirror 22: 0xef294c48 (stale) Mirror 23: 0xef294c48 (stale) Mirror 26: 0xef294c48 (stale) Mirror 27: 0xef294c48 (stale) Mirror 28: 0xef294c48 (stale) Mirror 29: 0xef294c48 (stale) Mirror 30: 0xef294c48 (stale) Mirror 31: 0xef294c48 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbaba): Mirror 16: 0xef294c48 (stale) Mirror 20: 0xef294c48 (stale) Mirror 22: 0xef294c48 (stale) Mirror 23: 0xef294c48 (stale) Mirror 26: 0xef294c48 (stale) Mirror 27: 0xef294c48 (stale) Mirror 28: 0xef294c48 (stale) Mirror 29: 0xef294c48 (stale) Mirror 30: 0xef294c48 (stale) Mirror 31: 0xef294c48 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbaba: skipped 000974 1789118870.943603 punch from 0x007b92 to 0x00ab15 (0x2f84 bytes) 000975 1789118870.948862 punch from 0x00b394 to 0x00bab9 (0x726 bytes) 000976 1789118870.954344 read 0x00ac3f thru 0x00bab9 (0xe7b bytes) 000977 1789118870.960980 trunc from 0x00baba to 0x03f6c8 (0x33c0f bytes) 000978 1789118870.987873 read 0x00daa3 thru 0x01042a (0x2988 bytes) 000979 1789118871.000964 trunc from 0x015058 to 0x016556 (0x14ff bytes) 000980 1789118871.006018 read 0x00329d thru 0x010d45 (0xdaa9 bytes) 000981 1789118871.006705 trunc from 0x03a30e to 0x03f6c8 (0x53bb bytes) 000982 1789118871.014280 trunc from 0x004854 to 0x00a79d (0x5f4a bytes) 000983 1789118871.025259 MIRROR_RESYNC 11 mirrors 000984 1789118872.628798 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000985 1789118873.296093 mapwrite 0x006424 thru 0x012a3f (0xc61c bytes) 000988 1789118873.452183 punch from 0x00fd24 to 0x015430 (0x570d bytes) 000989 1789118873.457970 mapwrite 0x037a0b thru 0x03fc41 (0x8237 bytes) 000990 1789118873.518498 MIRROR_RESYNC 10 mirrors 000991 1789118876.059125 fallocate 0x0081e4 thru 0x015914 (0xd731 bytes) 000992 1789118876.137267 punch from 0x023330 to 0x032090 (0xed61 bytes) 000993 1789118876.155847 write 0x024457 thru 0x0277a4 (0x334e bytes) 000994 1789118876.175966 punch from 0x03db72 to 0x03fc41 (0x20d0 bytes) 000995 1789118876.180242 mapwrite 0x03d35e thru 0x03ffff (0x2ca2 bytes) 000996 1789118876.297819 fallocate 0x034776 thru 0x03488b (0x116 bytes) 000997 1789118876.307685 mapread 0x03c27c thru 0x03ffff (0x3d84 bytes) 000998 1789118876.316422 read 0x028790 thru 0x02d6dc (0x4f4d bytes) 000999 1789118876.335145 trunc from 0x036072 to 0x03ae83 (0x4e12 bytes) 001000 1789118876.354273 punch from 0x00ca3d to 0x01361b (0x6bdf bytes) All operations completed A-OK! PASS 70a (159s) == sanity-flr test 71: check mirror extend parent fid ==== 05:28:02 (1789118882) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x55:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 05:28:13 (1789118893) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181243 s, 11.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882944 s, 11.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 05:28:22 (1789118902) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 921760 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1093920 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5915392 .. failed Extending file size to 3074848 .. Extending file size to 7844128 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3910944 .. Extending file size to 4753408 .. Extending file size to 6528576 .. Extending file size to 8647264 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8195392 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8331136 .. Extending file size to 8765504 .. Extending file size to 8927936 .. Extending file size to 8783200 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9038528 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9221888 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 62929 62930 62931 62933 62936 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) 192.168.202.159@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:29:44 (1789118984) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5633984 .. Extending file size to 4143520 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6992384 .. Extending file size to 7196832 .. Extending file size to 6986240 .. Extending file size to 8213792 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7249024 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8141056 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8399072 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8906496 .. Extending file size to 8785440 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 64528 64529 64530 64532 64533 failed 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) 192.168.202.159@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:31:05 (1789119065) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1268980 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2288 1269016 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17588 3589432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25324 3581696 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42912 7171128 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1268980 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2288 1269016 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17588 3589432 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25324 3581696 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42912 7171128 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0913388 s, 23.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.308257 s, 6.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xa6ff1292 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:31:31 (1789119091) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145578 s, 14.4 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 05:31:40 (1789119100) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:31:49 (1789119109) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130941 s, 16.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:31:57 (1789119117) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103877 s, 20.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:32:07 (1789119127) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178245 s, 11.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fe:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:32:42 (1789119162) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556711 s, 18.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x138:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:33:34 (1789119214) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126385 s, 16.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:34:22 (1789119262) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg259-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg259-server: Pool lustre.test_204f created Adding targets to pool oleg259-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg259-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122201 s, 17.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x139:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg259-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg259-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg259-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:35:43 (1789119343) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:35:52 (1789119352) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:36:00 (1789119360) oleg259-server: Pool lustre.test_206 created Adding targets to pool oleg259-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg259-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg259-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg259-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg259-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:36:29 (1789119389) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833753 s, 12.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085342 s, 12.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xef58e44e Mirror 3: 0xef58e44e PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:36:39 (1789119399) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:36:42 (1789119402) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:36:46 (1789119406) Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0631226 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1000000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1000000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1000000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x17a0000 Read this (4 bytes): BBBB 192.168.202.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg259-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:37:26 (1789119446) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0370375 s, 221 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:37:34 (1789119454) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044718 s, 23.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x143:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 05:37:36.000000000 -0400 Modify: 2026-09-11 05:37:36.000000000 -0400 Change: 2026-09-11 05:37:37.000000000 -0400 Birth: 2026-09-11 05:37:36.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:37:44 (1789119464) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031875 s, 32.9 MB/s oleg259-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg259-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg259-server: O/280000401/d4/144: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:38:07 (1789119487) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.227739 s, 180 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0213931 s, 191 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 05:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 05:38 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:38:15 (1789119495) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:38:23 (1789119503) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2059 sec =========== 05:38:33 (1789119513) osc.lustre-OST0000-osc-ffff93ea08e9d800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff93ea08e9d800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789119517.222300064 secs.nsecs start_time 1789117989.773646789 secs.nsecs elapsed_time 1527.448653275 secs.nsecs uncompressed_objects 3137 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789119517.222366177 secs.nsecs start_time 1789118037.683108725 secs.nsecs elapsed_time 1479.539257452 secs.nsecs uncompressed_objects 3115 samples [reqs] === sanity-flr: start cleanup 05:38:37 (1789119517) === === sanity-flr: finish cleanup 05:38:44 (1789119524) ===