-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 06:02:24 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg647-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:02:41 (1789120961) === oleg647-client.virtnet: executing check_config_client /mnt/lustre oleg647-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg647-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d05d8367800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d05d8367800.idle_timeout=debug disable quota as required oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Setting lod.lustre-MDT0001-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanity-flr: finish setup 06:03:25 (1789121005) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8253] == sanity-flr test 0a: lfs mirror create with -N option == 06:03:28 (1789121008) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:03:42 (1789121022) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:03:45 (1789121025) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:03:48 (1789121028) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:04:10 (1789121050) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:04:13 (1789121053) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:04:17 (1789121057) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9d05d8367800.stats=clear osc.lustre-OST0001-osc-ffff9d05d8367800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:04:24 (1789121064) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:04:34 (1789121074) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:04:40 (1789121080) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:04:42 (1789121082) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:04:49 (1789121089) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 06:04:58 (1789121098) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:05:06 (1789121106) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:05:13 (1789121113) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:05:20 (1789121120) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:05:28 (1789121128) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:05:34 (1789121134) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207472 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075125 s, 14.0 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:05:41 (1789121141) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.673199 s, 31.2 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:05:50 (1789121150) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:05:57 (1789121157) Stopping /mnt/lustre-ost1 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d05d8367800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d05d8367800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg647-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d05d8367800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d05d8367800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg647-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (61s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:06:58 (1789121218) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.483372 s, 19.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d05d8367800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d05d8367800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg647-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:07:37 (1789121257) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d05d8367800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d05d8367800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg647-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d05d8367800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d05d8367800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg647-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:08:53 (1789121333) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.1552 s, 24.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 06:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.746s user 0m0.001s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d05d8367800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d05d8367800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.890s user 0m0.000s sys 0m0.178s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg647-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg647-server oleg647-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d05d8367800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d05d8367800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.404s user 0m0.000s sys 0m0.203s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg647-server: oleg647-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg647-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg647-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d05d8367800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d05d8367800.ost_server_uuid in FULL state after 0 sec PASS 33b (83s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:10:16 (1789121416) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:10:20 (1789121420) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:10:22 (1789121422) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:10:25 (1789121425) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:10:32 (1789121432) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.741689 s, 17.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:10:44 (1789121444) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292931 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257995 s, 40.6 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:10:50 (1789121450) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:10:57 (1789121457) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00185839 s, 551 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00136206 s, 752 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000768828 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000719121 s, 1.4 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 06:11:05 (1789121465) create test files with size 15742755 .. 6c797cf9f0f8d9e4e3f23cbaa38f7719 - 7dced15e239811eee78c991a5151906e - 93ddd677b70f97a4c1f748fb9c4de1ad - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg647-client.virtnet /mnt/lustre (opts:) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 06:11:24 (1789121484) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:11:40 (1789121500) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:11:48 (1789121508) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592435 s, 17.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221045 s, 47.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 06:11:56 (1789121516) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114464 s, 18.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.216658 s, 19.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106434 s, 19.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 06:12:08 (1789121528) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:12:11 (1789121531) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:12:14 (1789121534) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036617 s, 28.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 43c: read prefer uncompressed mirror == 06:12:20 (1789121540) llite.lustre-ffff9d05e1317800.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 06:12:27 (1789121547) llite.lustre-ffff9d05e1317800.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395167 s, 26.5 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248946 s, 42.1 MB/s **verify components PASS 43d (6s) == sanity-flr test 44a: lfs mirror split check =========== 06:12:33 (1789121553) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.209816 s, 15.0 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:12:41 (1789121561) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:12:52 (1789121572) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.491422 s, 21.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg647-client.virtnet /mnt/lustre (opts:) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:13:03 (1789121583) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9d05d2ac3000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.16781 s, 62.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg647-client.virtnet /mnt/lustre (opts:) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9d05e01b3000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:13:19 (1789121599) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:14:02 (1789121642) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:14:40 (1789121680) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:14:44 (1789121684) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.203054 s, 15.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.19652 s, 16.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5944e7ae (nosync,stale) Mirror 2: 0x5944e7ae (nosync,stale) ** verify mirror 2 doesn't change original checksum: d4e2fa80539ea8e71dc851934e2cea5e - mirror 1 checksum: 0ed7f6c34ca072da5c1044ee2e219b1b - mirror 2 checksum: d4e2fa80539ea8e71dc851934e2cea5e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T06:14:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T06:14:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d4e2fa80539ea8e71dc851934e2cea5e - mirror 1 checksum: 0ed7f6c34ca072da5c1044ee2e219b1b - mirror 2 checksum: 0ed7f6c34ca072da5c1044ee2e219b1b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T06:14:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T06:14:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:14:57 (1789121697) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487961 s, 21.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:15:05 (1789121705) Failing mds1 on oleg647-server Stopping /mnt/lustre-mds1 (opts:) on oleg647-server 06:15:17 (1789121717) shut down facet: mds1 facet_host: oleg647-server facet_failover_host: oleg647-server Failover mds1 to oleg647-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg647-server: oleg647-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0000 06:15:41 (1789121741) targets are mounted 06:15:41 (1789121741) facet_failover done oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252825 s, 41.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:15:56 (1789121756) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0527461 s, 388 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0271826 s, 753 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:16:06 (1789121766) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0993767 s, 206 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0326787 s, 627 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:16:17 (1789121777) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00485281 s, 3.4 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:16:23 (1789121783) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0792192 s, 259 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:16:30 (1789121790) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0116271 s, 352 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0228649 s, 179 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:16:37 (1789121797) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:17:00 (1789121820) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:17:28 (1789121848) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:17:52 (1789121872) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:18:16 (1789121896) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 06:18:23 (1789121903) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 59121 mirror split pid 59123 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 59121 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 59123 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 06:19:30 (1789121970) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 1972 skipping zero length zero range 000002 1789121972.860527 mapwrite 0x02c4d5 thru 0x02e7ff (0x232b bytes) 000004 1789121972.911492 mapwrite 0x0247d4 thru 0x0301dd (0xba0a bytes) 000005 1789121972.983902 punch from 0x01987a to 0x01eeaa (0x5631 bytes) 000006 1789121972.995733 punch from 0x02ef95 to 0x0301dd (0x1249 bytes) 000007 1789121973.005369 mapwrite 0x01d493 thru 0x021478 (0x3fe6 bytes) 000009 1789121973.039724 read 0x019619 thru 0x01caf6 (0x34de bytes) truncating to largest ever: 0x2a5eb 000010 1789121973.041706 trunc from 0x0301de to 0x02a5ea (0x5bf3 bytes) 000012 1789121973.070149 punch from 0x0215e8 to 0x0225e7 (0x1000 bytes) 000013 1789121973.076494 mapread 0x01ed12 thru 0x0243b7 (0x56a6 bytes) 000014 1789121973.083362 punch from 0x0191c3 to 0x01e09e (0x4edc bytes) 000015 1789121973.089435 mapread 0x019309 thru 0x01a797 (0x148f bytes) 000016 1789121973.109452 punch from 0x00f4a3 to 0x01b14d (0xbcab bytes) 000017 1789121973.125823 write 0x012b01 thru 0x0227d7 (0xfcd7 bytes) 000018 1789121973.135782 trunc from 0x0179c5 to 0x01d583 (0x5bbf bytes) 000019 1789121973.161532 trunc from 0x019d59 to 0x020b25 (0x6dcd bytes) 000020 1789121973.167257 trunc from 0x0144c8 to 0x0206fb (0xc234 bytes) 000021 1789121973.174420 mapwrite 0x018c04 thru 0x028b49 (0xff46 bytes) 000022 1789121973.211398 fallocate 0x01b6e5 thru 0x01d465 (0x1d81 bytes) 000023 1789121973.217177 write 0x02e975 thru 0x03a41c (0xbaa8 bytes) 000024 1789121973.220140 write 0x016837 thru 0x016908 (0xd2 bytes) 000025 1789121973.221162 mapwrite 0x016c9a thru 0x01b6bd (0x4a24 bytes) 000026 1789121973.259581 punch from 0x023d8f to 0x02709c (0x330e bytes) 000027 1789121973.274115 MIRROR_EXTEND to 2 mirrors 000029 1789121974.225421 MIRROR_RESYNC 2 mirrors 000030 1789121974.893029 write 0x01e608 thru 0x028a6b (0xa464 bytes) 000031 1789121974.952308 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a41d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a41d): Mirror 2: 0x20be1e7c (stale) CRC-32 checksum value for chunk [0, 0x3a41d): Mirror 1: 0xf1d6c5d4 Mirror 3: 0xf1d6c5d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a41d: skipped fallocating to largest ever: 0x2d5c5 000032 1789121975.951202 fallocate 0x0223bd thru 0x02d5c4 (0xb208 bytes) 000033 1789121976.007055 read 0x0268ef thru 0x02b402 (0x4b14 bytes) 000034 1789121976.021655 punch from 0x02728a to 0x0353e9 (0xe160 bytes) 000035 1789121976.029933 mapwrite 0x03e610 thru 0x03ffff (0x19f0 bytes) 000036 1789121976.097986 trunc from 0x0064c4 to 0x0093d1 (0x2f0e bytes) 000037 1789121976.109374 punch from 0x025ce9 to 0x032391 (0xc6a9 bytes) 000038 1789121976.118506 mapread 0x014579 thru 0x015aa4 (0x152c bytes) 000041 1789121976.152404 trunc from 0x019a4c to 0x023f0f (0xa4c4 bytes) 000042 1789121976.166163 read 0x00b244 thru 0x011a1a (0x67d7 bytes) 000043 1789121976.196805 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe594e4b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe594e4b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x37830 000044 1789121976.685234 trunc from 0x040000 to 0x03782f (0x87d0 bytes) 000045 1789121976.711496 write 0x0321e0 thru 0x032843 (0x664 bytes) 000046 1789121976.724845 write 0x008517 thru 0x011cb3 (0x979d bytes) 000047 1789121976.748572 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37830): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37830): Mirror 2: 0xe361ce1d (stale) CRC-32 checksum value for chunk [0, 0x37830): Mirror 1: 0x716cddcc Mirror 3: 0x716cddcc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37830: skipped 000049 1789121977.866938 mapwrite 0x01ccb9 thru 0x0215d2 (0x491a bytes) fallocating to largest ever: 0x40000 000050 1789121978.011072 fallocate 0x03adb7 thru 0x03ffff (0x5249 bytes) 000051 1789121978.029429 punch from 0x000903 to 0x004b8c (0x428a bytes) 000052 1789121978.039558 trunc from 0x02564b to 0x0332ef (0xdca5 bytes) 000053 1789121978.056433 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37830): Mirror 3: 0x716cddcc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37830): Mirror 3: 0x716cddcc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37830: skipped 000054 1789121978.412196 mapread 0x02c9ea thru 0x03228c (0x58a3 bytes) 000055 1789121978.434670 trunc from 0x02a6ca to 0x03782f (0xd166 bytes) 000056 1789121978.440416 punch from 0x001e41 to 0x003a28 (0x1be8 bytes) 000057 1789121978.444940 read 0x0152cd thru 0x019ef6 (0x4c2a bytes) 000058 1789121978.453346 mapread 0x01ba66 thru 0x023b89 (0x8124 bytes) 000059 1789121978.465524 write 0x004585 thru 0x00ace5 (0x6761 bytes) 000060 1789121978.481140 mapwrite 0x03e7c5 thru 0x03ffff (0x183b bytes) 000061 1789121978.537960 MIRROR_RESYNC 2 mirrors 000062 1789121979.395169 fallocate 0x03d6a9 thru 0x03ffff (0x2957 bytes) 000063 1789121979.428372 trunc from 0x039ad0 to 0x03ffff (0x6530 bytes) 000064 1789121979.434112 trunc from 0x0163eb to 0x017a10 (0x1626 bytes) 000065 1789121979.438800 punch from 0x01ca4a to 0x01d2ff (0x8b6 bytes) 000066 1789121979.447061 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x821667d7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf7c09144 Mirror 4: 0xf7c09144 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000067 1789121980.336891 fallocate 0x01025a thru 0x016ad7 (0x687e bytes) 000068 1789121980.424901 punch from 0x01eae3 to 0x02bb04 (0xd022 bytes) 000069 1789121980.432101 mapwrite 0x03ca0c thru 0x03ffff (0x35f4 bytes) 000070 1789121980.506342 trunc from 0x040000 to 0x03f8c3 (0x73c bytes) 000071 1789121980.590372 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8c4): Mirror 4: 0x7ebf863b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8c4): Mirror 4: 0x7ebf863b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8c4: skipped 000072 1789121981.003429 write 0x012e9c thru 0x01c742 (0x98a7 bytes) 000073 1789121981.024910 trunc from 0x00ed5e to 0x0191a5 (0xa448 bytes) 000074 1789121981.053968 mapread 0x03b708 thru 0x03f8c3 (0x41bc bytes) 000075 1789121981.078023 MIRROR_RESYNC 2 mirrors 000076 1789121981.835394 read 0x01bae5 thru 0x023705 (0x7c21 bytes) 000077 1789121981.848698 MIRROR_RESYNC 2 mirrors 000078 1789121982.283206 write 0x01e8cd thru 0x02df8c (0xf6c0 bytes) 000079 1789121982.392062 trunc from 0x03f680 to 0x03f8c3 (0x244 bytes) 000080 1789121982.405055 punch from 0x02c8d9 to 0x03882d (0xbf55 bytes) 000081 1789121982.448280 punch from 0x02ddf2 to 0x035a48 (0x7c57 bytes) 000082 1789121982.459400 trunc from 0x028c21 to 0x02e5ea (0x59ca bytes) 000083 1789121982.484666 write 0x0181cd thru 0x0249e1 (0xc815 bytes) 000084 1789121982.496852 fallocate 0x00d3fb thru 0x00f7f9 (0x23ff bytes) 000085 1789121982.502165 write 0x007f96 thru 0x00c3d1 (0x443c bytes) 000086 1789121982.515370 fallocate 0x03fa3f thru 0x03ffff (0x5c1 bytes) 000087 1789121982.529681 read 0x00ba61 thru 0x01848b (0xca2b bytes) 000088 1789121982.555044 trunc from 0x040000 to 0x017133 (0x28ecc bytes) 000089 1789121982.631933 read 0x002d19 thru 0x00d87b (0xab63 bytes) 000090 1789121982.647237 mapread 0x0162d5 thru 0x017133 (0xe5f bytes) 000091 1789121982.650869 fallocate 0x00f2bb thru 0x013c7c (0x49c2 bytes) 000092 1789121982.658294 fallocate 0x020738 thru 0x023885 (0x314e bytes) 000093 1789121982.665053 fallocate 0x000c9f thru 0x0014fc (0x85e bytes) 000094 1789121982.681192 write 0x020018 thru 0x025f8f (0x5f78 bytes) 000095 1789121982.733938 trunc from 0x025f90 to 0x0178ba (0xe6d5 bytes) 000096 1789121982.773854 fallocate 0x01f248 thru 0x028c99 (0x9a52 bytes) 000097 1789121982.786286 mapread 0x02888b thru 0x028c99 (0x40f bytes) 000100 1789121982.843681 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000101 1789121983.054810 punch from 0x00866c to 0x013002 (0xa997 bytes) 000103 1789121983.061740 punch from 0x02601f to 0x028c99 (0x2c7b bytes) 000104 1789121983.066400 read 0x00806f thru 0x014026 (0xbfb8 bytes) 000105 1789121983.078183 trunc from 0x02351c to 0x027ad0 (0x45b5 bytes) 000106 1789121983.083084 punch from 0x003956 to 0x0083f4 (0x4a9f bytes) 000107 1789121983.088297 trunc from 0x0183e8 to 0x01e033 (0x5c4c bytes) 000109 1789121983.098296 MIRROR_EXTEND to 2 mirrors 000110 1789121984.169098 write 0x009533 thru 0x00d8fd (0x43cb bytes) 000111 1789121984.222341 punch from 0x01de40 to 0x028c99 (0xae5a bytes) 000113 1789121984.231000 mapwrite 0x00b4b9 thru 0x0150da (0x9c22 bytes) 000115 1789121984.287566 trunc from 0x026b0a to 0x028c99 (0x2190 bytes) 000116 1789121984.293027 trunc from 0x005885 to 0x006c93 (0x140f bytes) 000118 1789121984.298337 write 0x00e7d8 thru 0x010cbf (0x24e8 bytes) 000119 1789121984.299835 mapread 0x00510b thru 0x0106c4 (0xb5ba bytes) 000120 1789121984.309091 punch from 0x0064c8 to 0x00fbed (0x9726 bytes) 000121 1789121984.318412 fallocate 0x007983 thru 0x00c194 (0x4812 bytes) 000122 1789121984.332239 fallocate 0x01368e thru 0x016e08 (0x377b bytes) 000123 1789121984.340438 punch from 0x0153e7 to 0x018c57 (0x3871 bytes) 000124 1789121984.349255 trunc from 0x028c9a to 0x00b41b (0x1d87e bytes) 000126 1789121984.371008 mapread 0x003177 thru 0x00500c (0x1e96 bytes) 000127 1789121984.372018 trunc from 0x00b41c to 0x0101f3 (0x4dd8 bytes) 000128 1789121984.387966 punch from 0x00b1f0 to 0x0101f3 (0x5004 bytes) 000129 1789121984.396977 MIRROR_RESYNC 2 mirrors 000130 1789121985.145582 mapread 0x00e037 thru 0x0101f3 (0x21bd bytes) 000131 1789121985.179010 read 0x00eb90 thru 0x0101f3 (0x1664 bytes) 000132 1789121985.179784 read 0x00c395 thru 0x0101f3 (0x3e5f bytes) 000133 1789121985.185790 MIRROR_RESYNC 2 mirrors 000134 1789121985.541880 read 0x008027 thru 0x00a387 (0x2361 bytes) 000135 1789121985.550065 mapwrite 0x02c87a thru 0x02d292 (0xa19 bytes) 000136 1789121985.646380 MIRROR_RESYNC 2 mirrors 000137 1789121986.128021 write 0x00816e thru 0x010f55 (0x8de8 bytes) 000138 1789121986.184249 mapwrite 0x028c0f thru 0x033ed9 (0xb2cb bytes) 000140 1789121986.285176 read 0x02b631 thru 0x033ed9 (0x88a9 bytes) 000141 1789121986.286206 trunc from 0x033eda to 0x029f59 (0x9f80 bytes) 000142 1789121986.307352 mapread 0x0291b2 thru 0x029f59 (0xda8 bytes) 000143 1789121986.308436 fallocate 0x015791 thru 0x024973 (0xf1e3 bytes) 000145 1789121986.315571 mapread 0x00cb05 thru 0x00fbe5 (0x30e1 bytes) 000146 1789121986.317188 mapread 0x015427 thru 0x0219a1 (0xc57b bytes) 000147 1789121986.342969 mapread 0x0092d4 thru 0x018d2d (0xfa5a bytes) 000148 1789121986.344249 trunc from 0x010390 to 0x01765e (0x72cf bytes) 000149 1789121986.350123 write 0x02dfee thru 0x03c8d1 (0xe8e4 bytes) 000150 1789121986.353405 trunc from 0x03c8d2 to 0x024c25 (0x17cac bytes) 000151 1789121986.375760 write 0x02881e thru 0x0299e0 (0x11c3 bytes) 000152 1789121986.377850 fallocate 0x008704 thru 0x00f01c (0x6919 bytes) 000153 1789121986.383578 mapread 0x006df0 thru 0x00850f (0x1720 bytes) 000154 1789121986.395926 write 0x01a4e2 thru 0x02608b (0xbbaa bytes) 000156 1789121986.408696 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x299e1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x299e1): Mirror 2: 0xf66a92c (stale) CRC-32 checksum value for chunk [0, 0x299e1): Mirror 1: 0x670de4db Mirror 3: 0x670de4db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x299e1: skipped 000157 1789121987.478525 trunc from 0x01f1f4 to 0x0299e0 (0xa7ed bytes) 000158 1789121987.554220 mapread 0x01f4c6 thru 0x025f53 (0x6a8e bytes) 000159 1789121987.583271 trunc from 0x024444 to 0x0299e0 (0x559d bytes) 000160 1789121987.590197 read 0x008580 thru 0x016692 (0xe113 bytes) 000161 1789121987.607757 MIRROR_RESYNC 3 mirrors 000163 1789121988.567714 trunc from 0x0299e1 to 0x02422d (0x57b3 bytes) 000164 1789121988.621853 mapread 0x01d020 thru 0x02422d (0x720e bytes) 000165 1789121988.653383 fallocate 0x026b89 thru 0x02ed04 (0x817c bytes) 000166 1789121988.665099 read 0x009133 thru 0x015a94 (0xc962 bytes) 000167 1789121988.674866 write 0x03478e thru 0x03ffff (0xb872 bytes) 000168 1789121988.679413 mapwrite 0x00d34b thru 0x019af7 (0xc7ad bytes) 000169 1789121988.714547 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce8f04a1 (stale) Mirror 3: 0xce8f04a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x21641f06 Mirror 4: 0x21641f06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000170 1789121990.008200 trunc from 0x02f104 to 0x033f88 (0x4e85 bytes) 000171 1789121990.087770 fallocate 0x027d49 thru 0x02924e (0x1506 bytes) 000173 1789121990.098214 mapwrite 0x02e5f4 thru 0x03bd31 (0xd73e bytes) 000174 1789121990.149887 mapread 0x023f90 thru 0x029546 (0x55b7 bytes) 000176 1789121990.169831 MIRROR_RESYNC 4 mirrors 000177 1789121991.139795 write 0x02d856 thru 0x03c960 (0xf10b bytes) 000178 1789121991.186815 trunc from 0x040000 to 0x002538 (0x3dac7 bytes) 000179 1789121991.224503 trunc from 0x002539 to 0x017580 (0x15048 bytes) 000180 1789121991.238158 trunc from 0x017581 to 0x02e7b7 (0x17237 bytes) 000181 1789121991.257572 punch from 0x01f188 to 0x023613 (0x448c bytes) 000182 1789121991.273559 read 0x00a052 thru 0x01865d (0xe60c bytes) 000183 1789121991.284511 mapwrite 0x03c39c thru 0x03f086 (0x2ceb bytes) 000184 1789121991.361192 punch from 0x02d173 to 0x02ee58 (0x1ce6 bytes) 000186 1789121991.367823 read 0x014da6 thru 0x0167e0 (0x1a3b bytes) 000187 1789121991.373011 fallocate 0x001c89 thru 0x00c89f (0xac17 bytes) 000188 1789121991.379499 mapread 0x02ce7c thru 0x035ad8 (0x8c5d bytes) 000190 1789121991.410545 read 0x003a89 thru 0x0130be (0xf636 bytes) 000191 1789121991.412074 fallocate 0x01a9ee thru 0x028ac4 (0xe0d7 bytes) 000192 1789121991.421937 fallocate 0x018dc0 thru 0x01a347 (0x1588 bytes) 000193 1789121991.435045 mapwrite 0x00d06b thru 0x01c980 (0xf916 bytes) 000194 1789121991.483934 punch from 0x02eb1f to 0x02ffc2 (0x14a4 bytes) 000195 1789121991.497837 write 0x02a273 thru 0x0365c7 (0xc355 bytes) 000196 1789121991.499513 trunc from 0x03f087 to 0x009fe5 (0x350a1 bytes) 000197 1789121991.553007 punch from 0x00322a to 0x009fe5 (0x6dbc bytes) 000198 1789121991.568308 fallocate 0x01f78e thru 0x0279fc (0x826f bytes) 000199 1789121991.590420 punch from 0x024a10 to 0x0279fc (0x2fed bytes) 000201 1789121991.599594 read 0x00c6ef thru 0x012837 (0x6149 bytes) 000202 1789121991.611309 trunc from 0x0279fd to 0x02fb0f (0x8113 bytes) 000203 1789121991.635507 punch from 0x009079 to 0x010aab (0x7a33 bytes) 000204 1789121991.641726 fallocate 0x02754b thru 0x036b8c (0xf642 bytes) 000205 1789121991.659927 MIRROR_RESYNC 4 mirrors 000206 1789121992.677086 fallocate 0x01c06f thru 0x027600 (0xb592 bytes) 000207 1789121992.766598 write 0x032e84 thru 0x034bfb (0x1d78 bytes) 000208 1789121992.769149 trunc from 0x02cf03 to 0x032c02 (0x5d00 bytes) 000209 1789121992.790590 mapwrite 0x039207 thru 0x03d13c (0x3f36 bytes) 000210 1789121992.940134 trunc from 0x0016b6 to 0x00c7a9 (0xb0f4 bytes) 000211 1789121992.949123 fallocate 0x01fe92 thru 0x02bd9c (0xbf0b bytes) 000213 1789121992.960989 write 0x022290 thru 0x03199a (0xf70b bytes) 000214 1789121992.989312 trunc from 0x03d13d to 0x01e236 (0x1ef06 bytes) 000215 1789121993.033703 mapwrite 0x01536d thru 0x016540 (0x11d4 bytes) 000216 1789121993.082860 trunc from 0x01e237 to 0x020527 (0x22f1 bytes) 000218 1789121993.110260 trunc from 0x0041ef to 0x0077de (0x35f0 bytes) 000219 1789121993.115355 punch from 0x00943a to 0x00a651 (0x1218 bytes) 000220 1789121993.121251 trunc from 0x01d1b8 to 0x020527 (0x3370 bytes) 000221 1789121993.145443 mapwrite 0x01e7c9 thru 0x01fb8b (0x13c3 bytes) 000223 1789121993.194416 write 0x03a633 thru 0x03ffff (0x59cd bytes) 000224 1789121993.198082 mapread 0x01366b thru 0x01cb9a (0x9530 bytes) 000225 1789121993.200314 read 0x0324e7 thru 0x03ffff (0xdb19 bytes) 000226 1789121993.209986 read 0x00287b thru 0x003391 (0xb17 bytes) 000227 1789121993.224934 read 0x01aff3 thru 0x01e70d (0x371b bytes) 000228 1789121993.225849 read 0x020617 thru 0x025055 (0x4a3f bytes) 000229 1789121993.227634 punch from 0x01e679 to 0x02a80f (0xc197 bytes) 000230 1789121993.237002 trunc from 0x040000 to 0x031a39 (0xe5c6 bytes) 000231 1789121993.260288 mapread 0x012af0 thru 0x01a7b6 (0x7cc7 bytes) 000232 1789121993.261861 fallocate 0x01bc74 thru 0x0298a9 (0xdc36 bytes) 000233 1789121993.270037 mapread 0x011645 thru 0x01b690 (0xa04c bytes) 000234 1789121993.272223 trunc from 0x01f036 to 0x020938 (0x1903 bytes) 000235 1789121993.277968 trunc from 0x02c834 to 0x031a39 (0x5206 bytes) 000236 1789121993.284103 mapwrite 0x024648 thru 0x031d0e (0xd6c7 bytes) 000237 1789121993.373102 write 0x006423 thru 0x010aa9 (0xa687 bytes) 000238 1789121993.374687 fallocate 0x00a2ad thru 0x00baed (0x1841 bytes) 000239 1789121993.420543 trunc from 0x031d0f to 0x029a99 (0x8275 bytes) 000240 1789121993.450961 mapwrite 0x00d754 thru 0x01a777 (0xd024 bytes) 000241 1789121993.511544 MIRROR_RESYNC 4 mirrors 000242 1789121994.835809 mapread 0x021158 thru 0x022131 (0xfda bytes) 000243 1789121994.851191 punch from 0x0146f9 to 0x015499 (0xda1 bytes) 000244 1789121994.919051 trunc from 0x029a9a to 0x03a759 (0x10cc0 bytes) 000245 1789121994.962265 mapwrite 0x003674 thru 0x0112ad (0xdc3a bytes) 000247 1789121995.047287 mapread 0x00a672 thru 0x00ea55 (0x43e4 bytes) 000248 1789121995.054292 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a75a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a75a): Mirror 2: 0xabb14b41 (stale) Mirror 3: 0xabb14b41 (stale) Mirror 4: 0xabb14b41 (stale) CRC-32 checksum value for chunk [0, 0x3a75a): Mirror 1: 0xe4b8476f Mirror 5: 0xe4b8476f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a75a: skipped 000249 1789121996.065863 write 0x034de3 thru 0x039041 (0x425f bytes) 000250 1789121996.109027 write 0x00682b thru 0x006873 (0x49 bytes) 000251 1789121996.114911 trunc from 0x03a75a to 0x0164a0 (0x242b9 bytes) 000252 1789121996.171973 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x164a1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x164a1): Mirror 2: 0xb6cf3e05 (stale) Mirror 3: 0xb6cf3e05 (stale) Mirror 4: 0xb6cf3e05 (stale) Mirror 5: 0x631fd8a0 (stale) CRC-32 checksum value for chunk [0, 0x164a1): Mirror 1: 0xb02178ea Mirror 6: 0xb02178ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x164a1: skipped 000253 1789121997.157677 read 0x003c48 thru 0x00ee49 (0xb202 bytes) 000254 1789121997.172200 mapwrite 0x02c23f thru 0x030098 (0x3e5a bytes) 000255 1789121997.289268 fallocate 0x02e761 thru 0x030c3b (0x24db bytes) 000257 1789121997.296818 trunc from 0x00442f to 0x005daa (0x197c bytes) 000258 1789121997.302232 punch from 0x02d686 to 0x030098 (0x2a13 bytes) 000259 1789121997.312272 punch from 0x02811f to 0x02d246 (0x5128 bytes) 000260 1789121997.325913 trunc from 0x030099 to 0x02b327 (0x4d71 bytes) 000261 1789121997.349303 trunc from 0x02b328 to 0x0377b6 (0xc48f bytes) 000262 1789121997.367281 mapread 0x01e1a5 thru 0x02ae21 (0xcc7d bytes) 000263 1789121997.383901 trunc from 0x029f22 to 0x02da89 (0x3b68 bytes) 000264 1789121997.388332 mapwrite 0x020ab7 thru 0x02edfe (0xe348 bytes) 000265 1789121997.414343 trunc from 0x0006c8 to 0x00fd9b (0xf6d4 bytes) 000266 1789121997.421124 write 0x00dc2e thru 0x00f005 (0x13d8 bytes) 000267 1789121997.432931 write 0x035d94 thru 0x03da08 (0x7c75 bytes) 000268 1789121997.440395 fallocate 0x0364e8 thru 0x03baab (0x55c4 bytes) 000269 1789121997.446129 write 0x031578 thru 0x03bf21 (0xa9aa bytes) 000270 1789121997.448798 write 0x0085ac thru 0x00bb7b (0x35d0 bytes) 000271 1789121997.457822 fallocate 0x030516 thru 0x03fc15 (0xf700 bytes) 000272 1789121997.466653 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3da09): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da09): Mirror 2: 0x4a88a82d (stale) Mirror 3: 0x4a88a82d (stale) Mirror 4: 0x4a88a82d (stale) Mirror 5: 0x8de1649b (stale) Mirror 6: 0x2578d361 (stale) CRC-32 checksum value for chunk [0, 0x3da09): Mirror 1: 0xac272a52 Mirror 7: 0xac272a52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da09: skipped 000273 1789121998.422105 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3da09): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da09): Mirror 2: 0x4a88a82d (stale) Mirror 3: 0x4a88a82d (stale) Mirror 4: 0x4a88a82d (stale) Mirror 5: 0x8de1649b (stale) Mirror 6: 0x2578d361 (stale) CRC-32 checksum value for chunk [0, 0x3da09): Mirror 1: 0xac272a52 Mirror 7: 0xac272a52 Mirror 8: 0xac272a52 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da09: skipped 000274 1789121999.447758 write 0x014632 thru 0x01990b (0x52da bytes) 000275 1789121999.540228 punch from 0x036908 to 0x038b20 (0x2219 bytes) 000276 1789121999.554502 mapread 0x03167b thru 0x03a7d7 (0x915d bytes) 000277 1789121999.601696 punch from 0x03d4bd to 0x03da08 (0x54c bytes) 000278 1789121999.612358 mapwrite 0x0153fb thru 0x024516 (0xf11c bytes) 000279 1789121999.699886 mapwrite 0x00fbd7 thru 0x01d047 (0xd471 bytes) 000280 1789121999.764964 mapwrite 0x00827d thru 0x0170ad (0xee31 bytes) 000281 1789121999.802660 trunc from 0x03da09 to 0x011c5d (0x2bdab bytes) 000282 1789121999.839571 trunc from 0x011c5e to 0x0345c3 (0x22966 bytes) 000283 1789121999.864137 mapwrite 0x0041a1 thru 0x0141a0 (0x10000 bytes) 000285 1789121999.912276 punch from 0x019edc to 0x01b5b9 (0x16de bytes) 000286 1789121999.920164 trunc from 0x0345c4 to 0x02ec4c (0x5977 bytes) 000289 1789121999.944331 fallocate 0x02e19a thru 0x030d94 (0x2bfb bytes) 000290 1789121999.963379 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30d95): Mirror 3: 0xc00eb9e8 (stale) Mirror 4: 0xc00eb9e8 (stale) Mirror 5: 0xa4bc6113 (stale) Mirror 6: 0xabc5c83a (stale) Mirror 7: 0x5f684117 (stale) Mirror 8: 0x5f684117 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d95): Mirror 3: 0xc00eb9e8 (stale) Mirror 4: 0xc00eb9e8 (stale) Mirror 5: 0xa4bc6113 (stale) Mirror 6: 0xabc5c83a (stale) Mirror 7: 0x5f684117 (stale) Mirror 8: 0x5f684117 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d95: skipped 000291 1789122000.392838 MIRROR_RESYNC 7 mirrors 000294 1789122001.760951 trunc from 0x003611 to 0x00801b (0x4a0b bytes) 000295 1789122001.814378 punch from 0x012376 to 0x01720c (0x4e97 bytes) 000296 1789122001.821144 mapread 0x019fb5 thru 0x021b77 (0x7bc3 bytes) 000297 1789122001.843787 write 0x0179c8 thru 0x01e1b0 (0x67e9 bytes) 000298 1789122001.846204 read 0x01504a thru 0x01deb3 (0x8e6a bytes) 000299 1789122001.848130 mapread 0x01550a thru 0x018e70 (0x3967 bytes) 000300 1789122001.849790 trunc from 0x021e0e to 0x0243f2 (0x25e5 bytes) 000301 1789122001.855267 mapwrite 0x037213 thru 0x03ffff (0x8ded bytes) 000302 1789122001.913381 read 0x03237d thru 0x035ac9 (0x374d bytes) 000304 1789122001.917659 mapwrite 0x00aa4a thru 0x01969c (0xec53 bytes) 000305 1789122001.972421 fallocate 0x010f70 thru 0x01e561 (0xd5f2 bytes) 000306 1789122001.988780 mapwrite 0x007b39 thru 0x00adf9 (0x32c1 bytes) 000307 1789122002.001093 read 0x01a466 thru 0x029c12 (0xf7ad bytes) 000308 1789122002.002272 punch from 0x032c61 to 0x039ec5 (0x7265 bytes) 000309 1789122002.007830 mapwrite 0x01deff thru 0x02c20e (0xe310 bytes) 000310 1789122002.029971 punch from 0x023211 to 0x0243eb (0x11db bytes) 000311 1789122002.039479 read 0x0133a3 thru 0x022e8a (0xfae8 bytes) 000312 1789122002.040188 write 0x01d100 thru 0x022f99 (0x5e9a bytes) 000313 1789122002.041958 read 0x038c50 thru 0x03915d (0x50e bytes) 000314 1789122002.058568 mapwrite 0x00dd97 thru 0x01a6f5 (0xc95f bytes) 000315 1789122002.094009 write 0x0237d7 thru 0x029de2 (0x660c bytes) 000316 1789122002.097987 mapread 0x00d9a9 thru 0x00f361 (0x19b9 bytes) 000317 1789122002.099520 mapread 0x0088b5 thru 0x010d2c (0x8478 bytes) 000318 1789122002.100499 mapread 0x0007ad thru 0x00bfaf (0xb803 bytes) 000319 1789122002.110071 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x33504a0d (stale) Mirror 4: 0x33504a0d (stale) Mirror 5: 0x33504a0d (stale) Mirror 6: 0x33504a0d (stale) Mirror 7: 0x33504a0d (stale) Mirror 8: 0x33504a0d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa4527551 Mirror 9: 0xa4527551 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000320 1789122003.363164 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x33504a0d (stale) Mirror 5: 0x33504a0d (stale) Mirror 6: 0x33504a0d (stale) Mirror 7: 0x33504a0d (stale) Mirror 8: 0x33504a0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x33504a0d (stale) Mirror 5: 0x33504a0d (stale) Mirror 6: 0x33504a0d (stale) Mirror 7: 0x33504a0d (stale) Mirror 8: 0x33504a0d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa4527551 Mirror 9: 0xa4527551 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000321 1789122004.116885 fallocate 0x018c94 thru 0x01a745 (0x1ab2 bytes) 000322 1789122004.176552 trunc from 0x040000 to 0x007d3e (0x382c1 bytes) 000323 1789122004.240824 read 0x000825 thru 0x007d3e (0x751a bytes) 000324 1789122004.258541 punch from 0x002a64 to 0x007d3e (0x52db bytes) 000325 1789122004.269851 fallocate 0x01f8ea thru 0x026612 (0x6d29 bytes) 000326 1789122004.275905 mapwrite 0x003471 thru 0x00faca (0xc65a bytes) 000327 1789122004.337924 fallocate 0x024315 thru 0x03037c (0xc068 bytes) 000328 1789122004.355892 MIRROR_RESYNC 7 mirrors 000329 1789122005.894929 fallocate 0x0280d5 thru 0x02dfcd (0x5ef9 bytes) 000330 1789122005.942039 punch from 0x00f31a to 0x013631 (0x4318 bytes) 000331 1789122005.951244 read 0x023d97 thru 0x02bdfa (0x8064 bytes) 000332 1789122005.975264 MIRROR_RESYNC 7 mirrors 000333 1789122007.476598 mapwrite 0x02a3da thru 0x034f38 (0xab5f bytes) 000335 1789122007.585412 fallocate 0x03ea9a thru 0x03ffff (0x1566 bytes) 000336 1789122007.595787 mapread 0x0148fb thru 0x01810f (0x3815 bytes) 000337 1789122007.619248 MIRROR_RESYNC 7 mirrors 000338 1789122009.225466 trunc from 0x01badb to 0x01c833 (0xd59 bytes) 000339 1789122009.291305 fallocate 0x03fa2a thru 0x03ffff (0x5d6 bytes) 000340 1789122009.295347 punch from 0x005242 to 0x010a4a (0xb809 bytes) 000341 1789122009.315881 MIRROR_RESYNC 7 mirrors 000342 1789122010.780040 mapread 0x00c389 thru 0x00d780 (0x13f8 bytes) 000343 1789122010.788933 punch from 0x02be44 to 0x03174b (0x5908 bytes) 000344 1789122010.847124 trunc from 0x040000 to 0x03c2e9 (0x3d16 bytes) 000345 1789122010.873219 mapwrite 0x00376c thru 0x00e258 (0xaaed bytes) 000346 1789122010.927724 fallocate 0x00f8a6 thru 0x01a367 (0xaac2 bytes) 000347 1789122010.936888 read 0x01f6b2 thru 0x02f50b (0xfe5a bytes) 000348 1789122010.944597 trunc from 0x03147d to 0x03a9c3 (0x9547 bytes) 000349 1789122010.949208 punch from 0x019252 to 0x02402c (0xaddb bytes) 000350 1789122010.953274 trunc from 0x03c2ea to 0x002275 (0x3a074 bytes) 000351 1789122010.967758 trunc from 0x00092a to 0x002275 (0x194c bytes) 000352 1789122010.979170 mapread 0x00004f thru 0x002275 (0x2227 bytes) 000353 1789122010.985824 mapwrite 0x01e30e thru 0x01ede5 (0xad8 bytes) 000354 1789122011.035088 read 0x003f3c thru 0x00a5d8 (0x669d bytes) 000355 1789122011.046240 trunc from 0x01ede6 to 0x004207 (0x1abde bytes) 000356 1789122011.060384 write 0x028c1c thru 0x02d26a (0x464f bytes) 000357 1789122011.061638 punch from 0x017488 to 0x01da46 (0x65bf bytes) 000358 1789122011.066359 trunc from 0x01bcdb to 0x028c84 (0xcfaa bytes) 000359 1789122011.079203 mapwrite 0x03aa77 thru 0x03ffff (0x5589 bytes) 000360 1789122011.132558 mapread 0x031f86 thru 0x034c27 (0x2ca2 bytes) 000361 1789122011.133420 trunc from 0x040000 to 0x02bf7a (0x14085 bytes) 000362 1789122011.152226 read 0x01c310 thru 0x028db0 (0xcaa1 bytes) 000363 1789122011.167099 read 0x02016f thru 0x026435 (0x62c7 bytes) 000364 1789122011.167855 write 0x01f7ad thru 0x026e7e (0x76d2 bytes) 000365 1789122011.174422 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2bf7b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf7b): Mirror 4: 0xe88e6e50 (stale) Mirror 5: 0xe88e6e50 (stale) Mirror 6: 0xe88e6e50 (stale) Mirror 7: 0xe88e6e50 (stale) Mirror 8: 0xe88e6e50 (stale) Mirror 9: 0xe88e6e50 (stale) CRC-32 checksum value for chunk [0, 0x2bf7b): Mirror 1: 0xcf55783a Mirror 10: 0xcf55783a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf7b: skipped 000366 1789122012.121894 mapwrite 0x00060d thru 0x006f4f (0x6943 bytes) 000367 1789122012.313166 read 0x00d02e thru 0x00d7dd (0x7b0 bytes) 000368 1789122012.314158 fallocate 0x014f74 thru 0x01e495 (0x9522 bytes) 000369 1789122012.321166 mapwrite 0x002fbd thru 0x00486c (0x18b0 bytes) 000370 1789122012.343874 fallocate 0x0189ed thru 0x0189fc (0x10 bytes) 000371 1789122012.348582 trunc from 0x02876b to 0x0288d1 (0x167 bytes) 000372 1789122012.353121 trunc from 0x02bf7b to 0x02bb38 (0x442 bytes) 000373 1789122012.398069 mapread 0x008333 thru 0x009176 (0xe44 bytes) 000374 1789122012.400036 mapread 0x021edb thru 0x02658f (0x46b5 bytes) 000375 1789122012.408795 fallocate 0x02d13f thru 0x02efcb (0x1e8d bytes) 000377 1789122012.414431 mapwrite 0x0030bb thru 0x01218e (0xf0d4 bytes) 000378 1789122012.451072 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2bb39): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bb39): Mirror 4: 0xc4abd681 (stale) Mirror 5: 0xc4abd681 (stale) Mirror 6: 0xc4abd681 (stale) Mirror 7: 0xc4abd681 (stale) Mirror 8: 0xc4abd681 (stale) Mirror 9: 0xc4abd681 (stale) Mirror 10: 0xc1b46781 (stale) CRC-32 checksum value for chunk [0, 0x2bb39): Mirror 1: 0xd11da82d Mirror 11: 0xd11da82d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bb39: skipped 000379 1789122013.416396 trunc from 0x02a890 to 0x02bb38 (0x12a9 bytes) 000380 1789122013.471756 mapread 0x012008 thru 0x014d06 (0x2cff bytes) 000381 1789122013.512016 trunc from 0x004975 to 0x00ec48 (0xa2d4 bytes) 000382 1789122013.519027 read 0x01aef8 thru 0x01e23d (0x3346 bytes) 000383 1789122013.530757 mapread 0x0172ce thru 0x020e38 (0x9b6b bytes) 000384 1789122013.545791 fallocate 0x01072c thru 0x01b9a1 (0xb276 bytes) 000385 1789122013.550804 fallocate 0x034a7b thru 0x03b73f (0x6cc5 bytes) 000386 1789122013.559958 punch from 0x00fea3 to 0x01d622 (0xd780 bytes) 000388 1789122013.573274 trunc from 0x03587c to 0x03b73f (0x5ec4 bytes) 000389 1789122013.581577 trunc from 0x03b740 to 0x028026 (0x13719 bytes) 000390 1789122013.632233 trunc from 0x01a7d8 to 0x01a820 (0x49 bytes) 000391 1789122013.637320 trunc from 0x028027 to 0x03d468 (0x15442 bytes) 000392 1789122013.654937 mapwrite 0x004f9e thru 0x009141 (0x41a4 bytes) 000393 1789122013.682753 mapwrite 0x00a02e thru 0x00f6b5 (0x5688 bytes) 000394 1789122013.700209 punch from 0x00d201 to 0x016b3b (0x993b bytes) 000395 1789122013.705408 mapread 0x00b0b6 thru 0x01443a (0x9385 bytes) 000396 1789122013.718074 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d469): Mirror 4: 0x70c0e183 (stale) Mirror 5: 0x70c0e183 (stale) Mirror 6: 0x70c0e183 (stale) Mirror 8: 0x70c0e183 (stale) Mirror 9: 0x70c0e183 (stale) Mirror 10: 0x3fe69726 (stale) Mirror 11: 0x91fe7e99 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d469): Mirror 4: 0x70c0e183 (stale) Mirror 5: 0x70c0e183 (stale) Mirror 6: 0x70c0e183 (stale) Mirror 8: 0x70c0e183 (stale) Mirror 9: 0x70c0e183 (stale) Mirror 10: 0x3fe69726 (stale) Mirror 11: 0x91fe7e99 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d469: skipped 000397 1789122014.274154 read 0x025c6a thru 0x02d5fd (0x7994 bytes) 000398 1789122014.282195 trunc from 0x001951 to 0x00d446 (0xbaf6 bytes) 000399 1789122014.287992 punch from 0x03a59c to 0x03d468 (0x2ecd bytes) 000400 1789122014.292916 write 0x01a33d thru 0x021c5e (0x7922 bytes) 000401 1789122014.305839 trunc from 0x03c10f to 0x03d468 (0x135a bytes) 000402 1789122014.309855 write 0x036f43 thru 0x03ffff (0x90bd bytes) 000403 1789122014.315645 trunc from 0x040000 to 0x000312 (0x3fced bytes) 000404 1789122014.339674 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x313): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x313): Mirror 4: 0xf686a0f9 (stale) Mirror 5: 0xf686a0f9 (stale) Mirror 6: 0xf686a0f9 (stale) Mirror 8: 0xf686a0f9 (stale) Mirror 9: 0xf686a0f9 (stale) Mirror 10: 0xf686a0f9 (stale) Mirror 11: 0xf686a0f9 (stale) CRC-32 checksum value for chunk [0, 0x313): Mirror 1: 0xf686a0f9 Mirror 12: 0xf686a0f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x313: skipped 000405 1789122015.416282 mapwrite 0x011b4f thru 0x012930 (0xde2 bytes) 000406 1789122015.616438 trunc from 0x012931 to 0x028249 (0x15919 bytes) 000407 1789122015.646768 write 0x025449 thru 0x02f90c (0xa4c4 bytes) 000409 1789122015.656996 fallocate 0x024e2f thru 0x02a31c (0x54ee bytes) 000410 1789122015.662496 trunc from 0x02f90d to 0x00ed18 (0x20bf4 bytes) 000414 1789122015.684898 mapwrite 0x03b2db thru 0x03ffff (0x4d25 bytes) 000415 1789122015.725466 punch from 0x02feb2 to 0x03f478 (0xf5c7 bytes) 000416 1789122015.734595 trunc from 0x040000 to 0x0130c2 (0x2cf3d bytes) 000417 1789122015.756447 read 0x00d2a2 thru 0x0130c2 (0x5e21 bytes) 000418 1789122015.767816 trunc from 0x00bb47 to 0x0130c2 (0x757c bytes) 000419 1789122015.773978 fallocate 0x02b6b4 thru 0x03966f (0xdfbc bytes) 000420 1789122015.781139 mapread 0x00ff12 thru 0x0130c2 (0x31b1 bytes) 000421 1789122015.791430 mapwrite 0x015729 thru 0x01ffd3 (0xa8ab bytes) 000422 1789122015.852851 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1ffd4): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ffd4): Mirror 4: 0x92073355 (stale) Mirror 5: 0x92073355 (stale) Mirror 6: 0x92073355 (stale) Mirror 8: 0x92073355 (stale) Mirror 9: 0x92073355 (stale) Mirror 10: 0x1dd1f0aa (stale) Mirror 11: 0x82cc94a4 (stale) Mirror 12: 0xcb8fee3b (stale) CRC-32 checksum value for chunk [0, 0x1ffd4): Mirror 1: 0x34018504 Mirror 13: 0x34018504 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ffd4: skipped 000423 1789122016.982318 write 0x013641 thru 0x018764 (0x5124 bytes) 000424 1789122017.063833 write 0x022451 thru 0x0262e1 (0x3e91 bytes) 000425 1789122017.066180 fallocate 0x001cd5 thru 0x0062bd (0x45e9 bytes) 000426 1789122017.084278 read 0x003ade thru 0x00fce6 (0xc209 bytes) 000428 1789122017.109928 write 0x02aa3c thru 0x02c30c (0x18d1 bytes) 000429 1789122017.113860 fallocate 0x00d59c thru 0x00ee27 (0x188c bytes) 000430 1789122017.124021 mapread 0x009e57 thru 0x014b17 (0xacc1 bytes) 000431 1789122017.165031 trunc from 0x02c30d to 0x03cf42 (0x10c36 bytes) 000432 1789122017.209405 MIRROR_RESYNC 10 mirrors 000433 1789122020.557584 fallocate 0x03b271 thru 0x03ffff (0x4d8f bytes) 000434 1789122020.621405 punch from 0x036166 to 0x03fc68 (0x9b03 bytes) 000435 1789122020.627207 trunc from 0x040000 to 0x0368f6 (0x9709 bytes) 000436 1789122020.646445 mapread 0x029c54 thru 0x02ff35 (0x62e2 bytes) 000437 1789122020.665582 trunc from 0x0368f7 to 0x006c2d (0x2fcc9 bytes) 000438 1789122020.681895 trunc from 0x00143d to 0x006c2d (0x57f1 bytes) 000439 1789122020.687271 trunc from 0x0062c6 to 0x006c2d (0x968 bytes) 000440 1789122020.692857 fallocate 0x018627 thru 0x023e05 (0xb7df bytes) 000441 1789122020.698043 punch from 0x00699f to 0x006c2d (0x28f bytes) 000442 1789122020.703058 mapread 0x00375c thru 0x006c2d (0x34d2 bytes) 000443 1789122020.716760 MIRROR_RESYNC 10 mirrors 000444 1789122022.131665 write 0x02c1cc thru 0x02ca04 (0x839 bytes) 000445 1789122022.177177 trunc from 0x02ca05 to 0x015937 (0x170cd bytes) 000446 1789122022.209113 trunc from 0x003eb4 to 0x012e2d (0xef7a bytes) 000447 1789122022.227771 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x15938): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15938): Mirror 4: 0x323b8366 (stale) Mirror 5: 0x323b8366 (stale) Mirror 6: 0x323b8366 (stale) Mirror 8: 0x323b8366 (stale) Mirror 9: 0x323b8366 (stale) Mirror 10: 0x323b8366 (stale) Mirror 11: 0x323b8366 (stale) Mirror 12: 0x323b8366 (stale) Mirror 13: 0x323b8366 (stale) CRC-32 checksum value for chunk [0, 0x15938): Mirror 1: 0x323b8366 Mirror 14: 0x323b8366 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15938: skipped 000448 1789122023.207373 mapread 0x001a9d thru 0x010b43 (0xf0a7 bytes) 000449 1789122023.224938 trunc from 0x001525 to 0x002a22 (0x14fe bytes) 000450 1789122023.271149 fallocate 0x028516 thru 0x02a496 (0x1f81 bytes) 000451 1789122023.276554 mapwrite 0x02a525 thru 0x03642a (0xbf06 bytes) 000452 1789122023.358339 read 0x01ae35 thru 0x026dfe (0xbfca bytes) 000454 1789122023.373331 trunc from 0x00f4ba to 0x0101b4 (0xcfb bytes) 000455 1789122023.383535 MIRROR_RESYNC 11 mirrors 000457 1789122024.953815 fallocate 0x00678a thru 0x00d160 (0x69d7 bytes) 000458 1789122025.010262 mapread 0x02c08d thru 0x03642a (0xa39e bytes) 000459 1789122025.046145 mapwrite 0x00f8c8 thru 0x019b10 (0xa249 bytes) 000461 1789122025.123213 write 0x03ef93 thru 0x03f903 (0x971 bytes) 000462 1789122025.126168 trunc from 0x02d243 to 0x030c17 (0x39d5 bytes) 000463 1789122025.133878 trunc from 0x0027fa to 0x0032fb (0xb02 bytes) 000465 1789122025.139947 fallocate 0x00a7ca thru 0x00efe7 (0x481e bytes) 000466 1789122025.149965 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3f904): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f904): Mirror 4: 0x2911371c (stale) Mirror 5: 0x2911371c (stale) Mirror 6: 0x2911371c (stale) Mirror 8: 0x2911371c (stale) Mirror 9: 0x2911371c (stale) Mirror 10: 0x2911371c (stale) Mirror 11: 0x2911371c (stale) Mirror 12: 0x2911371c (stale) Mirror 13: 0x2911371c (stale) Mirror 14: 0x2911371c (stale) CRC-32 checksum value for chunk [0, 0x3f904): Mirror 1: 0xff17413b Mirror 15: 0xff17413b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f904: skipped 000467 1789122026.374506 read 0x011725 thru 0x011b24 (0x400 bytes) 000468 1789122026.381001 mapwrite 0x033222 thru 0x0392e5 (0x60c4 bytes) 000470 1789122026.515640 trunc from 0x03f904 to 0x01ba56 (0x23ead bytes) 000471 1789122026.573270 trunc from 0x01ba57 to 0x01a49d (0x15b9 bytes) 000472 1789122026.590676 mapwrite 0x00d4b3 thru 0x01055f (0x30ad bytes) 000473 1789122026.620567 fallocate 0x00535d thru 0x0057eb (0x48f bytes) 000474 1789122026.635147 read 0x00e59e thru 0x019edb (0xb93e bytes) 000475 1789122026.654872 read 0x018ee7 thru 0x01a49d (0x15b7 bytes) 000476 1789122026.655984 mapwrite 0x023734 thru 0x02edb7 (0xb684 bytes) 000477 1789122026.770175 punch from 0x009bf2 to 0x00a19a (0x5a9 bytes) 000478 1789122026.783476 fallocate 0x01189e thru 0x01fb0c (0xe26f bytes) 000479 1789122026.794920 trunc from 0x0110a0 to 0x01c531 (0xb492 bytes) 000480 1789122026.803217 write 0x03c40c thru 0x03ffff (0x3bf4 bytes) 000481 1789122026.806462 mapwrite 0x03d38e thru 0x03ffff (0x2c72 bytes) 000483 1789122026.830977 MIRROR_RESYNC 12 mirrors 000484 1789122029.774252 trunc from 0x002c73 to 0x00ba01 (0x8d8f bytes) 000485 1789122029.830681 trunc from 0x040000 to 0x011e89 (0x2e176 bytes) 000486 1789122029.856860 mapwrite 0x00d198 thru 0x01d0e3 (0xff4c bytes) 000487 1789122029.934194 trunc from 0x01d0e4 to 0x00eb31 (0xe5b2 bytes) 000489 1789122029.949454 trunc from 0x00406b to 0x00c255 (0x81eb bytes) 000490 1789122029.956285 mapread 0x00a313 thru 0x00eb31 (0x481f bytes) 000491 1789122029.972946 trunc from 0x009b72 to 0x00c9e4 (0x2e73 bytes) 000492 1789122029.980023 fallocate 0x037015 thru 0x03ffff (0x8feb bytes) 000493 1789122029.987724 mapread 0x0145ea thru 0x020b9f (0xc5b6 bytes) 000494 1789122030.010226 write 0x033e3a thru 0x03ffff (0xc1c6 bytes) 000495 1789122030.017112 write 0x02e4e9 thru 0x03e34d (0xfe65 bytes) 000496 1789122030.022965 mapread 0x017efa thru 0x01c525 (0x462c bytes) 000498 1789122030.024545 mapread 0x038166 thru 0x03fffe (0x7e99 bytes) 000499 1789122030.026413 trunc from 0x040000 to 0x03364e (0xc9b1 bytes) 000501 1789122030.048857 punch from 0x00ccbd to 0x01a90d (0xdc51 bytes) 000502 1789122030.054011 mapwrite 0x010d4b thru 0x01bdc1 (0xb077 bytes) 000503 1789122030.133706 fallocate 0x03f058 thru 0x03ffff (0xfa8 bytes) 000504 1789122030.146949 fallocate 0x025b76 thru 0x02c89a (0x6d25 bytes) 000505 1789122030.154150 mapread 0x008469 thru 0x00a13f (0x1cd7 bytes) 000506 1789122030.155049 trunc from 0x040000 to 0x014fce (0x2b031 bytes) 000507 1789122030.185850 trunc from 0x00b3cd to 0x014fce (0x9c02 bytes) 000508 1789122030.198472 mapwrite 0x039d19 thru 0x03ffff (0x62e7 bytes) 000509 1789122030.285816 mapread 0x00ae4d thru 0x016bd0 (0xbd84 bytes) 000510 1789122030.321235 read 0x034f40 thru 0x03fddf (0xaea0 bytes) 000511 1789122030.329536 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9b3fe0d7 (stale) Mirror 5: 0x9b3fe0d7 (stale) Mirror 6: 0x9b3fe0d7 (stale) Mirror 8: 0x9b3fe0d7 (stale) Mirror 10: 0x9b3fe0d7 (stale) Mirror 11: 0x9b3fe0d7 (stale) Mirror 12: 0x9b3fe0d7 (stale) Mirror 13: 0x9b3fe0d7 (stale) Mirror 14: 0x9b3fe0d7 (stale) Mirror 15: 0x9b3fe0d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x9b3fe0d7 (stale) Mirror 5: 0x9b3fe0d7 (stale) Mirror 6: 0x9b3fe0d7 (stale) Mirror 8: 0x9b3fe0d7 (stale) Mirror 10: 0x9b3fe0d7 (stale) Mirror 11: 0x9b3fe0d7 (stale) Mirror 12: 0x9b3fe0d7 (stale) Mirror 13: 0x9b3fe0d7 (stale) Mirror 14: 0x9b3fe0d7 (stale) Mirror 15: 0x9b3fe0d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000512 1789122030.947199 mapwrite 0x03913d thru 0x03ec1a (0x5ade bytes) 000513 1789122030.987133 punch from 0x00d163 to 0x0197a2 (0xc640 bytes) 000514 1789122030.992210 mapread 0x0141e3 thru 0x01df2c (0x9d4a bytes) 000515 1789122031.010812 punch from 0x02a15d to 0x02eaac (0x4950 bytes) 000517 1789122031.015833 trunc from 0x040000 to 0x026b6f (0x19490 bytes) 000518 1789122031.035026 mapwrite 0x010481 thru 0x01445c (0x3fdc bytes) 000519 1789122031.052454 mapwrite 0x037d6b thru 0x03f318 (0x75ae bytes) 000520 1789122031.103026 fallocate 0x00b3f6 thru 0x00cae3 (0x16ee bytes) 000521 1789122031.107839 mapwrite 0x00e5c1 thru 0x019679 (0xb0b9 bytes) 000522 1789122031.130410 MIRROR_RESYNC 11 mirrors 000523 1789122033.522824 read 0x02950a thru 0x038371 (0xee68 bytes) 000524 1789122033.537146 mapread 0x01f97c thru 0x023ead (0x4532 bytes) 000525 1789122033.561847 write 0x0160d0 thru 0x01dbe8 (0x7b19 bytes) 000526 1789122033.609525 punch from 0x026ca2 to 0x0284a8 (0x1807 bytes) 000527 1789122033.615006 punch from 0x0331fb to 0x03eeb1 (0xbcb7 bytes) 000528 1789122033.619345 trunc from 0x036c6b to 0x03f318 (0x86ae bytes) 000529 1789122033.624184 mapwrite 0x024159 thru 0x0293d2 (0x527a bytes) 000530 1789122033.659491 mapread 0x00682c thru 0x00f9db (0x91b0 bytes) 000531 1789122033.676968 mapwrite 0x02668a thru 0x031fc6 (0xb93d bytes) 000532 1789122033.704728 trunc from 0x03f319 to 0x01fa04 (0x1f914 bytes) 000533 1789122033.731903 trunc from 0x01be29 to 0x01fa04 (0x3bdc bytes) 000535 1789122033.751782 mapread 0x00cc9a thru 0x011fec (0x5353 bytes) 000536 1789122033.759769 read 0x01ed00 thru 0x01fa04 (0xd05 bytes) 000537 1789122033.771793 fallocate 0x00511d thru 0x008d15 (0x3bf9 bytes) 000540 1789122033.783169 trunc from 0x00e583 to 0x0194fe (0xaf7c bytes) 000541 1789122033.789053 read 0x018887 thru 0x01fa04 (0x717e bytes) 000542 1789122033.805955 punch from 0x011af7 to 0x012e9f (0x13a9 bytes) 000543 1789122033.818591 read 0x019d48 thru 0x01fa04 (0x5cbd bytes) 000544 1789122033.822849 trunc from 0x0142f4 to 0x01f326 (0xb033 bytes) 000545 1789122033.836553 trunc from 0x01fa05 to 0x016e68 (0x8b9c bytes) 000546 1789122033.863911 write 0x003cc6 thru 0x00b3da (0x7715 bytes) 000547 1789122033.868534 mapwrite 0x01d37d thru 0x02902d (0xbcb1 bytes) 000548 1789122033.928021 trunc from 0x011be8 to 0x01f72b (0xdb44 bytes) 000549 1789122033.945262 trunc from 0x009868 to 0x018dd9 (0xf572 bytes) 000550 1789122033.960120 punch from 0x0211f0 to 0x02902d (0x7e3e bytes) 000551 1789122033.977070 mapwrite 0x0211c4 thru 0x024887 (0x36c4 bytes) 000552 1789122034.033556 fallocate 0x030e2d thru 0x03f22b (0xe3ff bytes) 000554 1789122034.038590 write 0x028cf8 thru 0x037baa (0xeeb3 bytes) 000555 1789122034.050440 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x37bab): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bab): Mirror 4: 0xc7d62bbf (stale) Mirror 5: 0xc7d62bbf (stale) Mirror 6: 0xc7d62bbf (stale) Mirror 8: 0xc7d62bbf (stale) Mirror 10: 0xc7d62bbf (stale) Mirror 11: 0xc7d62bbf (stale) Mirror 12: 0xc7d62bbf (stale) Mirror 13: 0xc7d62bbf (stale) Mirror 14: 0xc7d62bbf (stale) Mirror 15: 0xc7d62bbf (stale) CRC-32 checksum value for chunk [0, 0x37bab): Mirror 1: 0x5791fb7a Mirror 16: 0x5791fb7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bab: skipped 000556 1789122035.267379 punch from 0x01dd2b to 0x01ff4e (0x2224 bytes) 000557 1789122035.318634 write 0x00c556 thru 0x00d4cf (0xf7a bytes) 000558 1789122035.347511 trunc from 0x037bab to 0x00caca (0x2b0e0 bytes) 000559 1789122035.372278 mapwrite 0x03007c thru 0x03d8e4 (0xd869 bytes) 000560 1789122035.469206 trunc from 0x033e61 to 0x03c649 (0x87e9 bytes) 000561 1789122035.474682 mapread 0x013244 thru 0x015fe4 (0x2da1 bytes) 000562 1789122035.500284 MIRROR_RESYNC 12 mirrors 000563 1789122038.872056 punch from 0x007994 to 0x00e5e5 (0x6c52 bytes) 000564 1789122038.922104 write 0x0034d4 thru 0x005763 (0x2290 bytes) 000565 1789122038.938600 punch from 0x026b55 to 0x028785 (0x1c31 bytes) 000566 1789122038.945739 trunc from 0x03d8e5 to 0x0075dc (0x36308 bytes) 000567 1789122038.984246 trunc from 0x0075dd to 0x027642 (0x20066 bytes) 000568 1789122039.009266 mapwrite 0x038499 thru 0x03d403 (0x4f6b bytes) 000570 1789122039.092193 read 0x02a178 thru 0x03132e (0x71b7 bytes) 000571 1789122039.101725 punch from 0x032138 to 0x03bfef (0x9eb8 bytes) 000572 1789122039.109085 mapread 0x00ff86 thru 0x0167e5 (0x6860 bytes) 000573 1789122039.129934 trunc from 0x03d404 to 0x00cce7 (0x3071c bytes) 000575 1789122039.167858 trunc from 0x00cce8 to 0x017ad0 (0xade9 bytes) 000576 1789122039.189775 punch from 0x00703f to 0x00ba09 (0x49cb bytes) 000577 1789122039.194788 write 0x03864b thru 0x03ffff (0x79b5 bytes) 000578 1789122039.197049 mapwrite 0x00cc29 thru 0x01a209 (0xd5e1 bytes) 000579 1789122039.239418 fallocate 0x00da51 thru 0x0123c5 (0x4975 bytes) 000580 1789122039.245720 write 0x003e67 thru 0x011ed2 (0xe06c bytes) 000581 1789122039.249315 trunc from 0x00dbf8 to 0x018da1 (0xb1aa bytes) 000582 1789122039.267130 read 0x021a95 thru 0x029c99 (0x8205 bytes) 000583 1789122039.277225 trunc from 0x028310 to 0x03029d (0x7f8e bytes) 000584 1789122039.281309 fallocate 0x00be41 thru 0x011106 (0x52c6 bytes) 000585 1789122039.286233 punch from 0x029dbd to 0x02f616 (0x585a bytes) 000586 1789122039.291160 punch from 0x02a245 to 0x034228 (0x9fe4 bytes) 000587 1789122039.295798 fallocate 0x00674a thru 0x00f6ef (0x8fa6 bytes) 000588 1789122039.301762 write 0x00d2ec thru 0x0170a8 (0x9dbd bytes) 000589 1789122039.306352 trunc from 0x040000 to 0x005027 (0x3afd8 bytes) 000590 1789122039.328229 write 0x026bc7 thru 0x02c734 (0x5b6e bytes) 000591 1789122039.329582 fallocate 0x02fdd5 thru 0x03f8cb (0xfaf7 bytes) 000592 1789122039.342186 MIRROR_RESYNC 12 mirrors 000593 1789122041.927528 trunc from 0x012126 to 0x013b4f (0x1a2a bytes) 000594 1789122041.995181 mapread 0x02c02a thru 0x02c734 (0x70b bytes) 000596 1789122042.005047 write 0x00c6c1 thru 0x01353d (0x6e7d bytes) 000597 1789122042.019959 trunc from 0x00da9d to 0x01742d (0x9991 bytes) 000598 1789122042.039577 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2c735): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c735): Mirror 4: 0x9358d9bb (stale) Mirror 5: 0x9358d9bb (stale) Mirror 6: 0x9358d9bb (stale) Mirror 8: 0x9358d9bb (stale) Mirror 10: 0x9358d9bb (stale) Mirror 11: 0x9358d9bb (stale) Mirror 12: 0x9358d9bb (stale) Mirror 13: 0x9358d9bb (stale) Mirror 14: 0x9358d9bb (stale) Mirror 15: 0x9358d9bb (stale) Mirror 16: 0x9358d9bb (stale) CRC-32 checksum value for chunk [0, 0x2c735): Mirror 1: 0xf076b672 Mirror 17: 0xf076b672 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c735: skipped 000599 1789122043.249800 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c735): Mirror 4: 0x9358d9bb (stale) Mirror 5: 0x9358d9bb (stale) Mirror 6: 0x9358d9bb (stale) Mirror 8: 0x9358d9bb (stale) Mirror 10: 0x9358d9bb (stale) Mirror 11: 0x9358d9bb (stale) Mirror 12: 0x9358d9bb (stale) Mirror 13: 0x9358d9bb (stale) Mirror 14: 0x9358d9bb (stale) Mirror 15: 0x9358d9bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c735): Mirror 4: 0x9358d9bb (stale) Mirror 5: 0x9358d9bb (stale) Mirror 6: 0x9358d9bb (stale) Mirror 8: 0x9358d9bb (stale) Mirror 10: 0x9358d9bb (stale) Mirror 11: 0x9358d9bb (stale) Mirror 12: 0x9358d9bb (stale) Mirror 13: 0x9358d9bb (stale) Mirror 14: 0x9358d9bb (stale) Mirror 15: 0x9358d9bb (stale) CRC-32 checksum value for chunk [0, 0x2c735): Mirror 1: 0xf076b672 Mirror 17: 0xf076b672 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c735: skipped 000601 1789122044.236276 fallocate 0x000b88 thru 0x00907b (0x84f4 bytes) 000602 1789122044.304062 read 0x02603e thru 0x02c6e1 (0x66a4 bytes) 000603 1789122044.320091 trunc from 0x02c735 to 0x001b98 (0x2ab9c bytes) 000604 1789122044.374799 fallocate 0x0216d2 thru 0x02353b (0x1e6a bytes) 000605 1789122044.387600 read 0x00f7c8 thru 0x0173e9 (0x7c22 bytes) 000606 1789122044.412080 read 0x0049b1 thru 0x012329 (0xd979 bytes) 000607 1789122044.412741 trunc from 0x02353c to 0x021aa6 (0x1a95 bytes) 000608 1789122044.449052 trunc from 0x010a9c to 0x013d29 (0x328e bytes) 000609 1789122044.453275 mapwrite 0x01d370 thru 0x02d139 (0xfdca bytes) 000610 1789122044.548203 trunc from 0x022ef0 to 0x02d139 (0xa24a bytes) 000611 1789122044.554974 trunc from 0x020a3f to 0x026269 (0x582b bytes) 000612 1789122044.560404 punch from 0x011e42 to 0x018fce (0x718d bytes) 000613 1789122044.565706 write 0x03163f thru 0x03d00f (0xb9d1 bytes) 000614 1789122044.567516 read 0x01b192 thru 0x01b88a (0x6f9 bytes) 000615 1789122044.573504 trunc from 0x03d010 to 0x034d08 (0x8307 bytes) 000616 1789122044.595755 trunc from 0x034d09 to 0x013be2 (0x21126 bytes) 000617 1789122044.616206 write 0x03ef1c thru 0x03ffff (0x10e4 bytes) 000618 1789122044.619816 mapread 0x0380b0 thru 0x03a19e (0x20ef bytes) 000619 1789122044.639745 mapread 0x03752f thru 0x03ffff (0x8ad1 bytes) 000620 1789122044.641413 fallocate 0x036096 thru 0x037ec5 (0x1e30 bytes) 000622 1789122044.659190 fallocate 0x023560 thru 0x025bed (0x268e bytes) 000623 1789122044.673820 MIRROR_RESYNC 12 mirrors 000624 1789122046.762253 trunc from 0x040000 to 0x03ef47 (0x10b8 bytes) 000625 1789122046.854922 read 0x026478 thru 0x0336ef (0xd278 bytes) 000626 1789122046.871400 MIRROR_RESYNC 12 mirrors 000627 1789122048.552787 write 0x0217dd thru 0x024620 (0x2e44 bytes) 000628 1789122048.632011 fallocate 0x030b39 thru 0x0399ea (0x8eb2 bytes) 000629 1789122048.638864 fallocate 0x02d2a1 thru 0x02ff7d (0x2cdd bytes) 000630 1789122048.646818 trunc from 0x03ef48 to 0x0157ae (0x29799 bytes) 000632 1789122048.719017 trunc from 0x00611b to 0x00c565 (0x644b bytes) 000633 1789122048.724421 punch from 0x0027a7 to 0x003e28 (0x1682 bytes) 000634 1789122048.730903 write 0x01d1f9 thru 0x02619e (0x8fa6 bytes) 000636 1789122048.734538 fallocate 0x016533 thru 0x020d88 (0xa856 bytes) 000637 1789122048.738967 trunc from 0x02619f to 0x007ff9 (0x1e1a5 bytes) 000638 1789122048.760755 punch from 0x005783 to 0x007ff9 (0x2877 bytes) 000639 1789122048.765733 mapwrite 0x007e03 thru 0x00dca7 (0x5ea5 bytes) 000640 1789122048.825300 trunc from 0x00dca8 to 0x00e287 (0x5e0 bytes) 000641 1789122048.856121 mapwrite 0x033ae3 thru 0x03d343 (0x9861 bytes) 000642 1789122048.922660 punch from 0x006bed to 0x00e8e9 (0x7cfd bytes) 000643 1789122048.931923 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d344): Mirror 4: 0xc70c3913 (stale) Mirror 5: 0xc70c3913 (stale) Mirror 8: 0xc70c3913 (stale) Mirror 10: 0xc70c3913 (stale) Mirror 11: 0xc70c3913 (stale) Mirror 12: 0xc70c3913 (stale) Mirror 13: 0xc70c3913 (stale) Mirror 14: 0xc70c3913 (stale) Mirror 15: 0xc70c3913 (stale) Mirror 17: 0xc70c3913 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d344): Mirror 4: 0xc70c3913 (stale) Mirror 5: 0xc70c3913 (stale) Mirror 8: 0xc70c3913 (stale) Mirror 10: 0xc70c3913 (stale) Mirror 11: 0xc70c3913 (stale) Mirror 12: 0xc70c3913 (stale) Mirror 13: 0xc70c3913 (stale) Mirror 14: 0xc70c3913 (stale) Mirror 15: 0xc70c3913 (stale) Mirror 17: 0xc70c3913 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d344: skipped 000644 1789122049.912250 mapwrite 0x02c187 thru 0x03a1e5 (0xe05f bytes) 000645 1789122050.004563 mapread 0x0337ef thru 0x034567 (0xd79 bytes) 000646 1789122050.005655 fallocate 0x02dd6a thru 0x03b427 (0xd6be bytes) 000647 1789122050.041132 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3d344): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d344): Mirror 4: 0xc70c3913 (stale) Mirror 5: 0xc70c3913 (stale) Mirror 8: 0xc70c3913 (stale) Mirror 10: 0xc70c3913 (stale) Mirror 11: 0xc70c3913 (stale) Mirror 12: 0xc70c3913 (stale) Mirror 13: 0xc70c3913 (stale) Mirror 14: 0xc70c3913 (stale) Mirror 15: 0xc70c3913 (stale) Mirror 17: 0xc70c3913 (stale) CRC-32 checksum value for chunk [0, 0x3d344): Mirror 1: 0xb236f96f Mirror 18: 0xb236f96f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d344: skipped 000648 1789122051.382960 mapread 0x02b4d0 thru 0x032bf2 (0x7723 bytes) 000649 1789122051.407456 read 0x016246 thru 0x01dd6f (0x7b2a bytes) 000650 1789122051.417362 punch from 0x0081c9 to 0x008365 (0x19d bytes) 000651 1789122051.465497 trunc from 0x03d344 to 0x01c10c (0x21237 bytes) 000652 1789122051.488955 trunc from 0x01c10d to 0x0316aa (0x1559e bytes) 000653 1789122051.504046 mapwrite 0x03bd14 thru 0x03ffff (0x42ec bytes) 000656 1789122051.542394 trunc from 0x040000 to 0x03eff1 (0x100e bytes) 000657 1789122051.557228 punch from 0x00eadd to 0x01bb24 (0xd048 bytes) 000658 1789122051.562974 write 0x016644 thru 0x0189c2 (0x237f bytes) 000659 1789122051.573802 fallocate 0x0114c3 thru 0x013ce9 (0x2827 bytes) 000660 1789122051.579322 trunc from 0x03eff2 to 0x01faf0 (0x1f501 bytes) 000661 1789122051.597682 mapread 0x01c7ad thru 0x01e9e4 (0x2238 bytes) 000662 1789122051.608485 punch from 0x00fb14 to 0x01acee (0xb1db bytes) 000663 1789122051.619457 mapread 0x001157 thru 0x00ee8f (0xdd39 bytes) 000665 1789122051.632048 trunc from 0x005f88 to 0x00b967 (0x59e0 bytes) 000666 1789122051.637497 read 0x0119f4 thru 0x01e5dc (0xcbe9 bytes) 000667 1789122051.648334 mapwrite 0x00c2b9 thru 0x00eff8 (0x2d40 bytes) 000668 1789122051.661851 trunc from 0x01faf1 to 0x0343be (0x148ce bytes) 000669 1789122051.684537 punch from 0x00f942 to 0x015f67 (0x6626 bytes) 000670 1789122051.689922 mapwrite 0x029eff thru 0x039203 (0xf305 bytes) 000671 1789122051.731395 punch from 0x00a566 to 0x00d511 (0x2fac bytes) 000673 1789122051.737062 trunc from 0x024eef to 0x0273e0 (0x24f2 bytes) 000674 1789122051.744702 fallocate 0x00c100 thru 0x012351 (0x6252 bytes) 000675 1789122051.750136 trunc from 0x0211ef to 0x02d56c (0xc37e bytes) 000676 1789122051.756100 write 0x03e9f6 thru 0x03ffff (0x160a bytes) 000677 1789122051.758548 trunc from 0x0068f4 to 0x0085ba (0x1cc7 bytes) 000678 1789122051.763864 trunc from 0x040000 to 0x03e74e (0x18b1 bytes) 000680 1789122051.787085 mapread 0x012a28 thru 0x021afc (0xf0d5 bytes) 000681 1789122051.828286 write 0x02ecfc thru 0x0309f4 (0x1cf9 bytes) 000682 1789122051.837978 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3e74f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e74f): Mirror 4: 0xddc25fab (stale) Mirror 5: 0xddc25fab (stale) Mirror 8: 0xddc25fab (stale) Mirror 10: 0xddc25fab (stale) Mirror 11: 0xddc25fab (stale) Mirror 12: 0xddc25fab (stale) Mirror 13: 0xddc25fab (stale) Mirror 14: 0xddc25fab (stale) Mirror 15: 0xddc25fab (stale) Mirror 17: 0xddc25fab (stale) Mirror 18: 0x4257c3dc (stale) CRC-32 checksum value for chunk [0, 0x3e74f): Mirror 1: 0xaa5b1ed9 Mirror 19: 0xaa5b1ed9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e74f: skipped 000683 1789122053.438065 write 0x02405a thru 0x02c213 (0x81ba bytes) 000684 1789122053.505518 mapread 0x022205 thru 0x02f037 (0xce33 bytes) 000685 1789122053.523055 punch from 0x003e06 to 0x00adea (0x6fe5 bytes) 000686 1789122053.527820 read 0x0010f6 thru 0x008b33 (0x7a3e bytes) 000687 1789122053.536338 trunc from 0x03e74f to 0x01356d (0x2b1e1 bytes) 000688 1789122053.566648 trunc from 0x003a20 to 0x00fcea (0xc2cb bytes) 000689 1789122053.572262 punch from 0x010718 to 0x01356d (0x2e56 bytes) 000690 1789122053.593961 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1356e): Mirror 4: 0x8b3724b9 (stale) Mirror 5: 0x8b3724b9 (stale) Mirror 8: 0x8b3724b9 (stale) Mirror 10: 0x8b3724b9 (stale) Mirror 11: 0x8b3724b9 (stale) Mirror 12: 0x8b3724b9 (stale) Mirror 13: 0x8b3724b9 (stale) Mirror 14: 0x8b3724b9 (stale) Mirror 17: 0x8b3724b9 (stale) Mirror 18: 0x8b3724b9 (stale) Mirror 19: 0xd0084cd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1356e): Mirror 4: 0x8b3724b9 (stale) Mirror 5: 0x8b3724b9 (stale) Mirror 8: 0x8b3724b9 (stale) Mirror 10: 0x8b3724b9 (stale) Mirror 11: 0x8b3724b9 (stale) Mirror 12: 0x8b3724b9 (stale) Mirror 13: 0x8b3724b9 (stale) Mirror 14: 0x8b3724b9 (stale) Mirror 17: 0x8b3724b9 (stale) Mirror 18: 0x8b3724b9 (stale) Mirror 19: 0xd0084cd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1356e: skipped 000691 1789122054.373524 mapread 0x00211c thru 0x009b52 (0x7a37 bytes) 000692 1789122054.438298 write 0x012c0f thru 0x01fd8a (0xd17c bytes) 000693 1789122054.464481 read 0x00e21c thru 0x01593a (0x771f bytes) 000694 1789122054.474068 mapread 0x006c8d thru 0x014ee0 (0xe254 bytes) 000695 1789122054.475312 punch from 0x013c9e to 0x018de9 (0x514c bytes) 000696 1789122054.491046 trunc from 0x014cbc to 0x0179eb (0x2d30 bytes) 000697 1789122054.495473 mapread 0x002d54 thru 0x007743 (0x49f0 bytes) 000698 1789122054.497190 mapread 0x01cb63 thru 0x01fd8a (0x3228 bytes) 000699 1789122054.502579 MIRROR_RESYNC 12 mirrors 000700 1789122057.493853 fallocate 0x01aa9c thru 0x025318 (0xa87d bytes) 000701 1789122057.587673 read 0x008bb3 thru 0x016ea1 (0xe2ef bytes) 000702 1789122057.608795 trunc from 0x011f36 to 0x0173c7 (0x5492 bytes) 000703 1789122057.619374 mapwrite 0x03b154 thru 0x03c9cc (0x1879 bytes) 000704 1789122057.756307 trunc from 0x03c9cd to 0x006ad1 (0x35efb bytes) 000705 1789122057.793000 trunc from 0x006ad2 to 0x0066af (0x422 bytes) 000706 1789122057.860862 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x66b0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x66b0): Mirror 4: 0xb4459219 (stale) Mirror 5: 0xb4459219 (stale) Mirror 8: 0xb4459219 (stale) Mirror 10: 0xb4459219 (stale) Mirror 11: 0xb4459219 (stale) Mirror 12: 0xb4459219 (stale) Mirror 13: 0xb4459219 (stale) Mirror 14: 0xb4459219 (stale) Mirror 17: 0xb4459219 (stale) Mirror 18: 0xb4459219 (stale) Mirror 19: 0xb4459219 (stale) CRC-32 checksum value for chunk [0, 0x66b0): Mirror 1: 0xb4459219 Mirror 20: 0xb4459219 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x66b0: skipped 000707 1789122059.512943 read 0x00225a thru 0x004297 (0x203e bytes) 000708 1789122059.521355 read 0x002d92 thru 0x0066af (0x391e bytes) 000709 1789122059.535070 punch from 0x00133f to 0x00643e (0x5100 bytes) 000710 1789122059.619006 mapread 0x003e2c thru 0x0066af (0x2884 bytes) 000711 1789122059.655020 fallocate 0x028357 thru 0x0288a9 (0x553 bytes) 000712 1789122059.661791 trunc from 0x01875d to 0x01f226 (0x6aca bytes) 000713 1789122059.679739 trunc from 0x015369 to 0x01a096 (0x4d2e bytes) 000714 1789122059.699782 write 0x011863 thru 0x014d50 (0x34ee bytes) 000715 1789122059.733737 MIRROR_RESYNC 13 mirrors 000716 1789122063.049289 write 0x02249d thru 0x030976 (0xe4da bytes) 000717 1789122063.091106 trunc from 0x030977 to 0x00e1c7 (0x227af bytes) 000718 1789122063.130262 mapread 0x00c308 thru 0x00e1c7 (0x1ec0 bytes) 000719 1789122063.164797 write 0x035c51 thru 0x037f30 (0x22e0 bytes) 000720 1789122063.177232 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37f31): Mirror 5: 0xa082e5f2 (stale) Mirror 8: 0xa082e5f2 (stale) Mirror 10: 0xa082e5f2 (stale) Mirror 11: 0xa082e5f2 (stale) Mirror 12: 0xa082e5f2 (stale) Mirror 13: 0xa082e5f2 (stale) Mirror 14: 0xa082e5f2 (stale) Mirror 17: 0xa082e5f2 (stale) Mirror 18: 0xa082e5f2 (stale) Mirror 19: 0xa082e5f2 (stale) Mirror 20: 0xa082e5f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f31): Mirror 5: 0xa082e5f2 (stale) Mirror 8: 0xa082e5f2 (stale) Mirror 10: 0xa082e5f2 (stale) Mirror 11: 0xa082e5f2 (stale) Mirror 12: 0xa082e5f2 (stale) Mirror 13: 0xa082e5f2 (stale) Mirror 14: 0xa082e5f2 (stale) Mirror 17: 0xa082e5f2 (stale) Mirror 18: 0xa082e5f2 (stale) Mirror 19: 0xa082e5f2 (stale) Mirror 20: 0xa082e5f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f31: skipped 000721 1789122063.967313 trunc from 0x02b83a to 0x02cddd (0x15a4 bytes) 000722 1789122063.973904 fallocate 0x01b9ff thru 0x02b622 (0xfc24 bytes) 000723 1789122063.985651 mapread 0x00bd2d thru 0x01151a (0x57ee bytes) 000724 1789122064.016016 read 0x02a7e7 thru 0x02a964 (0x17e bytes) 000725 1789122064.023574 trunc from 0x037f31 to 0x00446d (0x33ac3 bytes) 000726 1789122064.068024 mapread 0x004383 thru 0x00446d (0xeb bytes) 000727 1789122064.081746 write 0x02acf1 thru 0x02fad0 (0x4de0 bytes) 000728 1789122064.083320 trunc from 0x003aea to 0x006790 (0x2ca7 bytes) 000729 1789122064.097452 trunc from 0x023f8e to 0x028dae (0x4e21 bytes) 000730 1789122064.110909 trunc from 0x02fad1 to 0x027057 (0x8a79 bytes) 000731 1789122064.128435 fallocate 0x01f305 thru 0x0285cb (0x92c7 bytes) 000732 1789122064.134521 read 0x00a723 thru 0x00e242 (0x3b20 bytes) 000733 1789122064.148819 trunc from 0x004cf3 to 0x008ad1 (0x3ddf bytes) 000735 1789122064.157154 trunc from 0x0285cc to 0x02cd74 (0x47a9 bytes) 000736 1789122064.183296 read 0x020afd thru 0x029269 (0x876d bytes) 000737 1789122064.192726 punch from 0x01c9ca to 0x02a149 (0xd780 bytes) 000738 1789122064.209887 MIRROR_RESYNC 12 mirrors 000739 1789122065.981690 write 0x02c858 thru 0x030900 (0x40a9 bytes) 000740 1789122066.049823 trunc from 0x030901 to 0x00c589 (0x24377 bytes) 000741 1789122066.112726 punch from 0x00b86c to 0x00c589 (0xd1e bytes) 000742 1789122066.130991 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc58a): Mirror 5: 0xc19edcab (stale) Mirror 8: 0xc19edcab (stale) Mirror 10: 0xc19edcab (stale) Mirror 11: 0xc19edcab (stale) Mirror 12: 0xc19edcab (stale) Mirror 14: 0xc19edcab (stale) Mirror 17: 0xc19edcab (stale) Mirror 18: 0xc19edcab (stale) Mirror 19: 0xc19edcab (stale) Mirror 20: 0xc19edcab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc58a): Mirror 5: 0xc19edcab (stale) Mirror 8: 0xc19edcab (stale) Mirror 10: 0xc19edcab (stale) Mirror 11: 0xc19edcab (stale) Mirror 12: 0xc19edcab (stale) Mirror 14: 0xc19edcab (stale) Mirror 17: 0xc19edcab (stale) Mirror 18: 0xc19edcab (stale) Mirror 19: 0xc19edcab (stale) Mirror 20: 0xc19edcab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc58a: skipped 000743 1789122066.942885 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc58a): Mirror 5: 0xc19edcab (stale) Mirror 8: 0xc19edcab (stale) Mirror 10: 0xc19edcab (stale) Mirror 11: 0xc19edcab (stale) Mirror 12: 0xc19edcab (stale) Mirror 14: 0xc19edcab (stale) Mirror 17: 0xc19edcab (stale) Mirror 19: 0xc19edcab (stale) Mirror 20: 0xc19edcab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc58a): Mirror 5: 0xc19edcab (stale) Mirror 8: 0xc19edcab (stale) Mirror 10: 0xc19edcab (stale) Mirror 11: 0xc19edcab (stale) Mirror 12: 0xc19edcab (stale) Mirror 14: 0xc19edcab (stale) Mirror 17: 0xc19edcab (stale) Mirror 19: 0xc19edcab (stale) Mirror 20: 0xc19edcab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc58a: skipped 000744 1789122067.717574 MIRROR_RESYNC 10 mirrors 000745 1789122069.195309 mapread 0x0001e9 thru 0x003258 (0x3070 bytes) 000746 1789122069.215903 trunc from 0x00c58a to 0x00e785 (0x21fc bytes) 000747 1789122069.260472 mapread 0x00c7d7 thru 0x00e785 (0x1faf bytes) 000748 1789122069.271266 trunc from 0x008112 to 0x00e785 (0x6674 bytes) 000749 1789122069.276471 write 0x0216f7 thru 0x02fd34 (0xe63e bytes) 000750 1789122069.278214 read 0x01b236 thru 0x01c430 (0x11fb bytes) 000751 1789122069.285810 trunc from 0x02fd35 to 0x0031fe (0x2cb36 bytes) 000752 1789122069.319018 fallocate 0x0278fa thru 0x02822a (0x931 bytes) 000755 1789122069.324781 mapread 0x0008c7 thru 0x0031fe (0x2938 bytes) 000756 1789122069.337671 mapwrite 0x038a2d thru 0x038c0b (0x1df bytes) 000757 1789122069.439709 mapread 0x00c489 thru 0x018a02 (0xc57a bytes) 000758 1789122069.495409 read 0x02fde0 thru 0x034840 (0x4a61 bytes) 000759 1789122069.524433 fallocate 0x005bc4 thru 0x013986 (0xddc3 bytes) 000760 1789122069.530352 punch from 0x015874 to 0x016be7 (0x1374 bytes) 000761 1789122069.537122 trunc from 0x01884d to 0x028531 (0xfce5 bytes) 000762 1789122069.542523 fallocate 0x006216 thru 0x007f3b (0x1d26 bytes) 000763 1789122069.548441 trunc from 0x01ede8 to 0x0273c3 (0x85dc bytes) 000764 1789122069.555125 mapread 0x038690 thru 0x038c0b (0x57c bytes) 000765 1789122069.560766 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c0c): Mirror 5: 0x8c9abdd0 (stale) Mirror 8: 0x8c9abdd0 (stale) Mirror 10: 0x8c9abdd0 (stale) Mirror 11: 0x8c9abdd0 (stale) Mirror 12: 0x8c9abdd0 (stale) Mirror 14: 0x8c9abdd0 (stale) Mirror 17: 0x8c9abdd0 (stale) Mirror 19: 0x8c9abdd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c0c): Mirror 5: 0x8c9abdd0 (stale) Mirror 8: 0x8c9abdd0 (stale) Mirror 10: 0x8c9abdd0 (stale) Mirror 11: 0x8c9abdd0 (stale) Mirror 12: 0x8c9abdd0 (stale) Mirror 14: 0x8c9abdd0 (stale) Mirror 17: 0x8c9abdd0 (stale) Mirror 19: 0x8c9abdd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c0c: skipped 000766 1789122070.156114 trunc from 0x0151ef to 0x01582c (0x63e bytes) 000767 1789122070.174944 punch from 0x010ca1 to 0x01d809 (0xcb69 bytes) 000768 1789122070.179915 fallocate 0x013d5f thru 0x013f07 (0x1a9 bytes) 000769 1789122070.187279 write 0x029012 thru 0x0329c9 (0x99b8 bytes) 000770 1789122070.208316 trunc from 0x01e5a1 to 0x02891a (0xa37a bytes) 000771 1789122070.215993 read 0x013a6a thru 0x01d20e (0x97a5 bytes) 000772 1789122070.225814 fallocate 0x00d856 thru 0x0110cb (0x3876 bytes) 000773 1789122070.235965 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38c0c): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c0c): Mirror 5: 0x8c9abdd0 (stale) Mirror 8: 0x8c9abdd0 (stale) Mirror 10: 0x8c9abdd0 (stale) Mirror 11: 0x8c9abdd0 (stale) Mirror 12: 0x8c9abdd0 (stale) Mirror 14: 0x8c9abdd0 (stale) Mirror 17: 0x8c9abdd0 (stale) Mirror 19: 0x8c9abdd0 (stale) CRC-32 checksum value for chunk [0, 0x38c0c): Mirror 1: 0x794f6783 Mirror 20: 0x794f6783 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c0c: skipped 000774 1789122071.262416 fallocate 0x009455 thru 0x011552 (0x80fe bytes) 000775 1789122071.313877 write 0x00bc14 thru 0x01ab68 (0xef55 bytes) 000776 1789122071.344582 trunc from 0x038c0c to 0x0237f8 (0x15413 bytes) 000777 1789122071.394092 MIRROR_RESYNC 10 mirrors 000778 1789122074.104373 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000779 1789122074.688760 write 0x006b61 thru 0x0076a5 (0xb45 bytes) 000780 1789122074.748426 trunc from 0x00d657 to 0x011fd4 (0x497e bytes) 000781 1789122074.754183 trunc from 0x0064ea to 0x00a071 (0x3b88 bytes) 000782 1789122074.767023 write 0x025d1b thru 0x029af9 (0x3ddf bytes) 000783 1789122074.768347 write 0x01b337 thru 0x0298e2 (0xe5ac bytes) 000784 1789122074.777798 fallocate 0x031c39 thru 0x03698a (0x4d52 bytes) 000785 1789122074.783738 mapread 0x00c953 thru 0x01c029 (0xf6d7 bytes) 000786 1789122074.812108 read 0x00ccce thru 0x01a329 (0xd65c bytes) 000787 1789122074.812976 trunc from 0x029afa to 0x03f06e (0x15575 bytes) 000788 1789122074.841793 write 0x00f173 thru 0x0121ff (0x308d bytes) 000790 1789122074.845895 mapread 0x027741 thru 0x0363ce (0xec8e bytes) 000791 1789122074.864912 trunc from 0x03f06f to 0x00aeb6 (0x341b8 bytes) 000792 1789122074.881297 mapwrite 0x022f2b thru 0x0235b6 (0x68c bytes) 000793 1789122074.927236 punch from 0x01b0b7 to 0x0235b6 (0x8500 bytes) 000794 1789122074.935186 read 0x01f68a thru 0x0235b6 (0x3f2d bytes) 000795 1789122074.955097 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x235b7): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235b7): Mirror 5: 0x6282ba8e (stale) Mirror 8: 0x6282ba8e (stale) Mirror 10: 0x6282ba8e (stale) Mirror 12: 0x6282ba8e (stale) Mirror 14: 0x6282ba8e (stale) Mirror 17: 0x6282ba8e (stale) Mirror 19: 0x6282ba8e (stale) Mirror 20: 0x6282ba8e (stale) CRC-32 checksum value for chunk [0, 0x235b7): Mirror 1: 0xcdb8ca45 Mirror 21: 0xcdb8ca45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235b7: skipped 000796 1789122076.124132 trunc from 0x0235b7 to 0x036051 (0x12a9b bytes) 000797 1789122076.202963 punch from 0x022d96 to 0x02c9fd (0x9c68 bytes) 000798 1789122076.212212 write 0x02bd90 thru 0x0354c6 (0x9737 bytes) 000800 1789122076.227499 mapwrite 0x02d49b thru 0x02ea1a (0x1580 bytes) 000802 1789122076.258407 write 0x0163d6 thru 0x0202c4 (0x9eef bytes) 000803 1789122076.276256 read 0x0116c7 thru 0x018e1a (0x7754 bytes) 000805 1789122076.298264 read 0x023344 thru 0x02ca7f (0x973c bytes) 000806 1789122076.314297 trunc from 0x036052 to 0x0359ee (0x663 bytes) 000807 1789122076.348424 trunc from 0x00ba25 to 0x018d84 (0xd360 bytes) 000808 1789122076.369716 trunc from 0x0359ef to 0x015ac7 (0x1ff27 bytes) 000809 1789122076.387651 read 0x001a9b thru 0x00b8bf (0x9e25 bytes) 000810 1789122076.407660 read 0x00b0b0 thru 0x00d356 (0x22a7 bytes) 000811 1789122076.408252 mapread 0x00a915 thru 0x01482f (0x9f1b bytes) 000812 1789122076.409187 fallocate 0x03444e thru 0x03ffff (0xbbb2 bytes) 000813 1789122076.418193 punch from 0x0150fa to 0x019ce1 (0x4be8 bytes) 000814 1789122076.427986 mapread 0x004971 thru 0x009a35 (0x50c5 bytes) 000815 1789122076.447890 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8ef7f26d (stale) Mirror 8: 0x8ef7f26d (stale) Mirror 10: 0x8ef7f26d (stale) Mirror 12: 0x8ef7f26d (stale) Mirror 14: 0x8ef7f26d (stale) Mirror 17: 0x8ef7f26d (stale) Mirror 19: 0x8ef7f26d (stale) Mirror 20: 0x8ef7f26d (stale) Mirror 21: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 22: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000817 1789122077.540947 trunc from 0x040000 to 0x00e489 (0x31b76 bytes) 000819 1789122077.674773 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0xe48a): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe48a): Mirror 5: 0xdd8afc41 (stale) Mirror 8: 0xdd8afc41 (stale) Mirror 10: 0xdd8afc41 (stale) Mirror 12: 0xdd8afc41 (stale) Mirror 14: 0xdd8afc41 (stale) Mirror 17: 0xdd8afc41 (stale) Mirror 19: 0xdd8afc41 (stale) Mirror 20: 0xdd8afc41 (stale) Mirror 21: 0x5447656f (stale) Mirror 22: 0x5447656f (stale) CRC-32 checksum value for chunk [0, 0xe48a): Mirror 1: 0x5447656f Mirror 23: 0x5447656f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe48a: skipped 000820 1789122079.039588 trunc from 0x002745 to 0x00a2e4 (0x7ba0 bytes) 000821 1789122079.078134 read 0x00d730 thru 0x00e489 (0xd5a bytes) 000822 1789122079.084334 mapwrite 0x00e79c thru 0x00eec9 (0x72e bytes) 000823 1789122079.133673 MIRROR_RESYNC 12 mirrors 000824 1789122081.915051 trunc from 0x00eeca to 0x03a7c5 (0x2b8fc bytes) 000825 1789122081.969297 fallocate 0x000f64 thru 0x009aeb (0x8b88 bytes) 000826 1789122081.975377 trunc from 0x03a7c6 to 0x0093f8 (0x313cd bytes) 000827 1789122081.996834 trunc from 0x0036fb to 0x0093f8 (0x5cfe bytes) 000828 1789122082.001413 trunc from 0x00852e to 0x0093f8 (0xecb bytes) 000829 1789122082.006294 fallocate 0x009046 thru 0x010772 (0x772d bytes) 000830 1789122082.010914 fallocate 0x0002fb thru 0x000b36 (0x83c bytes) 000831 1789122082.015219 trunc from 0x008e60 to 0x0093f8 (0x599 bytes) 000832 1789122082.021901 trunc from 0x0043d3 to 0x0093f8 (0x5026 bytes) 000833 1789122082.026702 punch from 0x003fa5 to 0x0093f8 (0x5454 bytes) 000834 1789122082.035962 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x93f9): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x93f9): Mirror 5: 0x7e219e3d (stale) Mirror 8: 0x7e219e3d (stale) Mirror 10: 0x7e219e3d (stale) Mirror 12: 0x7e219e3d (stale) Mirror 14: 0x7e219e3d (stale) Mirror 17: 0x7e219e3d (stale) Mirror 19: 0x7e219e3d (stale) Mirror 20: 0x7e219e3d (stale) Mirror 21: 0x7e219e3d (stale) Mirror 22: 0x7e219e3d (stale) Mirror 23: 0x7e219e3d (stale) CRC-32 checksum value for chunk [0, 0x93f9): Mirror 1: 0x7e219e3d Mirror 24: 0x7e219e3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x93f9: skipped 000835 1789122083.051336 mapwrite 0x01e506 thru 0x0256b6 (0x71b1 bytes) 000836 1789122083.145364 mapread 0x01213f thru 0x01dc3b (0xbafd bytes) 000837 1789122083.161855 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x256b7): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x256b7): Mirror 5: 0x8f68df42 (stale) Mirror 8: 0x8f68df42 (stale) Mirror 10: 0x8f68df42 (stale) Mirror 12: 0x8f68df42 (stale) Mirror 14: 0x8f68df42 (stale) Mirror 17: 0x8f68df42 (stale) Mirror 19: 0x8f68df42 (stale) Mirror 20: 0x8f68df42 (stale) Mirror 21: 0x8f68df42 (stale) Mirror 22: 0x8f68df42 (stale) Mirror 23: 0x8f68df42 (stale) Mirror 24: 0x3b5eba20 (stale) CRC-32 checksum value for chunk [0, 0x256b7): Mirror 1: 0xa428f5b1 Mirror 25: 0xa428f5b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x256b7: skipped 000838 1789122084.118169 mapwrite 0x0083fd thru 0x01421d (0xbe21 bytes) 000839 1789122084.237030 punch from 0x020a77 to 0x0256b6 (0x4c40 bytes) 000840 1789122084.248020 MIRROR_RESYNC 14 mirrors 000841 1789122088.038199 punch from 0x00ccbb to 0x0159b3 (0x8cf9 bytes) 000842 1789122088.087768 write 0x006805 thru 0x00aeb5 (0x46b1 bytes) 000843 1789122088.106798 mapread 0x008948 thru 0x01597d (0xd036 bytes) 000845 1789122088.137033 write 0x002318 thru 0x00e0d1 (0xbdba bytes) 000846 1789122088.150842 write 0x025144 thru 0x027b9e (0x2a5b bytes) 000847 1789122088.161868 fallocate 0x025f71 thru 0x02e157 (0x81e7 bytes) 000848 1789122088.171324 fallocate 0x005bc0 thru 0x012678 (0xcab9 bytes) 000850 1789122088.179334 punch from 0x006773 to 0x012b1c (0xc3aa bytes) 000851 1789122088.197091 trunc from 0x027b9f to 0x02aec6 (0x3328 bytes) 000852 1789122088.223924 write 0x02ec8d thru 0x032320 (0x3694 bytes) 000853 1789122088.227448 write 0x02d72e thru 0x038ccc (0xb59f bytes) 000854 1789122088.235699 mapread 0x036f63 thru 0x037458 (0x4f6 bytes) 000855 1789122088.237215 fallocate 0x03ae3f thru 0x03bde4 (0xfa6 bytes) 000857 1789122088.248120 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ccd): Mirror 5: 0xdde38459 (stale) Mirror 8: 0xdde38459 (stale) Mirror 10: 0xdde38459 (stale) Mirror 12: 0xdde38459 (stale) Mirror 14: 0xdde38459 (stale) Mirror 17: 0xdde38459 (stale) Mirror 20: 0xdde38459 (stale) Mirror 21: 0xdde38459 (stale) Mirror 22: 0xdde38459 (stale) Mirror 23: 0xdde38459 (stale) Mirror 24: 0xdde38459 (stale) Mirror 25: 0xdde38459 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ccd): Mirror 5: 0xdde38459 (stale) Mirror 8: 0xdde38459 (stale) Mirror 10: 0xdde38459 (stale) Mirror 12: 0xdde38459 (stale) Mirror 14: 0xdde38459 (stale) Mirror 17: 0xdde38459 (stale) Mirror 20: 0xdde38459 (stale) Mirror 21: 0xdde38459 (stale) Mirror 22: 0xdde38459 (stale) Mirror 23: 0xdde38459 (stale) Mirror 24: 0xdde38459 (stale) Mirror 25: 0xdde38459 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ccd: skipped 000858 1789122089.018371 read 0x03826b thru 0x038ccc (0xa62 bytes) 000859 1789122089.027399 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x38ccd): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ccd): Mirror 5: 0xdde38459 (stale) Mirror 8: 0xdde38459 (stale) Mirror 10: 0xdde38459 (stale) Mirror 12: 0xdde38459 (stale) Mirror 14: 0xdde38459 (stale) Mirror 17: 0xdde38459 (stale) Mirror 20: 0xdde38459 (stale) Mirror 21: 0xdde38459 (stale) Mirror 22: 0xdde38459 (stale) Mirror 23: 0xdde38459 (stale) Mirror 24: 0xdde38459 (stale) Mirror 25: 0xdde38459 (stale) CRC-32 checksum value for chunk [0, 0x38ccd): Mirror 1: 0xf7afe755 Mirror 26: 0xf7afe755 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ccd: skipped 000860 1789122090.350727 mapread 0x00d4b4 thru 0x016a4b (0x9598 bytes) 000861 1789122090.378744 fallocate 0x01ca06 thru 0x02870f (0xbd0a bytes) 000862 1789122090.436862 mapread 0x01ab2f thru 0x02a251 (0xf723 bytes) 000863 1789122090.468165 trunc from 0x038ccd to 0x003695 (0x35637 bytes) 000864 1789122090.527422 fallocate 0x0274f1 thru 0x032597 (0xb0a7 bytes) 000865 1789122090.537139 fallocate 0x019a78 thru 0x02744b (0xd9d4 bytes) 000866 1789122090.545432 fallocate 0x01a9ca thru 0x0210c6 (0x66fd bytes) 000867 1789122090.553160 fallocate 0x013dc5 thru 0x015d5e (0x1f9a bytes) 000868 1789122090.558149 read 0x0292fc thru 0x032597 (0x929c bytes) 000869 1789122090.580844 punch from 0x014163 to 0x018747 (0x45e5 bytes) 000870 1789122090.585868 fallocate 0x030cf1 thru 0x0352f1 (0x4601 bytes) 000871 1789122090.599233 punch from 0x02b70e to 0x032597 (0x6e8a bytes) 000872 1789122090.612185 trunc from 0x032598 to 0x00975e (0x28e39 bytes) 000873 1789122090.642982 trunc from 0x00975f to 0x011c7c (0x851e bytes) 000874 1789122090.666216 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11c7d): Mirror 5: 0x4ce82cd0 (stale) Mirror 10: 0x4ce82cd0 (stale) Mirror 12: 0x4ce82cd0 (stale) Mirror 14: 0x4ce82cd0 (stale) Mirror 17: 0x4ce82cd0 (stale) Mirror 20: 0x4ce82cd0 (stale) Mirror 21: 0x4ce82cd0 (stale) Mirror 22: 0x4ce82cd0 (stale) Mirror 23: 0x4ce82cd0 (stale) Mirror 24: 0x4ce82cd0 (stale) Mirror 25: 0x4ce82cd0 (stale) Mirror 26: 0x513cab3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c7d): Mirror 5: 0x4ce82cd0 (stale) Mirror 10: 0x4ce82cd0 (stale) Mirror 12: 0x4ce82cd0 (stale) Mirror 14: 0x4ce82cd0 (stale) Mirror 17: 0x4ce82cd0 (stale) Mirror 20: 0x4ce82cd0 (stale) Mirror 21: 0x4ce82cd0 (stale) Mirror 22: 0x4ce82cd0 (stale) Mirror 23: 0x4ce82cd0 (stale) Mirror 24: 0x4ce82cd0 (stale) Mirror 25: 0x4ce82cd0 (stale) Mirror 26: 0x513cab3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c7d: skipped 000875 1789122091.374025 mapread 0x00c0aa thru 0x011c7c (0x5bd3 bytes) 000876 1789122091.405138 read 0x0088d7 thru 0x00ecd2 (0x63fc bytes) 000877 1789122091.406803 fallocate 0x00301f thru 0x006b72 (0x3b54 bytes) 000879 1789122091.416458 write 0x026331 thru 0x02e2a6 (0x7f76 bytes) 000881 1789122091.427020 trunc from 0x02e2a7 to 0x01aaef (0x137b7 bytes) 000882 1789122091.450796 write 0x03c215 thru 0x03ffff (0x3deb bytes) 000883 1789122091.452767 punch from 0x0121c3 to 0x0166ad (0x44eb bytes) 000884 1789122091.462602 mapwrite 0x03e133 thru 0x03ffff (0x1ecd bytes) 000885 1789122091.481413 mapwrite 0x0147d5 thru 0x01762c (0x2e58 bytes) 000886 1789122091.510253 trunc from 0x0336ea to 0x03d748 (0xa05f bytes) 000887 1789122091.516410 mapwrite 0x02ca5b thru 0x0380e5 (0xb68b bytes) 000889 1789122091.563981 trunc from 0x040000 to 0x00943c (0x36bc3 bytes) 000890 1789122091.580468 mapwrite 0x00fc3d thru 0x014d1e (0x50e2 bytes) 000891 1789122091.629954 mapread 0x00e43f thru 0x014d1e (0x68e0 bytes) 000892 1789122091.632081 mapwrite 0x0305f1 thru 0x031457 (0xe67 bytes) 000893 1789122091.667883 mapread 0x012097 thru 0x0197f4 (0x775e bytes) 000894 1789122091.679411 write 0x0055bf thru 0x00ba31 (0x6473 bytes) 000895 1789122091.687804 trunc from 0x02ed38 to 0x02f209 (0x4d2 bytes) 000896 1789122091.693017 punch from 0x02b662 to 0x0313c2 (0x5d61 bytes) 000898 1789122091.700334 read 0x00cb0c thru 0x01c59d (0xfa92 bytes) 000899 1789122091.707045 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x31458): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31458): Mirror 5: 0xb62dd2fb (stale) Mirror 10: 0xb62dd2fb (stale) Mirror 12: 0xb62dd2fb (stale) Mirror 14: 0xb62dd2fb (stale) Mirror 17: 0xb62dd2fb (stale) Mirror 20: 0xb62dd2fb (stale) Mirror 21: 0xb62dd2fb (stale) Mirror 22: 0xb62dd2fb (stale) Mirror 23: 0xb62dd2fb (stale) Mirror 24: 0xb62dd2fb (stale) Mirror 25: 0xb62dd2fb (stale) Mirror 26: 0x261bb193 (stale) CRC-32 checksum value for chunk [0, 0x31458): Mirror 1: 0x8eae5e11 Mirror 27: 0x8eae5e11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31458: skipped 000900 1789122093.271317 punch from 0x01ce45 to 0x02a873 (0xda2f bytes) 000901 1789122093.305570 punch from 0x018f14 to 0x021ab7 (0x8ba4 bytes) 000902 1789122093.310177 punch from 0x003e35 to 0x010c8e (0xce5a bytes) 000903 1789122093.315853 write 0x008c07 thru 0x010fc8 (0x83c2 bytes) 000904 1789122093.333748 fallocate 0x0323b4 thru 0x03a3d0 (0x801d bytes) 000905 1789122093.343447 trunc from 0x03a3d1 to 0x030731 (0x9c9f bytes) 000906 1789122093.373847 mapwrite 0x006694 thru 0x012014 (0xb981 bytes) 000907 1789122093.448318 write 0x02ffc7 thru 0x03e7dc (0xe816 bytes) 000908 1789122093.474017 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3e7dd): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e7dd): Mirror 5: 0x25fb6cc4 (stale) Mirror 10: 0x25fb6cc4 (stale) Mirror 12: 0x25fb6cc4 (stale) Mirror 14: 0x25fb6cc4 (stale) Mirror 17: 0x25fb6cc4 (stale) Mirror 20: 0x25fb6cc4 (stale) Mirror 21: 0x25fb6cc4 (stale) Mirror 22: 0x25fb6cc4 (stale) Mirror 23: 0x25fb6cc4 (stale) Mirror 24: 0x25fb6cc4 (stale) Mirror 25: 0x25fb6cc4 (stale) Mirror 26: 0xa6c2d6ac (stale) Mirror 27: 0xd2279c14 (stale) CRC-32 checksum value for chunk [0, 0x3e7dd): Mirror 1: 0x381f7aed Mirror 28: 0x381f7aed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e7dd: skipped 000909 1789122094.848804 mapwrite 0x020726 thru 0x02ac5f (0xa53a bytes) 000910 1789122094.958493 mapwrite 0x03d659 thru 0x03ffff (0x29a7 bytes) 000911 1789122095.073977 trunc from 0x040000 to 0x024a75 (0x1b58a bytes) 000913 1789122095.092363 read 0x0173cf thru 0x01c1a1 (0x4dd3 bytes) 000914 1789122095.098292 read 0x01e81d thru 0x024a75 (0x6259 bytes) 000915 1789122095.098904 fallocate 0x014229 thru 0x01cd26 (0x8afe bytes) 000916 1789122095.109835 trunc from 0x007884 to 0x00c1e2 (0x495f bytes) 000917 1789122095.114873 trunc from 0x024a76 to 0x033c54 (0xf1df bytes) 000918 1789122095.128564 trunc from 0x00d74f to 0x016cf6 (0x95a8 bytes) 000919 1789122095.134561 mapread 0x00f2f8 thru 0x017f41 (0x8c4a bytes) 000920 1789122095.151793 MIRROR_RESYNC 15 mirrors 000921 1789122101.075308 fallocate 0x036140 thru 0x03ffff (0x9ec0 bytes) 000922 1789122101.126729 fallocate 0x0075b1 thru 0x016946 (0xf396 bytes) 000923 1789122101.138652 mapwrite 0x020aa5 thru 0x02c2a4 (0xb800 bytes) 000924 1789122101.247023 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x33c55): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c55): Mirror 5: 0x49c34642 (stale) Mirror 10: 0x49c34642 (stale) Mirror 12: 0x49c34642 (stale) Mirror 14: 0x49c34642 (stale) Mirror 17: 0x49c34642 (stale) Mirror 20: 0x49c34642 (stale) Mirror 21: 0x49c34642 (stale) Mirror 22: 0x49c34642 (stale) Mirror 23: 0x49c34642 (stale) Mirror 24: 0x49c34642 (stale) Mirror 25: 0x49c34642 (stale) Mirror 26: 0x49c34642 (stale) Mirror 27: 0x49c34642 (stale) Mirror 28: 0x49c34642 (stale) CRC-32 checksum value for chunk [0, 0x33c55): Mirror 1: 0x7e222169 Mirror 29: 0x7e222169 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c55: skipped 000925 1789122102.540329 read 0x0190c6 thru 0x022ca5 (0x9be0 bytes) 000926 1789122102.549176 write 0x01a136 thru 0x01c40f (0x22da bytes) 000927 1789122102.601650 punch from 0x00aa82 to 0x010d20 (0x629f bytes) 000928 1789122102.607242 mapread 0x0213f0 thru 0x02e0bc (0xcccd bytes) 000929 1789122102.639114 trunc from 0x033c55 to 0x037b97 (0x3f43 bytes) 000931 1789122102.697419 read 0x009f18 thru 0x00f74a (0x5833 bytes) 000932 1789122102.744116 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37b98): Mirror 5: 0xea806358 (stale) Mirror 10: 0xea806358 (stale) Mirror 12: 0xea806358 (stale) Mirror 14: 0xea806358 (stale) Mirror 17: 0xea806358 (stale) Mirror 20: 0xea806358 (stale) Mirror 21: 0xea806358 (stale) Mirror 22: 0xea806358 (stale) Mirror 23: 0xea806358 (stale) Mirror 24: 0xea806358 (stale) Mirror 26: 0xea806358 (stale) Mirror 27: 0xea806358 (stale) Mirror 28: 0xea806358 (stale) Mirror 29: 0xb6e5fc7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b98): Mirror 5: 0xea806358 (stale) Mirror 10: 0xea806358 (stale) Mirror 12: 0xea806358 (stale) Mirror 14: 0xea806358 (stale) Mirror 17: 0xea806358 (stale) Mirror 20: 0xea806358 (stale) Mirror 21: 0xea806358 (stale) Mirror 22: 0xea806358 (stale) Mirror 23: 0xea806358 (stale) Mirror 24: 0xea806358 (stale) Mirror 26: 0xea806358 (stale) Mirror 27: 0xea806358 (stale) Mirror 28: 0xea806358 (stale) Mirror 29: 0xb6e5fc7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b98: skipped 000935 1789122103.747635 mapread 0x036ae7 thru 0x037b97 (0x10b1 bytes) 000936 1789122103.765598 punch from 0x02b2b3 to 0x037b97 (0xc8e5 bytes) 000937 1789122103.771753 punch from 0x008f7e to 0x0170a5 (0xe128 bytes) 000938 1789122103.776316 mapread 0x014b05 thru 0x01840c (0x3908 bytes) 000940 1789122103.792869 trunc from 0x00c877 to 0x013770 (0x6efa bytes) 000942 1789122103.802992 fallocate 0x02445c thru 0x0324d6 (0xe07b bytes) 000943 1789122103.808805 trunc from 0x0374a8 to 0x037b97 (0x6f0 bytes) 000944 1789122103.816930 write 0x00fb27 thru 0x01d97c (0xde56 bytes) 000945 1789122103.842173 fallocate 0x03620d thru 0x03ffff (0x9df3 bytes) 000946 1789122103.852505 punch from 0x028f3f to 0x02d3ab (0x446d bytes) 000947 1789122103.864298 read 0x014553 thru 0x016f23 (0x29d1 bytes) 000948 1789122103.864906 trunc from 0x0051e7 to 0x01300f (0xde29 bytes) 000949 1789122103.885681 punch from 0x0217ad to 0x0235bf (0x1e13 bytes) 000950 1789122103.891417 fallocate 0x017122 thru 0x01fa23 (0x8902 bytes) 000951 1789122103.904014 MIRROR_SPLIT mirror 7 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37b98): Mirror 5: 0xea806358 (stale) Mirror 10: 0xea806358 (stale) Mirror 12: 0xea806358 (stale) Mirror 14: 0xea806358 (stale) Mirror 17: 0xea806358 (stale) Mirror 20: 0xea806358 (stale) Mirror 22: 0xea806358 (stale) Mirror 23: 0xea806358 (stale) Mirror 24: 0xea806358 (stale) Mirror 26: 0xea806358 (stale) Mirror 27: 0xea806358 (stale) Mirror 28: 0xea806358 (stale) Mirror 29: 0xb6e5fc7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b98): Mirror 5: 0xea806358 (stale) Mirror 10: 0xea806358 (stale) Mirror 12: 0xea806358 (stale) Mirror 14: 0xea806358 (stale) Mirror 17: 0xea806358 (stale) Mirror 20: 0xea806358 (stale) Mirror 22: 0xea806358 (stale) Mirror 23: 0xea806358 (stale) Mirror 24: 0xea806358 (stale) Mirror 26: 0xea806358 (stale) Mirror 27: 0xea806358 (stale) Mirror 28: 0xea806358 (stale) Mirror 29: 0xb6e5fc7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b98: skipped 000952 1789122104.755917 punch from 0x03586a to 0x037b97 (0x232e bytes) 000954 1789122104.763559 mapwrite 0x00c3b7 thru 0x0152d0 (0x8f1a bytes) 000955 1789122104.844438 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x37b98): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b98): Mirror 5: 0xea806358 (stale) Mirror 10: 0xea806358 (stale) Mirror 12: 0xea806358 (stale) Mirror 14: 0xea806358 (stale) Mirror 17: 0xea806358 (stale) Mirror 20: 0xea806358 (stale) Mirror 22: 0xea806358 (stale) Mirror 23: 0xea806358 (stale) Mirror 24: 0xea806358 (stale) Mirror 26: 0xea806358 (stale) Mirror 27: 0xea806358 (stale) Mirror 28: 0xea806358 (stale) Mirror 29: 0xb6e5fc7c (stale) CRC-32 checksum value for chunk [0, 0x37b98): Mirror 1: 0x5e225859 Mirror 30: 0x5e225859 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b98: skipped 000956 1789122106.548879 MIRROR_RESYNC 15 mirrors 000957 1789122112.034242 mapwrite 0x03b894 thru 0x03ffff (0x476c bytes) 000958 1789122112.125103 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6fca7f85 (stale) Mirror 10: 0x6fca7f85 (stale) Mirror 12: 0x6fca7f85 (stale) Mirror 14: 0x6fca7f85 (stale) Mirror 17: 0x6fca7f85 (stale) Mirror 20: 0x6fca7f85 (stale) Mirror 22: 0x6fca7f85 (stale) Mirror 23: 0x6fca7f85 (stale) Mirror 24: 0x6fca7f85 (stale) Mirror 26: 0x6fca7f85 (stale) Mirror 27: 0x6fca7f85 (stale) Mirror 28: 0x6fca7f85 (stale) Mirror 29: 0x6fca7f85 (stale) Mirror 30: 0x6fca7f85 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x376c8e4 Mirror 31: 0x376c8e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000959 1789122113.526208 trunc from 0x009cce to 0x00b776 (0x1aa9 bytes) 000960 1789122113.570560 trunc from 0x040000 to 0x032250 (0xddaf bytes) 000961 1789122113.619759 write 0x01c01e thru 0x01d116 (0x10f9 bytes) 000962 1789122113.629142 mapread 0x01e4fd thru 0x02a1cb (0xbccf bytes) 000963 1789122113.657213 trunc from 0x032251 to 0x0038b8 (0x2e998 bytes) 000964 1789122113.671544 read 0x000264 thru 0x0028bb (0x2658 bytes) 000965 1789122113.677312 mapwrite 0x01e43a thru 0x0297a4 (0xb36b bytes) 000966 1789122113.731526 MIRROR_SPLIT mirror 13 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x297a5): Mirror 5: 0x98114338 (stale) Mirror 10: 0x98114338 (stale) Mirror 12: 0x98114338 (stale) Mirror 14: 0x98114338 (stale) Mirror 17: 0x98114338 (stale) Mirror 20: 0x98114338 (stale) Mirror 22: 0x98114338 (stale) Mirror 23: 0x98114338 (stale) Mirror 24: 0x98114338 (stale) Mirror 26: 0x98114338 (stale) Mirror 27: 0x98114338 (stale) Mirror 28: 0x98114338 (stale) Mirror 30: 0x98114338 (stale) Mirror 31: 0x98114338 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297a5): Mirror 5: 0x98114338 (stale) Mirror 10: 0x98114338 (stale) Mirror 12: 0x98114338 (stale) Mirror 14: 0x98114338 (stale) Mirror 17: 0x98114338 (stale) Mirror 20: 0x98114338 (stale) Mirror 22: 0x98114338 (stale) Mirror 23: 0x98114338 (stale) Mirror 24: 0x98114338 (stale) Mirror 26: 0x98114338 (stale) Mirror 27: 0x98114338 (stale) Mirror 28: 0x98114338 (stale) Mirror 30: 0x98114338 (stale) Mirror 31: 0x98114338 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297a5: skipped 000967 1789122114.741284 trunc from 0x0297a5 to 0x012193 (0x17611 bytes) 000968 1789122114.765484 write 0x02a8a4 thru 0x032d62 (0x84bf bytes) 000969 1789122114.768569 trunc from 0x032d63 to 0x03d3cc (0xa66a bytes) 000970 1789122114.788504 punch from 0x02c5c3 to 0x036eac (0xa8ea bytes) 000972 1789122114.804892 punch from 0x01edcd to 0x02d949 (0xeb7d bytes) 000974 1789122114.810536 mapread 0x008df9 thru 0x00a933 (0x1b3b bytes) 000975 1789122114.829395 fallocate 0x014326 thru 0x01d5fb (0x92d6 bytes) 000976 1789122114.836205 write 0x003d53 thru 0x00cdcd (0x907b bytes) 000977 1789122114.838970 fallocate 0x03bb8c thru 0x03ffff (0x4474 bytes) 000979 1789122114.844980 trunc from 0x03b30e to 0x03ffff (0x4cf2 bytes) 000981 1789122114.850203 punch from 0x0046a2 to 0x00af9e (0x68fd bytes) 000982 1789122114.863361 punch from 0x039d91 to 0x03ffff (0x626f bytes) 000983 1789122114.867846 mapwrite 0x001b19 thru 0x00989d (0x7d85 bytes) 000984 1789122114.905746 punch from 0x00d0e4 to 0x00f54c (0x2469 bytes) 000985 1789122114.917197 read 0x0370d9 thru 0x03af12 (0x3e3a bytes) 000986 1789122114.927648 read 0x00528b thru 0x00554e (0x2c4 bytes) 000987 1789122114.936100 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6fca7f85 (stale) Mirror 10: 0x6fca7f85 (stale) Mirror 12: 0x6fca7f85 (stale) Mirror 14: 0x6fca7f85 (stale) Mirror 17: 0x6fca7f85 (stale) Mirror 20: 0x6fca7f85 (stale) Mirror 22: 0x6fca7f85 (stale) Mirror 23: 0x6fca7f85 (stale) Mirror 24: 0x6fca7f85 (stale) Mirror 26: 0x6fca7f85 (stale) Mirror 28: 0x6fca7f85 (stale) Mirror 30: 0x6fca7f85 (stale) Mirror 31: 0x376c8e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6fca7f85 (stale) Mirror 10: 0x6fca7f85 (stale) Mirror 12: 0x6fca7f85 (stale) Mirror 14: 0x6fca7f85 (stale) Mirror 17: 0x6fca7f85 (stale) Mirror 20: 0x6fca7f85 (stale) Mirror 22: 0x6fca7f85 (stale) Mirror 23: 0x6fca7f85 (stale) Mirror 24: 0x6fca7f85 (stale) Mirror 26: 0x6fca7f85 (stale) Mirror 28: 0x6fca7f85 (stale) Mirror 30: 0x6fca7f85 (stale) Mirror 31: 0x376c8e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000988 1789122116.092546 fallocate 0x017de7 thru 0x019e93 (0x20ad bytes) 000989 1789122116.110220 mapread 0x03c957 thru 0x03ffff (0x36a9 bytes) 000990 1789122116.134150 trunc from 0x040000 to 0x0268de (0x19721 bytes) 000991 1789122116.155386 fallocate 0x01f43c thru 0x021452 (0x2017 bytes) 000992 1789122116.161566 punch from 0x01c277 to 0x0268de (0xa668 bytes) 000995 1789122116.178523 read 0x000439 thru 0x000f66 (0xb2e bytes) 000996 1789122116.203420 trunc from 0x008036 to 0x00e8df (0x68aa bytes) 000997 1789122116.218005 trunc from 0x01c513 to 0x0268de (0xa3cc bytes) 000998 1789122116.233512 write 0x0165b5 thru 0x01af44 (0x4990 bytes) 001000 1789122116.286736 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x268df): Mirror 5: 0x83643121 (stale) Mirror 10: 0x83643121 (stale) Mirror 12: 0x83643121 (stale) Mirror 14: 0x83643121 (stale) Mirror 17: 0x83643121 (stale) Mirror 20: 0x83643121 (stale) Mirror 22: 0x83643121 (stale) Mirror 23: 0x83643121 (stale) Mirror 24: 0x83643121 (stale) Mirror 26: 0x83643121 (stale) Mirror 30: 0x83643121 (stale) Mirror 31: 0x83643121 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268df): Mirror 5: 0x83643121 (stale) Mirror 10: 0x83643121 (stale) Mirror 12: 0x83643121 (stale) Mirror 14: 0x83643121 (stale) Mirror 17: 0x83643121 (stale) Mirror 20: 0x83643121 (stale) Mirror 22: 0x83643121 (stale) Mirror 23: 0x83643121 (stale) Mirror 24: 0x83643121 (stale) Mirror 26: 0x83643121 (stale) Mirror 30: 0x83643121 (stale) Mirror 31: 0x83643121 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268df: skipped All operations completed A-OK! PASS 70a (152s) == sanity-flr test 71: check mirror extend parent fid ==== 06:22:02 (1789122122) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x5e:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 06:22:14 (1789122134) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28505 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160213 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 06:22:24 (1789122144) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3138592 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5946592 .. failed Extending file size to 8084672 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8859264 .. Extending file size to 8202784 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8523424 .. Extending file size to 9130784 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9203168 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62915 62916 62919 62920 62922 done 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) 192.168.206.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:23:45 (1789122225) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8967424 .. Extending file size to 3417760 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5443264 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6324352 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6897280 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7220768 .. failed Extending file size to 9128256 .. Extending file size to 8350400 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8517888 .. Extending file size to 8673216 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9046624 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 64576 64578 64579 64580 64581 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) 192.168.206.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:25:08 (1789122308) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1268788 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2168 1269136 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24652 3582368 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18776 3588244 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43428 7170612 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2516 1268788 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2168 1269136 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24652 3582368 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18776 3588244 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43428 7170612 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130244 s, 16.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233029 s, 9.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xb06f9465 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:25:30 (1789122330) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10163 s, 20.6 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 06:25:38 (1789122338) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x304:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x304:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x305:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x302:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:25:45 (1789122345) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.097697 s, 21.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:25:52 (1789122352) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104555 s, 20.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:25:59 (1789122359) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0783772 s, 26.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x305:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x305:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x306:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:26:32 (1789122392) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg647-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173448 s, 6.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:27:20 (1789122440) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg647-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0999161 s, 21.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:28:08 (1789122488) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg647-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg647-server: Pool lustre.test_204f created Adding targets to pool oleg647-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg647-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10078 s, 20.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg647-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg647-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg647-server: Pool lustre.test_204f destroyed PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:29:16 (1789122556) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:29:25 (1789122565) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:29:32 (1789122572) oleg647-server: Pool lustre.test_206 created Adding targets to pool oleg647-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg647-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg647-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg647-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg647-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:30:00 (1789122600) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964438 s, 10.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325706 s, 32.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd0517322 Mirror 3: 0xd0517322 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:30:11 (1789122611) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:30:15 (1789122615) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:30:19 (1789122619) Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0446559 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ba0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x6b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x6b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x6b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xdf0000 Read this (4 bytes): BBBB 192.168.206.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg647-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:30:56 (1789122656) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0221629 s, 370 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:31:04 (1789122664) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789996 s, 13.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1e4 lmm_fid: [0x200000405:0x1e4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1e4 lmm_fid: [0x200000405:0x1e4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x311:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611620 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 06:31:06.000000000 -0400 Modify: 2026-09-11 06:31:06.000000000 -0400 Change: 2026-09-11 06:31:07.000000000 -0400 Birth: 2026-09-11 06:31:06.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:31:13 (1789122673) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674206 s, 15.6 MB/s oleg647-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg647-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg647-server: O/280000401/d18/312: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:31:37 (1789122697) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.270703 s, 151 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.018986 s, 216 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 06:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e9:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 06:31 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:31:46 (1789122706) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:31:53 (1789122713) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1776 sec =========== 06:32:02 (1789122722) osc.lustre-OST0000-osc-ffff9d05e01b3000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff9d05e01b3000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789122726.329223928 secs.nsecs start_time 1789121357.654054357 secs.nsecs elapsed_time 1368.675169571 secs.nsecs uncompressed_objects 3588 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789122726.329312063 secs.nsecs start_time 1789121392.057145416 secs.nsecs elapsed_time 1334.272166647 secs.nsecs uncompressed_objects 3603 samples [reqs] === sanity-flr: start cleanup 06:32:05 (1789122725) === === sanity-flr: finish cleanup 06:32:14 (1789122734) ===