-----============= acceptance-small: sanity-flr ============----- Fri Oct 2 09:11:53 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg605-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:12:10 (1790946730) === oleg605-client.virtnet: executing check_config_client /mnt/lustre oleg605-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg605-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c0bc61a2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c0bc61a2000.idle_timeout=debug disable quota as required oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:12:41 (1790946761) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8208] == sanity-flr test 0a: lfs mirror create with -N option == 09:12:43 (1790946763) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:12:58 (1790946778) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:13:01 (1790946781) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:13:04 (1790946784) PASS 0d (18s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:13:22 (1790946802) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:13:25 (1790946805) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 09:13:27 (1790946807) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c0bc61a2000.stats=clear osc.lustre-OST0001-osc-ffff8c0bc61a2000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:13:33 (1790946813) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:13:41 (1790946821) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:13:47 (1790946827) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:13:50 (1790946830) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:13:56 (1790946836) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 09:14:03 (1790946843) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:14:09 (1790946849) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:14:15 (1790946855) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:14:21 (1790946861) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:14:29 (1790946869) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:14:34 (1790946874) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.873952 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529 s, 19.8 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:14:41 (1790946881) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.220296 s, 33.3 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:14:47 (1790946887) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:14:55 (1790946895) Stopping /mnt/lustre-ost1 (opts:) on oleg605-server oleg605-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c0bc61a2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c0bc61a2000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg605-server: oleg605-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg605-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg605-server oleg605-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c0bc61a2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c0bc61a2000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg605-server: oleg605-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg605-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (57s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:15:53 (1790946953) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.202785 s, 20.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg605-server oleg605-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c0bc61a2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c0bc61a2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg605-server: oleg605-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg605-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (36s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:16:28 (1790946988) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg605-server oleg605-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c0bc61a2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c0bc61a2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg605-server: oleg605-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg605-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg605-server oleg605-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c0bc61a2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c0bc61a2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg605-server: oleg605-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg605-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (60s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:17:28 (1790947048) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.18617 s, 24.0 MB/s -rw-r--r-- 1 root root 52428800 Oct 2 09:17 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.671s user 0m0.001s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg605-server oleg605-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c0bc61a2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c0bc61a2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.756s user 0m0.001s sys 0m0.170s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg605-server: oleg605-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg605-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg605-server oleg605-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c0bc61a2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c0bc61a2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.992s user 0m0.001s sys 0m0.182s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg605-server: oleg605-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg605-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg605-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c0bc61a2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c0bc61a2000.ost_server_uuid in FULL state after 0 sec PASS 33b (76s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:18:45 (1790947125) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:18:48 (1790947128) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:18:51 (1790947131) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 09:18:54 (1790947134) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 09:19:00 (1790947140) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.393454 s, 32.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:19:09 (1790947149) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231596 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350627 s, 29.9 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:19:15 (1790947155) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:19:22 (1790947162) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011674 s, 877 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000898392 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00159807 s, 641 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000701949 s, 1.5 MB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 09:19:29 (1790947169) create test files with size 15743824 .. 8fc5151c7270f5ffe9d0523153c8cd3e - 6baa5bca793961b1211d1b4abd32071d - c82fe46f44dc20690b901bcc10bdd412 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg605-client.virtnet /mnt/lustre (opts:) Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre PASS 37 (18s) == sanity-flr test 38: resync ============================ 09:19:47 (1790947187) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:20:00 (1790947200) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:20:06 (1790947206) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828323 s, 12.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249744 s, 42.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc3:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc4:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 09:20:14 (1790947214) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0844032 s, 24.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.305628 s, 13.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14095 s, 14.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 09:20:26 (1790947226) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:20:29 (1790947229) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:20:32 (1790947232) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259657 s, 40.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 09:20:39 (1790947239) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.20952 s, 15.0 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:20:46 (1790947246) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc0:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:20:54 (1790947254) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.238464 s, 44.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg605-client.virtnet /mnt/lustre (opts:) Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:21:03 (1790947263) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8c0bc7257800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.298216 s, 35.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg605-client.virtnet /mnt/lustre (opts:) Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8c0bd33c1000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 09:21:17 (1790947277) error lfs split: create victim file failed: File exists PASS 44f (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:21:31 (1790947291) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:22:05 (1790947325) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:22:39 (1790947359) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 09:22:41 (1790947361) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.174468 s, 18.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.158522 s, 19.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x567c441b (nosync,stale) Mirror 2: 0x567c441b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 332a45dcf03da87da77e1864b28bc0f7 - mirror 1 checksum: c110ec6022d170c0ff46ee0841add00f - mirror 2 checksum: 332a45dcf03da87da77e1864b28bc0f7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-02T09:22:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x102:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-02T09:22:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x103:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 332a45dcf03da87da77e1864b28bc0f7 - mirror 1 checksum: c110ec6022d170c0ff46ee0841add00f - mirror 2 checksum: c110ec6022d170c0ff46ee0841add00f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-02T09:22:44-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x102:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-02T09:22:44-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x103:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:22:49 (1790947369) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340818 s, 30.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:22:56 (1790947376) Failing mds1 on oleg605-server Stopping /mnt/lustre-mds1 (opts:) on oleg605-server 09:23:01 (1790947381) shut down facet: mds1 facet_host: oleg605-server facet_failover_host: oleg605-server Failover mds1 to oleg605-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg605-server: oleg605-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg605-client: oleg605-server: ssh exited with exit code 1 Started lustre-MDT0000 09:23:23 (1790947403) targets are mounted 09:23:23 (1790947403) facet_failover done oleg605-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x9:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327668 s, 32.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000404 lmm_object_id: 0x9 lmm_fid: [0x240000404:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21:0x0] } PASS 50A (40s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:23:37 (1790947417) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.026574 s, 771 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0128179 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (7s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:23:45 (1790947425) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0503649 s, 407 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0105135 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (6s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:23:51 (1790947431) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0060969 s, 2.7 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:23:57 (1790947437) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0260453 s, 786 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:24:02 (1790947442) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0118241 s, 346 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0408734 s, 100 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:24:09 (1790947449) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) PASS 60b (17s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:24:26 (1790947466) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:24:53 (1790947493) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:25:15 (1790947515) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:25:35 (1790947535) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 09:25:40 (1790947540) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58194 mirror split pid 58195 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58194 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58195 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 09:26:45 (1790947605) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7607 000001 1790947607.214274 write 0x029000 thru 0x02cc78 (0x3c79 bytes) fallocating to largest ever: 0x249d9 000002 1790947607.222874 fallocate 0x02075f thru 0x0249d8 (0x427a bytes) 000003 1790947607.230884 punch from 0x013912 to 0x022262 (0xe951 bytes) 000004 1790947607.234307 mapread 0x027bf2 thru 0x02cc78 (0x5087 bytes) 000005 1790947607.242953 fallocate 0x00a8a0 thru 0x019d26 (0xf487 bytes) 000006 1790947607.248272 punch from 0x02375a to 0x02cc78 (0x951f bytes) 000008 1790947607.260793 MIRROR_EXTEND to 2 mirrors 000009 1790947607.688337 read 0x005861 thru 0x00e12f (0x88cf bytes) fallocating to largest ever: 0x3ea89 000010 1790947607.698385 fallocate 0x03aac1 thru 0x03ea88 (0x3fc8 bytes) 000012 1790947607.729115 mapread 0x020b01 thru 0x02d263 (0xc763 bytes) 000013 1790947607.751615 trunc from 0x0302f6 to 0x03e095 (0xdda0 bytes) 000014 1790947607.758106 trunc from 0x03ea89 to 0x028cf7 (0x15d91 bytes) 000015 1790947607.793031 mapread 0x002c72 thru 0x00c6ec (0x9a7b bytes) 000018 1790947607.804328 trunc from 0x028cf8 to 0x03b788 (0x12a91 bytes) 000019 1790947607.825493 punch from 0x032f1b to 0x0397c6 (0x68ac bytes) 000020 1790947607.833385 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b789): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b789): Mirror 2: 0xeefb574d (stale) CRC-32 checksum value for chunk [0, 0x3b789): Mirror 1: 0xeefb574d Mirror 3: 0xeefb574d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b789: skipped 000021 1790947608.429387 trunc from 0x039bf6 to 0x03b788 (0x1b93 bytes) 000022 1790947608.456013 read 0x021b3e thru 0x02f9b2 (0xde75 bytes) 000023 1790947608.465931 mapread 0x038cd2 thru 0x03b788 (0x2ab7 bytes) 000024 1790947608.485164 mapread 0x007d38 thru 0x011493 (0x975c bytes) 000025 1790947608.499146 write 0x0076e7 thru 0x014058 (0xc972 bytes) 000026 1790947608.502807 trunc from 0x035886 to 0x03b788 (0x5f03 bytes) 000027 1790947608.509613 fallocate 0x035d90 thru 0x03a0ed (0x435e bytes) 000028 1790947608.514978 read 0x02ac9c thru 0x02eda0 (0x4105 bytes) 000029 1790947608.515864 write 0x00b5fb thru 0x00b9ad (0x3b3 bytes) 000030 1790947608.516549 punch from 0x023323 to 0x029d17 (0x69f5 bytes) 000031 1790947608.520900 mapwrite 0x016894 thru 0x022047 (0xb7b4 bytes) 000032 1790947608.550329 trunc from 0x02eb1e to 0x03b788 (0xcc6b bytes) 000033 1790947608.554076 read 0x01d252 thru 0x02450d (0x72bc bytes) 000034 1790947608.555128 punch from 0x00f9a4 to 0x010943 (0xfa0 bytes) 000035 1790947608.563209 read 0x001ff1 thru 0x00ba23 (0x9a33 bytes) 000036 1790947608.564017 read 0x0200b5 thru 0x022049 (0x1f95 bytes) 000037 1790947608.565403 trunc from 0x03b789 to 0x00c685 (0x2f103 bytes) 000038 1790947608.584566 trunc from 0x00c686 to 0x003e04 (0x8881 bytes) 000039 1790947608.606950 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e05): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e05): Mirror 2: 0xe114df03 (stale) Mirror 3: 0xe114df03 (stale) CRC-32 checksum value for chunk [0, 0x3e05): Mirror 1: 0xe114df03 Mirror 4: 0xe114df03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e05: skipped 000041 1790947609.139548 fallocate 0x01196b thru 0x014d18 (0x33ae bytes) 000042 1790947609.171110 write 0x011ccb thru 0x0123ec (0x722 bytes) 000043 1790947609.184046 write 0x022bdc thru 0x02a2a2 (0x76c7 bytes) 000044 1790947609.185340 write 0x013690 thru 0x0169e3 (0x3354 bytes) 000045 1790947609.193261 trunc from 0x01c102 to 0x026f5b (0xae5a bytes) 000046 1790947609.210101 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a2a3): Mirror 2: 0xde57fb4d (stale) Mirror 3: 0xde57fb4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2a3): Mirror 2: 0xde57fb4d (stale) Mirror 3: 0xde57fb4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2a3: skipped 000047 1790947609.505530 fallocate 0x01f814 thru 0x023429 (0x3c16 bytes) 000049 1790947609.513904 trunc from 0x004764 to 0x0108e4 (0xc181 bytes) 000050 1790947609.518626 mapwrite 0x018a29 thru 0x023aeb (0xb0c3 bytes) 000051 1790947609.569154 fallocate 0x02f967 thru 0x035ab2 (0x614c bytes) 000052 1790947609.579728 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35ab3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ab3): Mirror 2: 0x9524c056 (stale) Mirror 3: 0x9524c056 (stale) CRC-32 checksum value for chunk [0, 0x35ab3): Mirror 1: 0x2f2e7b6a Mirror 4: 0x2f2e7b6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ab3: skipped 000053 1790947610.219996 write 0x037d39 thru 0x03ffff (0x82c7 bytes) 000054 1790947610.248301 punch from 0x033e5a to 0x03cb72 (0x8d19 bytes) 000055 1790947610.264732 fallocate 0x02927f thru 0x036631 (0xd3b3 bytes) 000056 1790947610.268037 mapread 0x0314af thru 0x03375d (0x22af bytes) 000057 1790947610.296527 read 0x003b93 thru 0x00751a (0x3988 bytes) 000058 1790947610.305115 punch from 0x01bdd0 to 0x02516a (0x939b bytes) 000059 1790947610.310404 punch from 0x02a3e5 to 0x0394bd (0xf0d9 bytes) 000060 1790947610.315114 mapread 0x007e5c thru 0x00cd1d (0x4ec2 bytes) 000061 1790947610.331498 MIRROR_RESYNC 4 mirrors 000062 1790947611.204965 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) 000063 1790947611.443892 mapread 0x01dd7c thru 0x02021c (0x24a1 bytes) 000064 1790947611.477098 fallocate 0x0270a4 thru 0x02f213 (0x8170 bytes) 000065 1790947611.522879 read 0x02c8c8 thru 0x031e76 (0x55af bytes) 000067 1790947611.534380 write 0x00956a thru 0x010339 (0x6dd0 bytes) 000068 1790947611.553751 mapread 0x024572 thru 0x02a1a0 (0x5c2f bytes) 000069 1790947611.562366 trunc from 0x040000 to 0x01e4cc (0x21b33 bytes) 000070 1790947611.602501 trunc from 0x01e4cd to 0x01f9cc (0x1500 bytes) 000071 1790947611.628231 write 0x01af84 thru 0x029b91 (0xec0e bytes) 000072 1790947611.648290 trunc from 0x015566 to 0x0155d7 (0x72 bytes) 000074 1790947611.652839 punch from 0x01e1b0 to 0x01f43e (0x128f bytes) 000075 1790947611.670615 punch from 0x016a6a to 0x01c0db (0x5672 bytes) 000076 1790947611.675473 punch from 0x01c85c to 0x01f6cd (0x2e72 bytes) fallocating to largest ever: 0x40000 000077 1790947611.680481 fallocate 0x03dc9e thru 0x03ffff (0x2362 bytes) 000078 1790947611.684882 trunc from 0x01d33e to 0x029b91 (0xc854 bytes) 000079 1790947611.688803 mapread 0x009c8c thru 0x010a81 (0x6df6 bytes) 000080 1790947611.689911 fallocate 0x01ce68 thru 0x021cca (0x4e63 bytes) 000081 1790947611.695023 punch from 0x00bdab to 0x00dd9a (0x1ff0 bytes) 000082 1790947611.700476 mapwrite 0x018179 thru 0x0252aa (0xd132 bytes) 000083 1790947611.728120 read 0x02653d thru 0x029b91 (0x3655 bytes) 000084 1790947611.728728 trunc from 0x029b92 to 0x01f17b (0xaa16 bytes) 000085 1790947611.741385 read 0x000303 thru 0x00d544 (0xd242 bytes) 000087 1790947611.758051 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f17c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f17c): Mirror 3: 0xba94c87a (stale) Mirror 4: 0xba94c87a (stale) CRC-32 checksum value for chunk [0, 0x1f17c): Mirror 1: 0x73b51789 Mirror 5: 0x73b51789 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f17c: skipped 000088 1790947612.359952 MIRROR_RESYNC 4 mirrors 000089 1790947613.017465 trunc from 0x0110c4 to 0x015e72 (0x4daf bytes) 000090 1790947613.059085 trunc from 0x0028a3 to 0x00c468 (0x9bc6 bytes) 000091 1790947613.065473 punch from 0x01d3a8 to 0x01f17b (0x1dd4 bytes) 000092 1790947613.073301 MIRROR_RESYNC 4 mirrors 000094 1790947613.970218 trunc from 0x007546 to 0x00b315 (0x3dd0 bytes) 000095 1790947614.000851 mapread 0x019d16 thru 0x01ccdf (0x2fca bytes) 000096 1790947614.025172 mapread 0x01bb3a thru 0x01f17b (0x3642 bytes) 000097 1790947614.026179 read 0x01d1b6 thru 0x01f17b (0x1fc6 bytes) 000098 1790947614.026713 punch from 0x016f9d to 0x01f17b (0x81df bytes) 000099 1790947614.035171 mapread 0x00a0cf thru 0x0161f1 (0xc123 bytes) 000100 1790947614.061564 trunc from 0x01f17c to 0x0148cf (0xa8ac bytes) 000101 1790947614.098918 punch from 0x0023d8 to 0x008ccb (0x68f4 bytes) 000103 1790947614.109353 mapwrite 0x026941 thru 0x029dd8 (0x3498 bytes) 000105 1790947614.162380 mapwrite 0x017ce4 thru 0x01a78c (0x2aa9 bytes) 000106 1790947614.187795 fallocate 0x01bfae thru 0x024ff1 (0x9044 bytes) 000107 1790947614.200582 trunc from 0x015e60 to 0x0219c9 (0xbb6a bytes) 000108 1790947614.208165 write 0x0364b2 thru 0x03ffa1 (0x9af0 bytes) 000109 1790947614.210046 mapwrite 0x039e3e thru 0x03f869 (0x5a2c bytes) 000110 1790947614.231618 mapread 0x031447 thru 0x03c962 (0xb51c bytes) 000111 1790947614.243925 trunc from 0x01ef76 to 0x021e41 (0x2ecc bytes) 000112 1790947614.250991 mapread 0x03cef7 thru 0x03ffa1 (0x30ab bytes) 000113 1790947614.252434 trunc from 0x00e2dd to 0x01a2fd (0xc021 bytes) 000114 1790947614.259132 punch from 0x028242 to 0x02db8f (0x594e bytes) 000115 1790947614.270503 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ffa2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffa2): Mirror 3: 0x8438e473 (stale) Mirror 4: 0x8438e473 (stale) Mirror 5: 0x8438e473 (stale) CRC-32 checksum value for chunk [0, 0x3ffa2): Mirror 1: 0x3ed427bb Mirror 6: 0x3ed427bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffa2: skipped 000117 1790947614.947134 read 0x0313c1 thru 0x03c30a (0xaf4a bytes) 000119 1790947614.954178 read 0x02daa9 thru 0x034ff7 (0x754f bytes) 000120 1790947614.962792 read 0x03402c thru 0x03c232 (0x8207 bytes) 000121 1790947614.973251 read 0x00c1d4 thru 0x014d2f (0x8b5c bytes) 000122 1790947614.973693 punch from 0x036447 to 0x03fd33 (0x98ed bytes) 000123 1790947615.008955 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ffa2): Mirror 4: 0x8438e473 (stale) Mirror 5: 0x8438e473 (stale) Mirror 6: 0x3ed427bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffa2): Mirror 4: 0x8438e473 (stale) Mirror 5: 0x8438e473 (stale) Mirror 6: 0x3ed427bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffa2: skipped 000124 1790947615.311441 trunc from 0x009ac3 to 0x00aac4 (0x1002 bytes) 000125 1790947615.320287 read 0x028667 thru 0x030292 (0x7c2c bytes) 000126 1790947615.331595 mapwrite 0x004ba9 thru 0x01268b (0xdae3 bytes) 000127 1790947615.382539 fallocate 0x004a59 thru 0x013699 (0xec41 bytes) 000128 1790947615.394210 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ffa2): Mirror 4: 0x8438e473 (stale) Mirror 5: 0x8438e473 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffa2): Mirror 4: 0x8438e473 (stale) Mirror 5: 0x8438e473 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffa2: skipped 000129 1790947615.692114 fallocate 0x0308b9 thru 0x0351f2 (0x493a bytes) 000130 1790947615.699905 punch from 0x010619 to 0x01a960 (0xa348 bytes) 000131 1790947615.704183 mapread 0x012105 thru 0x014cee (0x2bea bytes) 000132 1790947615.721190 trunc from 0x03ffa2 to 0x010a3f (0x2f562 bytes) 000133 1790947615.753032 fallocate 0x034f63 thru 0x03ceb3 (0x7f51 bytes) 000134 1790947615.758638 punch from 0x00f4eb to 0x010a3f (0x1555 bytes) 000135 1790947615.765373 mapwrite 0x03b90e thru 0x03ffff (0x46f2 bytes) 000136 1790947615.799703 trunc from 0x040000 to 0x00a439 (0x35bc6 bytes) 000137 1790947615.821418 MIRROR_RESYNC 3 mirrors 000138 1790947616.424131 trunc from 0x0080b7 to 0x00a439 (0x2383 bytes) 000139 1790947616.478253 mapread 0x009bb7 thru 0x00a439 (0x883 bytes) 000140 1790947616.495131 read 0x009814 thru 0x00a439 (0xc26 bytes) 000141 1790947616.495803 read 0x007591 thru 0x00a439 (0x2ea9 bytes) 000142 1790947616.510048 punch from 0x000420 to 0x000fbb (0xb9c bytes) 000143 1790947616.517007 trunc from 0x001f60 to 0x00a439 (0x84da bytes) 000145 1790947616.525193 punch from 0x001957 to 0x0091ff (0x78a9 bytes) 000146 1790947616.534314 trunc from 0x00a43a to 0x01b320 (0x10ee7 bytes) 000147 1790947616.557446 write 0x004b26 thru 0x013b96 (0xf071 bytes) 000148 1790947616.569113 fallocate 0x02a236 thru 0x030335 (0x6100 bytes) 000149 1790947616.573758 punch from 0x018e09 to 0x01b320 (0x2518 bytes) 000150 1790947616.578024 fallocate 0x0231f5 thru 0x02aaa5 (0x78b1 bytes) 000151 1790947616.583003 write 0x007c60 thru 0x0113a3 (0x9744 bytes) 000152 1790947616.585325 punch from 0x01caea to 0x0235a5 (0x6abc bytes) 000153 1790947616.589448 write 0x02f20c thru 0x0332cd (0x40c2 bytes) 000154 1790947616.593918 MIRROR_RESYNC 3 mirrors 000155 1790947617.220511 mapread 0x027842 thru 0x0332cd (0xba8c bytes) 000156 1790947617.254255 MIRROR_RESYNC 3 mirrors 000157 1790947617.582094 punch from 0x00e499 to 0x0142b6 (0x5e1e bytes) 000158 1790947617.632606 mapwrite 0x01c3aa thru 0x021b3e (0x5795 bytes) 000159 1790947617.690823 read 0x02a380 thru 0x02a878 (0x4f9 bytes) 000160 1790947617.697200 trunc from 0x0332ce to 0x00c97f (0x2694e bytes) 000161 1790947617.739876 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xc980): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc980): Mirror 4: 0x27906e4e (stale) Mirror 5: 0x27906e4e (stale) CRC-32 checksum value for chunk [0, 0xc980): Mirror 1: 0x27906e4e Mirror 6: 0x27906e4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc980: skipped 000162 1790947618.499711 MIRROR_RESYNC 4 mirrors 000163 1790947619.049012 mapwrite 0x00406c thru 0x013197 (0xf12c bytes) 000164 1790947619.175962 MIRROR_RESYNC 4 mirrors 000165 1790947619.771435 MIRROR_EXTEND to 5 mirrors 000166 1790947620.272727 punch from 0x010d95 to 0x013197 (0x2403 bytes) 000167 1790947620.308461 write 0x007b1f thru 0x015c17 (0xe0f9 bytes) 000168 1790947620.321806 mapwrite 0x02b98c thru 0x035494 (0x9b09 bytes) 000169 1790947620.404044 mapwrite 0x02c85e thru 0x037d6b (0xb50e bytes) 000170 1790947620.433499 trunc from 0x01bb7f to 0x02296b (0x6ded bytes) 000171 1790947620.437717 mapwrite 0x012816 thru 0x013263 (0xa4e bytes) 000172 1790947620.448534 write 0x02d7be thru 0x0372e0 (0x9b23 bytes) 000173 1790947620.449976 fallocate 0x026287 thru 0x0283ef (0x2169 bytes) 000174 1790947620.467066 read 0x023c9c thru 0x0319ba (0xdd1f bytes) 000175 1790947620.475993 trunc from 0x0169db to 0x01d1c9 (0x67ef bytes) 000176 1790947620.483881 mapread 0x02466e thru 0x031454 (0xcde7 bytes) 000177 1790947620.486091 trunc from 0x037d6c to 0x0310b9 (0x6cb2 bytes) 000178 1790947620.510990 punch from 0x002a39 to 0x00b222 (0x87ea bytes) 000179 1790947620.516810 punch from 0x012e9b to 0x016031 (0x3197 bytes) 000181 1790947620.521369 mapread 0x009867 thru 0x0151ef (0xb989 bytes) 000182 1790947620.537960 fallocate 0x0391a9 thru 0x03ffff (0x6e57 bytes) 000183 1790947620.543988 write 0x00debb thru 0x01b5e3 (0xd729 bytes) 000184 1790947620.549892 write 0x0204fd thru 0x024d02 (0x4806 bytes) 000185 1790947620.555455 mapwrite 0x01d70b thru 0x0273f5 (0x9ceb bytes) 000186 1790947620.600097 trunc from 0x0310ba to 0x0011cc (0x2feed bytes) 000187 1790947620.617006 trunc from 0x0011cd to 0x01a214 (0x19048 bytes) 000188 1790947620.629634 trunc from 0x0086f0 to 0x010580 (0x7e91 bytes) 000190 1790947620.636637 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1a215): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a215): Mirror 4: 0x5a2ddab2 (stale) Mirror 5: 0x5a2ddab2 (stale) Mirror 6: 0x5a2ddab2 (stale) Mirror 7: 0x5a2ddab2 (stale) CRC-32 checksum value for chunk [0, 0x1a215): Mirror 1: 0x342fde7d Mirror 8: 0x342fde7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a215: skipped 000191 1790947621.247056 fallocate 0x019510 thru 0x023b13 (0xa604 bytes) 000192 1790947621.291000 write 0x00ee94 thru 0x01dfd6 (0xf143 bytes) 000193 1790947621.308543 trunc from 0x023b14 to 0x03db6d (0x1a05a bytes) 000194 1790947621.342594 mapwrite 0x010553 thru 0x020367 (0xfe15 bytes) 000195 1790947621.371301 read 0x02eb64 thru 0x038b1f (0x9fbc bytes) 000196 1790947621.378828 read 0x022096 thru 0x026d09 (0x4c74 bytes) 000197 1790947621.379528 read 0x037c92 thru 0x03ab9c (0x2f0b bytes) 000199 1790947621.387511 mapwrite 0x03c0d8 thru 0x03dfa4 (0x1ecd bytes) 000200 1790947621.406614 punch from 0x0363ad to 0x037e0c (0x1a60 bytes) 000201 1790947621.410605 mapwrite 0x020313 thru 0x021113 (0xe01 bytes) 000203 1790947621.427293 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dfa5): Mirror 4: 0x431d29a7 (stale) Mirror 6: 0x431d29a7 (stale) Mirror 7: 0x431d29a7 (stale) Mirror 8: 0x730549f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfa5): Mirror 4: 0x431d29a7 (stale) Mirror 6: 0x431d29a7 (stale) Mirror 7: 0x431d29a7 (stale) Mirror 8: 0x730549f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfa5: skipped 000204 1790947621.736543 trunc from 0x00b701 to 0x00de72 (0x2772 bytes) 000205 1790947621.742006 write 0x01886d thru 0x02157a (0x8d0e bytes) 000206 1790947621.779240 MIRROR_RESYNC 5 mirrors 000207 1790947622.803100 mapread 0x027e49 thru 0x03181f (0x99d7 bytes) 000208 1790947622.833003 punch from 0x011997 to 0x014fe8 (0x3652 bytes) 000209 1790947622.889072 fallocate 0x03a51c thru 0x03f5a5 (0x508a bytes) 000211 1790947622.907406 read 0x027168 thru 0x027b87 (0xa20 bytes) 000212 1790947622.923122 read 0x017559 thru 0x0259cd (0xe475 bytes) 000213 1790947622.961516 fallocate 0x00b06c thru 0x014c92 (0x9c27 bytes) 000214 1790947622.972871 write 0x031265 thru 0x03ffff (0xed9b bytes) 000215 1790947622.976554 trunc from 0x040000 to 0x008372 (0x37c8d bytes) 000216 1790947623.013152 mapwrite 0x0234bd thru 0x024ccc (0x1810 bytes) 000217 1790947623.064901 fallocate 0x03a6a1 thru 0x03ffff (0x595f bytes) 000218 1790947623.073647 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24ccd): Mirror 6: 0xbd52112a (stale) Mirror 7: 0xbd52112a (stale) Mirror 8: 0xbd52112a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ccd): Mirror 6: 0xbd52112a (stale) Mirror 7: 0xbd52112a (stale) Mirror 8: 0xbd52112a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ccd: skipped 000219 1790947623.386599 read 0x01a325 thru 0x01c3f3 (0x20cf bytes) 000220 1790947623.394639 mapread 0x020bcc thru 0x024ccc (0x4101 bytes) 000221 1790947623.417923 fallocate 0x0368c1 thru 0x03f6d7 (0x8e17 bytes) 000222 1790947623.428153 mapread 0x003b01 thru 0x00b4e4 (0x79e4 bytes) 000223 1790947623.438949 write 0x03fd2d thru 0x03ffff (0x2d3 bytes) 000224 1790947623.443011 punch from 0x01c925 to 0x026d38 (0xa414 bytes) 000225 1790947623.449416 trunc from 0x040000 to 0x00e39d (0x31c62 bytes) 000226 1790947623.460460 write 0x003b9c thru 0x009c95 (0x60fa bytes) 000227 1790947623.461699 read 0x002591 thru 0x00e39d (0xbe0d bytes) 000228 1790947623.462881 read 0x00c694 thru 0x00e39d (0x1d0a bytes) 000229 1790947623.463482 trunc from 0x0092bb to 0x00e39d (0x50e3 bytes) 000230 1790947623.473760 fallocate 0x03c325 thru 0x03ffff (0x3cdb bytes) 000231 1790947623.482260 MIRROR_RESYNC 4 mirrors 000232 1790947624.053193 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) 000233 1790947624.305275 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) 000234 1790947624.557647 mapwrite 0x01f3b2 thru 0x0293e2 (0xa031 bytes) 000235 1790947624.642522 punch from 0x00e003 to 0x013d95 (0x5d93 bytes) 000236 1790947624.651916 read 0x0263f9 thru 0x02ead3 (0x86db bytes) 000237 1790947624.653604 mapwrite 0x020c04 thru 0x023651 (0x2a4e bytes) 000238 1790947624.675346 mapread 0x022d10 thru 0x02d64a (0xa93b bytes) 000239 1790947624.676886 write 0x035b42 thru 0x03ffff (0xa4be bytes) 000240 1790947624.690157 mapread 0x03d888 thru 0x03ffff (0x2778 bytes) 000241 1790947624.691017 trunc from 0x040000 to 0x005990 (0x3a66f bytes) 000242 1790947624.727909 MIRROR_RESYNC 2 mirrors 000243 1790947625.142350 read 0x005051 thru 0x005990 (0x940 bytes) 000244 1790947625.145805 mapwrite 0x00ca81 thru 0x00d69b (0xc1b bytes) 000245 1790947625.202019 fallocate 0x0211e7 thru 0x0247ef (0x3609 bytes) 000246 1790947625.207932 read 0x00c90b thru 0x00d69b (0xd91 bytes) 000247 1790947625.209097 trunc from 0x00a1bd to 0x00d69b (0x34df bytes) 000248 1790947625.218113 write 0x029e02 thru 0x02f098 (0x5297 bytes) 000249 1790947625.219628 trunc from 0x02ab2f to 0x02f098 (0x456a bytes) 000250 1790947625.233551 read 0x0153ec thru 0x0214c1 (0xc0d6 bytes) 000251 1790947625.249481 punch from 0x01b0dc to 0x0201a5 (0x50ca bytes) 000252 1790947625.264511 trunc from 0x00ad61 to 0x018010 (0xd2b0 bytes) 000253 1790947625.270570 mapwrite 0x0207b0 thru 0x024b2e (0x437f bytes) 000254 1790947625.289210 write 0x00d6fa thru 0x00f405 (0x1d0c bytes) 000255 1790947625.301633 fallocate 0x00eed9 thru 0x015fef (0x7117 bytes) 000256 1790947625.305456 write 0x01973d thru 0x022040 (0x8904 bytes) 000258 1790947625.307398 write 0x015020 thru 0x018a79 (0x3a5a bytes) 000259 1790947625.323633 read 0x00cf30 thru 0x019614 (0xc6e5 bytes) 000260 1790947625.334962 punch from 0x021f6d to 0x02f014 (0xd0a8 bytes) 000261 1790947625.352622 write 0x039fa1 thru 0x03ffff (0x605f bytes) 000262 1790947625.359456 MIRROR_RESYNC 2 mirrors 000263 1790947625.878307 punch from 0x029e91 to 0x031632 (0x77a2 bytes) 000264 1790947625.906575 read 0x02530f thru 0x02f2a4 (0x9f96 bytes) 000265 1790947625.913222 write 0x02245f thru 0x02d7b2 (0xb354 bytes) 000266 1790947625.918064 punch from 0x0261e6 to 0x031e97 (0xbcb2 bytes) 000267 1790947625.940015 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) 000268 1790947626.061964 read 0x0219c6 thru 0x0305e7 (0xec22 bytes) 000269 1790947626.074224 read 0x03e44e thru 0x03ffff (0x1bb2 bytes) 000270 1790947626.091958 MIRROR_EXTEND to 2 mirrors 000271 1790947626.628563 mapread 0x02b4fd thru 0x034296 (0x8d9a bytes) 000272 1790947626.655417 mapread 0x00e1f2 thru 0x01b659 (0xd468 bytes) 000273 1790947626.686993 MIRROR_EXTEND to 3 mirrors 000274 1790947627.247054 mapwrite 0x03422a thru 0x03f708 (0xb4df bytes) 000275 1790947627.343389 trunc from 0x040000 to 0x01d8ff (0x22700 bytes) 000276 1790947627.380309 mapread 0x00de2f thru 0x00f949 (0x1b1b bytes) 000277 1790947627.393633 mapwrite 0x00debb thru 0x0112ee (0x3434 bytes) 000278 1790947627.422141 MIRROR_RESYNC 3 mirrors 000279 1790947628.251272 mapread 0x014f5a thru 0x015af4 (0xb9b bytes) 000280 1790947628.263976 trunc from 0x01d900 to 0x00a725 (0x131da bytes) 000281 1790947628.328771 mapwrite 0x009893 thru 0x00e95b (0x50c9 bytes) 000282 1790947628.426887 write 0x03fb25 thru 0x03ffff (0x4db bytes) 000283 1790947628.428768 read 0x035a1e thru 0x03ffff (0xa5e2 bytes) 000284 1790947628.435273 trunc from 0x024e1e to 0x02ef7c (0xa15f bytes) 000285 1790947628.439217 trunc from 0x02de2b to 0x031a02 (0x3bd8 bytes) 000286 1790947628.443435 read 0x01f147 thru 0x020242 (0x10fc bytes) 000287 1790947628.453277 mapread 0x028967 thru 0x02e1a0 (0x583a bytes) 000288 1790947628.454569 trunc from 0x040000 to 0x03c492 (0x3b6d bytes) 000289 1790947628.471221 trunc from 0x03c493 to 0x016f28 (0x2556a bytes) 000290 1790947628.489716 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x16f29): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16f29): Mirror 2: 0x213115d4 (stale) Mirror 3: 0x213115d4 (stale) CRC-32 checksum value for chunk [0, 0x16f29): Mirror 1: 0x5f8908ca Mirror 4: 0x5f8908ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16f29: skipped 000291 1790947629.247699 mapwrite 0x00556b thru 0x011ef7 (0xc98d bytes) 000292 1790947629.363608 fallocate 0x02e67d thru 0x03d9ba (0xf33e bytes) 000293 1790947629.367910 mapread 0x0125a8 thru 0x0160ba (0x3b13 bytes) 000294 1790947629.369057 read 0x0168f9 thru 0x016f28 (0x630 bytes) 000295 1790947629.369651 mapwrite 0x00c4ac thru 0x010ce1 (0x4836 bytes) 000296 1790947629.389258 read 0x002a45 thru 0x00f843 (0xcdff bytes) 000297 1790947629.389964 mapread 0x00e964 thru 0x016f28 (0x85c5 bytes) 000298 1790947629.390901 trunc from 0x0088e7 to 0x010723 (0x7e3d bytes) 000299 1790947629.397857 mapwrite 0x0385f7 thru 0x03f1e8 (0x6bf2 bytes) 000301 1790947629.460555 read 0x01c313 thru 0x0207f1 (0x44df bytes) 000303 1790947629.461797 trunc from 0x03f1e9 to 0x02be65 (0x13383 bytes) 000304 1790947629.476119 MIRROR_RESYNC 4 mirrors 000305 1790947630.388560 fallocate 0x00fa9d thru 0x01c2c5 (0xc829 bytes) 000306 1790947630.431732 write 0x0298fd thru 0x02b8dd (0x1fe1 bytes) 000307 1790947630.449085 punch from 0x006f39 to 0x00877c (0x1844 bytes) 000308 1790947630.455474 punch from 0x0182fe to 0x018bc5 (0x8c8 bytes) 000309 1790947630.462969 mapwrite 0x033fb1 thru 0x03768c (0x36dc bytes) 000310 1790947630.519056 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3768d): Mirror 2: 0x23158292 (stale) Mirror 4: 0x23158292 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3768d): Mirror 2: 0x23158292 (stale) Mirror 4: 0x23158292 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3768d: skipped 000311 1790947630.767756 mapwrite 0x01ef0e thru 0x02a74a (0xb83d bytes) 000313 1790947630.836080 fallocate 0x0385c9 thru 0x03ffff (0x7a37 bytes) 000314 1790947630.841354 write 0x02e43c thru 0x02f0b6 (0xc7b bytes) 000315 1790947630.849585 punch from 0x027bd6 to 0x02b92f (0x3d5a bytes) 000316 1790947630.856984 punch from 0x01abe1 to 0x021fd4 (0x73f4 bytes) 000317 1790947630.862747 mapread 0x03e46f thru 0x03ffff (0x1b91 bytes) 000318 1790947630.879260 punch from 0x0177db to 0x01a78e (0x2fb4 bytes) 000319 1790947630.884610 read 0x01a1c7 thru 0x02245a (0x8294 bytes) 000320 1790947630.894080 punch from 0x0291c5 to 0x02a2ed (0x1129 bytes) 000321 1790947630.899722 trunc from 0x040000 to 0x01f5a7 (0x20a58 bytes) 000322 1790947630.917585 write 0x007da2 thru 0x0167b7 (0xea16 bytes) 000323 1790947630.929574 mapread 0x01d24e thru 0x01f5a7 (0x235a bytes) 000324 1790947630.931056 mapwrite 0x00b0ed thru 0x00ef79 (0x3e8d bytes) 000325 1790947630.959118 trunc from 0x01f5a8 to 0x0181f4 (0x73b3 bytes) 000326 1790947630.986829 punch from 0x016bd3 to 0x0181f4 (0x1622 bytes) 000327 1790947630.992144 read 0x00bda8 thru 0x016967 (0xabc0 bytes) 000328 1790947631.012402 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x181f5): Mirror 2: 0x4105fe41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x181f5): Mirror 2: 0x4105fe41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x181f5: skipped 000329 1790947631.243999 trunc from 0x001a62 to 0x010cdf (0xf27e bytes) 000330 1790947631.248854 fallocate 0x020650 thru 0x02cb7a (0xc52b bytes) 000331 1790947631.252509 write 0x02357f thru 0x02721c (0x3c9e bytes) 000332 1790947631.264782 punch from 0x005a88 to 0x010e53 (0xb3cc bytes) 000333 1790947631.272855 trunc from 0x02cb7b to 0x038b27 (0xbfad bytes) 000334 1790947631.295058 read 0x033a8f thru 0x038b27 (0x5099 bytes) 000335 1790947631.303921 punch from 0x031ed4 to 0x035090 (0x31bd bytes) 000336 1790947631.307456 trunc from 0x038b28 to 0x00d2bc (0x2b86b bytes) 000337 1790947631.329805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xd2bd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd2bd): Mirror 2: 0x80a3737a (stale) CRC-32 checksum value for chunk [0, 0xd2bd): Mirror 1: 0x42c537a5 Mirror 3: 0x42c537a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd2bd: skipped 000338 1790947631.899069 trunc from 0x00d2bd to 0x02a64c (0x1d390 bytes) 000340 1790947631.988624 read 0x01f2bd thru 0x028217 (0x8f5b bytes) 000341 1790947631.995432 read 0x02521f thru 0x02a64c (0x542e bytes) 000342 1790947632.004618 trunc from 0x022aea to 0x02a64c (0x7b63 bytes) 000344 1790947632.010197 read 0x01dfc0 thru 0x0235de (0x561f bytes) 000346 1790947632.020583 mapread 0x00376f thru 0x00ff17 (0xc7a9 bytes) 000347 1790947632.027085 MIRROR_RESYNC 3 mirrors 000348 1790947632.455953 fallocate 0x03cd88 thru 0x03ffff (0x3278 bytes) 000349 1790947632.494013 mapwrite 0x027edb thru 0x02cbec (0x4d12 bytes) 000351 1790947632.568466 write 0x01ed8d thru 0x020cd2 (0x1f46 bytes) 000352 1790947632.585050 read 0x02bf52 thru 0x02cbec (0xc9b bytes) 000353 1790947632.585805 fallocate 0x016e3c thru 0x02547a (0xe63f bytes) 000354 1790947632.598777 MIRROR_RESYNC 3 mirrors 000355 1790947633.104372 fallocate 0x026fd6 thru 0x02db15 (0x6b40 bytes) 000356 1790947633.145044 write 0x02186d thru 0x029158 (0x78ec bytes) 000358 1790947633.157261 write 0x00287f thru 0x004de8 (0x256a bytes) 000359 1790947633.163472 read 0x017ee6 thru 0x01eb88 (0x6ca3 bytes) 000360 1790947633.171191 mapwrite 0x01599c thru 0x022d3c (0xd3a1 bytes) 000361 1790947633.203702 mapwrite 0x023e30 thru 0x0265af (0x2780 bytes) 000362 1790947633.216390 mapread 0x012478 thru 0x018a1a (0x65a3 bytes) 000363 1790947633.223273 trunc from 0x01d255 to 0x0216c7 (0x4473 bytes) 000364 1790947633.227031 trunc from 0x02db16 to 0x02cc2e (0xee7 bytes) 000365 1790947633.263534 mapwrite 0x03d15e thru 0x03ffff (0x2ea2 bytes) 000366 1790947633.297254 mapwrite 0x02ce12 thru 0x0349ba (0x7ba9 bytes) 000368 1790947633.323413 write 0x03c095 thru 0x03ffff (0x3f6b bytes) 000369 1790947633.324441 mapwrite 0x031851 thru 0x03ffff (0xe7af bytes) 000370 1790947633.339297 trunc from 0x040000 to 0x02cf75 (0x1308a bytes) 000371 1790947633.354351 write 0x01978f thru 0x0224e8 (0x8d5a bytes) 000372 1790947633.355692 trunc from 0x02cf76 to 0x0077c6 (0x257af bytes) 000373 1790947633.374611 mapread 0x0024e7 thru 0x0077c6 (0x52e0 bytes) 000374 1790947633.381990 read 0x00700d thru 0x0077c6 (0x7ba bytes) 000375 1790947633.383040 mapread 0x005a00 thru 0x0077c6 (0x1dc7 bytes) 000376 1790947633.383879 trunc from 0x0038b9 to 0x0077c6 (0x3f0e bytes) 000377 1790947633.388174 mapwrite 0x0288d8 thru 0x02d7d5 (0x4efe bytes) 000379 1790947633.420187 punch from 0x028d16 to 0x02d7d5 (0x4ac0 bytes) 000380 1790947633.425760 trunc from 0x02d7d6 to 0x02647f (0x7356 bytes) 000381 1790947633.437070 read 0x00d5ae thru 0x019d99 (0xc7ec bytes) 000382 1790947633.438037 trunc from 0x026480 to 0x03025e (0x9ddf bytes) 000383 1790947633.448601 read 0x00589f thru 0x0147c0 (0xef22 bytes) 000384 1790947633.449318 punch from 0x018142 to 0x02478f (0xc64e bytes) 000385 1790947633.453683 read 0x02ff99 thru 0x03025e (0x2c6 bytes) 000386 1790947633.459297 mapread 0x025d9d thru 0x029483 (0x36e7 bytes) 000387 1790947633.460192 fallocate 0x008282 thru 0x016cf9 (0xea78 bytes) 000388 1790947633.468582 mapwrite 0x032bfb thru 0x03942d (0x6833 bytes) 000389 1790947633.528574 fallocate 0x01fcf5 thru 0x02476d (0x4a79 bytes) 000390 1790947633.541444 trunc from 0x03942e to 0x02757f (0x11eae bytes) 000392 1790947633.569784 read 0x02462e thru 0x02757f (0x2f52 bytes) 000393 1790947633.570369 mapwrite 0x02f52b thru 0x031bfa (0x26d0 bytes) 000394 1790947633.613027 trunc from 0x030c07 to 0x031bfa (0xff4 bytes) 000395 1790947633.622837 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31bfb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31bfb): Mirror 2: 0xdc591f39 (stale) Mirror 3: 0xdc591f39 (stale) CRC-32 checksum value for chunk [0, 0x31bfb): Mirror 1: 0xe92800cc Mirror 4: 0xe92800cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31bfb: skipped 000396 1790947634.285652 fallocate 0x03fd71 thru 0x03ffff (0x28f bytes) 000397 1790947634.336835 mapread 0x033406 thru 0x039dbe (0x69b9 bytes) 000399 1790947634.374547 trunc from 0x01d5e0 to 0x01db1a (0x53b bytes) 000400 1790947634.379013 trunc from 0x034533 to 0x03ffff (0xbacd bytes) 000401 1790947634.396435 trunc from 0x040000 to 0x0342c2 (0xbd3d bytes) 000402 1790947634.453577 mapwrite 0x0255da thru 0x033fe7 (0xea0e bytes) 000403 1790947634.498047 trunc from 0x02d094 to 0x0342c2 (0x722f bytes) 000404 1790947634.502506 trunc from 0x00c9be to 0x015957 (0x8f9a bytes) 000405 1790947634.507764 write 0x030cff thru 0x03fb59 (0xee5b bytes) 000406 1790947634.523810 trunc from 0x03fb5a to 0x010ce1 (0x2ee78 bytes) 000408 1790947634.556655 trunc from 0x00bca7 to 0x010ce1 (0x503b bytes) 000409 1790947634.569923 trunc from 0x010ce2 to 0x007281 (0x9a60 bytes) 000410 1790947634.600188 mapwrite 0x00a98a thru 0x013d87 (0x93fe bytes) 000411 1790947634.716648 mapread 0x005936 thru 0x013d87 (0xe452 bytes) 000412 1790947634.718025 trunc from 0x0125f4 to 0x013d87 (0x1794 bytes) 000413 1790947634.729733 write 0x01d57b thru 0x027de0 (0xa866 bytes) 000414 1790947634.732934 write 0x03ff0b thru 0x03ffff (0xf5 bytes) 000415 1790947634.735020 fallocate 0x018692 thru 0x025f0f (0xd87e bytes) 000416 1790947634.744481 fallocate 0x0285b3 thru 0x03143c (0x8e8a bytes) 000417 1790947634.752886 mapwrite 0x004272 thru 0x011150 (0xcedf bytes) 000418 1790947634.794422 fallocate 0x0343e6 thru 0x03ffff (0xbc1a bytes) 000419 1790947634.800614 write 0x03cd41 thru 0x03ffff (0x32bf bytes) 000420 1790947634.817429 MIRROR_RESYNC 4 mirrors 000421 1790947636.055103 MIRROR_EXTEND to 5 mirrors 000422 1790947636.770238 MIRROR_RESYNC 5 mirrors 000423 1790947637.265276 trunc from 0x00e070 to 0x01a2a1 (0xc232 bytes) 000424 1790947637.326461 read 0x02152e thru 0x02ba97 (0xa56a bytes) 000425 1790947637.337589 read 0x01948f thru 0x01dacb (0x463d bytes) 000426 1790947637.351644 trunc from 0x0304ca to 0x035500 (0x5037 bytes) 000427 1790947637.357090 fallocate 0x01e52b thru 0x01f466 (0xf3c bytes) 000428 1790947637.362189 mapwrite 0x0019c4 thru 0x00e325 (0xc962 bytes) 000429 1790947637.408183 mapread 0x01942c thru 0x026ff9 (0xdbce bytes) 000430 1790947637.409224 punch from 0x018ec5 to 0x01b6cd (0x2809 bytes) 000431 1790947637.414906 trunc from 0x040000 to 0x03797d (0x8682 bytes) 000432 1790947637.443538 write 0x038d3e thru 0x03ffff (0x72c2 bytes) 000433 1790947637.445485 read 0x009942 thru 0x00da6d (0x412c bytes) 000434 1790947637.446320 write 0x03fde4 thru 0x03ffff (0x21c bytes) 000435 1790947637.446964 punch from 0x030846 to 0x03d61b (0xcdd6 bytes) 000437 1790947637.458753 write 0x01598e thru 0x020735 (0xada8 bytes) 000440 1790947637.466718 punch from 0x00090c to 0x003fa8 (0x369d bytes) 000441 1790947637.480632 punch from 0x03404b to 0x03ec69 (0xac1f bytes) 000442 1790947637.487872 read 0x01fc54 thru 0x025833 (0x5be0 bytes) 000443 1790947637.489215 read 0x03c9a0 thru 0x03ffff (0x3660 bytes) 000445 1790947637.517467 trunc from 0x040000 to 0x007f91 (0x3806e bytes) 000446 1790947637.544347 read 0x00586c thru 0x007e76 (0x260b bytes) 000447 1790947637.544947 mapread 0x0000bf thru 0x007f91 (0x7ed3 bytes) 000448 1790947637.545842 trunc from 0x007f92 to 0x029973 (0x219e2 bytes) 000449 1790947637.593357 trunc from 0x029974 to 0x003230 (0x26743 bytes) 000450 1790947637.631185 read 0x001ff4 thru 0x003230 (0x123d bytes) 000451 1790947637.633382 punch from 0x000d67 to 0x003230 (0x24ca bytes) 000452 1790947637.642352 trunc from 0x003231 to 0x02ce53 (0x29c23 bytes) 000453 1790947637.680930 trunc from 0x00f92a to 0x01cab1 (0xd188 bytes) 000454 1790947637.691850 write 0x03def6 thru 0x03ffff (0x210a bytes) 000455 1790947637.693850 punch from 0x03a621 to 0x03cf87 (0x2967 bytes) 000456 1790947637.725714 mapwrite 0x03fd33 thru 0x03ffff (0x2cd bytes) 000457 1790947637.749363 write 0x016531 thru 0x01659e (0x6e bytes) 000458 1790947637.779926 write 0x02c724 thru 0x030777 (0x4054 bytes) 000459 1790947637.780973 read 0x01563c thru 0x01617f (0xb44 bytes) 000460 1790947637.781491 punch from 0x00c1e8 to 0x0158a7 (0x96c0 bytes) 000461 1790947637.789729 trunc from 0x006cfb to 0x00c723 (0x5a29 bytes) 000462 1790947637.795410 trunc from 0x040000 to 0x0317e2 (0xe81d bytes) 000463 1790947637.824810 write 0x019bbd thru 0x027a5c (0xdea0 bytes) 000464 1790947637.826184 trunc from 0x0317e3 to 0x00d010 (0x247d2 bytes) 000465 1790947637.843642 fallocate 0x00783e thru 0x008806 (0xfc9 bytes) 000466 1790947637.847982 write 0x02f5e2 thru 0x037f50 (0x896f bytes) 000467 1790947637.850372 fallocate 0x00e295 thru 0x01e062 (0xfdce bytes) 000468 1790947637.854901 write 0x02f0f2 thru 0x03661a (0x7529 bytes) 000469 1790947637.856403 mapwrite 0x003999 thru 0x00dd48 (0xa3b0 bytes) 000470 1790947637.909896 MIRROR_RESYNC 5 mirrors 000471 1790947639.057384 mapwrite 0x022cc5 thru 0x02a5b3 (0x78ef bytes) 000472 1790947639.152714 write 0x00708a thru 0x01032f (0x92a6 bytes) 000473 1790947639.176832 read 0x02b32e thru 0x02c100 (0xdd3 bytes) 000474 1790947639.189835 read 0x007891 thru 0x014721 (0xce91 bytes) 000476 1790947639.211459 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37f51): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f51): Mirror 2: 0x14d2c099 (stale) Mirror 3: 0x14d2c099 (stale) Mirror 4: 0x14d2c099 (stale) Mirror 5: 0x14d2c099 (stale) CRC-32 checksum value for chunk [0, 0x37f51): Mirror 1: 0xc6846458 Mirror 6: 0xc6846458 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f51: skipped 000477 1790947640.368993 trunc from 0x037f51 to 0x01e298 (0x19cb8 bytes) 000478 1790947640.438953 mapread 0x007b12 thru 0x00fc1d (0x810c bytes) 000479 1790947640.454810 write 0x00c71e thru 0x01877e (0xc061 bytes) 000480 1790947640.459921 punch from 0x01b42f to 0x01e298 (0x2e6a bytes) 000481 1790947640.463881 mapread 0x00f464 thru 0x015277 (0x5e14 bytes) 000482 1790947640.464827 trunc from 0x008d74 to 0x0166e3 (0xd970 bytes) 000483 1790947640.479216 trunc from 0x01d5ea to 0x01e298 (0xcaf bytes) 000484 1790947640.483474 mapread 0x0058df thru 0x00d80f (0x7f31 bytes) 000486 1790947640.491601 mapwrite 0x0167d4 thru 0x0183af (0x1bdc bytes) 000487 1790947640.508375 read 0x01a859 thru 0x01e298 (0x3a40 bytes) 000488 1790947640.512822 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1e299): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e299): Mirror 2: 0xce981428 (stale) Mirror 3: 0xce981428 (stale) Mirror 4: 0xce981428 (stale) Mirror 5: 0xce981428 (stale) Mirror 6: 0x2829765 (stale) CRC-32 checksum value for chunk [0, 0x1e299): Mirror 1: 0x18005265 Mirror 7: 0x18005265 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e299: skipped 000489 1790947641.333743 punch from 0x012f8e to 0x016616 (0x3689 bytes) 000490 1790947641.405861 trunc from 0x008958 to 0x0162da (0xd983 bytes) 000491 1790947641.413474 trunc from 0x0008de to 0x00b422 (0xab45 bytes) 000493 1790947641.423704 trunc from 0x01e299 to 0x00a750 (0x13b48 bytes) 000494 1790947641.475412 read 0x0033fe thru 0x0064dd (0x30e0 bytes) 000495 1790947641.481514 fallocate 0x019986 thru 0x01d583 (0x3bfe bytes) 000496 1790947641.486981 mapread 0x008e3b thru 0x00a745 (0x190b bytes) 000497 1790947641.501195 mapwrite 0x03c79f thru 0x03ffff (0x3861 bytes) 000498 1790947641.529702 trunc from 0x040000 to 0x02c6be (0x13941 bytes) 000499 1790947641.543458 trunc from 0x02c6bf to 0x002fb8 (0x29706 bytes) 000500 1790947641.577979 trunc from 0x002fb9 to 0x034cc7 (0x31d0f bytes) 000501 1790947641.607150 fallocate 0x011e8a thru 0x012deb (0xf62 bytes) 000502 1790947641.615260 fallocate 0x030018 thru 0x03b3e1 (0xb3ca bytes) 000503 1790947641.623156 mapread 0x00a9c6 thru 0x0156ac (0xace7 bytes) 000504 1790947641.649820 mapwrite 0x01e963 thru 0x02e6b2 (0xfd50 bytes) 000505 1790947641.721512 trunc from 0x03b3e2 to 0x01fe8e (0x1b553 bytes) 000506 1790947641.748207 mapwrite 0x007e80 thru 0x014840 (0xc9c1 bytes) 000507 1790947641.765886 write 0x01c250 thru 0x02a228 (0xdfd9 bytes) 000508 1790947641.767523 write 0x01231e thru 0x018fb4 (0x6c97 bytes) 000509 1790947641.769789 trunc from 0x01d734 to 0x029471 (0xbd3e bytes) 000510 1790947641.782122 mapwrite 0x006ce9 thru 0x010dfd (0xa115 bytes) 000511 1790947641.799813 mapwrite 0x001202 thru 0x0106ea (0xf4e9 bytes) 000512 1790947641.833230 trunc from 0x02a229 to 0x01db4b (0xc6dd bytes) 000513 1790947641.852921 write 0x0331c9 thru 0x03554d (0x2385 bytes) 000514 1790947641.853894 mapwrite 0x0126ce thru 0x01e7b3 (0xc0e6 bytes) 000516 1790947641.888331 write 0x03fd89 thru 0x03ffff (0x277 bytes) 000517 1790947641.893364 MIRROR_RESYNC 7 mirrors 000518 1790947643.980144 mapwrite 0x00bfc4 thru 0x010a8d (0x4aca bytes) 000519 1790947644.161065 mapread 0x03d87d thru 0x03ffff (0x2783 bytes) 000520 1790947644.179128 mapread 0x01d39f thru 0x0212a1 (0x3f03 bytes) 000521 1790947644.190613 trunc from 0x01ca98 to 0x027ed0 (0xb439 bytes) 000523 1790947644.196976 fallocate 0x01566e thru 0x01691b (0x12ae bytes) 000524 1790947644.204872 mapwrite 0x02b551 thru 0x03645f (0xaf0f bytes) 000525 1790947644.306538 mapread 0x00ea99 thru 0x00fae1 (0x1049 bytes) 000526 1790947644.312573 MIRROR_RESYNC 7 mirrors 000527 1790947646.214149 trunc from 0x040000 to 0x036a64 (0x959b bytes) 000528 1790947646.289595 read 0x00b67a thru 0x014796 (0x911d bytes) 000529 1790947646.301568 write 0x01eec0 thru 0x022ece (0x400f bytes) 000530 1790947646.314995 write 0x014ed5 thru 0x018374 (0x34a0 bytes) 000531 1790947646.329593 MIRROR_RESYNC 7 mirrors 000532 1790947648.955145 read 0x001fe9 thru 0x0112a0 (0xf2b8 bytes) 000533 1790947648.962802 trunc from 0x0163bc to 0x017b98 (0x17dd bytes) 000534 1790947649.031230 mapread 0x00b348 thru 0x0144a8 (0x9161 bytes) 000535 1790947649.075128 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36a65): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a65): Mirror 2: 0x9252486d (stale) Mirror 3: 0x9252486d (stale) Mirror 4: 0x9252486d (stale) Mirror 5: 0x9252486d (stale) Mirror 6: 0x9252486d (stale) Mirror 7: 0x9252486d (stale) CRC-32 checksum value for chunk [0, 0x36a65): Mirror 1: 0xfcc0dc59 Mirror 8: 0xfcc0dc59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a65: skipped 000536 1790947650.638130 write 0x01dbad thru 0x021bfa (0x404e bytes) 000537 1790947650.709982 read 0x00965f thru 0x00b1d9 (0x1b7b bytes) 000538 1790947650.723419 write 0x03888e thru 0x03a840 (0x1fb3 bytes) 000539 1790947650.726011 punch from 0x01b789 to 0x0223af (0x6c27 bytes) 000540 1790947650.745435 mapwrite 0x031d60 thru 0x03a55f (0x8800 bytes) 000541 1790947650.870844 write 0x027ed0 thru 0x02b773 (0x38a4 bytes) 000542 1790947650.910679 write 0x02b887 thru 0x03909e (0xd818 bytes) 000543 1790947650.917251 trunc from 0x02805e to 0x028f2c (0xecf bytes) 000544 1790947650.968399 trunc from 0x00b4a8 to 0x012c05 (0x775e bytes) 000545 1790947650.978826 trunc from 0x03a841 to 0x017e72 (0x229ce bytes) 000547 1790947651.025461 trunc from 0x017e73 to 0x00a529 (0xd949 bytes) 000548 1790947651.043424 read 0x00683c thru 0x00a529 (0x3cee bytes) 000549 1790947651.052557 trunc from 0x00a52a to 0x0162e3 (0xbdba bytes) 000550 1790947651.071836 fallocate 0x012306 thru 0x01496f (0x266a bytes) 000551 1790947651.077526 trunc from 0x00103a to 0x00306c (0x2033 bytes) 000552 1790947651.086266 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x162e4): Mirror 2: 0x6abf04b (stale) Mirror 3: 0x6abf04b (stale) Mirror 4: 0x6abf04b (stale) Mirror 5: 0x6abf04b (stale) Mirror 6: 0x6abf04b (stale) Mirror 7: 0x6abf04b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x162e4): Mirror 2: 0x6abf04b (stale) Mirror 3: 0x6abf04b (stale) Mirror 4: 0x6abf04b (stale) Mirror 5: 0x6abf04b (stale) Mirror 6: 0x6abf04b (stale) Mirror 7: 0x6abf04b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x162e4: skipped 000553 1790947651.460009 read 0x001c54 thru 0x0087b7 (0x6b64 bytes) 000554 1790947651.485497 mapread 0x00e109 thru 0x0162e3 (0x81db bytes) 000555 1790947651.558982 read 0x00f2cf thru 0x01214c (0x2e7e bytes) 000557 1790947651.561266 trunc from 0x004ceb to 0x007ea7 (0x31bd bytes) 000558 1790947651.575311 trunc from 0x0162e4 to 0x002e4c (0x13497 bytes) 000559 1790947651.654882 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e4d): Mirror 2: 0x821576a3 (stale) Mirror 3: 0x821576a3 (stale) Mirror 4: 0x821576a3 (stale) Mirror 6: 0x821576a3 (stale) Mirror 7: 0x821576a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e4d): Mirror 2: 0x821576a3 (stale) Mirror 3: 0x821576a3 (stale) Mirror 4: 0x821576a3 (stale) Mirror 6: 0x821576a3 (stale) Mirror 7: 0x821576a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e4d: skipped 000560 1790947652.287239 write 0x00419b thru 0x008a82 (0x48e8 bytes) 000561 1790947652.293315 trunc from 0x005e3e to 0x008a82 (0x2c45 bytes) 000563 1790947652.309860 read 0x0071f1 thru 0x0088c7 (0x16d7 bytes) 000565 1790947652.315526 mapread 0x002684 thru 0x008a82 (0x63ff bytes) 000566 1790947652.328981 fallocate 0x03b682 thru 0x03f33c (0x3cbb bytes) 000567 1790947652.333439 trunc from 0x008a83 to 0x0175b8 (0xeb36 bytes) 000568 1790947652.344577 punch from 0x01031a to 0x0141d9 (0x3ec0 bytes) 000569 1790947652.349105 mapread 0x00ce54 thru 0x012e90 (0x603d bytes) 000570 1790947652.358211 trunc from 0x00af8a to 0x0175b8 (0xc62f bytes) 000571 1790947652.362302 write 0x020a22 thru 0x030785 (0xfd64 bytes) 000572 1790947652.364152 fallocate 0x01cc06 thru 0x01f0ce (0x24c9 bytes) 000573 1790947652.373144 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30786): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30786): Mirror 2: 0x7e62ee2e (stale) Mirror 3: 0x7e62ee2e (stale) Mirror 4: 0x7e62ee2e (stale) Mirror 6: 0x7e62ee2e (stale) Mirror 7: 0x7e62ee2e (stale) CRC-32 checksum value for chunk [0, 0x30786): Mirror 1: 0x109379a4 Mirror 8: 0x109379a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30786: skipped 000574 1790947653.358928 mapread 0x004165 thru 0x00c324 (0x81c0 bytes) 000575 1790947653.386433 trunc from 0x030786 to 0x02bcae (0x4ad7 bytes) 000576 1790947653.499791 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bcaf): Mirror 2: 0xb45777ee (stale) Mirror 3: 0xb45777ee (stale) Mirror 4: 0xb45777ee (stale) Mirror 6: 0xb45777ee (stale) Mirror 7: 0xb45777ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bcaf): Mirror 2: 0xb45777ee (stale) Mirror 3: 0xb45777ee (stale) Mirror 4: 0xb45777ee (stale) Mirror 6: 0xb45777ee (stale) Mirror 7: 0xb45777ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bcaf: skipped 000577 1790947653.962477 trunc from 0x003e62 to 0x00a2a1 (0x6440 bytes) 000578 1790947653.971608 trunc from 0x01b72b to 0x02120b (0x5ae1 bytes) 000580 1790947653.977816 write 0x01ce2a thru 0x01e80b (0x19e2 bytes) 000581 1790947653.991758 fallocate 0x029c91 thru 0x02f71c (0x5a8c bytes) 000582 1790947653.999473 trunc from 0x02af86 to 0x02bcae (0xd29 bytes) 000583 1790947654.004503 mapread 0x011227 thru 0x017cae (0x6a88 bytes) 000584 1790947654.023957 mapwrite 0x03ff66 thru 0x03ffff (0x9a bytes) 000585 1790947654.077931 trunc from 0x01c534 to 0x020daf (0x487c bytes) 000586 1790947654.086991 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8e742d39 (stale) Mirror 3: 0x8e742d39 (stale) Mirror 4: 0x8e742d39 (stale) Mirror 6: 0x8e742d39 (stale) Mirror 7: 0x8e742d39 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc3caa76c Mirror 8: 0xc3caa76c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1790947655.171105 trunc from 0x040000 to 0x02c8fd (0x13702 bytes) 000588 1790947655.266306 read 0x01cf50 thru 0x01dffd (0x10ae bytes) 000589 1790947655.274088 trunc from 0x02c8fe to 0x0302d7 (0x39da bytes) 000590 1790947655.293560 write 0x033a36 thru 0x03ffff (0xc5ca bytes) 000591 1790947655.296582 mapwrite 0x0137dd thru 0x016683 (0x2ea7 bytes) 000592 1790947655.328096 mapwrite 0x03bc9c thru 0x03ffff (0x4364 bytes) 000594 1790947655.342169 read 0x022a1a thru 0x0236b6 (0xc9d bytes) 000596 1790947655.363793 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8e742d39 (stale) Mirror 3: 0x8e742d39 (stale) Mirror 4: 0x8e742d39 (stale) Mirror 6: 0x8e742d39 (stale) Mirror 7: 0x8e742d39 (stale) Mirror 8: 0xc3caa76c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x14d45235 Mirror 9: 0x14d45235 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1790947656.437631 punch from 0x00c0e1 to 0x00f129 (0x3049 bytes) 000598 1790947656.468219 trunc from 0x0090a1 to 0x00adde (0x1d3e bytes) 000599 1790947656.471599 read 0x02c5db thru 0x02f582 (0x2fa8 bytes) 000600 1790947656.476111 write 0x00ced4 thru 0x01a664 (0xd791 bytes) 000601 1790947656.486610 write 0x0026c9 thru 0x010727 (0xe05f bytes) 000602 1790947656.492520 mapread 0x00464a thru 0x00b65b (0x7012 bytes) 000603 1790947656.493531 mapwrite 0x0023f1 thru 0x002eaa (0xaba bytes) 000606 1790947656.530013 trunc from 0x040000 to 0x023ee3 (0x1c11c bytes) 000607 1790947656.581947 trunc from 0x023ee4 to 0x028298 (0x43b5 bytes) 000608 1790947656.599620 fallocate 0x020e63 thru 0x02d455 (0xc5f3 bytes) 000609 1790947656.603860 write 0x0065c3 thru 0x00cc20 (0x665e bytes) 000610 1790947656.605061 mapread 0x01af18 thru 0x01e26b (0x3354 bytes) 000611 1790947656.620320 mapwrite 0x02dfe4 thru 0x036a01 (0x8a1e bytes) 000612 1790947656.661516 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36a02): Mirror 2: 0x8de50173 (stale) Mirror 3: 0x8de50173 (stale) Mirror 4: 0x8de50173 (stale) Mirror 6: 0x8de50173 (stale) Mirror 7: 0x8de50173 (stale) Mirror 9: 0x8ff8d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a02): Mirror 2: 0x8de50173 (stale) Mirror 3: 0x8de50173 (stale) Mirror 4: 0x8de50173 (stale) Mirror 6: 0x8de50173 (stale) Mirror 7: 0x8de50173 (stale) Mirror 9: 0x8ff8d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a02: skipped 000613 1790947657.142224 punch from 0x0257a5 to 0x02b732 (0x5f8e bytes) 000614 1790947657.148002 trunc from 0x02dc78 to 0x032a2f (0x4db8 bytes) 000615 1790947657.158414 write 0x024793 thru 0x031a98 (0xd306 bytes) 000616 1790947657.202645 trunc from 0x036a02 to 0x0110a7 (0x2595a bytes) 000617 1790947657.232298 write 0x03a194 thru 0x03ffff (0x5e6c bytes) 000618 1790947657.238401 mapwrite 0x034370 thru 0x036147 (0x1dd8 bytes) 000619 1790947657.334791 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8e742d39 (stale) Mirror 4: 0x8e742d39 (stale) Mirror 6: 0x8e742d39 (stale) Mirror 7: 0x8e742d39 (stale) Mirror 9: 0x14d45235 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8e742d39 (stale) Mirror 4: 0x8e742d39 (stale) Mirror 6: 0x8e742d39 (stale) Mirror 7: 0x8e742d39 (stale) Mirror 9: 0x14d45235 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000620 1790947657.855824 fallocate 0x032197 thru 0x039a3c (0x78a6 bytes) 000622 1790947657.860749 write 0x01e96c thru 0x025f02 (0x7597 bytes) 000624 1790947657.883925 trunc from 0x040000 to 0x03af46 (0x50b9 bytes) 000625 1790947657.896960 read 0x011db5 thru 0x01f207 (0xd453 bytes) 000626 1790947657.909858 read 0x01ec0c thru 0x020d66 (0x215b bytes) 000627 1790947657.910426 read 0x039ff3 thru 0x03af46 (0xf54 bytes) 000628 1790947657.923288 write 0x012aca thru 0x01670a (0x3c41 bytes) 000629 1790947657.924315 punch from 0x01ef8f to 0x02d381 (0xe3f3 bytes) 000630 1790947657.936324 mapread 0x00731b thru 0x00de69 (0x6b4f bytes) 000631 1790947657.937291 trunc from 0x03af47 to 0x01ce76 (0x1e0d0 bytes) 000632 1790947657.954302 read 0x0153aa thru 0x01ce76 (0x7acd bytes) 000633 1790947657.955371 trunc from 0x0010d2 to 0x00da85 (0xc9b4 bytes) 000634 1790947657.960434 punch from 0x01bca9 to 0x01ce76 (0x11ce bytes) 000635 1790947657.965942 mapwrite 0x00ca94 thru 0x0187ca (0xbd37 bytes) 000636 1790947658.001316 fallocate 0x00eed9 thru 0x01aea2 (0xbfca bytes) 000637 1790947658.005590 write 0x02799f thru 0x02e8b2 (0x6f14 bytes) 000638 1790947658.010244 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e8b3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e8b3): Mirror 3: 0xad88889f (stale) Mirror 4: 0xad88889f (stale) Mirror 6: 0xad88889f (stale) Mirror 7: 0xad88889f (stale) Mirror 9: 0x5186b6ee (stale) CRC-32 checksum value for chunk [0, 0x2e8b3): Mirror 1: 0xf22426ad Mirror 10: 0xf22426ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e8b3: skipped 000640 1790947658.816416 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e8b3): Mirror 4: 0xad88889f (stale) Mirror 6: 0xad88889f (stale) Mirror 7: 0xad88889f (stale) Mirror 9: 0x5186b6ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e8b3): Mirror 4: 0xad88889f (stale) Mirror 6: 0xad88889f (stale) Mirror 7: 0xad88889f (stale) Mirror 9: 0x5186b6ee (stale) CRC-32 checksum value for chunk [0, 0x2e8b3): Mirror 1: 0xf22426ad Mirror 10: 0xf22426ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e8b3: skipped 000641 1790947659.492313 fallocate 0x016713 thru 0x017d58 (0x1646 bytes) 000642 1790947659.571274 mapwrite 0x005601 thru 0x0151ff (0xfbff bytes) 000643 1790947659.660103 mapwrite 0x01a763 thru 0x024a04 (0xa2a2 bytes) 000644 1790947659.706247 trunc from 0x025f33 to 0x02e8b2 (0x8980 bytes) 000646 1790947659.715478 read 0x0124ca thru 0x01d391 (0xaec8 bytes) 000647 1790947659.718077 mapwrite 0x02fe55 thru 0x032a83 (0x2c2f bytes) 000648 1790947659.780640 mapwrite 0x02ba13 thru 0x03a141 (0xe72f bytes) 000649 1790947659.840274 mapread 0x009624 thru 0x014da9 (0xb786 bytes) 000650 1790947659.841317 trunc from 0x036570 to 0x03976f (0x3200 bytes) 000651 1790947659.849221 mapread 0x0092ce thru 0x00fc3a (0x696d bytes) 000652 1790947659.853855 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a142): Mirror 4: 0x2bc86835 (stale) Mirror 7: 0x2bc86835 (stale) Mirror 9: 0x760be6ab (stale) Mirror 10: 0x69e47df7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a142): Mirror 4: 0x2bc86835 (stale) Mirror 7: 0x2bc86835 (stale) Mirror 9: 0x760be6ab (stale) Mirror 10: 0x69e47df7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a142: skipped 000653 1790947660.372575 fallocate 0x028f3b thru 0x02bd39 (0x2dff bytes) 000654 1790947660.378947 punch from 0x003fca to 0x00858a (0x45c1 bytes) 000655 1790947660.393250 read 0x00258b thru 0x00e710 (0xc186 bytes) 000656 1790947660.402315 mapwrite 0x018229 thru 0x021426 (0x91fe bytes) 000657 1790947660.449348 write 0x01feed thru 0x0279ea (0x7afe bytes) 000658 1790947660.459941 write 0x02dbd2 thru 0x0377be (0x9bed bytes) 000659 1790947660.467355 fallocate 0x037235 thru 0x03ffff (0x8dcb bytes) 000660 1790947660.471071 punch from 0x000798 to 0x000b12 (0x37b bytes) 000661 1790947660.475311 read 0x00a5f0 thru 0x0157c8 (0xb1d9 bytes) 000662 1790947660.483297 punch from 0x013873 to 0x0210d8 (0xd866 bytes) 000663 1790947660.492832 trunc from 0x028aa9 to 0x02e949 (0x5ea1 bytes) 000664 1790947660.497960 write 0x004ebf thru 0x008dfc (0x3f3e bytes) 000665 1790947660.499406 trunc from 0x01d376 to 0x022563 (0x51ee bytes) 000667 1790947660.504505 write 0x002742 thru 0x00b300 (0x8bbf bytes) 000668 1790947660.505714 mapread 0x008f31 thru 0x0163af (0xd47f bytes) 000669 1790947660.530082 mapwrite 0x0178ee thru 0x01c2f9 (0x4a0c bytes) 000670 1790947660.579893 trunc from 0x01a9a9 to 0x02a900 (0xff58 bytes) 000671 1790947660.587240 mapread 0x02bc23 thru 0x02d2ee (0x16cc bytes) 000672 1790947660.601702 trunc from 0x0351b2 to 0x0387ef (0x363e bytes) 000673 1790947660.607341 trunc from 0x03a142 to 0x01653e (0x23c03 bytes) 000674 1790947660.621055 mapread 0x010ce8 thru 0x01653e (0x5857 bytes) 000675 1790947660.622461 fallocate 0x02653c thru 0x026e4e (0x913 bytes) 000676 1790947660.627582 punch from 0x004a65 to 0x0097da (0x4d76 bytes) 000677 1790947660.636736 MIRROR_RESYNC 5 mirrors 000679 1790947661.938794 trunc from 0x026e4f to 0x01e331 (0x8b1d bytes) 000680 1790947661.995030 punch from 0x0007ec to 0x00bbb6 (0xb3cb bytes) 000681 1790947662.005743 mapread 0x005851 thru 0x00ac11 (0x53c1 bytes) 000682 1790947662.066900 fallocate 0x0274b0 thru 0x0328bc (0xb40d bytes) 000683 1790947662.071880 mapread 0x017c8f thru 0x01ca98 (0x4e0a bytes) 000684 1790947662.097413 mapread 0x014483 thru 0x01b903 (0x7481 bytes) 000685 1790947662.101262 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1e332): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e332): Mirror 4: 0xb198f7c5 (stale) Mirror 7: 0xb198f7c5 (stale) Mirror 9: 0xb198f7c5 (stale) Mirror 10: 0xb198f7c5 (stale) CRC-32 checksum value for chunk [0, 0x1e332): Mirror 1: 0x5cea9da6 Mirror 11: 0x5cea9da6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e332: skipped 000686 1790947663.136569 write 0x01b64b thru 0x025fb2 (0xa968 bytes) 000687 1790947663.192404 trunc from 0x01bbd3 to 0x021e91 (0x62bf bytes) 000688 1790947663.219457 MIRROR_RESYNC 6 mirrors 000690 1790947664.801448 punch from 0x007a6b to 0x00c3ab (0x4941 bytes) 000691 1790947664.848265 trunc from 0x002d00 to 0x00c7c7 (0x9ac8 bytes) 000692 1790947664.853768 mapread 0x00b0de thru 0x011087 (0x5faa bytes) 000693 1790947664.868273 fallocate 0x02b6a8 thru 0x037093 (0xb9ec bytes) 000694 1790947664.875330 punch from 0x016a5e to 0x01a113 (0x36b6 bytes) 000695 1790947664.879311 mapwrite 0x037370 thru 0x03ffff (0x8c90 bytes) 000696 1790947664.924425 punch from 0x022c95 to 0x027798 (0x4b04 bytes) 000697 1790947664.929557 mapwrite 0x01b8a9 thru 0x029bd2 (0xe32a bytes) 000698 1790947664.978226 mapwrite 0x03374b thru 0x03ffff (0xc8b5 bytes) 000699 1790947665.013987 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x79362c54 (stale) Mirror 7: 0x79362c54 (stale) Mirror 10: 0x79362c54 (stale) Mirror 11: 0x79362c54 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x79362c54 (stale) Mirror 7: 0x79362c54 (stale) Mirror 10: 0x79362c54 (stale) Mirror 11: 0x79362c54 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000700 1790947665.370718 write 0x02d056 thru 0x03bb84 (0xeb2f bytes) 000701 1790947665.387238 write 0x014e32 thru 0x024110 (0xf2df bytes) 000703 1790947665.409428 mapwrite 0x028898 thru 0x031e08 (0x9571 bytes) 000704 1790947665.459438 punch from 0x02f5fb to 0x038d47 (0x974d bytes) 000705 1790947665.466197 mapread 0x01f01a thru 0x021e95 (0x2e7c bytes) 000706 1790947665.467134 read 0x01afb9 thru 0x01d62a (0x2672 bytes) 000707 1790947665.471961 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x79362c54 (stale) Mirror 10: 0x79362c54 (stale) Mirror 11: 0x79362c54 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x79362c54 (stale) Mirror 10: 0x79362c54 (stale) Mirror 11: 0x79362c54 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000708 1790947665.796164 trunc from 0x040000 to 0x00d3f8 (0x32c07 bytes) 000709 1790947665.842991 punch from 0x00657a to 0x00d07c (0x6b03 bytes) 000711 1790947665.851821 mapread 0x0035eb thru 0x00d3f8 (0x9e0e bytes) 000712 1790947665.878557 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xd3f9): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd3f9): Mirror 7: 0x769905f7 (stale) Mirror 10: 0x769905f7 (stale) Mirror 11: 0x769905f7 (stale) CRC-32 checksum value for chunk [0, 0xd3f9): Mirror 1: 0x3f50f410 Mirror 12: 0x3f50f410 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd3f9: skipped 000713 1790947666.618505 punch from 0x0024c0 to 0x00a51a (0x805b bytes) 000714 1790947666.691623 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd3f9): Mirror 7: 0x769905f7 (stale) Mirror 10: 0x769905f7 (stale) Mirror 11: 0x769905f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd3f9): Mirror 7: 0x769905f7 (stale) Mirror 10: 0x769905f7 (stale) Mirror 11: 0x769905f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd3f9: skipped 000715 1790947667.072435 fallocate 0x02adc8 thru 0x039b21 (0xed5a bytes) 000716 1790947667.079864 mapread 0x00cc16 thru 0x00d3f8 (0x7e3 bytes) 000717 1790947667.103559 fallocate 0x012145 thru 0x012d5b (0xc17 bytes) 000718 1790947667.108618 trunc from 0x00e957 to 0x012d5b (0x4405 bytes) 000719 1790947667.114238 trunc from 0x0120bc to 0x012d5b (0xca0 bytes) 000721 1790947667.120839 read 0x005dc4 thru 0x00789e (0x1adb bytes) 000722 1790947667.127132 mapwrite 0x01022c thru 0x01eeaa (0xec7f bytes) 000723 1790947667.205822 MIRROR_RESYNC 4 mirrors 000724 1790947668.130563 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) 000726 1790947668.373359 write 0x00ef43 thru 0x01bb0c (0xcbca bytes) 000727 1790947668.423313 punch from 0x015ae9 to 0x016d47 (0x125f bytes) 000728 1790947668.449366 fallocate 0x016ad7 thru 0x01e4a2 (0x79cc bytes) 000729 1790947668.453899 read 0x015157 thru 0x0174e7 (0x2391 bytes) 000730 1790947668.461577 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1eeab): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eeab): Mirror 10: 0xeff64b5e (stale) Mirror 11: 0xeff64b5e (stale) CRC-32 checksum value for chunk [0, 0x1eeab): Mirror 1: 0x467c5b00 Mirror 12: 0x467c5b00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eeab: skipped 000731 1790947669.367147 read 0x0029b2 thru 0x008a56 (0x60a5 bytes) 000732 1790947669.407725 MIRROR_RESYNC 4 mirrors 000733 1790947670.272208 punch from 0x012010 to 0x016818 (0x4809 bytes) 000734 1790947670.315904 trunc from 0x008343 to 0x00b117 (0x2dd5 bytes) 000735 1790947670.320420 mapwrite 0x01d73e thru 0x02c7e3 (0xf0a6 bytes) 000736 1790947670.404500 mapread 0x00f44f thru 0x0151c9 (0x5d7b bytes) 000737 1790947670.416571 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c7e4): Mirror 10: 0x7c090d28 (stale) Mirror 11: 0x7c090d28 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c7e4): Mirror 10: 0x7c090d28 (stale) Mirror 11: 0x7c090d28 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c7e4: skipped 000738 1790947670.760285 fallocate 0x015d39 thru 0x01efee (0x92b6 bytes) 000739 1790947670.764293 mapread 0x01bd90 thru 0x0262e2 (0xa553 bytes) 000740 1790947670.794432 mapread 0x00bcd5 thru 0x01722f (0xb55b bytes) 000741 1790947670.820257 punch from 0x0274c8 to 0x02a6be (0x31f7 bytes) 000742 1790947670.834383 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c7e4): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c7e4): Mirror 10: 0x7c090d28 (stale) Mirror 11: 0x7c090d28 (stale) CRC-32 checksum value for chunk [0, 0x2c7e4): Mirror 1: 0xa1de2055 Mirror 12: 0xa1de2055 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c7e4: skipped 000743 1790947672.086574 fallocate 0x03e93b thru 0x03ffff (0x16c5 bytes) 000744 1790947672.125832 trunc from 0x040000 to 0x016568 (0x29a97 bytes) 000745 1790947672.146253 trunc from 0x016569 to 0x01c1ba (0x5c52 bytes) 000746 1790947672.161149 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1c1bb): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c1bb): Mirror 10: 0x585172d4 (stale) Mirror 11: 0x585172d4 (stale) Mirror 12: 0xca2d9601 (stale) CRC-32 checksum value for chunk [0, 0x1c1bb): Mirror 1: 0x2825bc4 Mirror 13: 0x2825bc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c1bb: skipped 000748 1790947672.892835 punch from 0x0078a6 to 0x00c1b7 (0x4912 bytes) 000749 1790947672.931347 fallocate 0x03baca thru 0x03ffff (0x4536 bytes) 000750 1790947672.936762 write 0x03def9 thru 0x03ffff (0x2107 bytes) 000751 1790947672.953214 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xc8fd0c35 (stale) Mirror 11: 0xc8fd0c35 (stale) Mirror 12: 0xc2008a6c (stale) Mirror 13: 0x446a6644 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4015614 Mirror 14: 0x4015614 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000752 1790947673.900304 trunc from 0x01ac69 to 0x0289b2 (0xdd4a bytes) 000753 1790947673.939064 mapread 0x02c6e0 thru 0x03b9f5 (0xf316 bytes) 000754 1790947673.967908 read 0x0034ad thru 0x00c6dd (0x9231 bytes) 000755 1790947673.975858 punch from 0x008583 to 0x00d5d7 (0x5055 bytes) 000756 1790947673.984110 MIRROR_RESYNC 6 mirrors 000758 1790947675.484324 trunc from 0x040000 to 0x0335d8 (0xca27 bytes) 000759 1790947675.533799 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x335d9): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335d9): Mirror 10: 0xf6d4b3b1 (stale) Mirror 11: 0xf6d4b3b1 (stale) Mirror 12: 0xf6d4b3b1 (stale) Mirror 13: 0xf6d4b3b1 (stale) Mirror 14: 0xf6d4b3b1 (stale) CRC-32 checksum value for chunk [0, 0x335d9): Mirror 1: 0xf6d4b3b1 Mirror 15: 0xf6d4b3b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335d9: skipped 000760 1790947676.466870 mapread 0x0148a8 thru 0x01b2af (0x6a08 bytes) 000761 1790947676.489000 trunc from 0x00dc76 to 0x0129cf (0x4d5a bytes) 000762 1790947676.534867 punch from 0x0207d0 to 0x023ad4 (0x3305 bytes) 000763 1790947676.540456 read 0x0023e0 thru 0x00584c (0x346d bytes) 000764 1790947676.549646 punch from 0x006b7b to 0x0142e6 (0xd76c bytes) 000765 1790947676.556717 mapread 0x0080a4 thru 0x01051f (0x847c bytes) 000766 1790947676.567140 mapwrite 0x01615b thru 0x01e287 (0x812d bytes) 000767 1790947676.611524 fallocate 0x01bb53 thru 0x021c3a (0x60e8 bytes) 000768 1790947676.617775 fallocate 0x0011f1 thru 0x011120 (0xff30 bytes) 000769 1790947676.625430 fallocate 0x016d18 thru 0x025ec8 (0xf1b1 bytes) 000770 1790947676.631381 punch from 0x030b16 to 0x0335d8 (0x2ac3 bytes) 000771 1790947676.635655 read 0x014d22 thru 0x024cf3 (0xffd2 bytes) 000772 1790947676.645360 write 0x014e35 thru 0x02093d (0xbb09 bytes) 000773 1790947676.650768 mapread 0x0296b9 thru 0x031e1e (0x8766 bytes) 000774 1790947676.655720 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x335d9): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335d9): Mirror 10: 0xf6d4b3b1 (stale) Mirror 11: 0xf6d4b3b1 (stale) Mirror 12: 0xf6d4b3b1 (stale) Mirror 13: 0xf6d4b3b1 (stale) Mirror 14: 0xf6d4b3b1 (stale) Mirror 15: 0xf6d4b3b1 (stale) CRC-32 checksum value for chunk [0, 0x335d9): Mirror 1: 0x5e2ca57f Mirror 16: 0x5e2ca57f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335d9: skipped 000775 1790947677.496493 read 0x0186ef thru 0x02507f (0xc991 bytes) 000777 1790947677.522574 mapwrite 0x039cf8 thru 0x03ffff (0x6308 bytes) 000778 1790947677.638291 trunc from 0x040000 to 0x0159a1 (0x2a65e bytes) 000779 1790947677.672614 punch from 0x010bce to 0x0159a1 (0x4dd4 bytes) 000780 1790947677.681026 mapwrite 0x0101c6 thru 0x01f811 (0xf64c bytes) 000781 1790947677.743490 fallocate 0x00ae03 thru 0x013c0a (0x8e08 bytes) 000782 1790947677.749677 read 0x008b5d thru 0x010ab4 (0x7f58 bytes) 000783 1790947677.758251 fallocate 0x0302c5 thru 0x03dfee (0xdd2a bytes) 000784 1790947677.763266 trunc from 0x01f812 to 0x01333c (0xc4d5 bytes) 000785 1790947677.779114 punch from 0x00aeef to 0x0112bc (0x63ce bytes) 000786 1790947677.787762 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1333d): Mirror 11: 0xf4f3e879 (stale) Mirror 12: 0xf4f3e879 (stale) Mirror 13: 0xf4f3e879 (stale) Mirror 14: 0xf4f3e879 (stale) Mirror 15: 0xf4f3e879 (stale) Mirror 16: 0xb3b9c4d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1333d): Mirror 11: 0xf4f3e879 (stale) Mirror 12: 0xf4f3e879 (stale) Mirror 13: 0xf4f3e879 (stale) Mirror 14: 0xf4f3e879 (stale) Mirror 15: 0xf4f3e879 (stale) Mirror 16: 0xb3b9c4d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1333d: skipped 000787 1790947678.220856 mapread 0x0130ba thru 0x01333c (0x283 bytes) 000788 1790947678.238882 fallocate 0x00d6ed thru 0x00fa34 (0x2348 bytes) 000789 1790947678.251357 trunc from 0x00c8a6 to 0x01333c (0x6a97 bytes) 000790 1790947678.260211 MIRROR_RESYNC 7 mirrors 000791 1790947679.283478 read 0x007224 thru 0x00b617 (0x43f4 bytes) 000792 1790947679.295242 punch from 0x006b67 to 0x00db3c (0x6fd6 bytes) 000794 1790947679.332955 read 0x008708 thru 0x00f1d6 (0x6acf bytes) 000795 1790947679.340286 trunc from 0x00cec9 to 0x0108bb (0x39f3 bytes) 000796 1790947679.345157 mapwrite 0x005a59 thru 0x00c362 (0x690a bytes) 000797 1790947679.365166 mapread 0x00190b thru 0x009bef (0x82e5 bytes) 000798 1790947679.370215 MIRROR_RESYNC 7 mirrors 000799 1790947680.593385 trunc from 0x000362 to 0x00d3d8 (0xd077 bytes) 000800 1790947680.625400 read 0x0108e7 thru 0x013289 (0x29a3 bytes) 000801 1790947680.631790 trunc from 0x000832 to 0x008281 (0x7a50 bytes) 000802 1790947680.637477 read 0x004c27 thru 0x00b668 (0x6a42 bytes) 000804 1790947680.646429 fallocate 0x00e8e3 thru 0x015325 (0x6a43 bytes) 000805 1790947680.650987 write 0x00d0a0 thru 0x016bc7 (0x9b28 bytes) 000806 1790947680.652846 trunc from 0x014af4 to 0x0159ee (0xefb bytes) 000807 1790947680.664611 read 0x00f3a4 thru 0x012e54 (0x3ab1 bytes) 000808 1790947680.665656 read 0x01689c thru 0x016bc7 (0x32c bytes) 000809 1790947680.669857 MIRROR_RESYNC 7 mirrors 000810 1790947681.605427 write 0x035ef8 thru 0x039f16 (0x401f bytes) 000811 1790947681.643569 trunc from 0x039f17 to 0x00e3c8 (0x2bb4e bytes) 000812 1790947681.666498 trunc from 0x005ca5 to 0x00e3c8 (0x8724 bytes) 000813 1790947681.670934 read 0x008338 thru 0x00e3c8 (0x6091 bytes) 000814 1790947681.679355 trunc from 0x00b978 to 0x00e3c8 (0x2a51 bytes) 000815 1790947681.683840 mapwrite 0x020756 thru 0x024722 (0x3fcd bytes) 000818 1790947681.738637 trunc from 0x024723 to 0x007a10 (0x1cd12 bytes) 000819 1790947681.786052 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x7a11): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7a11): Mirror 11: 0xfe77326 (stale) Mirror 12: 0xfe77326 (stale) Mirror 13: 0xfe77326 (stale) Mirror 14: 0xfe77326 (stale) Mirror 15: 0xfe77326 (stale) Mirror 16: 0xfe77326 (stale) CRC-32 checksum value for chunk [0, 0x7a11): Mirror 1: 0xfe77326 Mirror 17: 0xfe77326 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7a11: skipped 000820 1790947682.472888 trunc from 0x003764 to 0x007a10 (0x42ad bytes) 000821 1790947682.508138 mapwrite 0x0394a7 thru 0x03ffff (0x6b59 bytes) 000822 1790947682.610577 read 0x008c99 thru 0x016cf4 (0xe05c bytes) 000823 1790947682.623160 trunc from 0x01f15b to 0x028a87 (0x992d bytes) 000824 1790947682.630603 punch from 0x004278 to 0x009332 (0x50bb bytes) 000825 1790947682.634363 mapread 0x026884 thru 0x0312f9 (0xaa76 bytes) 000826 1790947682.650870 mapread 0x00c18f thru 0x010c05 (0x4a77 bytes) 000827 1790947682.654514 mapread 0x007f6f thru 0x00ff85 (0x8017 bytes) 000828 1790947682.666539 mapread 0x012671 thru 0x0199bb (0x734b bytes) 000829 1790947682.682810 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7e7c3342 (stale) Mirror 12: 0x7e7c3342 (stale) Mirror 14: 0x7e7c3342 (stale) Mirror 15: 0x7e7c3342 (stale) Mirror 16: 0x7e7c3342 (stale) Mirror 17: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7e7c3342 (stale) Mirror 12: 0x7e7c3342 (stale) Mirror 14: 0x7e7c3342 (stale) Mirror 15: 0x7e7c3342 (stale) Mirror 16: 0x7e7c3342 (stale) Mirror 17: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000830 1790947683.086439 punch from 0x033ac9 to 0x03dc70 (0xa1a8 bytes) 000831 1790947683.090005 read 0x02b1dd thru 0x02e766 (0x358a bytes) 000832 1790947683.099828 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7e7c3342 (stale) Mirror 14: 0x7e7c3342 (stale) Mirror 15: 0x7e7c3342 (stale) Mirror 16: 0x7e7c3342 (stale) Mirror 17: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7e7c3342 (stale) Mirror 14: 0x7e7c3342 (stale) Mirror 15: 0x7e7c3342 (stale) Mirror 16: 0x7e7c3342 (stale) Mirror 17: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000833 1790947683.463253 trunc from 0x00e31f to 0x019f6d (0xbc4f bytes) 000834 1790947683.470134 write 0x01a83f thru 0x02a376 (0xfb38 bytes) 000835 1790947683.484783 read 0x0090d8 thru 0x00aaa8 (0x19d1 bytes) 000836 1790947683.491107 punch from 0x0303dc to 0x03f7da (0xf3ff bytes) 000837 1790947683.497124 trunc from 0x02f668 to 0x0379f4 (0x838d bytes) 000838 1790947683.501611 mapwrite 0x02abc2 thru 0x037de4 (0xd223 bytes) 000839 1790947683.566231 punch from 0x035a4b to 0x03bbaf (0x6165 bytes) 000840 1790947683.576296 read 0x00d40d thru 0x0149f3 (0x75e7 bytes) 000842 1790947683.589926 fallocate 0x03c4e2 thru 0x03ec20 (0x273f bytes) 000843 1790947683.603311 trunc from 0x023f9a to 0x02ed9b (0xae02 bytes) 000844 1790947683.615770 fallocate 0x001ebf thru 0x00c670 (0xa7b2 bytes) 000845 1790947683.627224 write 0x00b41c thru 0x0127c9 (0x73ae bytes) 000846 1790947683.629459 write 0x03b903 thru 0x03ffff (0x46fd bytes) 000847 1790947683.636468 trunc from 0x03a69e to 0x03ffff (0x5962 bytes) 000848 1790947683.653143 trunc from 0x02ea3f to 0x039ba1 (0xb163 bytes) 000849 1790947683.657753 mapread 0x035297 thru 0x03f818 (0xa582 bytes) 000850 1790947683.713082 read 0x01e420 thru 0x029a8c (0xb66d bytes) 000851 1790947683.722221 fallocate 0x02a015 thru 0x038e43 (0xee2f bytes) 000852 1790947683.729971 MIRROR_RESYNC 6 mirrors 000854 1790947685.446096 read 0x02fe4d thru 0x03d604 (0xd7b8 bytes) 000855 1790947685.457426 mapwrite 0x036292 thru 0x03ffff (0x9d6e bytes) 000856 1790947685.577845 trunc from 0x040000 to 0x038d2d (0x72d2 bytes) 000857 1790947685.611621 punch from 0x02c5aa to 0x0367e4 (0xa23b bytes) 000858 1790947685.618488 read 0x02e042 thru 0x038d2d (0xacec bytes) 000859 1790947685.633979 mapread 0x01e6d6 thru 0x01fdb8 (0x16e3 bytes) 000860 1790947685.635895 trunc from 0x038d2e to 0x00e8c4 (0x2a469 bytes) 000861 1790947685.656388 read 0x00a43d thru 0x00e8c4 (0x4488 bytes) 000862 1790947685.658192 trunc from 0x002dad to 0x008f28 (0x617c bytes) 000863 1790947685.663221 write 0x0250c2 thru 0x028d03 (0x3c42 bytes) 000864 1790947685.664948 read 0x007c02 thru 0x017688 (0xfa87 bytes) 000865 1790947685.686015 punch from 0x009952 to 0x00b871 (0x1f20 bytes) 000866 1790947685.700486 punch from 0x01d829 to 0x02855e (0xad36 bytes) 000867 1790947685.711254 read 0x00427f thru 0x00562e (0x13b0 bytes) 000868 1790947685.711945 mapwrite 0x003e33 thru 0x00ae85 (0x7053 bytes) 000869 1790947685.733232 mapwrite 0x00a035 thru 0x00b88f (0x185b bytes) 000870 1790947685.742878 trunc from 0x028d04 to 0x01223d (0x16ac6 bytes) 000871 1790947685.762212 read 0x009cdb thru 0x01223d (0x8563 bytes) 000873 1790947685.763276 punch from 0x0106ec to 0x01223d (0x1b52 bytes) 000875 1790947685.769306 trunc from 0x01223e to 0x019cc7 (0x7a8a bytes) 000876 1790947685.795218 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x19cc8): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19cc8): Mirror 12: 0x16465d44 (stale) Mirror 14: 0x16465d44 (stale) Mirror 15: 0x16465d44 (stale) Mirror 16: 0x16465d44 (stale) Mirror 17: 0x16465d44 (stale) CRC-32 checksum value for chunk [0, 0x19cc8): Mirror 1: 0x4eab8707 Mirror 18: 0x4eab8707 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19cc8: skipped 000877 1790947686.617816 mapwrite 0x00b7e5 thru 0x0150c1 (0x98dd bytes) 000878 1790947686.745487 mapwrite 0x0026ae thru 0x005140 (0x2a93 bytes) 000879 1790947686.772376 punch from 0x0050b4 to 0x010ef9 (0xbe46 bytes) 000880 1790947686.788407 mapwrite 0x0331f2 thru 0x03ffff (0xce0e bytes) 000882 1790947686.861176 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa6034401 (stale) Mirror 14: 0xa6034401 (stale) Mirror 15: 0xa6034401 (stale) Mirror 16: 0xa6034401 (stale) Mirror 17: 0xa6034401 (stale) Mirror 18: 0xd8614938 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc1af0f9 Mirror 19: 0xc1af0f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000884 1790947687.733098 trunc from 0x040000 to 0x0095f2 (0x36a0d bytes) 000885 1790947687.778825 trunc from 0x0092ed to 0x0095f2 (0x306 bytes) 000886 1790947687.785606 mapwrite 0x00e0d2 thru 0x00f7a2 (0x16d1 bytes) 000887 1790947687.827590 trunc from 0x005057 to 0x00be90 (0x6e3a bytes) 000888 1790947687.834762 trunc from 0x004a13 to 0x007d5d (0x334b bytes) 000889 1790947687.839148 mapread 0x008408 thru 0x00f7a2 (0x739b bytes) 000890 1790947687.847152 mapread 0x005f88 thru 0x00f7a2 (0x981b bytes) 000891 1790947687.848030 write 0x01b643 thru 0x02951b (0xded9 bytes) 000892 1790947687.850336 mapwrite 0x037c00 thru 0x03ffff (0x8400 bytes) 000893 1790947687.893612 mapwrite 0x01fc23 thru 0x0254b5 (0x5893 bytes) 000894 1790947687.910381 fallocate 0x022d2f thru 0x02e53a (0xb80c bytes) 000895 1790947687.914384 read 0x01e73a thru 0x027a10 (0x92d7 bytes) 000896 1790947687.915394 trunc from 0x040000 to 0x02b912 (0x146ed bytes) 000897 1790947687.928954 trunc from 0x029816 to 0x029875 (0x60 bytes) 000898 1790947687.934167 trunc from 0x009332 to 0x00e333 (0x5002 bytes) 000899 1790947687.938621 mapread 0x024aa5 thru 0x02b912 (0x6e6e bytes) 000900 1790947687.944725 mapwrite 0x031147 thru 0x039ac9 (0x8983 bytes) 000901 1790947688.009594 mapwrite 0x037185 thru 0x03ffff (0x8e7b bytes) 000902 1790947688.114153 trunc from 0x040000 to 0x02ddd6 (0x12229 bytes) 000903 1790947688.137264 read 0x013b97 thru 0x016713 (0x2b7d bytes) 000904 1790947688.162967 trunc from 0x02ddd7 to 0x003337 (0x2aa9f bytes) 000905 1790947688.180636 mapwrite 0x00b574 thru 0x01a6d6 (0xf163 bytes) 000906 1790947688.370481 mapread 0x008a5e thru 0x00ba30 (0x2fd3 bytes) 000907 1790947688.396552 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a6d7): Mirror 12: 0xf8a2ac19 (stale) Mirror 15: 0xf8a2ac19 (stale) Mirror 16: 0xf8a2ac19 (stale) Mirror 17: 0xf8a2ac19 (stale) Mirror 18: 0xdfbfa5d1 (stale) Mirror 19: 0x20bff912 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a6d7): Mirror 12: 0xf8a2ac19 (stale) Mirror 15: 0xf8a2ac19 (stale) Mirror 16: 0xf8a2ac19 (stale) Mirror 17: 0xf8a2ac19 (stale) Mirror 18: 0xdfbfa5d1 (stale) Mirror 19: 0x20bff912 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a6d7: skipped 000908 1790947688.829304 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a6d7): Mirror 12: 0xf8a2ac19 (stale) Mirror 16: 0xf8a2ac19 (stale) Mirror 17: 0xf8a2ac19 (stale) Mirror 18: 0xdfbfa5d1 (stale) Mirror 19: 0x20bff912 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a6d7): Mirror 12: 0xf8a2ac19 (stale) Mirror 16: 0xf8a2ac19 (stale) Mirror 17: 0xf8a2ac19 (stale) Mirror 18: 0xdfbfa5d1 (stale) Mirror 19: 0x20bff912 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a6d7: skipped 000909 1790947689.178286 fallocate 0x00d5d6 thru 0x011f47 (0x4972 bytes) 000910 1790947689.182401 mapwrite 0x011ff6 thru 0x0120a8 (0xb3 bytes) 000911 1790947689.200383 fallocate 0x010207 thru 0x01f84a (0xf644 bytes) 000912 1790947689.204059 mapwrite 0x0194e5 thru 0x020920 (0x743c bytes) 000913 1790947689.261511 write 0x00bc9f thru 0x017236 (0xb598 bytes) 000914 1790947689.275448 mapwrite 0x019487 thru 0x01c58e (0x3108 bytes) 000915 1790947689.296447 read 0x01e244 thru 0x01e5f2 (0x3af bytes) 000917 1790947689.299014 punch from 0x0113a9 to 0x01e26a (0xcec2 bytes) 000918 1790947689.307249 punch from 0x007eb6 to 0x00d1b9 (0x5304 bytes) 000919 1790947689.313459 write 0x02f079 thru 0x03239c (0x3324 bytes) 000920 1790947689.316784 read 0x00f63a thru 0x01c5f8 (0xcfbf bytes) 000921 1790947689.327448 trunc from 0x01b12e to 0x029824 (0xe6f7 bytes) 000922 1790947689.333102 trunc from 0x024389 to 0x0275eb (0x3263 bytes) 000923 1790947689.337239 mapread 0x026595 thru 0x03239c (0xbe08 bytes) 000924 1790947689.363231 mapwrite 0x00020b thru 0x000d9e (0xb94 bytes) 000925 1790947689.391338 read 0x02022f thru 0x02998e (0x9760 bytes) 000927 1790947689.421983 mapread 0x0000cc thru 0x00a631 (0xa566 bytes) 000928 1790947689.426243 read 0x008fd2 thru 0x00cda9 (0x3dd8 bytes) 000929 1790947689.427597 punch from 0x00a255 to 0x0196ca (0xf476 bytes) 000930 1790947689.438342 trunc from 0x03239d to 0x005f81 (0x2c41b bytes) 000931 1790947689.459722 trunc from 0x00100b to 0x005f81 (0x4f77 bytes) 000932 1790947689.469134 trunc from 0x005f82 to 0x00e71f (0x879e bytes) 000933 1790947689.497961 write 0x03a583 thru 0x03bfd3 (0x1a51 bytes) 000936 1790947689.505651 fallocate 0x0360b8 thru 0x03ffff (0x9f48 bytes) 000937 1790947689.520170 write 0x03e479 thru 0x03ffff (0x1b87 bytes) 000938 1790947689.540440 read 0x03dec1 thru 0x03e0e9 (0x229 bytes) 000939 1790947689.541048 trunc from 0x0276c9 to 0x02d05a (0x5992 bytes) 000940 1790947689.551451 fallocate 0x01831b thru 0x0279b3 (0xf699 bytes) 000941 1790947689.556506 punch from 0x035328 to 0x03ffff (0xacd8 bytes) 000942 1790947689.573082 read 0x0208e3 thru 0x02f9d6 (0xf0f4 bytes) 000943 1790947689.584266 mapread 0x0346c9 thru 0x03b38a (0x6cc2 bytes) 000944 1790947689.585354 read 0x01d29b thru 0x01e1e0 (0xf46 bytes) 000946 1790947689.587410 read 0x00c2b1 thru 0x01027a (0x3fca bytes) 000947 1790947689.588420 trunc from 0x040000 to 0x02566a (0x1a995 bytes) 000948 1790947689.609328 MIRROR_RESYNC 6 mirrors 000949 1790947690.478533 trunc from 0x0220f7 to 0x02566a (0x3574 bytes) 000950 1790947690.538347 fallocate 0x023f66 thru 0x02cff7 (0x9092 bytes) 000951 1790947690.544705 read 0x000c07 thru 0x00125b (0x655 bytes) 000952 1790947690.556600 mapread 0x015ca5 thru 0x01c643 (0x699f bytes) 000953 1790947690.576333 read 0x00a617 thru 0x0125e5 (0x7fcf bytes) 000954 1790947690.586565 trunc from 0x020170 to 0x02566a (0x54fb bytes) 000955 1790947690.590947 mapread 0x00fb65 thru 0x011ce7 (0x2183 bytes) 000956 1790947690.592492 fallocate 0x00c7bd thru 0x01325c (0x6aa0 bytes) 000957 1790947690.597634 read 0x00daf1 thru 0x013d0d (0x621d bytes) 000958 1790947690.598587 trunc from 0x00e34f to 0x01d539 (0xf1eb bytes) 000960 1790947690.605305 fallocate 0x00cfdb thru 0x0113ac (0x43d2 bytes) 000961 1790947690.610927 mapread 0x013b2c thru 0x022468 (0xe93d bytes) 000962 1790947690.624017 mapread 0x0133fd thru 0x01aa08 (0x760c bytes) 000963 1790947690.628446 punch from 0x019dac to 0x02440b (0xa660 bytes) 000965 1790947690.638774 write 0x025a3f thru 0x02cd64 (0x7326 bytes) 000966 1790947690.641068 mapread 0x025a2a thru 0x027026 (0x15fd bytes) 000967 1790947690.643261 mapwrite 0x021d2e thru 0x02d356 (0xb629 bytes) 000968 1790947690.730409 mapread 0x01f8e8 thru 0x02b4b6 (0xbbcf bytes) 000969 1790947690.742190 punch from 0x01588f to 0x018577 (0x2ce9 bytes) 000971 1790947690.747393 trunc from 0x00dcf5 to 0x00f4c4 (0x17d0 bytes) 000972 1790947690.753212 trunc from 0x00187b to 0x002d9a (0x1520 bytes) 000973 1790947690.758325 trunc from 0x00e9fa to 0x0177c0 (0x8dc7 bytes) 000974 1790947690.763626 mapread 0x00b514 thru 0x0165c2 (0xb0af bytes) 000975 1790947690.804503 punch from 0x00f552 to 0x016d38 (0x77e7 bytes) 000977 1790947690.809127 mapread 0x023220 thru 0x0266d6 (0x34b7 bytes) 000978 1790947690.809994 read 0x02b8b7 thru 0x02d356 (0x1aa0 bytes) 000979 1790947690.810528 trunc from 0x00de54 to 0x01bd60 (0xdf0d bytes) 000980 1790947690.814436 write 0x02a36a thru 0x038028 (0xdcbf bytes) 000981 1790947690.816111 punch from 0x031246 to 0x038028 (0x6de3 bytes) 000982 1790947690.830428 read 0x0011e2 thru 0x002f69 (0x1d88 bytes) 000983 1790947690.838585 trunc from 0x038029 to 0x004381 (0x33ca7 bytes) 000984 1790947690.850572 mapread 0x003b79 thru 0x004381 (0x809 bytes) 000985 1790947690.852069 fallocate 0x01d6ab thru 0x023896 (0x61ec bytes) 000986 1790947690.857190 fallocate 0x02ddfb thru 0x0346df (0x68e5 bytes) 000987 1790947690.861196 write 0x02c54b thru 0x02f7d2 (0x3288 bytes) 000989 1790947690.866921 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f7d3): Mirror 12: 0x9436d9a8 (stale) Mirror 16: 0x9436d9a8 (stale) Mirror 18: 0x9436d9a8 (stale) Mirror 19: 0x9436d9a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f7d3): Mirror 12: 0x9436d9a8 (stale) Mirror 16: 0x9436d9a8 (stale) Mirror 18: 0x9436d9a8 (stale) Mirror 19: 0x9436d9a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f7d3: skipped 000990 1790947691.195355 trunc from 0x01ee00 to 0x026ae8 (0x7ce9 bytes) 000991 1790947691.205168 MIRROR_RESYNC 5 mirrors 000992 1790947692.126174 write 0x036fb9 thru 0x03ffff (0x9047 bytes) 000994 1790947692.173311 punch from 0x00df38 to 0x019ab5 (0xbb7e bytes) 000995 1790947692.180412 trunc from 0x040000 to 0x02a25c (0x15da3 bytes) 000996 1790947692.213385 read 0x0021da thru 0x00b9d6 (0x97fd bytes) 000997 1790947692.228172 read 0x01d48c thru 0x01f7d9 (0x234e bytes) 000998 1790947692.241465 MIRROR_RESYNC 5 mirrors 000999 1790947692.915490 write 0x0292cb thru 0x031408 (0x813e bytes) 001000 1790947692.979358 mapread 0x0225f0 thru 0x02ff17 (0xd928 bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 09:28:17 (1790947697) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x22b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 09:28:24 (1790947704) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168344 s, 12.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108612 s, 9.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 09:28:31 (1790947711) Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre2 Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5587808 .. Extending file size to 4874144 .. Extending file size to 8016896 .. Extending file size to 7339680 .. failed Extending file size to 8469184 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8521408 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8735936 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8669888 .. Extending file size to 9287552 .. Extending file size to 9325056 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62342 62343 62346 62348 62349 done 192.168.206.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg605-client.virtnet /mnt/lustre2 (opts:) 192.168.206.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg605-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:29:46 (1790947786) Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre2 Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6799328 .. Extending file size to 2114368 .. failed Extending file size to 7434368 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7953280 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8555616 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8185952 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8682816 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9231264 .. failed Extending file size to 8933024 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9171840 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9272224 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64077 64078 64079 64082 64083 done 192.168.206.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg605-client.virtnet /mnt/lustre2 (opts:) 192.168.206.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg605-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:31:03 (1790947863) Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1268868 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2164 1269140 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24828 3582192 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18972 3588048 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43800 7170240 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1268868 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2164 1269140 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24828 3582192 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18972 3588048 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43800 7170240 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143887 s, 14.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131413 s, 16.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6788dc9b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg605-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:31:18 (1790947878) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0994817 s, 21.1 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 09:31:24 (1790947884) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xeb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x170:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x16e:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:31:31 (1790947891) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0976489 s, 21.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:31:38 (1790947898) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110137 s, 19.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x171:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x172:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x171:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x172:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x170:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:31:46 (1790947906) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0770863 s, 27.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x173:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x171:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x173:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x171:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x172:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:32:17 (1790947937) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg605-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647158 s, 16.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c8:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:33:02 (1790947982) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg605-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x173:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x174:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0815106 s, 25.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x173:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x174:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x174:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x173:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x174:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x174:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:33:47 (1790948027) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg605-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg605-server: Pool lustre.test_204f created Adding targets to pool oleg605-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg605-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0767144 s, 27.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x175:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x175:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x175:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x175:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg605-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg605-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg605-server: Pool lustre.test_204f destroyed PASS 204f (67s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:34:55 (1790948095) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:35:01 (1790948101) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:35:08 (1790948108) oleg605-server: Pool lustre.test_206 created Adding targets to pool oleg605-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg605-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg605-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg605-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg605-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:35:32 (1790948132) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931535 s, 11.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366868 s, 28.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9b95aaef Mirror 3: 0x9b95aaef PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:35:42 (1790948142) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:35:45 (1790948145) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:35:48 (1790948148) Starting client: oleg605-client.virtnet: -o user_xattr,flock 192.168.206.105@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0404815 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1680000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1780000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1780000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1780000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1440000 Read this (4 bytes): BBBB 192.168.206.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg605-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:36:21 (1790948181) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.011677 s, 702 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:36:28 (1790948188) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426001 s, 24.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10c lmm_fid: [0x200000405:0x10c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10c lmm_fid: [0x200000405:0x10c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x17f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611404 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 09:36:29.000000000 -0400 Modify: 2026-10-02 09:36:29.000000000 -0400 Change: 2026-10-02 09:36:31.000000000 -0400 Birth: 2026-10-02 09:36:30.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:36:35 (1790948195) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550079 s, 19.1 MB/s oleg605-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg605-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg605-server: O/280000401/d0/180: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:36:54 (1790948214) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.221356 s, 185 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0261186 s, 157 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 2 09:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x111:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 2 09:36 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:37:02 (1790948222) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:37:08 (1790948228) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1521 sec =========== 09:37:16 (1790948236) === sanity-flr: start cleanup 09:37:18 (1790948238) === === sanity-flr: finish cleanup 09:37:25 (1790948245) ===