-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 13:50:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg422-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:50:51 (1787853051) === oleg422-client.virtnet: executing check_config_client /mnt/lustre oleg422-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg422-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff902111fd2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff902111fd2000.idle_timeout=debug disable quota as required oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:51:41 (1787853101) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8225] == sanity-flr test 0a: lfs mirror create with -N option == 13:51:44 (1787853104) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:52:06 (1787853126) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:52:11 (1787853131) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:52:16 (1787853136) PASS 0d (31s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:52:48 (1787853168) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:52:52 (1787853172) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 13:52:55 (1787853175) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff902111fd2000.stats=clear osc.lustre-OST0001-osc-ffff902111fd2000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:53:04 (1787853184) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:53:16 (1787853196) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:53:25 (1787853205) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:53:29 (1787853209) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:53:40 (1787853220) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 13:53:50 (1787853230) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:53:58 (1787853238) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:54:06 (1787853246) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:54:15 (1787853255) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:54:27 (1787853267) PASS 7 (10s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:54:37 (1787853277) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.782678 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155712 s, 6.7 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:54:50 (1787853290) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.324069 s, 22.6 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:55:01 (1787853301) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:55:13 (1787853313) Stopping /mnt/lustre-ost1 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff902111fd2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff902111fd2000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg422-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff902111fd2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff902111fd2000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg422-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:56:35 (1787853395) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.61681 s, 17.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff902111fd2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff902111fd2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg422-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:57:21 (1787853441) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff902111fd2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff902111fd2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg422-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff902111fd2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff902111fd2000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg422-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (95s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:58:56 (1787853536) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.29401 s, 15.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 13:59 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.244s user 0m0.000s sys 0m0.162s Stopping /mnt/lustre-ost1 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff902111fd2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff902111fd2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.920s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg422-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg422-server oleg422-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff902111fd2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff902111fd2000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.253s user 0m0.002s sys 0m0.173s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg422-server: oleg422-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg422-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff902111fd2000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff902111fd2000.ost_server_uuid in FULL state after 0 sec PASS 33b (111s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:00:47 (1787853647) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:00:52 (1787853652) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:00:56 (1787853656) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 14:01:00 (1787853660) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 14:01:09 (1787853669) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.835503 s, 15.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:01:24 (1787853684) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608595 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635165 s, 16.5 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:01:33 (1787853693) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:01:44 (1787853704) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00136333 s, 751 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113334 s, 904 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00285206 s, 359 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00144576 s, 708 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 14:01:56 (1787853716) create test files with size 15742844 .. bc90a68280b5580accc2d4e7d4755117 - 09676724e105ccbf04eab4c4bcc5a88d - fc758ced0362e4eae4be67a39d055c59 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg422-client.virtnet /mnt/lustre (opts:) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 14:02:25 (1787853745) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:02:40 (1787853760) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:02:48 (1787853768) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974292 s, 10.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309357 s, 33.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 14:02:58 (1787853778) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160127 s, 13.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.376554 s, 11.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169071 s, 12.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 14:03:18 (1787853798) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 14:03:22 (1787853802) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:03:25 (1787853805) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318 s, 33.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 43c: read prefer uncompressed mirror == 14:03:35 (1787853815) llite.lustre-ffff902107684000.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 14:03:44 (1787853824) llite.lustre-ffff902107684000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425451 s, 24.6 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453596 s, 23.1 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 14:03:53 (1787853833) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.449711 s, 7.0 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:04:04 (1787853844) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:04:17 (1787853857) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.493794 s, 21.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg422-client.virtnet /mnt/lustre (opts:) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:04:31 (1787853871) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9021082f9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.278628 s, 37.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg422-client.virtnet /mnt/lustre (opts:) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff90210822f800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:04:48 (1787853888) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:05:31 (1787853931) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:06:09 (1787853969) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:06:13 (1787853973) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.309337 s, 10.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.251361 s, 12.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7227767 (nosync,stale) Mirror 2: 0x7227767 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 000334ac345a203f9d5ae9d86d7bdf35 - mirror 1 checksum: 996007f564ae515d38a28809616f6e2a - mirror 2 checksum: 000334ac345a203f9d5ae9d86d7bdf35 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:06:16-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:06:16-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 000334ac345a203f9d5ae9d86d7bdf35 - mirror 1 checksum: 996007f564ae515d38a28809616f6e2a - mirror 2 checksum: 996007f564ae515d38a28809616f6e2a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:06:16-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:06:16-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:06:24 (1787853984) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599327 s, 17.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:06:32 (1787853992) Failing mds1 on oleg422-server Stopping /mnt/lustre-mds1 (opts:) on oleg422-server 14:06:43 (1787854003) shut down facet: mds1 facet_host: oleg422-server facet_failover_host: oleg422-server Failover mds1 to oleg422-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 14:07:08 (1787854028) targets are mounted 14:07:08 (1787854028) facet_failover done oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045293 s, 23.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:07:24 (1787854044) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0298239 s, 687 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0141402 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:07:34 (1787854054) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0418016 s, 490 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0127817 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:07:43 (1787854063) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0129885 s, 1.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:07:51 (1787854071) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0579902 s, 353 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:08:00 (1787854080) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0157428 s, 260 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0287357 s, 143 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:08:07 (1787854087) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:08:31 (1787854111) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:08:59 (1787854139) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:09:23 (1787854163) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:09:47 (1787854187) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 14:09:55 (1787854195) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58867 mirror split pid 58868 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58867 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58868 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 14:11:02 (1787854262) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 4265 skipping zero size read skipping zero size read 000004 1787854265.526337 mapwrite 0x032d87 thru 0x03ffff (0xd279 bytes) 000005 1787854265.659050 read 0x02c370 thru 0x02f1a7 (0x2e38 bytes) 000006 1787854265.673943 write 0x018bfa thru 0x020a1f (0x7e26 bytes) 000007 1787854265.696602 punch from 0x0061c8 to 0x00caf8 (0x6931 bytes) fallocating to largest ever: 0x40000 000009 1787854265.702469 fallocate 0x03be80 thru 0x03ffff (0x4180 bytes) 000010 1787854265.717637 fallocate 0x00d04d thru 0x0130e5 (0x6099 bytes) 000011 1787854265.727964 mapread 0x03d3bb thru 0x03ffff (0x2c45 bytes) 000012 1787854265.732136 punch from 0x02c1a5 to 0x032da9 (0x6c05 bytes) 000013 1787854265.743337 MIRROR_EXTEND to 2 mirrors 000014 1787854266.835790 write 0x01ab9c thru 0x01eaaf (0x3f14 bytes) 000015 1787854266.892746 mapread 0x0289aa thru 0x0307cb (0x7e22 bytes) 000016 1787854266.941192 mapread 0x039cc4 thru 0x03ffff (0x633c bytes) 000017 1787854266.961939 read 0x012a9a thru 0x01b70c (0x8c73 bytes) 000018 1787854266.985565 fallocate 0x019064 thru 0x01f35a (0x62f7 bytes) 000019 1787854266.997969 read 0x003a5b thru 0x00f77a (0xbd20 bytes) 000020 1787854267.011097 mapwrite 0x02c6e7 thru 0x034d98 (0x86b2 bytes) 000021 1787854267.052158 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000024 1787854267.288409 fallocate 0x02de73 thru 0x0332a2 (0x5430 bytes) 000025 1787854267.295873 mapread 0x00f5cf thru 0x010ed7 (0x1909 bytes) 000026 1787854267.316965 read 0x0260ac thru 0x02874c (0x26a1 bytes) 000027 1787854267.325814 mapread 0x02a372 thru 0x030105 (0x5d94 bytes) 000028 1787854267.340579 write 0x015bd0 thru 0x01d673 (0x7aa4 bytes) 000029 1787854267.359764 mapread 0x03e7f7 thru 0x03f67b (0xe85 bytes) 000030 1787854267.373196 read 0x01b806 thru 0x0234d5 (0x7cd0 bytes) 000031 1787854267.388765 fallocate 0x02d00a thru 0x0359c3 (0x89ba bytes) 000032 1787854267.396296 trunc from 0x040000 to 0x027485 (0x18b7a bytes) 000033 1787854267.414455 mapwrite 0x014357 thru 0x01ebe8 (0xa892 bytes) 000034 1787854267.443156 mapread 0x001547 thru 0x00da57 (0xc511 bytes) 000035 1787854267.445143 trunc from 0x02024f to 0x020f6c (0xd1e bytes) 000036 1787854267.452337 read 0x025820 thru 0x026435 (0xc16 bytes) 000038 1787854267.454498 mapwrite 0x01ad8c thru 0x020c66 (0x5edb bytes) 000040 1787854267.502995 mapwrite 0x03cf22 thru 0x03ffff (0x30de bytes) 000041 1787854267.576749 read 0x003b00 thru 0x008af0 (0x4ff1 bytes) 000042 1787854267.577983 trunc from 0x040000 to 0x00e729 (0x318d6 bytes) 000043 1787854267.599512 write 0x03e0a0 thru 0x03ffff (0x1f60 bytes) 000044 1787854267.601868 read 0x0157d6 thru 0x0169c1 (0x11ec bytes) 000046 1787854267.612421 trunc from 0x00f2e1 to 0x0115b4 (0x22d4 bytes) 000047 1787854267.618219 punch from 0x028ff9 to 0x02dff6 (0x4ffe bytes) 000048 1787854267.623829 trunc from 0x0010aa to 0x00885c (0x77b3 bytes) 000049 1787854267.628788 mapread 0x02c2bd thru 0x0354a2 (0x91e6 bytes) 000050 1787854267.639671 mapwrite 0x0332aa thru 0x03339c (0xf3 bytes) 000051 1787854267.654737 trunc from 0x040000 to 0x0391ae (0x6e51 bytes) 000052 1787854267.671321 punch from 0x02e3d5 to 0x02e5b2 (0x1de bytes) 000053 1787854267.676809 mapread 0x02d89e thru 0x033cf9 (0x645c bytes) 000054 1787854267.687792 MIRROR_EXTEND to 2 mirrors 000055 1787854268.593836 write 0x0315ae thru 0x039dbe (0x8811 bytes) 000056 1787854268.649809 read 0x026f51 thru 0x0333cb (0xc47b bytes) 000057 1787854268.660128 trunc from 0x039dbf to 0x00d78f (0x2c62f bytes) 000058 1787854268.680370 fallocate 0x017b6c thru 0x02573c (0xdbd1 bytes) 000059 1787854268.685899 trunc from 0x02573d to 0x01e511 (0x722b bytes) 000060 1787854268.710605 mapwrite 0x007930 thru 0x0126ed (0xadbe bytes) 000061 1787854268.774253 write 0x03c476 thru 0x03ffff (0x3b8a bytes) 000063 1787854268.776115 mapwrite 0x00422d thru 0x012d20 (0xeaf4 bytes) 000064 1787854268.804588 read 0x025dff thru 0x02ba1d (0x5c1f bytes) 000065 1787854268.821797 mapread 0x00071b thru 0x00a38a (0x9c70 bytes) 000066 1787854268.823007 write 0x03254c thru 0x037d32 (0x57e7 bytes) 000067 1787854268.825269 mapread 0x026d34 thru 0x034724 (0xd9f1 bytes) 000068 1787854268.835427 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbb4ff2c9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x963856b Mirror 3: 0x963856b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000069 1787854270.157017 mapwrite 0x037445 thru 0x03a35d (0x2f19 bytes) 000070 1787854270.328795 write 0x000e46 thru 0x00adc3 (0x9f7e bytes) 000071 1787854270.370239 read 0x006eef thru 0x016779 (0xf88b bytes) 000072 1787854270.405160 write 0x024b24 thru 0x031c38 (0xd115 bytes) 000073 1787854270.417623 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x963856b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x963856b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000074 1787854270.870928 mapread 0x03c8ae thru 0x03ffff (0x3752 bytes) 000075 1787854270.916725 trunc from 0x02a404 to 0x033027 (0x8c24 bytes) 000076 1787854270.924545 fallocate 0x01f4e9 thru 0x02877f (0x9297 bytes) 000077 1787854270.931971 trunc from 0x024220 to 0x0261f8 (0x1fd9 bytes) 000078 1787854270.943821 write 0x02313a thru 0x0246f9 (0x15c0 bytes) 000079 1787854270.981147 write 0x00c5d5 thru 0x00f2ca (0x2cf6 bytes) 000080 1787854271.008006 write 0x0127b3 thru 0x019837 (0x7085 bytes) 000081 1787854271.032047 punch from 0x02fc35 to 0x03339e (0x376a bytes) 000083 1787854271.055023 trunc from 0x040000 to 0x012430 (0x2dbcf bytes) 000085 1787854271.092808 mapread 0x0099a5 thru 0x00f421 (0x5a7d bytes) 000086 1787854271.138948 mapread 0x0091be thru 0x012420 (0x9263 bytes) 000087 1787854271.141937 write 0x025fef thru 0x027d06 (0x1d18 bytes) 000088 1787854271.144640 read 0x003728 thru 0x00ce44 (0x971d bytes) 000089 1787854271.152482 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000090 1787854271.398027 read 0x013621 thru 0x01e80f (0xb1ef bytes) 000091 1787854271.424602 mapwrite 0x013d9e thru 0x01e466 (0xa6c9 bytes) 000092 1787854271.449597 fallocate 0x02b267 thru 0x034c01 (0x999b bytes) 000093 1787854271.455244 trunc from 0x034c02 to 0x00a870 (0x2a391 bytes) 000094 1787854271.481922 read 0x00a092 thru 0x00a870 (0x7df bytes) 000097 1787854271.495032 punch from 0x00748e to 0x00a089 (0x2bfc bytes) 000098 1787854271.500590 punch from 0x0047d5 to 0x00a870 (0x609c bytes) 000099 1787854271.507239 mapread 0x005033 thru 0x00a870 (0x583e bytes) 000100 1787854271.553962 write 0x000b74 thru 0x003b9d (0x302a bytes) 000101 1787854271.556903 read 0x001fdd thru 0x002e68 (0xe8c bytes) 000102 1787854271.561044 mapwrite 0x02640e thru 0x0328fb (0xc4ee bytes) 000103 1787854271.618858 mapread 0x00901c thru 0x0147fe (0xb7e3 bytes) 000104 1787854271.620765 fallocate 0x02ea2f thru 0x02f85e (0xe30 bytes) 000105 1787854271.627016 write 0x005e46 thru 0x00acb9 (0x4e74 bytes) 000107 1787854271.631850 punch from 0x0185be to 0x028036 (0xfa79 bytes) 000109 1787854271.638005 trunc from 0x02d047 to 0x0328fb (0x58b5 bytes) 000111 1787854271.645997 read 0x00a35e thru 0x0173bf (0xd062 bytes) 000112 1787854271.647589 mapread 0x030fe4 thru 0x0328fb (0x1918 bytes) 000113 1787854271.669882 read 0x02b19c thru 0x0328fb (0x7760 bytes) 000114 1787854271.670592 read 0x01b791 thru 0x0226b1 (0x6f21 bytes) 000115 1787854271.671196 trunc from 0x0015dd to 0x00f9b4 (0xe3d8 bytes) 000116 1787854271.687255 read 0x00a484 thru 0x0177cd (0xd34a bytes) 000118 1787854271.697802 trunc from 0x00a14b to 0x00bb07 (0x19bd bytes) 000119 1787854271.704931 read 0x002daa thru 0x00553d (0x2794 bytes) 000120 1787854271.706236 trunc from 0x0127cb to 0x022211 (0xfa47 bytes) 000121 1787854271.715627 MIRROR_EXTEND to 2 mirrors 000122 1787854272.630517 trunc from 0x02c9cc to 0x0328fb (0x5f30 bytes) 000123 1787854272.693019 punch from 0x02e229 to 0x0328fb (0x46d3 bytes) 000124 1787854272.704052 trunc from 0x02ccde to 0x0328fb (0x5c1e bytes) 000125 1787854272.718845 write 0x03d357 thru 0x03ffff (0x2ca9 bytes) 000126 1787854272.725870 mapread 0x003729 thru 0x00dcf2 (0xa5ca bytes) 000127 1787854272.761293 read 0x01194e thru 0x019c1c (0x82cf bytes) 000128 1787854272.773013 read 0x03d737 thru 0x03ffff (0x28c9 bytes) 000129 1787854272.789194 fallocate 0x002d45 thru 0x00a2f2 (0x75ae bytes) 000130 1787854272.804053 mapread 0x0270f4 thru 0x02e617 (0x7524 bytes) 000131 1787854272.822237 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbfe33e8d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfab2fafc Mirror 3: 0xfab2fafc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000133 1787854273.879487 mapread 0x00fdc5 thru 0x013f36 (0x4172 bytes) 000134 1787854273.921822 fallocate 0x030ac6 thru 0x03d13f (0xc67a bytes) 000135 1787854273.971525 mapwrite 0x0354df thru 0x03ffff (0xab21 bytes) 000136 1787854274.011615 punch from 0x00c28d to 0x015a94 (0x9808 bytes) 000137 1787854274.017805 write 0x000244 thru 0x009d4a (0x9b07 bytes) 000138 1787854274.041969 write 0x00e348 thru 0x01911b (0xadd4 bytes) 000139 1787854274.056392 write 0x00d0d3 thru 0x0163f4 (0x9322 bytes) 000140 1787854274.067294 mapwrite 0x038b5c thru 0x03ffff (0x74a4 bytes) 000141 1787854274.104135 write 0x031010 thru 0x03168a (0x67b bytes) 000142 1787854274.105137 mapread 0x020fd4 thru 0x02fbc9 (0xebf6 bytes) 000144 1787854274.153903 punch from 0x037c4d to 0x038c9a (0x104e bytes) 000145 1787854274.164572 mapwrite 0x03f7f1 thru 0x03ffff (0x80f bytes) 000146 1787854274.199085 mapwrite 0x03966a thru 0x03c581 (0x2f18 bytes) 000147 1787854274.222627 mapread 0x019b87 thru 0x0205ee (0x6a68 bytes) 000148 1787854274.223871 mapwrite 0x00a0b0 thru 0x00c778 (0x26c9 bytes) 000149 1787854274.279881 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfab2fafc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfab2fafc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1787854274.687777 fallocate 0x024c76 thru 0x0293c9 (0x4754 bytes) 000151 1787854274.692915 trunc from 0x02111e to 0x022b5e (0x1a41 bytes) 000152 1787854274.697178 trunc from 0x02383b to 0x02441b (0xbe1 bytes) 000154 1787854274.702724 trunc from 0x040000 to 0x0054fa (0x3ab05 bytes) 000155 1787854274.730614 mapread 0x003d3d thru 0x0054fa (0x17be bytes) 000156 1787854274.759467 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000157 1787854275.019182 read 0x0010b6 thru 0x004ddb (0x3d26 bytes) 000158 1787854275.053389 mapread 0x000d5d thru 0x002b84 (0x1e28 bytes) 000159 1787854275.091115 punch from 0x000842 to 0x0054fa (0x4cb9 bytes) 000160 1787854275.105675 mapwrite 0x0295ab thru 0x030fae (0x7a04 bytes) 000162 1787854275.234995 mapwrite 0x006b94 thru 0x00bfa1 (0x540e bytes) 000163 1787854275.363416 write 0x02e9d2 thru 0x03425a (0x5889 bytes) 000164 1787854275.369785 read 0x02ab9a thru 0x03425a (0x96c1 bytes) 000165 1787854275.370444 trunc from 0x03425b to 0x037c89 (0x3a2f bytes) 000166 1787854275.400978 write 0x034cbf thru 0x03ffff (0xb341 bytes) 000167 1787854275.404718 trunc from 0x0193d9 to 0x01c879 (0x34a1 bytes) 000168 1787854275.413332 trunc from 0x040000 to 0x02901b (0x16fe4 bytes) 000169 1787854275.466563 punch from 0x0073a8 to 0x00b7e1 (0x443a bytes) 000170 1787854275.475591 trunc from 0x00775e to 0x00a5b9 (0x2e5c bytes) 000171 1787854275.484897 fallocate 0x01dae5 thru 0x022a9a (0x4fb6 bytes) 000172 1787854275.494091 fallocate 0x002f47 thru 0x007d4f (0x4e09 bytes) 000173 1787854275.503283 trunc from 0x02901c to 0x0036f8 (0x25923 bytes) 000174 1787854275.559011 trunc from 0x0036f9 to 0x034f9e (0x318a6 bytes) 000176 1787854275.611885 trunc from 0x034f9f to 0x004a9c (0x30502 bytes) 000177 1787854275.637600 mapread 0x0026d5 thru 0x004a9c (0x23c8 bytes) 000178 1787854275.661922 fallocate 0x0057b7 thru 0x01048e (0xacd8 bytes) 000179 1787854275.670617 fallocate 0x00f8d2 thru 0x016589 (0x6cb8 bytes) 000180 1787854275.679878 mapwrite 0x0012d6 thru 0x01093c (0xf667 bytes) 000181 1787854275.760668 read 0x009bde thru 0x01093c (0x6d5f bytes) 000182 1787854275.761396 write 0x03aa85 thru 0x03ffff (0x557b bytes) 000184 1787854275.777810 read 0x0082c9 thru 0x00b911 (0x3649 bytes) 000185 1787854275.778925 trunc from 0x01672a to 0x01c7c6 (0x609d bytes) 000186 1787854275.783991 mapwrite 0x007726 thru 0x0161fe (0xead9 bytes) 000187 1787854275.862885 fallocate 0x030325 thru 0x03fd16 (0xf9f2 bytes) 000189 1787854275.877870 punch from 0x00dc8b to 0x0130ee (0x5464 bytes) 000190 1787854275.898352 trunc from 0x03f286 to 0x03ffff (0xd7a bytes) 000191 1787854275.908248 mapwrite 0x039202 thru 0x03ffff (0x6dfe bytes) 000192 1787854275.944160 read 0x00f9f9 thru 0x012d6b (0x3373 bytes) 000193 1787854275.971269 punch from 0x00a45b to 0x01367a (0x9220 bytes) 000194 1787854275.986885 fallocate 0x011d02 thru 0x0138c7 (0x1bc6 bytes) 000195 1787854275.994146 trunc from 0x02cee0 to 0x02d8d1 (0x9f2 bytes) 000197 1787854276.015895 MIRROR_EXTEND to 2 mirrors 000199 1787854277.274687 fallocate 0x01e6cd thru 0x02ac52 (0xc586 bytes) 000201 1787854277.338361 mapwrite 0x0353af thru 0x03dcc2 (0x8914 bytes) 000202 1787854277.380227 mapread 0x003ddb thru 0x00ae27 (0x704d bytes) 000203 1787854277.390537 trunc from 0x03a70d to 0x03ffff (0x58f3 bytes) 000204 1787854277.395644 fallocate 0x008a60 thru 0x00aa1a (0x1fbb bytes) 000206 1787854277.407589 mapread 0x0175fe thru 0x01f824 (0x8227 bytes) 000207 1787854277.420594 write 0x01c156 thru 0x0219a9 (0x5854 bytes) 000208 1787854277.435734 mapwrite 0x0006e9 thru 0x00786a (0x7182 bytes) 000209 1787854277.463049 trunc from 0x040000 to 0x02ef18 (0x110e7 bytes) 000210 1787854277.485890 mapwrite 0x003be9 thru 0x00f685 (0xba9d bytes) 000211 1787854277.514603 punch from 0x013b89 to 0x0225ff (0xea77 bytes) 000212 1787854277.526615 mapwrite 0x02b2aa thru 0x02e343 (0x309a bytes) 000213 1787854277.562895 trunc from 0x02ef19 to 0x00be4a (0x230ce bytes) 000214 1787854277.585687 mapwrite 0x02a6a8 thru 0x02d4f3 (0x2e4c bytes) 000215 1787854277.625987 fallocate 0x03818b thru 0x03cfb8 (0x4e2e bytes) 000216 1787854277.631116 punch from 0x012805 to 0x01de4b (0xb647 bytes) 000217 1787854277.635986 mapwrite 0x000088 thru 0x009c45 (0x9bbe bytes) 000218 1787854277.666565 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d4f4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d4f4): Mirror 2: 0xcb3a2a0a (stale) CRC-32 checksum value for chunk [0, 0x2d4f4): Mirror 1: 0x60d3f554 Mirror 3: 0x60d3f554 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d4f4: skipped 000219 1787854278.593442 mapread 0x013dc8 thru 0x021dfa (0xe033 bytes) 000220 1787854278.648295 mapwrite 0x02c1f0 thru 0x031639 (0x544a bytes) 000221 1787854278.825575 punch from 0x020ae3 to 0x02a589 (0x9aa7 bytes) 000222 1787854278.838888 read 0x0235f0 thru 0x030bc8 (0xd5d9 bytes) 000223 1787854278.880406 mapread 0x0000c2 thru 0x009584 (0x94c3 bytes) 000224 1787854278.895341 mapwrite 0x00e82e thru 0x014a04 (0x61d7 bytes) 000225 1787854278.919733 punch from 0x00c60b to 0x0194bf (0xceb5 bytes) 000226 1787854278.924695 fallocate 0x03eb44 thru 0x03ffff (0x14bc bytes) 000227 1787854278.938908 fallocate 0x000486 thru 0x00bc95 (0xb810 bytes) 000228 1787854278.963582 MIRROR_RESYNC 3 mirrors 000229 1787854279.852569 mapwrite 0x0387c4 thru 0x03d180 (0x49bd bytes) 000230 1787854279.953857 fallocate 0x02da6f thru 0x0380a5 (0xa637 bytes) 000231 1787854279.961678 mapwrite 0x03c154 thru 0x03ffff (0x3eac bytes) 000232 1787854280.029672 trunc from 0x03fac2 to 0x03ffff (0x53e bytes) 000233 1787854280.039343 trunc from 0x040000 to 0x0104e9 (0x2fb16 bytes) 000235 1787854280.065225 trunc from 0x00074e to 0x010387 (0xfc3a bytes) 000236 1787854280.072527 trunc from 0x00bc68 to 0x0104e9 (0x4882 bytes) 000237 1787854280.077624 mapwrite 0x03a1e8 thru 0x03ffff (0x5e18 bytes) 000238 1787854280.181790 mapread 0x0156c9 thru 0x01f0a3 (0x99db bytes) 000239 1787854280.224873 trunc from 0x040000 to 0x03337a (0xcc85 bytes) 000240 1787854280.281506 write 0x00d08d thru 0x0183cd (0xb341 bytes) 000241 1787854280.297193 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3337b): Mirror 3: 0x1f66281b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3337b): Mirror 3: 0x1f66281b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3337b: skipped 000242 1787854280.774618 trunc from 0x0159f5 to 0x025686 (0xfc92 bytes) 000243 1787854280.780788 mapread 0x0241eb thru 0x02ffdf (0xbdf5 bytes) 000244 1787854280.814462 punch from 0x002975 to 0x00e911 (0xbf9d bytes) 000245 1787854280.819843 trunc from 0x03337b to 0x01df92 (0x153e8 bytes) 000246 1787854280.845733 mapread 0x003687 thru 0x00fd5d (0xc6d7 bytes) 000247 1787854280.871750 fallocate 0x03e048 thru 0x03e901 (0x8ba bytes) 000248 1787854280.879018 read 0x00a728 thru 0x019b9d (0xf476 bytes) 000249 1787854280.898022 punch from 0x013349 to 0x01df92 (0xac4a bytes) 000250 1787854280.911527 punch from 0x0146b9 to 0x01df92 (0x98da bytes) 000251 1787854280.921045 trunc from 0x01df93 to 0x000b85 (0x1d40d bytes) 000253 1787854280.956040 read 0x000452 thru 0x000b85 (0x734 bytes) 000254 1787854280.965493 mapwrite 0x01e7ea thru 0x020ec7 (0x26de bytes) 000255 1787854281.016364 trunc from 0x020ec8 to 0x01aa19 (0x64ae bytes) 000256 1787854281.039800 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1aa1a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aa1a): Mirror 3: 0xc7e957c0 (stale) CRC-32 checksum value for chunk [0, 0x1aa1a): Mirror 1: 0xade28fa7 Mirror 4: 0xade28fa7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aa1a: skipped 000257 1787854281.947037 trunc from 0x0023dd to 0x00d189 (0xadad bytes) 000258 1787854281.988953 trunc from 0x0081a0 to 0x013489 (0xb2ea bytes) 000259 1787854281.998889 mapwrite 0x01ec2a thru 0x02b52c (0xc903 bytes) 000260 1787854282.116690 read 0x0126c5 thru 0x01baa5 (0x93e1 bytes) 000261 1787854282.136948 mapwrite 0x02b0ae thru 0x038e2d (0xdd80 bytes) 000262 1787854282.229466 trunc from 0x02d231 to 0x03375d (0x652d bytes) 000263 1787854282.237970 mapread 0x0008f1 thru 0x00ff93 (0xf6a3 bytes) 000264 1787854282.263025 fallocate 0x0389ea thru 0x038d47 (0x35e bytes) 000265 1787854282.268748 punch from 0x029411 to 0x0335e4 (0xa1d4 bytes) 000266 1787854282.281953 punch from 0x00afd1 to 0x010987 (0x59b7 bytes) 000267 1787854282.288577 mapwrite 0x02cb19 thru 0x0332c9 (0x67b1 bytes) 000268 1787854282.320156 fallocate 0x002ff7 thru 0x00f7e5 (0xc7ef bytes) 000269 1787854282.327688 punch from 0x02f48d to 0x036c32 (0x77a6 bytes) 000270 1787854282.350579 MIRROR_RESYNC 3 mirrors 000271 1787854283.268666 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000272 1787854283.634280 fallocate 0x00f627 thru 0x0162bb (0x6c95 bytes) 000273 1787854283.684376 mapwrite 0x002bed thru 0x00704f (0x4463 bytes) 000274 1787854283.768794 mapread 0x019007 thru 0x024150 (0xb14a bytes) 000275 1787854283.796847 fallocate 0x03cfa6 thru 0x03ffff (0x305a bytes) 000276 1787854283.802538 trunc from 0x015c29 to 0x016e7d (0x1255 bytes) 000277 1787854283.807580 write 0x0251f3 thru 0x02a1ed (0x4ffb bytes) 000278 1787854283.812896 read 0x00cce4 thru 0x00f21f (0x253c bytes) 000279 1787854283.813755 fallocate 0x00ea0a thru 0x0158ee (0x6ee5 bytes) 000280 1787854283.822456 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000281 1787854284.028993 punch from 0x0022e7 to 0x00a13d (0x7e57 bytes) 000282 1787854284.039948 write 0x02bd17 thru 0x02d041 (0x132b bytes) 000283 1787854284.062259 mapwrite 0x002779 thru 0x010923 (0xe1ab bytes) 000284 1787854284.147434 trunc from 0x040000 to 0x01bcbe (0x24341 bytes) 000285 1787854284.187313 punch from 0x012581 to 0x0196a4 (0x7124 bytes) 000286 1787854284.198277 read 0x012f89 thru 0x017660 (0x46d8 bytes) 000287 1787854284.210935 mapread 0x001214 thru 0x0055c7 (0x43b4 bytes) 000288 1787854284.214366 mapwrite 0x01da49 thru 0x02b447 (0xd9ff bytes) 000289 1787854284.293986 mapread 0x00eea1 thru 0x016d79 (0x7ed9 bytes) 000290 1787854284.303273 fallocate 0x01d366 thru 0x01db9a (0x835 bytes) 000291 1787854284.311761 mapread 0x014314 thru 0x01b7c0 (0x74ad bytes) 000292 1787854284.314012 fallocate 0x00a4d9 thru 0x010008 (0x5b30 bytes) 000294 1787854284.322865 mapread 0x013e80 thru 0x01862e (0x47af bytes) 000295 1787854284.332281 read 0x013cb1 thru 0x01c2e6 (0x8636 bytes) 000296 1787854284.335320 fallocate 0x028dee thru 0x032d29 (0x9f3c bytes) 000297 1787854284.345869 punch from 0x014cae to 0x020b01 (0xbe54 bytes) 000299 1787854284.352886 write 0x0235cd thru 0x023790 (0x1c4 bytes) 000300 1787854284.354023 mapwrite 0x0110fd thru 0x0137bc (0x26c0 bytes) 000301 1787854284.400200 write 0x03d2ae thru 0x03ffff (0x2d52 bytes) 000302 1787854284.403865 punch from 0x023a71 to 0x0327d1 (0xed61 bytes) 000303 1787854284.413999 trunc from 0x038f39 to 0x03ffff (0x70c7 bytes) 000304 1787854284.426877 read 0x013e8d thru 0x01fdc5 (0xbf39 bytes) 000305 1787854284.445441 read 0x01df86 thru 0x01e193 (0x20e bytes) 000306 1787854284.446668 write 0x02b843 thru 0x02fec4 (0x4682 bytes) 000307 1787854284.448563 mapread 0x01ccfb thru 0x024363 (0x7669 bytes) 000308 1787854284.450674 trunc from 0x040000 to 0x02b048 (0x14fb7 bytes) 000309 1787854284.473453 trunc from 0x02b049 to 0x014bfe (0x1644a bytes) 000310 1787854284.491131 write 0x014a58 thru 0x02330f (0xe8b8 bytes) 000311 1787854284.492930 read 0x02217e thru 0x022ac3 (0x946 bytes) 000312 1787854284.497372 read 0x017b20 thru 0x01b6fa (0x3bdb bytes) 000313 1787854284.498257 fallocate 0x013c17 thru 0x017737 (0x3b21 bytes) 000316 1787854284.504942 trunc from 0x01bcf0 to 0x02330f (0x7620 bytes) 000317 1787854284.529622 trunc from 0x01c320 to 0x02330f (0x6ff0 bytes) 000319 1787854284.556179 mapread 0x002f3c thru 0x009fbc (0x7081 bytes) 000320 1787854284.561768 write 0x03e985 thru 0x03ffff (0x167b bytes) 000321 1787854284.562910 fallocate 0x019abe thru 0x0208bf (0x6e02 bytes) 000322 1787854284.579202 mapread 0x027479 thru 0x0284b7 (0x103f bytes) 000323 1787854284.602515 mapread 0x03ffee thru 0x03ffff (0x12 bytes) 000325 1787854284.605851 fallocate 0x030c36 thru 0x034755 (0x3b20 bytes) 000326 1787854284.611632 write 0x0161ff thru 0x016efa (0xcfc bytes) 000327 1787854284.612847 read 0x01dcc6 thru 0x029fed (0xc328 bytes) 000328 1787854284.613925 punch from 0x024986 to 0x02b224 (0x689f bytes) 000329 1787854284.620145 fallocate 0x028dea thru 0x03368c (0xa8a3 bytes) 000330 1787854284.626308 mapwrite 0x00e382 thru 0x01cb46 (0xe7c5 bytes) 000331 1787854284.654188 trunc from 0x021443 to 0x02332c (0x1eea bytes) 000332 1787854284.661651 trunc from 0x040000 to 0x03bf2f (0x40d0 bytes) 000334 1787854284.697558 write 0x01689f thru 0x018449 (0x1bab bytes) 000335 1787854284.698688 read 0x02b81d thru 0x038217 (0xc9fb bytes) 000337 1787854284.715856 trunc from 0x032dc2 to 0x0379d8 (0x4c17 bytes) 000338 1787854284.721606 write 0x023641 thru 0x032ec1 (0xf881 bytes) 000339 1787854284.730799 mapwrite 0x0276ba thru 0x034c5f (0xd5a6 bytes) 000340 1787854284.773414 read 0x0167ae thru 0x0260e1 (0xf934 bytes) 000341 1787854284.774112 read 0x00c314 thru 0x00de27 (0x1b14 bytes) 000342 1787854284.774586 mapwrite 0x03651e thru 0x03cb48 (0x662b bytes) 000343 1787854284.834894 fallocate 0x02a70a thru 0x0326ff (0x7ff6 bytes) 000345 1787854284.844744 read 0x01ab90 thru 0x0206e6 (0x5b57 bytes) 000346 1787854284.845700 mapwrite 0x02eae9 thru 0x03e6c7 (0xfbdf bytes) 000348 1787854284.902674 trunc from 0x03e6c8 to 0x00b029 (0x3369e bytes) 000349 1787854284.932983 mapwrite 0x039c98 thru 0x03ffff (0x6368 bytes) 000350 1787854285.035511 trunc from 0x040000 to 0x00d354 (0x32cab bytes) 000351 1787854285.052694 write 0x00e704 thru 0x019caf (0xb5ac bytes) 000352 1787854285.055799 mapwrite 0x02df03 thru 0x03393c (0x5a3a bytes) 000353 1787854285.144632 punch from 0x0162b7 to 0x024e8f (0xebd9 bytes) 000354 1787854285.152612 fallocate 0x0069a8 thru 0x009775 (0x2dce bytes) 000355 1787854285.168716 punch from 0x0183ae to 0x026775 (0xe3c8 bytes) 000356 1787854285.174419 punch from 0x00203a to 0x008f10 (0x6ed7 bytes) 000357 1787854285.181827 read 0x0274d0 thru 0x02ac31 (0x3762 bytes) 000358 1787854285.201808 trunc from 0x02b4f4 to 0x03393c (0x8449 bytes) 000359 1787854285.214761 mapread 0x02beb3 thru 0x02d162 (0x12b0 bytes) 000360 1787854285.251233 fallocate 0x0222ca thru 0x027020 (0x4d57 bytes) 000361 1787854285.260121 write 0x0278bc thru 0x0332d5 (0xba1a bytes) 000362 1787854285.272276 fallocate 0x03f886 thru 0x03ffff (0x77a bytes) 000363 1787854285.290977 MIRROR_EXTEND to 2 mirrors 000364 1787854286.139544 trunc from 0x00099e to 0x00c556 (0xbbb9 bytes) 000365 1787854286.186247 mapread 0x015597 thru 0x0157da (0x244 bytes) 000366 1787854286.197345 MIRROR_RESYNC 2 mirrors 000367 1787854286.922664 read 0x002da7 thru 0x00b867 (0x8ac1 bytes) 000368 1787854286.933695 fallocate 0x010099 thru 0x0117b5 (0x171d bytes) 000369 1787854286.980625 read 0x00b116 thru 0x00dfea (0x2ed5 bytes) 000370 1787854287.009500 fallocate 0x0162fe thru 0x01a327 (0x402a bytes) 000372 1787854287.024883 trunc from 0x040000 to 0x007d3d (0x382c2 bytes) 000373 1787854287.057334 mapread 0x003974 thru 0x007d3d (0x43ca bytes) 000375 1787854287.058789 trunc from 0x007d3e to 0x0095e5 (0x18a8 bytes) 000376 1787854287.079294 write 0x014f16 thru 0x0159dc (0xac7 bytes) 000377 1787854287.080528 fallocate 0x035002 thru 0x03ffff (0xaffe bytes) 000378 1787854287.086438 mapwrite 0x0343c9 thru 0x03fa38 (0xb670 bytes) 000379 1787854287.178177 fallocate 0x02448f thru 0x02657d (0x20ef bytes) 000380 1787854287.186369 mapread 0x020164 thru 0x02637b (0x6218 bytes) 000381 1787854287.245454 mapwrite 0x02b6d6 thru 0x03156d (0x5e98 bytes) 000382 1787854287.304458 trunc from 0x040000 to 0x015da1 (0x2a25e bytes) 000383 1787854287.327656 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000384 1787854287.561706 fallocate 0x0309e1 thru 0x03931d (0x893d bytes) 000385 1787854287.569096 mapwrite 0x039c43 thru 0x03ffff (0x63bd bytes) 000386 1787854287.628545 trunc from 0x009544 to 0x01002f (0x6aec bytes) 000387 1787854287.638897 MIRROR_EXTEND to 2 mirrors 000388 1787854288.443701 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000389 1787854288.625186 read 0x03e8b9 thru 0x03ffff (0x1747 bytes) 000391 1787854288.637270 trunc from 0x032717 to 0x037f5e (0x5848 bytes) 000392 1787854288.652965 mapwrite 0x037f8c thru 0x03ffff (0x8074 bytes) 000393 1787854288.701025 trunc from 0x03b7f9 to 0x03ffff (0x4807 bytes) 000394 1787854288.709863 read 0x0358c0 thru 0x03ffff (0xa740 bytes) 000396 1787854288.724705 trunc from 0x02d2f0 to 0x02ddb2 (0xac3 bytes) 000397 1787854288.730169 write 0x01491f thru 0x0181c2 (0x38a4 bytes) 000398 1787854288.736581 trunc from 0x01dd58 to 0x022375 (0x461e bytes) 000399 1787854288.742178 fallocate 0x02659c thru 0x02b636 (0x509b bytes) 000400 1787854288.747359 write 0x022ad1 thru 0x02a962 (0x7e92 bytes) 000401 1787854288.753742 fallocate 0x038194 thru 0x03ffff (0x7e6c bytes) 000402 1787854288.758752 mapwrite 0x02ab90 thru 0x0305b4 (0x5a25 bytes) 000404 1787854288.783618 write 0x025a1e thru 0x02ad99 (0x537c bytes) 000405 1787854288.785712 trunc from 0x015193 to 0x020a8b (0xb8f9 bytes) 000406 1787854288.792258 fallocate 0x0172f3 thru 0x024d7a (0xda88 bytes) 000408 1787854288.797571 mapread 0x00dc9c thru 0x010da0 (0x3105 bytes) 000409 1787854288.800598 read 0x0399d3 thru 0x03ffff (0x662d bytes) 000410 1787854288.801656 trunc from 0x027018 to 0x02c392 (0x537b bytes) 000411 1787854288.817349 write 0x012ee1 thru 0x01ef16 (0xc036 bytes) 000412 1787854288.823690 mapwrite 0x0253fa thru 0x0336f1 (0xe2f8 bytes) 000413 1787854288.868397 trunc from 0x01e9b5 to 0x02a276 (0xb8c2 bytes) 000414 1787854288.879776 punch from 0x0142ed to 0x02153f (0xd253 bytes) 000415 1787854288.887491 mapread 0x01a934 thru 0x01b8a8 (0xf75 bytes) 000416 1787854288.906334 MIRROR_EXTEND to 2 mirrors 000417 1787854289.602023 trunc from 0x039b1c to 0x03e4e8 (0x49cd bytes) 000418 1787854289.647350 read 0x0101ea thru 0x014002 (0x3e19 bytes) 000419 1787854289.661613 mapwrite 0x00d7f6 thru 0x019406 (0xbc11 bytes) 000420 1787854289.714890 write 0x016565 thru 0x01b1a2 (0x4c3e bytes) 000422 1787854289.721410 read 0x03368d thru 0x03f8e8 (0xc25c bytes) 000423 1787854289.736966 punch from 0x01eea6 to 0x021d91 (0x2eec bytes) 000424 1787854289.742314 trunc from 0x040000 to 0x02bad5 (0x1452a bytes) 000425 1787854289.761484 trunc from 0x018c6d to 0x01da7e (0x4e12 bytes) 000426 1787854289.776686 trunc from 0x02bad6 to 0x02d917 (0x1e42 bytes) 000428 1787854289.795361 trunc from 0x02d918 to 0x02e9e1 (0x10ca bytes) 000429 1787854289.814640 read 0x004e00 thru 0x011465 (0xc666 bytes) 000430 1787854289.815689 read 0x00388e thru 0x011a4b (0xe1be bytes) 000431 1787854289.816852 trunc from 0x02e9e2 to 0x026f34 (0x7aad bytes) 000432 1787854289.830687 read 0x00faea thru 0x016276 (0x678d bytes) 000433 1787854289.832352 trunc from 0x026f35 to 0x01f6eb (0x7849 bytes) 000434 1787854289.850550 trunc from 0x01028a to 0x017877 (0x75ee bytes) 000435 1787854289.854947 trunc from 0x007254 to 0x013fa2 (0xcd4f bytes) 000436 1787854289.860491 fallocate 0x0364cd thru 0x03b183 (0x4cb7 bytes) 000437 1787854289.867941 mapread 0x01b5aa thru 0x01f6eb (0x4142 bytes) 000438 1787854289.911050 punch from 0x013786 to 0x01f1b6 (0xba31 bytes) 000439 1787854289.917864 write 0x0287ef thru 0x033376 (0xab88 bytes) 000440 1787854289.920511 punch from 0x026ec7 to 0x033376 (0xc4b0 bytes) 000441 1787854289.941746 mapwrite 0x01b23e thru 0x028db6 (0xdb79 bytes) 000442 1787854290.031378 read 0x02a064 thru 0x02bdc3 (0x1d60 bytes) 000443 1787854290.034299 punch from 0x0282d2 to 0x033376 (0xb0a5 bytes) 000444 1787854290.046332 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000445 1787854290.210132 trunc from 0x002bdc to 0x00b2c8 (0x86ed bytes) 000446 1787854290.216552 trunc from 0x004afc to 0x0131a7 (0xe6ac bytes) 000447 1787854290.221971 read 0x01c394 thru 0x020800 (0x446d bytes) 000448 1787854290.237736 punch from 0x013f1e to 0x014606 (0x6e9 bytes) 000449 1787854290.242543 read 0x010215 thru 0x01b6c0 (0xb4ac bytes) 000450 1787854290.261689 trunc from 0x0052a3 to 0x012e04 (0xdb62 bytes) 000451 1787854290.266397 trunc from 0x005c4f to 0x0062c8 (0x67a bytes) 000452 1787854290.271380 trunc from 0x033377 to 0x00784a (0x2bb2c bytes) 000454 1787854290.294383 write 0x017494 thru 0x0246a1 (0xd20e bytes) 000455 1787854290.297499 trunc from 0x019bc8 to 0x01c7fb (0x2c34 bytes) 000456 1787854290.312617 trunc from 0x019f78 to 0x01c0e3 (0x216c bytes) 000457 1787854290.317585 trunc from 0x0246a2 to 0x016dda (0xd8c7 bytes) 000458 1787854290.332554 trunc from 0x016ddb to 0x0226e6 (0xb90c bytes) 000459 1787854290.352461 trunc from 0x0205b2 to 0x0226e6 (0x2135 bytes) 000460 1787854290.357285 mapread 0x016393 thru 0x01d6ec (0x735a bytes) 000461 1787854290.402182 mapwrite 0x02ff86 thru 0x03e858 (0xe8d3 bytes) 000462 1787854290.505503 MIRROR_EXTEND to 2 mirrors 000463 1787854291.306600 punch from 0x008858 to 0x0146bd (0xbe66 bytes) 000465 1787854291.395117 mapwrite 0x008dd4 thru 0x014ce8 (0xbf15 bytes) 000468 1787854291.481299 MIRROR_RESYNC 2 mirrors 000469 1787854292.234353 read 0x00ffd2 thru 0x01547f (0x54ae bytes) 000470 1787854292.250393 trunc from 0x037df7 to 0x03e858 (0x6a62 bytes) 000471 1787854292.320918 fallocate 0x007fc6 thru 0x01523d (0xd278 bytes) 000472 1787854292.337358 trunc from 0x03e859 to 0x02c5e7 (0x12271 bytes) 000473 1787854292.401841 mapread 0x0162b9 thru 0x01e403 (0x814b bytes) 000474 1787854292.424666 fallocate 0x02886d thru 0x0338e3 (0xb077 bytes) 000475 1787854292.430395 read 0x009747 thru 0x00cfb2 (0x386c bytes) 000476 1787854292.452877 fallocate 0x025855 thru 0x0337c5 (0xdf71 bytes) 000477 1787854292.458460 fallocate 0x01c5f0 thru 0x029f99 (0xd9aa bytes) 000480 1787854292.468327 trunc from 0x0097dd to 0x0130af (0x98d3 bytes) 000481 1787854292.475144 mapwrite 0x02e7f5 thru 0x0311e5 (0x29f1 bytes) 000483 1787854292.496516 mapread 0x02a686 thru 0x031c38 (0x75b3 bytes) 000484 1787854292.497485 trunc from 0x0338e4 to 0x00ba4b (0x27e98 bytes) 000485 1787854292.522711 mapread 0x0065e4 thru 0x00ba4b (0x5468 bytes) 000486 1787854292.557481 read 0x0070dd thru 0x00ba4b (0x496f bytes) 000487 1787854292.558226 trunc from 0x00ba4c to 0x02140a (0x159bf bytes) 000488 1787854292.575840 read 0x0171dc thru 0x01bf46 (0x4d6b bytes) 000489 1787854292.594228 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2140b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2140b): Mirror 2: 0xf5dfcda2 (stale) CRC-32 checksum value for chunk [0, 0x2140b): Mirror 1: 0xad4863d6 Mirror 3: 0xad4863d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2140b: skipped 000490 1787854293.423354 trunc from 0x02140b to 0x022808 (0x13fe bytes) 000491 1787854293.544618 trunc from 0x00002d to 0x000a30 (0xa04 bytes) 000492 1787854293.551585 write 0x0038d6 thru 0x005f35 (0x2660 bytes) 000493 1787854293.562869 write 0x014277 thru 0x02419a (0xff24 bytes) 000494 1787854293.583239 trunc from 0x00388f to 0x00dafe (0xa270 bytes) 000495 1787854293.609774 fallocate 0x00de3e thru 0x015d86 (0x7f49 bytes) 000496 1787854293.621918 trunc from 0x02419b to 0x01f376 (0x4e24 bytes) 000497 1787854293.651564 write 0x019a5e thru 0x01ab99 (0x113c bytes) 000499 1787854293.658009 trunc from 0x01bd76 to 0x01e592 (0x281d bytes) 000500 1787854293.668005 mapread 0x01a980 thru 0x01f376 (0x49f7 bytes) 000501 1787854293.687523 trunc from 0x0105cb to 0x01e916 (0xe34c bytes) 000502 1787854293.706583 trunc from 0x00ce66 to 0x00f1d5 (0x2370 bytes) 000503 1787854293.712160 trunc from 0x01f377 to 0x02dfc3 (0xec4d bytes) 000504 1787854293.739868 read 0x010712 thru 0x01a818 (0xa107 bytes) 000505 1787854293.771103 punch from 0x00c63a to 0x0104df (0x3ea6 bytes) 000507 1787854293.782363 punch from 0x0155f5 to 0x01acc6 (0x56d2 bytes) 000508 1787854293.795137 trunc from 0x015dc7 to 0x022173 (0xc3ad bytes) 000509 1787854293.821749 write 0x031ec9 thru 0x039032 (0x716a bytes) 000510 1787854293.823866 write 0x02820a thru 0x02db8c (0x5983 bytes) 000511 1787854293.846550 trunc from 0x039033 to 0x0135fd (0x25a35 bytes) 000512 1787854293.877488 read 0x00cb2f thru 0x011423 (0x48f5 bytes) 000513 1787854293.891604 fallocate 0x03b8f0 thru 0x03fbb2 (0x42c3 bytes) 000514 1787854293.903754 mapwrite 0x017453 thru 0x0177a8 (0x356 bytes) 000515 1787854293.961337 trunc from 0x00fcf8 to 0x01dc1e (0xdf27 bytes) 000517 1787854293.972361 read 0x03a5ca thru 0x03fbb2 (0x55e9 bytes) 000518 1787854293.973364 fallocate 0x035bb3 thru 0x03bd86 (0x61d4 bytes) 000519 1787854293.981582 read 0x00b1ed thru 0x0160e7 (0xaefb bytes) 000520 1787854293.993568 write 0x005f92 thru 0x00aa84 (0x4af3 bytes) 000521 1787854293.996179 fallocate 0x037303 thru 0x03fddf (0x8add bytes) 000522 1787854294.004045 mapread 0x038d15 thru 0x03fbb2 (0x6e9e bytes) 000525 1787854294.007604 fallocate 0x03ca74 thru 0x03f9d8 (0x2f65 bytes) 000526 1787854294.023954 write 0x003340 thru 0x008ad3 (0x5794 bytes) 000528 1787854294.027229 punch from 0x031506 to 0x03b170 (0x9c6b bytes) 000529 1787854294.042442 write 0x012eb0 thru 0x0204ed (0xd63e bytes) 000530 1787854294.059596 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fbb3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbb3): Mirror 2: 0xd20a84d2 (stale) Mirror 3: 0xf164f917 (stale) CRC-32 checksum value for chunk [0, 0x3fbb3): Mirror 1: 0xec6473d0 Mirror 4: 0xec6473d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbb3: skipped 000531 1787854295.137475 MIRROR_RESYNC 4 mirrors 000532 1787854296.311473 MIRROR_RESYNC 4 mirrors 000533 1787854296.826400 fallocate 0x0189a5 thru 0x01d444 (0x4aa0 bytes) 000534 1787854296.894777 mapwrite 0x00a703 thru 0x00c244 (0x1b42 bytes) 000535 1787854296.967420 write 0x01bad2 thru 0x020875 (0x4da4 bytes) 000536 1787854297.002249 trunc from 0x01ce60 to 0x02a279 (0xd41a bytes) 000537 1787854297.030537 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fbb3): Mirror 2: 0xec6473d0 (stale) Mirror 3: 0xec6473d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fbb3): Mirror 2: 0xec6473d0 (stale) Mirror 3: 0xec6473d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fbb3: skipped 000538 1787854297.466670 write 0x010ac5 thru 0x01898f (0x7ecb bytes) 000539 1787854297.509299 fallocate 0x024d24 thru 0x02d452 (0x872f bytes) 000540 1787854297.520264 write 0x02ca2c thru 0x03101b (0x45f0 bytes) 000541 1787854297.550647 punch from 0x012ad1 to 0x01a1d6 (0x7706 bytes) 000542 1787854297.585343 mapread 0x029970 thru 0x0395ef (0xfc80 bytes) 000543 1787854297.621617 trunc from 0x03fbb3 to 0x029111 (0x16aa1 bytes) 000544 1787854297.641892 punch from 0x01d38a to 0x024e63 (0x7ada bytes) 000545 1787854297.647544 write 0x03b920 thru 0x03ffff (0x46e0 bytes) 000546 1787854297.649338 punch from 0x0008ca to 0x00728a (0x69c1 bytes) 000547 1787854297.658535 mapread 0x01bf24 thru 0x0214ff (0x55dc bytes) 000548 1787854297.668513 mapread 0x015d10 thru 0x022d8d (0xd07e bytes) 000549 1787854297.669994 trunc from 0x040000 to 0x02b3bb (0x14c44 bytes) 000550 1787854297.688478 write 0x00bc3f thru 0x00c6b1 (0xa73 bytes) 000552 1787854297.700501 mapwrite 0x03d999 thru 0x03ffff (0x2667 bytes) 000553 1787854297.782498 write 0x03e470 thru 0x03ffff (0x1b90 bytes) 000554 1787854297.783771 read 0x00b7f2 thru 0x00ed12 (0x3521 bytes) 000555 1787854297.796011 mapread 0x003451 thru 0x008db2 (0x5962 bytes) 000556 1787854297.814331 punch from 0x033c51 to 0x03a6a3 (0x6a53 bytes) 000557 1787854297.819318 read 0x012f9f thru 0x016051 (0x30b3 bytes) 000558 1787854297.833027 trunc from 0x01605b to 0x024d33 (0xecd9 bytes) 000559 1787854297.842522 MIRROR_RESYNC 3 mirrors 000560 1787854298.942322 read 0x036191 thru 0x03ffff (0x9e6f bytes) 000561 1787854298.957821 read 0x01fa3f thru 0x020e81 (0x1443 bytes) 000562 1787854298.980703 mapread 0x03bdeb thru 0x03f8b2 (0x3ac8 bytes) 000563 1787854298.988572 MIRROR_EXTEND to 4 mirrors 000564 1787854299.909396 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000565 1787854300.201050 punch from 0x018b08 to 0x01aec2 (0x23bb bytes) 000567 1787854300.256701 punch from 0x005aaa to 0x0087a8 (0x2cff bytes) 000568 1787854300.270519 MIRROR_RESYNC 3 mirrors 000570 1787854301.386763 punch from 0x02378a to 0x0309e8 (0xd25f bytes) 000571 1787854301.443500 mapwrite 0x03662c thru 0x03e2e7 (0x7cbc bytes) 000573 1787854301.495357 trunc from 0x040000 to 0x035aae (0xa551 bytes) 000574 1787854301.527509 fallocate 0x000e5f thru 0x001f90 (0x1132 bytes) 000575 1787854301.535607 punch from 0x035590 to 0x035aae (0x51f bytes) 000576 1787854301.547576 MIRROR_RESYNC 3 mirrors 000578 1787854302.693026 write 0x02ed05 thru 0x035a1c (0x6d18 bytes) 000579 1787854302.743630 mapread 0x029416 thru 0x0302ca (0x6eb5 bytes) 000580 1787854302.764628 write 0x0223b3 thru 0x027408 (0x5056 bytes) 000581 1787854302.771899 punch from 0x008ab9 to 0x009e5c (0x13a4 bytes) 000582 1787854302.777430 fallocate 0x03a988 thru 0x03ffff (0x5678 bytes) 000583 1787854302.783049 trunc from 0x035aaf to 0x0277a9 (0xe305 bytes) 000584 1787854302.805946 punch from 0x00a4a6 to 0x018a88 (0xe5e3 bytes) 000585 1787854302.811494 write 0x02b180 thru 0x036055 (0xaed6 bytes) 000586 1787854302.814434 write 0x0170e7 thru 0x02514f (0xe069 bytes) 000587 1787854302.820248 trunc from 0x036056 to 0x030f57 (0x50fe bytes) 000588 1787854302.846221 read 0x01ec47 thru 0x0270f4 (0x84ae bytes) 000590 1787854302.848374 mapwrite 0x02a898 thru 0x038391 (0xdafa bytes) 000592 1787854302.928163 write 0x019ed8 thru 0x020c44 (0x6d6d bytes) 000593 1787854302.930385 mapwrite 0x00800a thru 0x00ea3f (0x6a36 bytes) 000594 1787854302.983726 read 0x017d37 thru 0x0269e1 (0xecab bytes) 000595 1787854302.984577 fallocate 0x025ed2 thru 0x034c15 (0xed44 bytes) 000596 1787854302.994854 write 0x012507 thru 0x019c41 (0x773b bytes) 000597 1787854303.032278 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38392): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38392): Mirror 2: 0xace14860 (stale) Mirror 4: 0xace14860 (stale) CRC-32 checksum value for chunk [0, 0x38392): Mirror 1: 0x68a7bd9a Mirror 5: 0x68a7bd9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38392: skipped 000598 1787854304.238474 punch from 0x0247df to 0x024ad0 (0x2f2 bytes) 000599 1787854304.310154 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38392): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38392): Mirror 2: 0xace14860 (stale) Mirror 4: 0xace14860 (stale) Mirror 5: 0x68a7bd9a (stale) CRC-32 checksum value for chunk [0, 0x38392): Mirror 1: 0x83517b42 Mirror 6: 0x83517b42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38392: skipped 000600 1787854305.563830 fallocate 0x0299e6 thru 0x03505d (0xb678 bytes) 000601 1787854305.632509 mapwrite 0x03ec8f thru 0x03ffff (0x1371 bytes) 000602 1787854305.723017 write 0x01b431 thru 0x01cf28 (0x1af8 bytes) 000604 1787854305.735138 punch from 0x035b99 to 0x03ffff (0xa467 bytes) 000605 1787854305.742785 fallocate 0x013bfa thru 0x019b15 (0x5f1c bytes) 000607 1787854305.755274 trunc from 0x00cf87 to 0x018dc1 (0xbe3b bytes) 000608 1787854305.769696 read 0x00d15a thru 0x0116e4 (0x458b bytes) 000609 1787854305.794087 trunc from 0x040000 to 0x007b28 (0x384d7 bytes) 000610 1787854305.829572 trunc from 0x007b29 to 0x010838 (0x8d10 bytes) 000611 1787854305.887996 write 0x038f9d thru 0x03ffff (0x7063 bytes) 000612 1787854305.894156 write 0x0160f9 thru 0x01c988 (0x6890 bytes) 000613 1787854305.921587 punch from 0x004766 to 0x008c63 (0x44fe bytes) 000614 1787854305.931387 trunc from 0x040000 to 0x01bc3f (0x243c0 bytes) 000615 1787854305.990038 fallocate 0x017362 thru 0x025e46 (0xeae5 bytes) 000617 1787854306.029839 mapwrite 0x038e50 thru 0x03ffff (0x71b0 bytes) 000618 1787854306.098885 write 0x02c12c thru 0x030078 (0x3f4d bytes) 000619 1787854306.114414 trunc from 0x01c7ad to 0x01dc83 (0x14d7 bytes) 000620 1787854306.121534 write 0x007b26 thru 0x009202 (0x16dd bytes) 000621 1787854306.137835 punch from 0x038618 to 0x03b82c (0x3215 bytes) 000623 1787854306.144795 trunc from 0x00b1cc to 0x017808 (0xc63d bytes) 000624 1787854306.152290 mapread 0x0177d2 thru 0x025492 (0xdcc1 bytes) 000625 1787854306.172918 write 0x01edce thru 0x02ca40 (0xdc73 bytes) 000626 1787854306.175316 trunc from 0x00cbf4 to 0x015c7f (0x908c bytes) 000627 1787854306.180787 read 0x0134eb thru 0x01bd3c (0x8852 bytes) 000628 1787854306.202401 punch from 0x0109ab to 0x011d72 (0x13c8 bytes) 000629 1787854306.207425 trunc from 0x00cff7 to 0x0196a3 (0xc6ad bytes) 000630 1787854306.226285 MIRROR_RESYNC 5 mirrors 000631 1787854308.309929 read 0x031325 thru 0x03f299 (0xdf75 bytes) 000632 1787854308.328688 read 0x00fcdc thru 0x01b470 (0xb795 bytes) 000633 1787854308.374441 mapread 0x014a31 thru 0x01eebf (0xa48f bytes) 000636 1787854308.381159 mapwrite 0x03227c thru 0x03719e (0x4f23 bytes) 000637 1787854308.546401 mapread 0x013470 thru 0x01ec64 (0xb7f5 bytes) 000638 1787854308.623925 read 0x034d3d thru 0x03ffff (0xb2c3 bytes) 000639 1787854308.637717 fallocate 0x02aec3 thru 0x03984f (0xe98d bytes) 000640 1787854308.643031 trunc from 0x01ce81 to 0x01dddc (0xf5c bytes) 000641 1787854308.648859 mapread 0x011cfb thru 0x02190c (0xfc12 bytes) 000642 1787854308.660044 mapread 0x03bc94 thru 0x03ffff (0x436c bytes) 000643 1787854308.661853 mapread 0x036808 thru 0x03ffff (0x97f8 bytes) 000644 1787854308.663212 write 0x00fa6e thru 0x01ca04 (0xcf97 bytes) 000645 1787854308.666007 write 0x03ad20 thru 0x03b633 (0x914 bytes) 000646 1787854308.667807 mapread 0x00f912 thru 0x016eab (0x759a bytes) 000648 1787854308.670936 punch from 0x01d981 to 0x02c950 (0xefd0 bytes) 000649 1787854308.681877 read 0x036508 thru 0x03bf3c (0x5a35 bytes) 000650 1787854308.682958 punch from 0x03f680 to 0x03ffff (0x980 bytes) 000651 1787854308.690784 trunc from 0x0111d9 to 0x01ceb8 (0xbce0 bytes) 000652 1787854308.719807 fallocate 0x0187bd thru 0x01a14c (0x1990 bytes) 000653 1787854308.726836 read 0x014b27 thru 0x01e40f (0x98e9 bytes) 000654 1787854308.739645 read 0x03708d thru 0x039bf8 (0x2b6c bytes) 000655 1787854308.740312 mapwrite 0x01e527 thru 0x01e85a (0x334 bytes) 000656 1787854308.800026 MIRROR_RESYNC 5 mirrors 000657 1787854311.519721 trunc from 0x023e7b to 0x029ebd (0x6043 bytes) 000658 1787854311.572873 write 0x03a1a6 thru 0x03ffff (0x5e5a bytes) 000659 1787854311.587159 mapwrite 0x01d322 thru 0x024539 (0x7218 bytes) 000660 1787854311.675237 mapwrite 0x01c6d6 thru 0x02bbec (0xf517 bytes) 000661 1787854311.721925 trunc from 0x040000 to 0x024dac (0x1b253 bytes) 000662 1787854311.759559 write 0x01896d thru 0x025199 (0xc82d bytes) 000663 1787854311.762240 punch from 0x01deef to 0x025199 (0x72ab bytes) 000664 1787854311.799408 trunc from 0x004174 to 0x004cc7 (0xb54 bytes) 000665 1787854311.809665 write 0x00dd0a thru 0x0144f4 (0x67eb bytes) 000666 1787854311.824006 fallocate 0x0139cc thru 0x016fb8 (0x35ed bytes) 000667 1787854311.833780 trunc from 0x00a71c to 0x00aec3 (0x7a8 bytes) 000668 1787854311.838803 write 0x00ccc4 thru 0x017d73 (0xb0b0 bytes) 000669 1787854311.852718 mapread 0x00cdb9 thru 0x015b05 (0x8d4d bytes) 000671 1787854311.860527 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2519a): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2519a): Mirror 2: 0x2d2e06b0 (stale) Mirror 4: 0x2d2e06b0 (stale) Mirror 5: 0x2d2e06b0 (stale) Mirror 6: 0x2d2e06b0 (stale) CRC-32 checksum value for chunk [0, 0x2519a): Mirror 1: 0x361b32a6 Mirror 7: 0x361b32a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2519a: skipped 000672 1787854313.176006 punch from 0x00d515 to 0x01a57f (0xd06b bytes) 000673 1787854313.235226 write 0x0107f4 thru 0x01f561 (0xed6e bytes) 000675 1787854313.274269 punch from 0x00b46a to 0x012400 (0x6f97 bytes) 000676 1787854313.314469 trunc from 0x02519a to 0x00e066 (0x17133 bytes) 000677 1787854313.346741 read 0x007f4c thru 0x00e066 (0x611b bytes) 000678 1787854313.384448 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xe067): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe067): Mirror 2: 0x3b5292aa (stale) Mirror 4: 0x3b5292aa (stale) Mirror 5: 0x3b5292aa (stale) Mirror 6: 0x3b5292aa (stale) Mirror 7: 0x4919f80e (stale) CRC-32 checksum value for chunk [0, 0xe067): Mirror 1: 0x3b5292aa Mirror 8: 0x3b5292aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe067: skipped 000680 1787854314.345910 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xe067): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe067): Mirror 2: 0x3b5292aa (stale) Mirror 4: 0x3b5292aa (stale) Mirror 5: 0x3b5292aa (stale) Mirror 6: 0x3b5292aa (stale) Mirror 7: 0x4919f80e (stale) CRC-32 checksum value for chunk [0, 0xe067): Mirror 1: 0x3b5292aa Mirror 8: 0x3b5292aa Mirror 9: 0x3b5292aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe067: skipped 000681 1787854315.411184 fallocate 0x013204 thru 0x01ca69 (0x9866 bytes) 000682 1787854315.471996 punch from 0x00cfb5 to 0x017a96 (0xaae2 bytes) 000683 1787854315.477749 mapwrite 0x032954 thru 0x03ffff (0xd6ac bytes) 000684 1787854315.561947 mapread 0x026e2b thru 0x031569 (0xa73f bytes) 000686 1787854315.574956 mapread 0x022778 thru 0x027bfd (0x5486 bytes) 000687 1787854315.576025 read 0x02474d thru 0x02526a (0xb1e bytes) 000689 1787854315.578273 mapwrite 0x01bef1 thru 0x025d0a (0x9e1a bytes) 000690 1787854315.618243 punch from 0x02036b to 0x02487e (0x4514 bytes) 000691 1787854315.624152 punch from 0x02f2a7 to 0x03ee8b (0xfbe5 bytes) 000692 1787854315.643183 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5859b9b (stale) Mirror 4: 0x5859b9b (stale) Mirror 5: 0x5859b9b (stale) Mirror 6: 0x5859b9b (stale) Mirror 7: 0x437dcb1a (stale) Mirror 8: 0x37cf9326 (stale) Mirror 9: 0x37cf9326 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x13fadffc Mirror 10: 0x13fadffc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000693 1787854317.044943 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5859b9b (stale) Mirror 5: 0x5859b9b (stale) Mirror 6: 0x5859b9b (stale) Mirror 7: 0x437dcb1a (stale) Mirror 8: 0x37cf9326 (stale) Mirror 9: 0x37cf9326 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5859b9b (stale) Mirror 5: 0x5859b9b (stale) Mirror 6: 0x5859b9b (stale) Mirror 7: 0x437dcb1a (stale) Mirror 8: 0x37cf9326 (stale) Mirror 9: 0x37cf9326 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x13fadffc Mirror 10: 0x13fadffc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1787854318.000835 fallocate 0x03d722 thru 0x03f369 (0x1c48 bytes) 000695 1787854318.063292 MIRROR_RESYNC 8 mirrors 000696 1787854321.375235 read 0x039279 thru 0x03a245 (0xfcd bytes) 000697 1787854321.385045 MIRROR_RESYNC 8 mirrors 000698 1787854322.102584 MIRROR_EXTEND to 9 mirrors 000699 1787854323.113475 write 0x00ee60 thru 0x01c086 (0xd227 bytes) 000700 1787854323.178654 trunc from 0x000285 to 0x008710 (0x848c bytes) 000701 1787854323.191697 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x13fadffc (stale) Mirror 5: 0x13fadffc (stale) Mirror 6: 0x13fadffc (stale) Mirror 7: 0x13fadffc (stale) Mirror 8: 0x13fadffc (stale) Mirror 9: 0x13fadffc (stale) Mirror 10: 0x13fadffc (stale) Mirror 11: 0x13fadffc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9590fcc Mirror 12: 0x9590fcc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000702 1787854324.702423 trunc from 0x022f22 to 0x031818 (0xe8f7 bytes) 000703 1787854324.771605 read 0x02f673 thru 0x03b632 (0xbfc0 bytes) 000704 1787854324.789810 MIRROR_RESYNC 10 mirrors 000705 1787854327.702293 trunc from 0x040000 to 0x03ba24 (0x45db bytes) 000706 1787854327.753248 trunc from 0x03ba25 to 0x03e458 (0x2a34 bytes) 000708 1787854327.774815 trunc from 0x03e459 to 0x034238 (0xa220 bytes) 000709 1787854327.791519 trunc from 0x031890 to 0x034238 (0x29a9 bytes) 000710 1787854327.797804 read 0x013509 thru 0x022c88 (0xf780 bytes) 000711 1787854327.809705 write 0x023c44 thru 0x029a99 (0x5e56 bytes) 000712 1787854327.820835 read 0x015b1c thru 0x0225d6 (0xcabb bytes) 000713 1787854327.827434 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34239): Mirror 5: 0x61ccce1d (stale) Mirror 6: 0x61ccce1d (stale) Mirror 7: 0x61ccce1d (stale) Mirror 8: 0x61ccce1d (stale) Mirror 9: 0x61ccce1d (stale) Mirror 10: 0x61ccce1d (stale) Mirror 11: 0x61ccce1d (stale) Mirror 12: 0x61ccce1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34239): Mirror 5: 0x61ccce1d (stale) Mirror 6: 0x61ccce1d (stale) Mirror 7: 0x61ccce1d (stale) Mirror 8: 0x61ccce1d (stale) Mirror 9: 0x61ccce1d (stale) Mirror 10: 0x61ccce1d (stale) Mirror 11: 0x61ccce1d (stale) Mirror 12: 0x61ccce1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34239: skipped 000714 1787854328.441437 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x34239): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34239): Mirror 5: 0x61ccce1d (stale) Mirror 6: 0x61ccce1d (stale) Mirror 7: 0x61ccce1d (stale) Mirror 8: 0x61ccce1d (stale) Mirror 9: 0x61ccce1d (stale) Mirror 10: 0x61ccce1d (stale) Mirror 11: 0x61ccce1d (stale) Mirror 12: 0x61ccce1d (stale) CRC-32 checksum value for chunk [0, 0x34239): Mirror 1: 0x56f05169 Mirror 13: 0x56f05169 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34239: skipped 000715 1787854329.895864 trunc from 0x034239 to 0x01e3ef (0x15e49 bytes) 000716 1787854329.997010 mapwrite 0x038021 thru 0x03df01 (0x5ee1 bytes) 000717 1787854330.100453 write 0x02c6c4 thru 0x02e224 (0x1b61 bytes) 000718 1787854330.128710 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3df02): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df02): Mirror 5: 0xdd7c22bd (stale) Mirror 6: 0xdd7c22bd (stale) Mirror 7: 0xdd7c22bd (stale) Mirror 8: 0xdd7c22bd (stale) Mirror 9: 0xdd7c22bd (stale) Mirror 10: 0xdd7c22bd (stale) Mirror 11: 0xdd7c22bd (stale) Mirror 12: 0xdd7c22bd (stale) Mirror 13: 0xfb41d099 (stale) CRC-32 checksum value for chunk [0, 0x3df02): Mirror 1: 0xa974cd82 Mirror 14: 0xa974cd82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df02: skipped 000719 1787854331.651909 trunc from 0x03df02 to 0x0112dd (0x2cc24 bytes) 000720 1787854331.750971 fallocate 0x00a7e3 thru 0x00fad2 (0x52f0 bytes) 000721 1787854331.759946 trunc from 0x0112de to 0x01f7bd (0xe4e0 bytes) 000722 1787854331.787799 punch from 0x001b9d to 0x00a139 (0x859d bytes) 000723 1787854331.802395 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1f7be): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f7be): Mirror 5: 0xd1af171b (stale) Mirror 6: 0xd1af171b (stale) Mirror 7: 0xd1af171b (stale) Mirror 8: 0xd1af171b (stale) Mirror 9: 0xd1af171b (stale) Mirror 10: 0xd1af171b (stale) Mirror 11: 0xd1af171b (stale) Mirror 12: 0xd1af171b (stale) Mirror 13: 0xd1af171b (stale) Mirror 14: 0x5e27ae07 (stale) CRC-32 checksum value for chunk [0, 0x1f7be): Mirror 1: 0xb77115d0 Mirror 15: 0xb77115d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f7be: skipped 000724 1787854333.273867 read 0x00c036 thru 0x00dc8d (0x1c58 bytes) 000725 1787854333.282306 mapwrite 0x009f27 thru 0x018663 (0xe73d bytes) 000726 1787854333.443254 fallocate 0x02e7d0 thru 0x03471e (0x5f4f bytes) 000727 1787854333.451007 punch from 0x009623 to 0x00b8b9 (0x2297 bytes) 000728 1787854333.457035 read 0x00f5b8 thru 0x0116b7 (0x2100 bytes) 000729 1787854333.457980 trunc from 0x01f7be to 0x013c58 (0xbb65 bytes) 000730 1787854333.486347 read 0x00e799 thru 0x013c58 (0x54c0 bytes) 000731 1787854333.487519 mapwrite 0x0261b4 thru 0x0333b9 (0xd206 bytes) 000732 1787854333.537261 write 0x029139 thru 0x036eb8 (0xdd80 bytes) 000733 1787854333.543251 mapread 0x00d148 thru 0x00deb3 (0xd6c bytes) 000734 1787854333.544870 trunc from 0x025a08 to 0x02e096 (0x868f bytes) 000735 1787854333.569886 mapread 0x01e1be thru 0x023427 (0x526a bytes) 000736 1787854333.571509 trunc from 0x036eb9 to 0x011b29 (0x2538f bytes) 000737 1787854333.600126 trunc from 0x004488 to 0x0062a4 (0x1e1d bytes) 000738 1787854333.608681 fallocate 0x00b762 thru 0x00d931 (0x21d0 bytes) 000739 1787854333.617389 trunc from 0x00d7ab to 0x011b29 (0x437f bytes) 000740 1787854333.623530 mapwrite 0x02077e thru 0x028789 (0x800c bytes) 000741 1787854333.716560 write 0x02a828 thru 0x0302cc (0x5aa5 bytes) 000742 1787854333.719235 trunc from 0x0280f4 to 0x02bfdd (0x3eea bytes) 000743 1787854333.735801 mapwrite 0x00097c thru 0x00919a (0x881f bytes) 000744 1787854333.771752 fallocate 0x008d82 thru 0x00e791 (0x5a10 bytes) 000745 1787854333.776695 trunc from 0x0302cd to 0x00ddc9 (0x22503 bytes) 000746 1787854333.799721 trunc from 0x00ddca to 0x03c314 (0x2e54b bytes) 000747 1787854333.824870 MIRROR_RESYNC 12 mirrors 000748 1787854336.587555 mapwrite 0x00d776 thru 0x014ccf (0x755a bytes) 000749 1787854336.717387 punch from 0x004883 to 0x00850c (0x3c8a bytes) 000750 1787854336.727242 fallocate 0x0113a4 thru 0x017cff (0x695c bytes) 000751 1787854336.736249 mapread 0x003794 thru 0x006d44 (0x35b1 bytes) 000753 1787854336.762532 mapread 0x010ee8 thru 0x01cc60 (0xbd79 bytes) 000754 1787854336.775870 trunc from 0x03c315 to 0x01c39c (0x1ff78 bytes) 000755 1787854336.804631 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c39d): Mirror 5: 0xc5a695f9 (stale) Mirror 6: 0xc5a695f9 (stale) Mirror 7: 0xc5a695f9 (stale) Mirror 8: 0xc5a695f9 (stale) Mirror 9: 0xc5a695f9 (stale) Mirror 10: 0xc5a695f9 (stale) Mirror 11: 0xc5a695f9 (stale) Mirror 12: 0xc5a695f9 (stale) Mirror 13: 0xc5a695f9 (stale) Mirror 15: 0xc5a695f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c39d): Mirror 5: 0xc5a695f9 (stale) Mirror 6: 0xc5a695f9 (stale) Mirror 7: 0xc5a695f9 (stale) Mirror 8: 0xc5a695f9 (stale) Mirror 9: 0xc5a695f9 (stale) Mirror 10: 0xc5a695f9 (stale) Mirror 11: 0xc5a695f9 (stale) Mirror 12: 0xc5a695f9 (stale) Mirror 13: 0xc5a695f9 (stale) Mirror 15: 0xc5a695f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c39d: skipped 000756 1787854337.511851 trunc from 0x01c39d to 0x00478f (0x17c0d bytes) 000757 1787854337.557376 write 0x037692 thru 0x03fe62 (0x87d1 bytes) 000758 1787854337.558996 mapread 0x01c737 thru 0x023876 (0x7140 bytes) 000759 1787854337.583308 read 0x019436 thru 0x0283e0 (0xefab bytes) 000761 1787854337.593811 mapwrite 0x01ec2d thru 0x021ef2 (0x32c6 bytes) 000762 1787854337.646383 mapread 0x000ff3 thru 0x006064 (0x5072 bytes) 000764 1787854337.674259 punch from 0x02ebb9 to 0x0396eb (0xab33 bytes) 000765 1787854337.700743 read 0x026b48 thru 0x02b986 (0x4e3f bytes) 000766 1787854337.721793 mapwrite 0x013351 thru 0x019f96 (0x6c46 bytes) 000767 1787854337.749213 mapwrite 0x03970e thru 0x03ffff (0x68f2 bytes) 000768 1787854337.795736 mapread 0x029d98 thru 0x034e03 (0xb06c bytes) 000769 1787854337.798331 write 0x01d5ea thru 0x024815 (0x722c bytes) 000770 1787854337.800502 mapread 0x00b88c thru 0x00e920 (0x3095 bytes) 000771 1787854337.802146 read 0x010083 thru 0x010a2f (0x9ad bytes) 000773 1787854337.804880 fallocate 0x03ef46 thru 0x03ffff (0x10ba bytes) 000774 1787854337.819166 mapwrite 0x00f2d2 thru 0x010f88 (0x1cb7 bytes) 000775 1787854337.869692 fallocate 0x016aa5 thru 0x01b216 (0x4772 bytes) 000776 1787854337.888785 read 0x032fd4 thru 0x03a563 (0x7590 bytes) 000778 1787854337.890467 trunc from 0x040000 to 0x032e46 (0xd1b9 bytes) 000779 1787854337.930816 mapwrite 0x024f38 thru 0x02a43f (0x5508 bytes) 000780 1787854337.974670 read 0x020bce thru 0x023fd3 (0x3406 bytes) 000781 1787854337.975707 punch from 0x0262d9 to 0x02d338 (0x7060 bytes) 000782 1787854337.984851 punch from 0x011974 to 0x01f2c8 (0xd955 bytes) 000783 1787854337.991928 read 0x006b9e thru 0x009108 (0x256b bytes) 000784 1787854337.993043 mapread 0x02fed0 thru 0x032e46 (0x2f77 bytes) 000785 1787854337.996760 fallocate 0x01a1a6 thru 0x0286dd (0xe538 bytes) 000786 1787854338.003399 fallocate 0x01e216 thru 0x02600b (0x7df6 bytes) 000787 1787854338.015591 fallocate 0x024bad thru 0x02b746 (0x6b9a bytes) 000788 1787854338.024484 trunc from 0x032e47 to 0x03ecd1 (0xbe8b bytes) 000790 1787854338.044593 fallocate 0x02f67f thru 0x0385f3 (0x8f75 bytes) 000791 1787854338.051366 fallocate 0x027d8b thru 0x02cd2d (0x4fa3 bytes) 000792 1787854338.057559 mapread 0x00c107 thru 0x0158d8 (0x97d2 bytes) 000793 1787854338.073289 mapread 0x01f5a8 thru 0x02584d (0x62a6 bytes) 000794 1787854338.074958 mapread 0x03c328 thru 0x03ecd1 (0x29aa bytes) 000795 1787854338.076489 trunc from 0x02f6cd to 0x035acf (0x6403 bytes) 000796 1787854338.082509 mapwrite 0x02a9bf thru 0x0366cc (0xbd0e bytes) 000797 1787854338.155264 read 0x0133a1 thru 0x01c1d3 (0x8e33 bytes) 000799 1787854338.156789 punch from 0x01e7a0 to 0x021f5b (0x37bc bytes) 000800 1787854338.164213 trunc from 0x03ecd2 to 0x009ed1 (0x34e00 bytes) 000801 1787854338.183956 read 0x005fe5 thru 0x0075f9 (0x1615 bytes) 000802 1787854338.184624 mapread 0x000b6e thru 0x009ed1 (0x9364 bytes) 000803 1787854338.185560 trunc from 0x009ed2 to 0x025d43 (0x1be72 bytes) 000804 1787854338.205362 punch from 0x01ba8e to 0x022d96 (0x7309 bytes) 000806 1787854338.214506 read 0x01a316 thru 0x025d43 (0xba2e bytes) 000807 1787854338.223538 write 0x029b03 thru 0x02a23e (0x73c bytes) 000808 1787854338.229336 MIRROR_RESYNC 11 mirrors 000809 1787854340.465368 write 0x00043a thru 0x00d294 (0xce5b bytes) 000810 1787854340.517181 mapwrite 0x01fa9e thru 0x026022 (0x6585 bytes) 000811 1787854340.559404 read 0x00b7d2 thru 0x0102e1 (0x4b10 bytes) 000812 1787854340.565710 trunc from 0x016ada to 0x0224e7 (0xba0e bytes) 000814 1787854340.573458 trunc from 0x02a23f to 0x0078f6 (0x22948 bytes) 000816 1787854340.614887 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x78f7): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x78f7): Mirror 5: 0xb633f39d (stale) Mirror 6: 0xb633f39d (stale) Mirror 7: 0xb633f39d (stale) Mirror 8: 0xb633f39d (stale) Mirror 9: 0xb633f39d (stale) Mirror 10: 0xb633f39d (stale) Mirror 11: 0xb633f39d (stale) Mirror 12: 0xb633f39d (stale) Mirror 13: 0xb633f39d (stale) Mirror 15: 0xb633f39d (stale) CRC-32 checksum value for chunk [0, 0x78f7): Mirror 1: 0xab1bd8b1 Mirror 16: 0xab1bd8b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x78f7: skipped 000817 1787854341.865118 punch from 0x00423c to 0x0078f6 (0x36bb bytes) 000819 1787854341.979389 write 0x004f27 thru 0x00b611 (0x66eb bytes) 000820 1787854342.005011 read 0x006f90 thru 0x00b611 (0x4682 bytes) 000821 1787854342.005703 mapread 0x008ef2 thru 0x00b611 (0x2720 bytes) 000822 1787854342.009135 mapread 0x00042c thru 0x00b611 (0xb1e6 bytes) 000823 1787854342.033419 mapread 0x008000 thru 0x00b611 (0x3612 bytes) 000824 1787854342.047730 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb612): Mirror 5: 0x55669a83 (stale) Mirror 6: 0x55669a83 (stale) Mirror 7: 0x55669a83 (stale) Mirror 8: 0x55669a83 (stale) Mirror 9: 0x55669a83 (stale) Mirror 11: 0x55669a83 (stale) Mirror 12: 0x55669a83 (stale) Mirror 13: 0x55669a83 (stale) Mirror 15: 0x55669a83 (stale) Mirror 16: 0xf28d26aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb612): Mirror 5: 0x55669a83 (stale) Mirror 6: 0x55669a83 (stale) Mirror 7: 0x55669a83 (stale) Mirror 8: 0x55669a83 (stale) Mirror 9: 0x55669a83 (stale) Mirror 11: 0x55669a83 (stale) Mirror 12: 0x55669a83 (stale) Mirror 13: 0x55669a83 (stale) Mirror 15: 0x55669a83 (stale) Mirror 16: 0xf28d26aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb612: skipped 000825 1787854342.928718 trunc from 0x0018c5 to 0x008ef7 (0x7633 bytes) 000826 1787854342.935828 punch from 0x000b8d to 0x00747b (0x68ef bytes) 000827 1787854342.943478 trunc from 0x001a5c to 0x00b611 (0x9bb6 bytes) 000828 1787854342.950612 fallocate 0x013828 thru 0x01be38 (0x8611 bytes) 000829 1787854342.956158 mapwrite 0x0202b7 thru 0x02aee9 (0xac33 bytes) 000830 1787854343.062719 fallocate 0x01804a thru 0x026ac4 (0xea7b bytes) 000831 1787854343.070670 read 0x029e33 thru 0x02a4f5 (0x6c3 bytes) 000832 1787854343.072685 write 0x018ded thru 0x021b9d (0x8db1 bytes) 000833 1787854343.081370 mapread 0x01c770 thru 0x026e96 (0xa727 bytes) 000834 1787854343.082326 punch from 0x028954 to 0x02aee9 (0x2596 bytes) 000835 1787854343.091723 write 0x03ad7a thru 0x03ffff (0x5286 bytes) 000836 1787854343.094694 mapwrite 0x0202fb thru 0x02258f (0x2295 bytes) 000837 1787854343.135572 trunc from 0x040000 to 0x011fcf (0x2e030 bytes) 000839 1787854343.169675 read 0x0082b9 thru 0x00eb2c (0x6874 bytes) 000840 1787854343.189671 mapwrite 0x03597b thru 0x03cb02 (0x7188 bytes) 000842 1787854343.272163 write 0x000ece thru 0x00a745 (0x9878 bytes) 000843 1787854343.276325 punch from 0x0249a5 to 0x026a94 (0x20f0 bytes) 000844 1787854343.282576 punch from 0x024a02 to 0x02b9fc (0x6ffb bytes) 000845 1787854343.289099 trunc from 0x02ba21 to 0x02dafc (0x20dc bytes) 000846 1787854343.296577 trunc from 0x03cb03 to 0x02c585 (0x1057d bytes) 000847 1787854343.324387 write 0x0354de thru 0x03a6b7 (0x51da bytes) 000848 1787854343.336508 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a6b8): Mirror 5: 0x60b1715a (stale) Mirror 6: 0x60b1715a (stale) Mirror 7: 0x60b1715a (stale) Mirror 8: 0x60b1715a (stale) Mirror 9: 0x60b1715a (stale) Mirror 11: 0x60b1715a (stale) Mirror 12: 0x60b1715a (stale) Mirror 15: 0x60b1715a (stale) Mirror 16: 0xa72353e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a6b8): Mirror 5: 0x60b1715a (stale) Mirror 6: 0x60b1715a (stale) Mirror 7: 0x60b1715a (stale) Mirror 8: 0x60b1715a (stale) Mirror 9: 0x60b1715a (stale) Mirror 11: 0x60b1715a (stale) Mirror 12: 0x60b1715a (stale) Mirror 15: 0x60b1715a (stale) Mirror 16: 0xa72353e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a6b8: skipped 000849 1787854344.119770 write 0x031c3d thru 0x032afd (0xec1 bytes) 000850 1787854344.144323 mapread 0x01ea8e thru 0x023c63 (0x51d6 bytes) 000851 1787854344.175522 write 0x0050cd thru 0x010301 (0xb235 bytes) 000852 1787854344.188450 write 0x019cdb thru 0x027788 (0xdaae bytes) 000853 1787854344.191221 read 0x018eca thru 0x019187 (0x2be bytes) 000854 1787854344.196759 punch from 0x00a1ba to 0x01835a (0xe1a1 bytes) 000855 1787854344.226562 punch from 0x02f4d5 to 0x037887 (0x83b3 bytes) 000856 1787854344.233239 mapwrite 0x01bf65 thru 0x027517 (0xb5b3 bytes) 000857 1787854344.257166 trunc from 0x03a6b8 to 0x01e07f (0x1c638 bytes) 000858 1787854344.281846 trunc from 0x01a443 to 0x01e07f (0x3c3d bytes) 000859 1787854344.289828 punch from 0x014517 to 0x0188ab (0x4395 bytes) 000860 1787854344.296049 mapread 0x002b32 thru 0x00dbb4 (0xb083 bytes) 000861 1787854344.306469 punch from 0x01dcae to 0x01e07f (0x3d2 bytes) 000862 1787854344.311425 write 0x03bdb9 thru 0x03c83a (0xa82 bytes) 000863 1787854344.312595 fallocate 0x005162 thru 0x010f5a (0xbdf9 bytes) 000864 1787854344.321356 read 0x038626 thru 0x03c83a (0x4215 bytes) 000865 1787854344.339805 read 0x023ee9 thru 0x032136 (0xe24e bytes) 000866 1787854344.340605 fallocate 0x01b550 thru 0x0266e5 (0xb196 bytes) 000867 1787854344.345985 mapwrite 0x02acbe thru 0x02e131 (0x3474 bytes) 000868 1787854344.365295 trunc from 0x03c83b to 0x0148d2 (0x27f68 bytes) 000870 1787854344.384648 read 0x01197c thru 0x012f34 (0x15b9 bytes) 000871 1787854344.386181 trunc from 0x00efe1 to 0x0148d2 (0x58f2 bytes) 000872 1787854344.391619 write 0x02a7ef thru 0x036be1 (0xc3f3 bytes) 000873 1787854344.395039 punch from 0x011eaa to 0x01b4a9 (0x9600 bytes) 000874 1787854344.400218 trunc from 0x036be2 to 0x0088c1 (0x2e320 bytes) 000875 1787854344.420505 trunc from 0x005712 to 0x0088c1 (0x31b0 bytes) 000876 1787854344.426273 trunc from 0x003557 to 0x006eee (0x3998 bytes) 000877 1787854344.437801 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x88c2): Mirror 5: 0xe7d661b3 (stale) Mirror 6: 0xe7d661b3 (stale) Mirror 7: 0xe7d661b3 (stale) Mirror 8: 0xe7d661b3 (stale) Mirror 11: 0xe7d661b3 (stale) Mirror 12: 0xe7d661b3 (stale) Mirror 15: 0xe7d661b3 (stale) Mirror 16: 0x311263e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x88c2): Mirror 5: 0xe7d661b3 (stale) Mirror 6: 0xe7d661b3 (stale) Mirror 7: 0xe7d661b3 (stale) Mirror 8: 0xe7d661b3 (stale) Mirror 11: 0xe7d661b3 (stale) Mirror 12: 0xe7d661b3 (stale) Mirror 15: 0xe7d661b3 (stale) Mirror 16: 0x311263e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x88c2: skipped 000878 1787854345.096206 trunc from 0x0007ed to 0x00148d (0xca1 bytes) 000880 1787854345.109793 mapread 0x0036bd thru 0x0088c1 (0x5205 bytes) 000881 1787854345.171162 read 0x002045 thru 0x0088c1 (0x687d bytes) 000882 1787854345.173676 mapwrite 0x0239fb thru 0x02fc18 (0xc21e bytes) 000884 1787854345.265215 write 0x0377eb thru 0x03d910 (0x6126 bytes) 000885 1787854345.270466 punch from 0x033b54 to 0x036fc4 (0x3471 bytes) 000886 1787854345.281944 write 0x03e352 thru 0x03ffff (0x1cae bytes) 000887 1787854345.284999 trunc from 0x040000 to 0x01101a (0x2efe5 bytes) 000888 1787854345.319547 write 0x027359 thru 0x0291f0 (0x1e98 bytes) 000889 1787854345.322523 mapwrite 0x0006ea thru 0x000afa (0x411 bytes) 000890 1787854345.353287 punch from 0x022c4c to 0x02832b (0x56e0 bytes) 000891 1787854345.360700 write 0x023588 thru 0x030a23 (0xd49c bytes) 000892 1787854345.370802 mapread 0x001b40 thru 0x008490 (0x6951 bytes) 000893 1787854345.371780 fallocate 0x0379ca thru 0x03ffff (0x8636 bytes) 000894 1787854345.377595 read 0x02f5d0 thru 0x030a23 (0x1454 bytes) 000895 1787854345.378211 fallocate 0x0059a2 thru 0x01034b (0xa9aa bytes) 000896 1787854345.384392 write 0x00bbe1 thru 0x014f44 (0x9364 bytes) 000897 1787854345.394810 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30a24): Mirror 5: 0xa2e1eed4 (stale) Mirror 6: 0xa2e1eed4 (stale) Mirror 7: 0xa2e1eed4 (stale) Mirror 8: 0xa2e1eed4 (stale) Mirror 12: 0xa2e1eed4 (stale) Mirror 15: 0xa2e1eed4 (stale) Mirror 16: 0xeb10ce04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a24): Mirror 5: 0xa2e1eed4 (stale) Mirror 6: 0xa2e1eed4 (stale) Mirror 7: 0xa2e1eed4 (stale) Mirror 8: 0xa2e1eed4 (stale) Mirror 12: 0xa2e1eed4 (stale) Mirror 15: 0xa2e1eed4 (stale) Mirror 16: 0xeb10ce04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a24: skipped 000898 1787854345.974378 mapwrite 0x010c2a thru 0x01ca77 (0xbe4e bytes) 000899 1787854346.046319 read 0x02c127 thru 0x030a23 (0x48fd bytes) 000900 1787854346.056288 trunc from 0x009513 to 0x01569b (0xc189 bytes) 000902 1787854346.063185 read 0x0175a0 thru 0x02455f (0xcfc0 bytes) 000903 1787854346.078204 write 0x023992 thru 0x02dc7b (0xa2ea bytes) 000904 1787854346.093161 MIRROR_RESYNC 8 mirrors 000905 1787854348.859391 mapread 0x00939d thru 0x00ac76 (0x18da bytes) 000907 1787854348.872325 mapwrite 0x029a71 thru 0x037753 (0xdce3 bytes) 000908 1787854349.027652 trunc from 0x02b480 to 0x034602 (0x9183 bytes) 000909 1787854349.040188 trunc from 0x0219f9 to 0x0298dd (0x7ee5 bytes) 000910 1787854349.047606 fallocate 0x00823a thru 0x00d9b4 (0x577b bytes) 000911 1787854349.054969 trunc from 0x037754 to 0x004eef (0x32864 bytes) 000912 1787854349.088421 mapread 0x00225e thru 0x004eef (0x2c92 bytes) 000913 1787854349.097268 fallocate 0x001173 thru 0x00de88 (0xcd16 bytes) 000914 1787854349.109746 punch from 0x0039c2 to 0x004468 (0xaa7 bytes) 000915 1787854349.119233 punch from 0x00108a to 0x004eef (0x3e66 bytes) 000916 1787854349.130402 mapread 0x000881 thru 0x004eef (0x466f bytes) 000917 1787854349.160154 read 0x000ee2 thru 0x0034da (0x25f9 bytes) 000918 1787854349.161705 mapwrite 0x0388ae thru 0x03e8bd (0x6010 bytes) 000919 1787854349.277743 mapread 0x0033e5 thru 0x00fcb3 (0xc8cf bytes) 000920 1787854349.301235 write 0x02c2b6 thru 0x02dc29 (0x1974 bytes) 000921 1787854349.325585 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e8be): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e8be): Mirror 5: 0xfc9dce9c (stale) Mirror 6: 0xfc9dce9c (stale) Mirror 7: 0xfc9dce9c (stale) Mirror 8: 0xfc9dce9c (stale) Mirror 12: 0xfc9dce9c (stale) Mirror 15: 0xfc9dce9c (stale) Mirror 16: 0xfc9dce9c (stale) CRC-32 checksum value for chunk [0, 0x3e8be): Mirror 1: 0xddc3774c Mirror 17: 0xddc3774c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e8be: skipped 000922 1787854350.511527 trunc from 0x024659 to 0x029f46 (0x58ee bytes) 000923 1787854350.564458 punch from 0x02ebdb to 0x038ac5 (0x9eeb bytes) 000924 1787854350.576049 read 0x0127e7 thru 0x018d57 (0x6571 bytes) 000925 1787854350.587695 trunc from 0x03e8be to 0x009a80 (0x34e3d bytes) 000926 1787854350.639554 trunc from 0x006768 to 0x0081cd (0x1a66 bytes) 000927 1787854350.649650 write 0x021205 thru 0x025378 (0x4174 bytes) 000928 1787854350.651706 write 0x03d49c thru 0x03ffff (0x2b64 bytes) 000929 1787854350.665033 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x782bdc24 (stale) Mirror 6: 0x782bdc24 (stale) Mirror 7: 0x782bdc24 (stale) Mirror 8: 0x782bdc24 (stale) Mirror 12: 0x782bdc24 (stale) Mirror 15: 0x782bdc24 (stale) Mirror 16: 0x782bdc24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x782bdc24 (stale) Mirror 6: 0x782bdc24 (stale) Mirror 7: 0x782bdc24 (stale) Mirror 8: 0x782bdc24 (stale) Mirror 12: 0x782bdc24 (stale) Mirror 15: 0x782bdc24 (stale) Mirror 16: 0x782bdc24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000930 1787854351.314258 MIRROR_RESYNC 8 mirrors 000931 1787854353.717225 trunc from 0x03f552 to 0x03ffff (0xaae bytes) 000932 1787854353.789828 read 0x02ecac thru 0x039af1 (0xae46 bytes) 000933 1787854353.803570 punch from 0x023a6e to 0x030ccb (0xd25e bytes) 000934 1787854353.817194 mapread 0x0224fe thru 0x027415 (0x4f18 bytes) 000935 1787854353.852675 read 0x02aeea thru 0x02c65f (0x1776 bytes) 000936 1787854353.858807 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa66316e8 (stale) Mirror 6: 0xa66316e8 (stale) Mirror 7: 0xa66316e8 (stale) Mirror 12: 0xa66316e8 (stale) Mirror 15: 0xa66316e8 (stale) Mirror 16: 0xa66316e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa66316e8 (stale) Mirror 6: 0xa66316e8 (stale) Mirror 7: 0xa66316e8 (stale) Mirror 12: 0xa66316e8 (stale) Mirror 15: 0xa66316e8 (stale) Mirror 16: 0xa66316e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000937 1787854354.526396 trunc from 0x028575 to 0x02e88c (0x6318 bytes) 000938 1787854354.536838 trunc from 0x0357a6 to 0x03b5fb (0x5e56 bytes) 000939 1787854354.543995 read 0x02dcb1 thru 0x03824a (0xa59a bytes) 000940 1787854354.557472 mapwrite 0x01fe3a thru 0x02b94f (0xbb16 bytes) 000941 1787854354.625983 punch from 0x00fb11 to 0x01647e (0x696e bytes) 000942 1787854354.632745 fallocate 0x00a4fd thru 0x00eebb (0x49bf bytes) 000943 1787854354.639634 punch from 0x00d5a1 to 0x014a47 (0x74a7 bytes) 000944 1787854354.645190 trunc from 0x040000 to 0x003d12 (0x3c2ed bytes) 000946 1787854354.678758 punch from 0x0002e4 to 0x003d12 (0x3a2f bytes) 000947 1787854354.686931 write 0x02ad00 thru 0x02b348 (0x649 bytes) 000948 1787854354.690857 mapwrite 0x00dc20 thru 0x0162cb (0x86ac bytes) 000949 1787854354.729039 read 0x029659 thru 0x02b348 (0x1cf0 bytes) 000950 1787854354.746912 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b349): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b349): Mirror 5: 0x7d7f1c01 (stale) Mirror 6: 0x7d7f1c01 (stale) Mirror 7: 0x7d7f1c01 (stale) Mirror 12: 0x7d7f1c01 (stale) Mirror 15: 0x7d7f1c01 (stale) Mirror 16: 0x7d7f1c01 (stale) CRC-32 checksum value for chunk [0, 0x2b349): Mirror 1: 0x6cbe238e Mirror 17: 0x6cbe238e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b349: skipped 000951 1787854356.282529 write 0x0096e5 thru 0x00f172 (0x5a8e bytes) 000952 1787854356.367910 mapread 0x022e92 thru 0x025299 (0x2408 bytes) 000954 1787854356.417984 mapread 0x02914e thru 0x02b348 (0x21fb bytes) 000955 1787854356.421844 punch from 0x008a57 to 0x009bd6 (0x1180 bytes) 000956 1787854356.442055 punch from 0x025f77 to 0x028ff9 (0x3083 bytes) 000957 1787854356.451823 trunc from 0x00e0a9 to 0x01561e (0x7576 bytes) 000958 1787854356.458835 read 0x004bb6 thru 0x00d18f (0x85da bytes) 000959 1787854356.479239 read 0x01a0b2 thru 0x021ed7 (0x7e26 bytes) 000960 1787854356.491362 mapread 0x01ae12 thru 0x024236 (0x9425 bytes) 000961 1787854356.504963 MIRROR_RESYNC 8 mirrors 000962 1787854359.102526 mapwrite 0x01d1a5 thru 0x02ce4c (0xfca8 bytes) 000963 1787854359.268701 trunc from 0x02ce4d to 0x011726 (0x1b726 bytes) 000964 1787854359.344328 fallocate 0x016771 thru 0x01b702 (0x4f92 bytes) 000966 1787854359.354433 write 0x03cc8a thru 0x03f399 (0x2710 bytes) 000967 1787854359.360189 fallocate 0x01fcbe thru 0x027c5d (0x7fa0 bytes) 000968 1787854359.366352 mapwrite 0x001520 thru 0x00b0d4 (0x9bb5 bytes) 000969 1787854359.422469 write 0x02e7da thru 0x03980d (0xb034 bytes) 000970 1787854359.441984 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f39a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f39a): Mirror 5: 0xef98ec13 (stale) Mirror 6: 0xef98ec13 (stale) Mirror 7: 0xef98ec13 (stale) Mirror 12: 0xef98ec13 (stale) Mirror 15: 0xef98ec13 (stale) Mirror 16: 0xef98ec13 (stale) Mirror 17: 0xef98ec13 (stale) CRC-32 checksum value for chunk [0, 0x3f39a): Mirror 1: 0xddec23ff Mirror 18: 0xddec23ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f39a: skipped 000971 1787854360.592491 mapwrite 0x0078f4 thru 0x0131ac (0xb8b9 bytes) 000972 1787854360.748026 punch from 0x029ff1 to 0x0353e9 (0xb3f9 bytes) 000973 1787854360.755390 write 0x0122f9 thru 0x0215a2 (0xf2aa bytes) 000974 1787854360.777041 fallocate 0x008257 thru 0x00c6d5 (0x447f bytes) 000975 1787854360.796999 trunc from 0x03f39a to 0x00691f (0x38a7a bytes) 000976 1787854360.898670 fallocate 0x02ccc1 thru 0x02f421 (0x2761 bytes) 000979 1787854360.920963 MIRROR_RESYNC 9 mirrors 000980 1787854362.173249 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) 000981 1787854362.658453 punch from 0x006407 to 0x00691f (0x519 bytes) 000982 1787854362.718053 trunc from 0x006920 to 0x025a46 (0x1f127 bytes) 000983 1787854362.796220 write 0x01b654 thru 0x02828b (0xcc38 bytes) 000984 1787854362.804782 mapwrite 0x01fd63 thru 0x026e17 (0x70b5 bytes) 000985 1787854362.838950 mapread 0x024ae0 thru 0x02828b (0x37ac bytes) 000986 1787854362.852130 MIRROR_RESYNC 8 mirrors 000987 1787854364.677986 write 0x014756 thru 0x016add (0x2388 bytes) 000990 1787854364.756936 trunc from 0x02828c to 0x028330 (0xa5 bytes) 000992 1787854364.793166 trunc from 0x01f540 to 0x026a1b (0x74dc bytes) 000993 1787854364.799049 read 0x00a23e thru 0x012a2a (0x87ed bytes) 000994 1787854364.827243 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x17b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28331): Mirror 5: 0x45867936 (stale) Mirror 7: 0x45867936 (stale) Mirror 12: 0x45867936 (stale) Mirror 15: 0x45867936 (stale) Mirror 17: 0x45867936 (stale) Mirror 18: 0x45867936 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28331): Mirror 5: 0x45867936 (stale) Mirror 7: 0x45867936 (stale) Mirror 12: 0x45867936 (stale) Mirror 15: 0x45867936 (stale) Mirror 17: 0x45867936 (stale) Mirror 18: 0x45867936 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28331: skipped 000995 1787854365.428033 trunc from 0x010697 to 0x01922d (0x8b97 bytes) 000996 1787854365.433885 fallocate 0x005e05 thru 0x0099e0 (0x3bdc bytes) 000997 1787854365.442838 trunc from 0x028331 to 0x03c1af (0x13e7f bytes) 000998 1787854365.485049 trunc from 0x03c1b0 to 0x00a834 (0x3197b bytes) 000999 1787854365.503026 read 0x00460c thru 0x0069cf (0x23c4 bytes) 001000 1787854365.509765 trunc from 0x00a835 to 0x017818 (0xcfe4 bytes) All operations completed A-OK! PASS 70a (108s) == sanity-flr test 71: check mirror extend parent fid ==== 14:12:50 (1787854370) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1b5:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 14:13:01 (1787854381) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2031 s, 10.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859815 s, 12.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 14:13:09 (1787854389) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3400352 .. Extending file size to 7198272 .. Extending file size to 6044864 .. Extending file size to 8638656 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7627232 .. Extending file size to 7699872 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8038496 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8068832 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62569 62571 62572 62574 62576 done 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) 192.168.204.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:14:31 (1787854471) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6987584 .. Extending file size to 4861056 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6625600 .. Extending file size to 8708928 .. Extending file size to 8572864 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9078400 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9276608 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 64033 64034 64037 64039 64040 done 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) 192.168.204.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:15:52 (1787854552) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1268984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18960 3588060 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24812 3582208 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43772 7170268 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1268984 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18960 3588060 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24812 3582208 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43772 7170268 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104185 s, 20.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.262288 s, 8.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x73f733da (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:16:15 (1787854575) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174442 s, 12.0 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 14:16:24 (1787854584) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:16:34 (1787854594) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101574 s, 20.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x212:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x211:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x212:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:16:42 (1787854602) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104196 s, 20.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:16:51 (1787854611) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132589 s, 15.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x213:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x213:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x213:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x213:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x214:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:17:25 (1787854645) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg422-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615878 s, 17.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x214:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x215:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x214:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x215:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x215:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:18:18 (1787854698) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg422-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223208 s, 9.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x217:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x217:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:19:13 (1787854753) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg422-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg422-server: Pool lustre.test_204f created Adding targets to pool oleg422-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg422-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0985223 s, 21.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg422-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg422-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg422-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:20:28 (1787854828) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:20:37 (1787854837) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:20:46 (1787854846) oleg422-server: Pool lustre.test_206 created Adding targets to pool oleg422-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg422-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg422-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg422-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg422-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:21:15 (1787854875) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744816 s, 14.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411665 s, 25.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc1054ebb Mirror 3: 0xc1054ebb PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:21:26 (1787854886) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:21:30 (1787854890) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:21:34 (1787854894) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0731707 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x17c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x21d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x21d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x21d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1f90000 Read this (4 bytes): BBBB 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:22:14 (1787854934) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00945003 s, 867 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:22:21 (1787854941) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876285 s, 12.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x142:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 14:22:23.000000000 -0400 Modify: 2026-08-27 14:22:23.000000000 -0400 Change: 2026-08-27 14:22:25.000000000 -0400 Birth: 2026-08-27 14:22:23.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:22:30 (1787854950) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051096 s, 20.5 MB/s oleg422-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg422-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg422-server: O/280000401/d0/140: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:22:54 (1787854974) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.227613 s, 180 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0250477 s, 164 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 14:22 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x85:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 14:22 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:23:02 (1787854982) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:23:10 (1787854990) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1969 sec =========== 14:23:19 (1787854999) === sanity-flr: start cleanup 14:23:21 (1787855001) === === sanity-flr: finish cleanup 14:23:29 (1787855009) ===