-----============= acceptance-small: sanity-flr ============----- Tue May 19 10:48:27 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg335-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:48:50 (1779202130) === oleg335-client.virtnet: executing check_config_client /mnt/lustre oleg335-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg335-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c80c9f5f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c80c9f5f800.idle_timeout=debug disable quota as required oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:49:27 (1779202167) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7615] == sanity-flr test 0a: lfs mirror create with -N option == 10:49:30 (1779202170) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:49:50 (1779202190) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:49:54 (1779202194) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:49:57 (1779202197) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:50:24 (1779202224) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:50:27 (1779202227) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 10:50:31 (1779202231) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c80c9f5f800.stats=clear osc.lustre-OST0001-osc-ffff9c80c9f5f800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:50:39 (1779202239) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:50:52 (1779202252) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:51:00 (1779202260) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:51:03 (1779202263) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:51:12 (1779202272) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 10:51:23 (1779202283) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:51:33 (1779202293) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:51:37 (1779202297) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:51:46 (1779202306) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:51:58 (1779202318) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:52:05 (1779202325) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.57879 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21972 s, 4.8 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:52:21 (1779202341) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.841612 s, 24.9 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:52:33 (1779202353) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:52:44 (1779202364) Stopping /mnt/lustre-ost1 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c80c9f5f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c80c9f5f800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg335-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c80c9f5f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c80c9f5f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg335-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (66s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:53:50 (1779202430) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.352945 s, 17.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c80c9f5f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c80c9f5f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg335-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:54:33 (1779202473) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c80c9f5f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c80c9f5f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg335-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c80c9f5f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c80c9f5f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg335-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:55:50 (1779202550) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.15695 s, 10.2 MB/s -rw-r--r-- 1 root root 52428800 May 19 10:55 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.992s user 0m0.002s sys 0m0.150s Stopping /mnt/lustre-ost1 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c80c9f5f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c80c9f5f800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m1.481s user 0m0.001s sys 0m0.192s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg335-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg335-server oleg335-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c80c9f5f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c80c9f5f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.313s user 0m0.000s sys 0m0.202s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg335-server: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg335-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c80c9f5f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c80c9f5f800.ost_server_uuid in FULL state after 0 sec PASS 33b (87s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:57:17 (1779202637) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:57:21 (1779202641) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:57:24 (1779202644) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 10:57:28 (1779202648) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 10:57:37 (1779202657) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.54222 s, 23.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:57:49 (1779202669) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473619 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251845 s, 41.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:57:57 (1779202677) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:58:05 (1779202685) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000813453 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00368667 s, 278 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00082954 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011149 s, 918 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 10:58:15 (1779202695) create test files with size 15741597 .. 6898a3891ce8c3f415fa18064edda230 - 9008d3db645f2cb9b8383b37c1386777 - f1df0385792686ece0c6b5d8954e9399 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg335-client.virtnet /mnt/lustre (opts:) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 10:58:37 (1779202717) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:58:50 (1779202730) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:58:57 (1779202737) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168137 s, 6.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436667 s, 24.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 10:59:06 (1779202746) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152396 s, 13.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.42166 s, 9.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227891 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 10:59:24 (1779202764) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 10:59:28 (1779202768) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:59:32 (1779202772) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314005 s, 33.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 10:59:39 (1779202779) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:59:42 (1779202782) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:59:51 (1779202791) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.320052 s, 32.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg335-client.virtnet /mnt/lustre (opts:) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:00:03 (1779202803) llite.lustre-ffff9c80c3519800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.288567 s, 36.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg335-client.virtnet /mnt/lustre (opts:) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c80c8f7e800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:00:16 (1779202816) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:00:54 (1779202854) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:01:31 (1779202891) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 11:01:34 (1779202894) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.287817 s, 10.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.16464 s, 19.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x2b3deb02 (nosync,stale) Mirror 2: 0x2b3deb02 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7e32ef9a634a7671c571ca97a201ad19 - mirror 1 checksum: 1366d8bc14b228fc8c2c4161aadee9b0 - mirror 2 checksum: 7e32ef9a634a7671c571ca97a201ad19 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue May 19 11:01:39 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue May 19 11:01:39 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7e32ef9a634a7671c571ca97a201ad19 - mirror 1 checksum: 1366d8bc14b228fc8c2c4161aadee9b0 - mirror 2 checksum: 1366d8bc14b228fc8c2c4161aadee9b0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue May 19 11:01:39 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue May 19 11:01:39 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:01:48 (1779202908) Failing mds1 on oleg335-server Stopping /mnt/lustre-mds1 (opts:) on oleg335-server 11:02:02 (1779202922) shut down facet: mds1 facet_host: oleg335-server facet_failover_host: oleg335-server Failover mds1 to oleg335-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 11:02:37 (1779202957) targets are mounted 11:02:37 (1779202957) facet_failover done oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322044 s, 32.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 50A (65s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:02:53 (1779202973) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:02:57 (1779202977) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:03:00 (1779202980) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00662665 s, 2.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:03:09 (1779202989) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:03:13 (1779202993) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:03:17 (1779202997) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:03:44 (1779203024) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:04:13 (1779203053) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:04:38 (1779203078) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:05:03 (1779203103) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 11:05:13 (1779203113) mirror create pid 53592 mirror split pid 53595 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 53592 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 53595 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 11:06:21 (1779203181) Seed set to 3183 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1779203183.825933 MIRROR_EXTEND to 2 mirrors 000002 1779203184.504325 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000006 1779203184.787728 write 0x0147e2 thru 0x018fff (0x481e bytes) 000007 1779203184.789341 read 0x00084e thru 0x00bb65 (0xb318 bytes) 000010 1779203184.826565 read 0x002331 thru 0x00dfdc (0xbcac bytes) 000011 1779203184.855056 mapwrite 0x0208a0 thru 0x02b8eb (0xb04c bytes) 000013 1779203185.034381 mapwrite 0x03cdd3 thru 0x03ffff (0x322d bytes) truncating to largest ever: 0x29946 000022 1779203185.205114 trunc from 0x040000 to 0x029945 (0x166ba bytes) 000023 1779203185.247033 mapread 0x0157e2 thru 0x0211c6 (0xb9e5 bytes) 000024 1779203185.248894 write 0x0078af thru 0x00d157 (0x58a9 bytes) 000027 1779203185.251712 mapwrite 0x0126ad thru 0x0138dc (0x1230 bytes) 000028 1779203185.351496 MIRROR_EXTEND to 2 mirrors 000029 1779203186.133608 write 0x03e34a thru 0x03ffff (0x1cb6 bytes) 000031 1779203186.246209 mapwrite 0x032a5d thru 0x036ad9 (0x407d bytes) 000032 1779203186.484819 mapread 0x039405 thru 0x03ffff (0x6bfb bytes) 000035 1779203186.521199 write 0x008833 thru 0x0127f7 (0x9fc5 bytes) 000038 1779203186.578229 mapread 0x02e4d1 thru 0x03803d (0x9b6d bytes) 000039 1779203186.587349 mapwrite 0x01b7cb thru 0x02299e (0x71d4 bytes) 000040 1779203186.705098 mapwrite 0x034b63 thru 0x03ee45 (0xa2e3 bytes) 000042 1779203186.823924 MIRROR_RESYNC 2 mirrors 000045 1779203187.941303 mapwrite 0x027d60 thru 0x033bb6 (0xbe57 bytes) 000046 1779203188.277005 mapwrite 0x008351 thru 0x00d377 (0x5027 bytes) 000047 1779203188.362190 mapread 0x039bc8 thru 0x03ffff (0x6438 bytes) 000048 1779203188.405883 write 0x009680 thru 0x01135a (0x7cdb bytes) truncating to largest ever: 0x3ef84 000051 1779203188.422380 trunc from 0x040000 to 0x03ef83 (0x107c bytes) 000052 1779203188.478573 read 0x01523a thru 0x0175c3 (0x238a bytes) 000054 1779203188.520034 read 0x019a46 thru 0x022116 (0x86d1 bytes) 000056 1779203188.537682 mapwrite 0x03eacc thru 0x03ffff (0x1534 bytes) 000059 1779203188.679506 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000060 1779203188.846589 mapwrite 0x0300f7 thru 0x03bdad (0xbcb7 bytes) 000061 1779203189.022470 read 0x035fe6 thru 0x03f398 (0x93b3 bytes) 000062 1779203189.029772 read 0x0068f2 thru 0x007f75 (0x1684 bytes) 000063 1779203189.059047 read 0x0122a9 thru 0x021f4b (0xfca3 bytes) 000064 1779203189.060021 mapwrite 0x0381da thru 0x03ffff (0x7e26 bytes) 000067 1779203189.165648 write 0x02c197 thru 0x033648 (0x74b2 bytes) 000068 1779203189.167587 mapread 0x003bfe thru 0x00ff82 (0xc385 bytes) 000069 1779203189.171944 trunc from 0x040000 to 0x0344a3 (0xbb5c bytes) 000070 1779203189.206964 trunc from 0x0344a4 to 0x01d7fb (0x16ca8 bytes) 000072 1779203189.230212 mapread 0x016988 thru 0x01d7fb (0x6e74 bytes) truncating to largest ever: 0x3fde8 000076 1779203189.236780 trunc from 0x01d7fc to 0x03fde7 (0x225ec bytes) 000080 1779203189.255897 read 0x001ba3 thru 0x00415a (0x25b8 bytes) 000082 1779203189.258467 mapwrite 0x00986f thru 0x009b7d (0x30f bytes) 000083 1779203189.331471 MIRROR_EXTEND to 2 mirrors 000084 1779203190.258805 mapwrite 0x03a95c thru 0x03ffff (0x56a4 bytes) 000085 1779203190.502012 read 0x033450 thru 0x03ffff (0xcbb0 bytes) 000086 1779203190.511039 trunc from 0x040000 to 0x032f9e (0xd061 bytes) 000088 1779203190.540503 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32f9f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f9f): Mirror 2: 0x46876c90 (stale) CRC-32 checksum value for chunk [0, 0x32f9f): Mirror 1: 0x46876c90 Mirror 3: 0x46876c90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f9f: skipped 000091 1779203191.547885 mapwrite 0x021abc thru 0x02714a (0x568f bytes) 000092 1779203191.755052 trunc from 0x032f9f to 0x0196ac (0x198f2 bytes) 000093 1779203191.841753 read 0x017f7f thru 0x0196ac (0x172e bytes) 000095 1779203191.866426 mapread 0x005f6f thru 0x0071d6 (0x1268 bytes) 000096 1779203191.914991 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x196ad): Mirror 2: 0xe078d92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x196ad): Mirror 2: 0xe078d92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x196ad: skipped 000097 1779203192.340928 mapwrite 0x037cc8 thru 0x03ffff (0x8338 bytes) 000098 1779203192.484245 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe9de37a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc0f16844 Mirror 3: 0xc0f16844 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000099 1779203193.703620 trunc from 0x040000 to 0x01baf4 (0x2450b bytes) 000100 1779203193.872020 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1baf5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1baf5): Mirror 2: 0xfdf69f4 (stale) Mirror 3: 0xa985c059 (stale) CRC-32 checksum value for chunk [0, 0x1baf5): Mirror 1: 0xa985c059 Mirror 4: 0xa985c059 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1baf5: skipped 000101 1779203195.022964 mapwrite 0x031aab thru 0x03bf6f (0xa4c5 bytes) 000105 1779203195.219982 trunc from 0x03bf70 to 0x032f91 (0x8fde bytes) 000106 1779203195.256006 mapread 0x00e4b1 thru 0x01cfe0 (0xeb30 bytes) 000107 1779203195.281052 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f92): Mirror 3: 0xfc29adc1 (stale) Mirror 4: 0xfc29adc1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f92): Mirror 3: 0xfc29adc1 (stale) Mirror 4: 0xfc29adc1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f92: skipped 000108 1779203195.733571 read 0x007e0f thru 0x008957 (0xb49 bytes) 000109 1779203195.743448 mapread 0x02461a thru 0x032f91 (0xe978 bytes) 000111 1779203195.773639 mapread 0x00fb94 thru 0x01b067 (0xb4d4 bytes) 000112 1779203195.792855 read 0x01e9b4 thru 0x02e555 (0xfba2 bytes) 000113 1779203195.794068 write 0x0154da thru 0x0191fe (0x3d25 bytes) 000116 1779203195.796975 read 0x020640 thru 0x022022 (0x19e3 bytes) 000117 1779203195.798024 read 0x01f415 thru 0x025e15 (0x6a01 bytes) 000118 1779203195.799368 trunc from 0x032f92 to 0x0223b7 (0x10bda bytes) 000120 1779203195.814799 read 0x0186ba thru 0x01eadc (0x6423 bytes) 000121 1779203195.815722 trunc from 0x0223b8 to 0x005260 (0x1d157 bytes) 000122 1779203195.838158 read 0x000e5d thru 0x005260 (0x4404 bytes) 000123 1779203195.849305 write 0x03d02f thru 0x03ff81 (0x2f53 bytes) 000124 1779203195.851838 read 0x037e88 thru 0x03ff81 (0x80fa bytes) 000125 1779203195.860392 mapread 0x0021fe thru 0x00ea20 (0xc823 bytes) 000126 1779203195.862633 trunc from 0x03ff82 to 0x015e73 (0x2a10e bytes) 000128 1779203195.883368 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x15e74): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15e74): Mirror 3: 0x439354c5 (stale) Mirror 4: 0x439354c5 (stale) CRC-32 checksum value for chunk [0, 0x15e74): Mirror 1: 0x91b34f6c Mirror 5: 0x91b34f6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15e74: skipped 000129 1779203196.955467 read 0x00eab6 thru 0x015e73 (0x73be bytes) 000130 1779203196.984034 mapread 0x00c929 thru 0x012ba0 (0x6278 bytes) 000131 1779203197.061259 read 0x0134dc thru 0x014177 (0xc9c bytes) 000132 1779203197.073749 mapwrite 0x032133 thru 0x033bb6 (0x1a84 bytes) 000133 1779203197.410952 write 0x01243a thru 0x01d483 (0xb04a bytes) 000134 1779203197.413176 write 0x00f936 thru 0x01b6c7 (0xbd92 bytes) 000135 1779203197.416508 write 0x01a698 thru 0x02a12f (0xfa98 bytes) 000136 1779203197.419097 mapwrite 0x01d7c4 thru 0x029ecb (0xc708 bytes) 000138 1779203197.510085 mapread 0x0327d7 thru 0x033bb6 (0x13e0 bytes) 000142 1779203197.514236 read 0x00c4ab thru 0x010136 (0x3c8c bytes) 000143 1779203197.520248 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33bb7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33bb7): Mirror 3: 0x19405c8e (stale) Mirror 4: 0x19405c8e (stale) Mirror 5: 0x8679d4b2 (stale) CRC-32 checksum value for chunk [0, 0x33bb7): Mirror 1: 0xaccb0eda Mirror 6: 0xaccb0eda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33bb7: skipped 000144 1779203198.670621 mapread 0x01bbe2 thru 0x01efd2 (0x33f1 bytes) 000147 1779203198.760137 mapwrite 0x035dfb thru 0x03ffff (0xa205 bytes) 000148 1779203199.024962 mapwrite 0x039bd6 thru 0x03a6ee (0xb19 bytes) 000150 1779203199.113934 mapread 0x02a7c4 thru 0x02ace0 (0x51d bytes) 000154 1779203199.149830 trunc from 0x040000 to 0x03d71a (0x28e5 bytes) 000155 1779203199.190913 read 0x01df75 thru 0x023ae3 (0x5b6f bytes) 000156 1779203199.205244 write 0x003ae0 thru 0x00f92e (0xbe4f bytes) 000157 1779203199.223984 read 0x00c93e thru 0x01a199 (0xd85c bytes) 000158 1779203199.235573 read 0x01bd75 thru 0x0222c5 (0x6551 bytes) 000159 1779203199.241772 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d71b): Mirror 4: 0xb3b28ae5 (stale) Mirror 5: 0x9172ffd3 (stale) Mirror 6: 0x7331c7c7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d71b): Mirror 4: 0xb3b28ae5 (stale) Mirror 5: 0x9172ffd3 (stale) Mirror 6: 0x7331c7c7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d71b: skipped 000160 1779203199.792529 trunc from 0x03d71b to 0x032955 (0xadc5 bytes) 000162 1779203199.815906 mapread 0x031708 thru 0x0318c6 (0x1bf bytes) 000163 1779203199.823567 read 0x0176d1 thru 0x01efc3 (0x78f3 bytes) 000165 1779203199.857680 write 0x022ff8 thru 0x02ff47 (0xcf50 bytes) 000166 1779203199.881809 mapread 0x025d0d thru 0x0303a8 (0xa69c bytes) 000167 1779203199.895306 mapread 0x01d132 thru 0x02cb41 (0xfa10 bytes) 000168 1779203199.912994 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32956): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32956): Mirror 4: 0x20530873 (stale) Mirror 5: 0xb502a5a5 (stale) Mirror 6: 0x738f9d2e (stale) CRC-32 checksum value for chunk [0, 0x32956): Mirror 1: 0xd50f9085 Mirror 7: 0xd50f9085 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32956: skipped 000170 1779203201.334327 mapwrite 0x02c7c2 thru 0x02de37 (0x1676 bytes) 000171 1779203201.553441 read 0x01eec2 thru 0x02cb77 (0xdcb6 bytes) 000173 1779203201.566210 mapwrite 0x003e51 thru 0x0059b0 (0x1b60 bytes) 000176 1779203201.665071 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32956): Mirror 5: 0xb502a5a5 (stale) Mirror 6: 0x738f9d2e (stale) Mirror 7: 0xd50f9085 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32956): Mirror 5: 0xb502a5a5 (stale) Mirror 6: 0x738f9d2e (stale) Mirror 7: 0xd50f9085 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32956: skipped 000177 1779203202.390067 mapwrite 0x00eef6 thru 0x0147bf (0x58ca bytes) 000178 1779203202.537075 write 0x03d082 thru 0x03ffff (0x2f7e bytes) 000179 1779203202.540587 mapread 0x006e7a thru 0x00f902 (0x8a89 bytes) 000182 1779203202.562909 write 0x017e6b thru 0x01bcec (0x3e82 bytes) 000187 1779203202.591042 trunc from 0x040000 to 0x0120a1 (0x2df5e bytes) 000188 1779203202.649057 mapread 0x00dc82 thru 0x0120a1 (0x4420 bytes) 000190 1779203202.665366 MIRROR_RESYNC 4 mirrors 000194 1779203203.771203 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000196 1779203204.367855 mapwrite 0x01e59e thru 0x02d681 (0xf0e4 bytes) 000198 1779203204.608756 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d682): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d682): Mirror 6: 0x8b1eb036 (stale) Mirror 7: 0x8b1eb036 (stale) CRC-32 checksum value for chunk [0, 0x2d682): Mirror 1: 0xa874ebad Mirror 8: 0xa874ebad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d682: skipped 000200 1779203205.986798 mapread 0x00b890 thru 0x00f21f (0x3990 bytes) 000202 1779203206.089375 mapread 0x01317c thru 0x01459a (0x141f bytes) 000203 1779203206.111106 mapwrite 0x00d8e8 thru 0x010884 (0x2f9d bytes) 000207 1779203206.403239 mapwrite 0x0248e7 thru 0x02afa7 (0x66c1 bytes) 000208 1779203206.466549 mapwrite 0x02c98d thru 0x02d603 (0xc77 bytes) 000211 1779203206.531811 trunc from 0x02d682 to 0x029e52 (0x382f bytes) 000212 1779203206.559201 read 0x006f3d thru 0x00f321 (0x83e5 bytes) 000214 1779203206.565057 MIRROR_RESYNC 4 mirrors 000215 1779203207.600366 mapwrite 0x03f46d thru 0x03ffff (0xb93 bytes) 000216 1779203207.777873 read 0x035a3c thru 0x03b943 (0x5f08 bytes) 000218 1779203207.797262 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x83b02398 (stale) Mirror 7: 0x83b02398 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x83b02398 (stale) Mirror 7: 0x83b02398 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000219 1779203208.238349 mapread 0x00c993 thru 0x0161a7 (0x9815 bytes) 000220 1779203208.276471 mapread 0x026084 thru 0x02e74b (0x86c8 bytes) 000221 1779203208.295114 write 0x01cee1 thru 0x0283ff (0xb51f bytes) 000222 1779203208.314022 trunc from 0x040000 to 0x023056 (0x1cfa9 bytes) 000228 1779203208.348447 write 0x020aba thru 0x02ed2a (0xe271 bytes) 000229 1779203208.352044 read 0x02ae11 thru 0x02ed2a (0x3f1a bytes) 000230 1779203208.353186 trunc from 0x02ed2b to 0x008cf1 (0x26039 bytes) 000231 1779203208.385933 MIRROR_RESYNC 3 mirrors 000232 1779203209.312868 read 0x008310 thru 0x008cf1 (0x9e2 bytes) 000233 1779203209.320993 write 0x013ca5 thru 0x018dca (0x5126 bytes) 000234 1779203209.432398 MIRROR_RESYNC 3 mirrors 000236 1779203210.355861 write 0x00aab0 thru 0x011433 (0x6984 bytes) 000237 1779203210.493236 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18dcb): Mirror 7: 0xb0eac46b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18dcb): Mirror 7: 0xb0eac46b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18dcb: skipped 000239 1779203210.878933 mapread 0x009526 thru 0x00df49 (0x4a24 bytes) 000243 1779203210.917132 trunc from 0x018dcb to 0x00f6f7 (0x96d3 bytes) 000245 1779203210.946625 trunc from 0x00f6f8 to 0x00084e (0xeea9 bytes) 000246 1779203210.970409 write 0x019048 thru 0x019645 (0x5fe bytes) 000247 1779203210.975922 write 0x017e18 thru 0x01c31a (0x4503 bytes) 000248 1779203211.003040 mapwrite 0x005f00 thru 0x011172 (0xb273 bytes) 000249 1779203211.104303 write 0x0082de thru 0x0155be (0xd2e1 bytes) 000250 1779203211.107291 trunc from 0x01c31b to 0x014571 (0x7da9 bytes) 000251 1779203211.123926 write 0x01ca2d thru 0x022921 (0x5ef5 bytes) 000252 1779203211.128841 mapread 0x00f025 thru 0x00ff0f (0xeeb bytes) 000253 1779203211.130862 trunc from 0x022922 to 0x014c17 (0xdd0a bytes) 000254 1779203211.161621 read 0x01005c thru 0x011d33 (0x1cd8 bytes) 000255 1779203211.162318 mapwrite 0x034819 thru 0x03ffff (0xb7e7 bytes) 000256 1779203211.299202 mapread 0x02adcf thru 0x038cb7 (0xdee9 bytes) 000259 1779203211.314788 mapread 0x023139 thru 0x031a35 (0xe8fd bytes) 000261 1779203211.317036 mapread 0x003a3c thru 0x00e773 (0xad38 bytes) 000264 1779203211.322154 mapwrite 0x018317 thru 0x023c22 (0xb90c bytes) 000266 1779203211.412882 MIRROR_RESYNC 2 mirrors 000267 1779203212.580177 mapwrite 0x0113b3 thru 0x0194f7 (0x8145 bytes) 000269 1779203212.866121 mapwrite 0x02c052 thru 0x036f6d (0xaf1c bytes) 000272 1779203212.992351 read 0x0212a2 thru 0x02d18f (0xbeee bytes) 000273 1779203213.006981 MIRROR_RESYNC 2 mirrors 000274 1779203213.996902 mapwrite 0x032e1b thru 0x03a38d (0x7573 bytes) 000275 1779203214.211806 mapwrite 0x017272 thru 0x021764 (0xa4f3 bytes) 000277 1779203214.289121 mapread 0x0395fb thru 0x03e20c (0x4c12 bytes) 000278 1779203214.298214 trunc from 0x040000 to 0x0251ab (0x1ae54 bytes) 000283 1779203214.334929 write 0x022532 thru 0x02e818 (0xc2e7 bytes) 000284 1779203214.337743 read 0x01551f thru 0x0239fc (0xe4de bytes) 000285 1779203214.338894 read 0x027f3c thru 0x02e818 (0x68dd bytes) 000287 1779203214.346172 MIRROR_RESYNC 2 mirrors 000288 1779203215.185790 mapread 0x02b15e thru 0x02e818 (0x36bb bytes) 000289 1779203215.218350 write 0x016477 thru 0x023894 (0xd41e bytes) 000290 1779203215.345476 write 0x02dfe1 thru 0x03c5b0 (0xe5d0 bytes) 000294 1779203215.381576 mapwrite 0x0159c9 thru 0x0190e0 (0x3718 bytes) 000295 1779203215.518764 trunc from 0x03c5b1 to 0x0328a3 (0x9d0d bytes) 000296 1779203215.582091 read 0x02585f thru 0x027434 (0x1bd6 bytes) 000299 1779203215.601433 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000305 1779203215.820723 trunc from 0x0328a4 to 0x00dc9e (0x24c05 bytes) 000306 1779203215.845794 read 0x007e12 thru 0x00dc9e (0x5e8d bytes) 000309 1779203215.863610 write 0x004e0e thru 0x012b1a (0xdd0d bytes) 000311 1779203215.875175 trunc from 0x012b1b to 0x00d370 (0x57aa bytes) 000312 1779203215.904243 trunc from 0x00d371 to 0x026df3 (0x19a83 bytes) 000315 1779203215.933193 write 0x013863 thru 0x019a0e (0x61ac bytes) 000317 1779203215.949778 write 0x0181c7 thru 0x01b5ff (0x3439 bytes) 000319 1779203215.963128 write 0x015aff thru 0x024d89 (0xf28b bytes) 000320 1779203215.975835 mapwrite 0x03b6b7 thru 0x03ffff (0x4949 bytes) 000321 1779203216.107131 mapwrite 0x02722e thru 0x02e9b8 (0x778b bytes) 000324 1779203216.230697 trunc from 0x040000 to 0x0264b8 (0x19b47 bytes) 000325 1779203216.250434 read 0x00119d thru 0x009ea3 (0x8d07 bytes) 000326 1779203216.268020 trunc from 0x0264b9 to 0x008554 (0x1df64 bytes) 000329 1779203216.305988 read 0x00053d thru 0x00216a (0x1c2e bytes) 000331 1779203216.307180 read 0x000ccc thru 0x008554 (0x7889 bytes) 000335 1779203216.315935 write 0x014023 thru 0x018632 (0x4610 bytes) 000336 1779203216.318115 mapread 0x011d59 thru 0x018632 (0x68da bytes) 000337 1779203216.330128 mapread 0x00cf5c thru 0x017841 (0xa8e6 bytes) 000341 1779203216.346216 mapread 0x016180 thru 0x018632 (0x24b3 bytes) 000342 1779203216.347599 write 0x017301 thru 0x01b79a (0x449a bytes) 000343 1779203216.350104 mapread 0x012348 thru 0x0189e6 (0x669f bytes) 000344 1779203216.352197 mapread 0x01895d thru 0x01b79a (0x2e3e bytes) 000345 1779203216.369243 MIRROR_EXTEND to 2 mirrors 000347 1779203217.055685 read 0x0072e5 thru 0x01175f (0xa47b bytes) 000350 1779203217.076826 mapread 0x013fc2 thru 0x015173 (0x11b2 bytes) 000351 1779203217.095563 mapread 0x0139f0 thru 0x01b79a (0x7dab bytes) 000352 1779203217.096643 write 0x012c7a thru 0x0206cf (0xda56 bytes) 000353 1779203217.243708 trunc from 0x0206d0 to 0x0302c1 (0xfbf2 bytes) 000354 1779203217.306109 mapwrite 0x025188 thru 0x026b03 (0x197c bytes) 000355 1779203217.408924 mapread 0x00612f thru 0x009f8c (0x3e5e bytes) 000358 1779203217.428424 read 0x0038c4 thru 0x00f0d8 (0xb815 bytes) 000361 1779203217.430943 read 0x02aa54 thru 0x0302c1 (0x586e bytes) 000362 1779203217.453115 trunc from 0x0302c2 to 0x02ff19 (0x3a8 bytes) 000364 1779203217.489980 trunc from 0x02ff1a to 0x000f86 (0x2ef93 bytes) 000366 1779203217.532134 read 0x000cf2 thru 0x000f86 (0x295 bytes) 000370 1779203217.538764 mapwrite 0x01425a thru 0x01ba00 (0x77a7 bytes) 000371 1779203217.640371 write 0x007700 thru 0x01576a (0xe06b bytes) 000374 1779203217.654705 MIRROR_RESYNC 2 mirrors 000378 1779203218.530825 mapread 0x00a4fd thru 0x00badc (0x15e0 bytes) 000384 1779203218.552987 MIRROR_RESYNC 2 mirrors 000386 1779203218.973336 mapwrite 0x018083 thru 0x02524d (0xd1cb bytes) 000389 1779203219.229849 trunc from 0x02524e to 0x00d51d (0x17d30 bytes) 000390 1779203219.279193 write 0x03415d thru 0x03ab8d (0x6a31 bytes) 000391 1779203219.280768 read 0x02fb5a thru 0x032927 (0x2dce bytes) 000393 1779203219.315654 read 0x0092a2 thru 0x014809 (0xb568 bytes) 000396 1779203219.353317 mapread 0x01510b thru 0x01bbea (0x6ae0 bytes) 000397 1779203219.354754 mapread 0x02387f thru 0x02c9af (0x9131 bytes) 000398 1779203219.366965 read 0x02e0ba thru 0x02ff02 (0x1e49 bytes) 000401 1779203219.381398 read 0x026fa7 thru 0x036947 (0xf9a1 bytes) 000402 1779203219.382647 read 0x0009e6 thru 0x0069e9 (0x6004 bytes) 000403 1779203219.390723 trunc from 0x03ab8e to 0x022dcf (0x17dbe bytes) 000404 1779203219.470089 MIRROR_RESYNC 2 mirrors 000406 1779203220.529016 MIRROR_RESYNC 2 mirrors 000407 1779203221.193567 mapwrite 0x01d072 thru 0x0293eb (0xc37a bytes) 000408 1779203221.639185 MIRROR_RESYNC 2 mirrors 000409 1779203222.733613 write 0x002d01 thru 0x003536 (0x836 bytes) 000410 1779203222.866107 write 0x034c2f thru 0x03ffff (0xb3d1 bytes) 000412 1779203222.872330 write 0x01a093 thru 0x01abf8 (0xb66 bytes) 000414 1779203222.897235 read 0x02c88b thru 0x039b9a (0xd310 bytes) 000416 1779203222.916468 read 0x0197c7 thru 0x029450 (0xfc8a bytes) 000418 1779203222.967611 mapread 0x006828 thru 0x00cbe4 (0x63bd bytes) 000421 1779203222.975803 trunc from 0x040000 to 0x027f09 (0x180f6 bytes) 000422 1779203223.011146 read 0x00b72e thru 0x00beb1 (0x784 bytes) 000423 1779203223.012082 read 0x025ab0 thru 0x027f09 (0x245a bytes) 000424 1779203223.019644 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000427 1779203223.226738 write 0x00a2d8 thru 0x0128f3 (0x861c bytes) 000429 1779203223.242159 trunc from 0x027f0a to 0x001f03 (0x26006 bytes) 000430 1779203223.261067 mapwrite 0x00680d thru 0x00bc09 (0x53fd bytes) 000431 1779203223.366177 MIRROR_EXTEND to 2 mirrors 000432 1779203224.145070 trunc from 0x00bc0a to 0x02bb0c (0x1ff03 bytes) 000434 1779203224.355738 trunc from 0x02bb0d to 0x017e99 (0x13c73 bytes) 000435 1779203224.402219 trunc from 0x017e9a to 0x002537 (0x15962 bytes) 000436 1779203224.445011 mapread 0x000410 thru 0x002537 (0x2128 bytes) 000438 1779203224.482100 mapwrite 0x018603 thru 0x01f5e0 (0x6fde bytes) 000439 1779203224.632626 trunc from 0x01f5e1 to 0x03a92c (0x1b34c bytes) 000440 1779203224.686351 MIRROR_RESYNC 2 mirrors 000442 1779203225.611230 MIRROR_EXTEND to 3 mirrors 000444 1779203226.618798 mapwrite 0x0201e9 thru 0x024fde (0x4df6 bytes) 000445 1779203226.849176 write 0x0007fd thru 0x0077cd (0x6fd1 bytes) 000446 1779203226.869732 read 0x02506b thru 0x029d42 (0x4cd8 bytes) 000449 1779203226.871644 mapwrite 0x01d9cd thru 0x02ba78 (0xe0ac bytes) 000450 1779203226.989793 mapread 0x031d34 thru 0x03a92c (0x8bf9 bytes) 000452 1779203227.009808 write 0x020256 thru 0x02a2c7 (0xa072 bytes) 000453 1779203227.012624 read 0x02b18f thru 0x02c87d (0x16ef bytes) 000459 1779203227.019886 write 0x0126ac thru 0x014c5d (0x25b2 bytes) 000460 1779203227.030709 trunc from 0x03a92d to 0x00aac5 (0x2fe67 bytes) 000461 1779203227.059081 mapread 0x003c6e thru 0x00aac5 (0x6e58 bytes) 000462 1779203227.060902 write 0x03cbd7 thru 0x03ffff (0x3429 bytes) 000463 1779203227.063842 mapread 0x02ec81 thru 0x03c318 (0xd698 bytes) 000464 1779203227.155658 write 0x02b7cb thru 0x0317ba (0x5ff0 bytes) 000465 1779203227.163769 trunc from 0x040000 to 0x002414 (0x3dbeb bytes) 000466 1779203227.188894 mapread 0x00066d thru 0x002414 (0x1da8 bytes) 000468 1779203227.190970 mapread 0x000282 thru 0x002414 (0x2193 bytes) 000469 1779203227.200010 MIRROR_RESYNC 3 mirrors 000470 1779203228.239988 MIRROR_EXTEND to 4 mirrors 000471 1779203229.058055 mapwrite 0x011ed5 thru 0x01e725 (0xc851 bytes) 000472 1779203229.397179 mapread 0x00e7b5 thru 0x016467 (0x7cb3 bytes) 000475 1779203229.403381 trunc from 0x01e726 to 0x024c29 (0x6504 bytes) 000476 1779203229.439713 write 0x03e632 thru 0x03ffff (0x19ce bytes) 000483 1779203229.461220 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xda6ada2f (stale) Mirror 3: 0xda6ada2f (stale) Mirror 4: 0xda6ada2f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x797b13f Mirror 5: 0x797b13f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000485 1779203230.889164 trunc from 0x040000 to 0x0174c1 (0x28b3e bytes) 000486 1779203230.988563 trunc from 0x0174c2 to 0x02eac3 (0x17602 bytes) 000488 1779203231.020151 mapread 0x00d1a1 thru 0x01b3fe (0xe25e bytes) 000489 1779203231.100093 trunc from 0x02eac4 to 0x00668d (0x28436 bytes) 000490 1779203231.139301 mapwrite 0x03570c thru 0x03bb6c (0x6461 bytes) 000491 1779203231.388078 MIRROR_RESYNC 5 mirrors 000493 1779203232.672758 mapread 0x008dad thru 0x00b47c (0x26d0 bytes) 000497 1779203232.712754 read 0x00ef2e thru 0x01651c (0x75ef bytes) 000499 1779203232.721232 read 0x01d687 thru 0x02b9fe (0xe378 bytes) 000500 1779203232.758725 MIRROR_EXTEND to 6 mirrors 000501 1779203233.558535 read 0x001509 thru 0x002d76 (0x186e bytes) 000502 1779203233.566602 read 0x00d94e thru 0x0102e2 (0x2995 bytes) 000503 1779203233.585541 MIRROR_RESYNC 6 mirrors 000506 1779203234.280834 mapwrite 0x031395 thru 0x0323e1 (0x104d bytes) 000507 1779203234.466149 mapwrite 0x02b956 thru 0x02d817 (0x1ec2 bytes) 000510 1779203234.505836 write 0x015c3b thru 0x01d1a2 (0x7568 bytes) 000512 1779203234.518257 write 0x012240 thru 0x01b2b8 (0x9079 bytes) 000513 1779203234.520065 trunc from 0x03bb6d to 0x019f5b (0x21c11 bytes) 000516 1779203234.547564 mapread 0x00a070 thru 0x013d76 (0x9d07 bytes) 000517 1779203234.558580 trunc from 0x019f5c to 0x03f0b1 (0x25156 bytes) 000519 1779203234.588543 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f0b2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0b2): Mirror 2: 0xfe556358 (stale) Mirror 3: 0xfe556358 (stale) Mirror 4: 0xfe556358 (stale) Mirror 5: 0xfe556358 (stale) Mirror 6: 0xfe556358 (stale) CRC-32 checksum value for chunk [0, 0x3f0b2): Mirror 1: 0x8feafc33 Mirror 7: 0x8feafc33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0b2: skipped 000520 1779203235.903680 write 0x013b9d thru 0x019ad8 (0x5f3c bytes) 000521 1779203236.154346 mapwrite 0x029fb6 thru 0x03318f (0x91da bytes) 000524 1779203236.224009 mapwrite 0x032278 thru 0x033858 (0x15e1 bytes) 000525 1779203236.280700 read 0x017216 thru 0x018148 (0xf33 bytes) 000528 1779203236.282258 mapread 0x02d2f3 thru 0x032704 (0x5412 bytes) 000533 1779203236.285422 mapwrite 0x01d1eb thru 0x02ae2b (0xdc41 bytes) 000534 1779203236.345536 mapwrite 0x019fd1 thru 0x028bf2 (0xec22 bytes) 000535 1779203236.392768 mapwrite 0x037e92 thru 0x03cd3d (0x4eac bytes) 000536 1779203236.450557 mapwrite 0x006684 thru 0x00af98 (0x4915 bytes) 000537 1779203236.502028 mapread 0x034436 thru 0x036429 (0x1ff4 bytes) 000538 1779203236.503772 mapread 0x018b1a thru 0x01ad69 (0x2250 bytes) 000539 1779203236.505062 read 0x01f351 thru 0x02311e (0x3dce bytes) 000542 1779203236.512218 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f0b2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0b2): Mirror 2: 0xfe556358 (stale) Mirror 3: 0xfe556358 (stale) Mirror 4: 0xfe556358 (stale) Mirror 5: 0xfe556358 (stale) Mirror 6: 0xfe556358 (stale) Mirror 7: 0x8feafc33 (stale) CRC-32 checksum value for chunk [0, 0x3f0b2): Mirror 1: 0xbe4e6830 Mirror 8: 0xbe4e6830 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0b2: skipped 000543 1779203237.645681 trunc from 0x03f0b2 to 0x02dfa6 (0x1110b bytes) 000544 1779203237.800985 mapread 0x021613 thru 0x028a2b (0x7419 bytes) 000545 1779203237.824474 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dfa7): Mirror 2: 0xc89301e1 (stale) Mirror 3: 0xc89301e1 (stale) Mirror 4: 0xc89301e1 (stale) Mirror 5: 0xc89301e1 (stale) Mirror 6: 0xc89301e1 (stale) Mirror 7: 0xa5e0e408 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dfa7): Mirror 2: 0xc89301e1 (stale) Mirror 3: 0xc89301e1 (stale) Mirror 4: 0xc89301e1 (stale) Mirror 5: 0xc89301e1 (stale) Mirror 6: 0xc89301e1 (stale) Mirror 7: 0xa5e0e408 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dfa7: skipped 000547 1779203238.379935 write 0x01fa0c thru 0x0204ab (0xaa0 bytes) 000549 1779203238.403031 mapwrite 0x03b5df thru 0x03ffff (0x4a21 bytes) 000553 1779203238.606564 mapwrite 0x02738d thru 0x03176a (0xa3de bytes) 000557 1779203238.725750 mapwrite 0x0364f6 thru 0x03a4d4 (0x3fdf bytes) 000558 1779203238.830845 write 0x00a690 thru 0x018790 (0xe101 bytes) 000560 1779203238.876147 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8cf668cb (stale) Mirror 3: 0x8cf668cb (stale) Mirror 5: 0x8cf668cb (stale) Mirror 6: 0x8cf668cb (stale) Mirror 7: 0xdff4c3c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8cf668cb (stale) Mirror 3: 0x8cf668cb (stale) Mirror 5: 0x8cf668cb (stale) Mirror 6: 0x8cf668cb (stale) Mirror 7: 0xdff4c3c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000562 1779203239.499007 mapwrite 0x00fba5 thru 0x01a0d9 (0xa535 bytes) 000566 1779203239.682770 read 0x008777 thru 0x0134b1 (0xad3b bytes) 000567 1779203239.704032 MIRROR_RESYNC 6 mirrors 000571 1779203241.462410 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000572 1779203242.050054 read 0x005294 thru 0x00b052 (0x5dbf bytes) 000576 1779203242.066929 read 0x0000ce thru 0x003c8e (0x3bc1 bytes) 000577 1779203242.110009 write 0x034f03 thru 0x03ffff (0xb0fd bytes) 000578 1779203242.242022 trunc from 0x040000 to 0x036532 (0x9acd bytes) 000579 1779203242.276369 read 0x033621 thru 0x036532 (0x2f12 bytes) 000581 1779203242.304065 write 0x024c82 thru 0x02a540 (0x58bf bytes) 000585 1779203242.310161 trunc from 0x036533 to 0x0386e2 (0x21b0 bytes) 000586 1779203242.335876 mapwrite 0x01679c thru 0x018479 (0x1cde bytes) 000587 1779203242.435609 trunc from 0x0386e3 to 0x011cb2 (0x26a30 bytes) 000589 1779203242.467756 read 0x00beb1 thru 0x00ec0a (0x2d5a bytes) 000590 1779203242.468565 mapread 0x0040c7 thru 0x01057f (0xc4b9 bytes) 000592 1779203242.474984 write 0x009d03 thru 0x00ea8f (0x4d8d bytes) 000593 1779203242.481183 MIRROR_RESYNC 5 mirrors 000594 1779203244.342679 mapwrite 0x031617 thru 0x036508 (0x4ef2 bytes) 000595 1779203244.630818 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36509): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36509): Mirror 3: 0x673eaf84 (stale) Mirror 5: 0x673eaf84 (stale) Mirror 6: 0x673eaf84 (stale) Mirror 7: 0x673eaf84 (stale) CRC-32 checksum value for chunk [0, 0x36509): Mirror 1: 0xe86e32c6 Mirror 8: 0xe86e32c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36509: skipped 000598 1779203245.636840 write 0x02b656 thru 0x035fad (0xa958 bytes) 000599 1779203245.819613 write 0x0093c2 thru 0x01393f (0xa57e bytes) 000600 1779203245.849848 mapwrite 0x00c7a6 thru 0x0171e8 (0xaa43 bytes) 000602 1779203246.000142 MIRROR_RESYNC 6 mirrors 000603 1779203247.756172 trunc from 0x036509 to 0x01b3e4 (0x1b124 bytes) 000604 1779203247.973983 write 0x029dec thru 0x02d6ee (0x3903 bytes) 000605 1779203247.975502 read 0x00a311 thru 0x0116dd (0x73cd bytes) 000607 1779203248.022535 write 0x00e8f4 thru 0x01aba4 (0xc2b1 bytes) 000608 1779203248.036235 trunc from 0x02d6ef to 0x02bc49 (0x1aa5 bytes) 000609 1779203248.064420 read 0x017c8c thru 0x01988b (0x1c00 bytes) 000612 1779203248.066289 read 0x02627a thru 0x028346 (0x20cd bytes) 000613 1779203248.093871 mapwrite 0x03d761 thru 0x03ffff (0x289f bytes) 000614 1779203248.311986 MIRROR_RESYNC 6 mirrors 000616 1779203250.537630 write 0x03c2c6 thru 0x03ffff (0x3d3a bytes) 000617 1779203250.735405 read 0x00eb71 thru 0x01ce91 (0xe321 bytes) 000618 1779203250.822207 mapread 0x01b738 thru 0x01fa4a (0x4313 bytes) 000619 1779203250.864241 mapread 0x0116ca thru 0x019800 (0x8137 bytes) 000620 1779203250.868637 trunc from 0x040000 to 0x02e607 (0x119f8 bytes) 000623 1779203250.972563 trunc from 0x02e608 to 0x00a0c8 (0x2453f bytes) 000624 1779203251.034559 write 0x00dc0e thru 0x019732 (0xbb25 bytes) 000625 1779203251.037731 write 0x0065e1 thru 0x009c6f (0x368f bytes) 000627 1779203251.073168 read 0x004df0 thru 0x006db4 (0x1fc5 bytes) 000628 1779203251.080502 mapwrite 0x026a1b thru 0x03175b (0xad41 bytes) 000629 1779203251.463867 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3175c): Mirror 3: 0x446af5f9 (stale) Mirror 6: 0x446af5f9 (stale) Mirror 7: 0x446af5f9 (stale) Mirror 8: 0x446af5f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3175c): Mirror 3: 0x446af5f9 (stale) Mirror 6: 0x446af5f9 (stale) Mirror 7: 0x446af5f9 (stale) Mirror 8: 0x446af5f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3175c: skipped 000631 1779203252.340540 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3175c): Mirror 3: 0x446af5f9 (stale) Mirror 6: 0x446af5f9 (stale) Mirror 7: 0x446af5f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3175c): Mirror 3: 0x446af5f9 (stale) Mirror 6: 0x446af5f9 (stale) Mirror 7: 0x446af5f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3175c: skipped 000632 1779203252.905336 trunc from 0x03175c to 0x004c43 (0x2cb18 bytes) 000633 1779203252.934935 trunc from 0x004c44 to 0x03d677 (0x38a34 bytes) 000634 1779203252.959485 mapread 0x022bbd thru 0x02da57 (0xae9b bytes) 000635 1779203253.011013 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d678): Mirror 3: 0xe4df8a97 (stale) Mirror 7: 0xe4df8a97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d678): Mirror 3: 0xe4df8a97 (stale) Mirror 7: 0xe4df8a97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d678: skipped 000636 1779203253.534028 mapread 0x0378fc thru 0x03d677 (0x5d7c bytes) 000639 1779203253.556854 write 0x00a127 thru 0x01966c (0xf546 bytes) 000641 1779203253.572788 write 0x00c834 thru 0x010e58 (0x4625 bytes) 000643 1779203253.577434 mapwrite 0x02cf50 thru 0x02d4ca (0x57b bytes) 000647 1779203253.675491 mapread 0x017b72 thru 0x01df9d (0x642c bytes) 000648 1779203253.685557 mapread 0x00ab62 thru 0x00f51b (0x49ba bytes) 000649 1779203253.686730 mapwrite 0x0320d2 thru 0x03ffff (0xdf2e bytes) 000650 1779203253.767475 mapread 0x001fa5 thru 0x00867e (0x66da bytes) 000651 1779203253.780312 MIRROR_RESYNC 3 mirrors 000652 1779203255.060654 mapread 0x0367fa thru 0x03c7f6 (0x5ffd bytes) 000653 1779203255.158651 trunc from 0x040000 to 0x0272b9 (0x18d46 bytes) 000654 1779203255.339502 mapread 0x016541 thru 0x02613b (0xfbfb bytes) 000655 1779203255.384823 trunc from 0x0272ba to 0x00d95e (0x1995b bytes) 000657 1779203255.445468 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xd95f): Mirror 3: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd95f): Mirror 3: 0xa1f79f80 (stale) Mirror 7: 0xa1f79f80 (stale) CRC-32 checksum value for chunk [0, 0xd95f): Mirror 1: 0xa1f79f80 Mirror 8: 0xa1f79f80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd95f: skipped 000659 1779203256.922679 trunc from 0x00d95f to 0x019102 (0xb7a4 bytes) 000661 1779203257.143170 write 0x00c738 thru 0x01a2fb (0xdbc4 bytes) 000663 1779203257.168550 trunc from 0x01a2fc to 0x02b116 (0x10e1b bytes) 000665 1779203257.217586 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b117): Mirror 3: 0x100dff0d (stale) Mirror 7: 0x100dff0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b117): Mirror 3: 0x100dff0d (stale) Mirror 7: 0x100dff0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b117: skipped 000666 1779203257.817693 trunc from 0x02b117 to 0x030383 (0x526d bytes) 000667 1779203257.868537 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30384): Mirror 7: 0x26b0b8aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30384): Mirror 7: 0x26b0b8aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30384: skipped 000668 1779203258.389436 mapwrite 0x00dd31 thru 0x00fd40 (0x2010 bytes) 000671 1779203258.472960 read 0x00b78c thru 0x00e261 (0x2ad6 bytes) 000672 1779203258.473639 write 0x01a8b6 thru 0x021970 (0x70bb bytes) 000675 1779203258.517858 write 0x006c7b thru 0x008961 (0x1ce7 bytes) 000676 1779203258.532759 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000677 1779203258.855784 mapread 0x02fa86 thru 0x030383 (0x8fe bytes) 000679 1779203258.868331 trunc from 0x030384 to 0x014c84 (0x1b6ff bytes) 000680 1779203258.908175 trunc from 0x014c85 to 0x013b14 (0x1170 bytes) 000681 1779203258.924363 mapread 0x003813 thru 0x012376 (0xeb64 bytes) 000683 1779203258.946049 mapread 0x011c56 thru 0x0136cb (0x1a76 bytes) 000684 1779203258.949055 mapwrite 0x0308f8 thru 0x0330f6 (0x27ff bytes) 000685 1779203259.093444 write 0x01fdec thru 0x01febe (0xd3 bytes) 000686 1779203259.111850 MIRROR_EXTEND to 2 mirrors 000687 1779203260.145558 trunc from 0x0330f7 to 0x0099cd (0x29729 bytes) 000688 1779203260.322879 trunc from 0x0099ce to 0x02ee8c (0x254bf bytes) 000690 1779203260.379690 read 0x010015 thru 0x018ab4 (0x8aa0 bytes) 000692 1779203260.405585 write 0x00efda thru 0x01d6e5 (0xe70c bytes) 000693 1779203260.429970 read 0x028d34 thru 0x02ee8c (0x6159 bytes) 000695 1779203260.454748 mapread 0x00ef3d thru 0x019215 (0xa2d9 bytes) 000698 1779203260.458908 read 0x02b189 thru 0x02d165 (0x1fdd bytes) 000699 1779203260.461703 read 0x017c5f thru 0x01dd0d (0x60af bytes) 000700 1779203260.463685 write 0x00c6a3 thru 0x0169ed (0xa34b bytes) 000705 1779203260.468179 mapwrite 0x0167b7 thru 0x02660c (0xfe56 bytes) 000707 1779203260.617007 read 0x021931 thru 0x02845d (0x6b2d bytes) 000709 1779203260.618303 trunc from 0x02ee8d to 0x00f56f (0x1f91d bytes) 000713 1779203260.660304 mapread 0x001fa5 thru 0x007625 (0x5681 bytes) 000714 1779203260.670410 write 0x00521c thru 0x00a916 (0x56fb bytes) 000715 1779203260.671764 mapread 0x003563 thru 0x005235 (0x1cd3 bytes) 000716 1779203260.680195 read 0x000179 thru 0x006c08 (0x6a90 bytes) 000717 1779203260.686438 read 0x00cd81 thru 0x00f56f (0x27ef bytes) 000718 1779203260.694060 trunc from 0x00f570 to 0x02b7d5 (0x1c266 bytes) 000719 1779203260.738978 mapwrite 0x00d097 thru 0x0158e3 (0x884d bytes) 000720 1779203260.880591 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000721 1779203261.123463 mapread 0x023a12 thru 0x02b7d5 (0x7dc4 bytes) 000724 1779203261.176432 read 0x00bcdc thru 0x015971 (0x9c96 bytes) 000725 1779203261.198789 trunc from 0x02b7d6 to 0x01020c (0x1b5c9 bytes) 000726 1779203261.246177 trunc from 0x01020d to 0x01611e (0x5f12 bytes) 000727 1779203261.292617 MIRROR_EXTEND to 2 mirrors 000728 1779203262.229331 MIRROR_EXTEND to 3 mirrors 000729 1779203263.207131 trunc from 0x01611f to 0x03c63a (0x2651c bytes) 000732 1779203263.334463 read 0x00ff33 thru 0x01d3c1 (0xd48f bytes) 000735 1779203263.354614 MIRROR_RESYNC 3 mirrors 000741 1779203264.580136 read 0x01241f thru 0x01e2df (0xbec1 bytes) 000744 1779203264.605627 read 0x030f7e thru 0x036533 (0x55b6 bytes) 000746 1779203264.623010 trunc from 0x03c63b to 0x001460 (0x3b1da bytes) 000747 1779203264.742830 write 0x004b3c thru 0x0061a2 (0x1667 bytes) 000748 1779203264.745396 write 0x006d76 thru 0x0075e7 (0x872 bytes) 000749 1779203264.746740 write 0x00c382 thru 0x010fc0 (0x4c3f bytes) 000750 1779203264.749067 mapread 0x000571 thru 0x00570b (0x519b bytes) 000752 1779203264.792715 MIRROR_RESYNC 3 mirrors 000754 1779203265.977807 read 0x00260f thru 0x010298 (0xdc8a bytes) 000755 1779203265.990356 mapread 0x005626 thru 0x00f832 (0xa20d bytes) 000756 1779203265.995537 MIRROR_RESYNC 3 mirrors 000757 1779203266.733895 MIRROR_RESYNC 3 mirrors 000758 1779203267.305577 trunc from 0x010fc1 to 0x004f9f (0xc021 bytes) 000759 1779203267.478679 read 0x001f76 thru 0x004f9f (0x302a bytes) 000760 1779203267.498121 write 0x023827 thru 0x0316c1 (0xde9b bytes) 000762 1779203267.512184 read 0x0062e1 thru 0x014523 (0xe243 bytes) 000763 1779203267.543014 read 0x008762 thru 0x015e18 (0xd6b7 bytes) 000764 1779203267.547963 write 0x00d5a7 thru 0x017cb0 (0xa70a bytes) 000765 1779203267.555411 mapwrite 0x03b191 thru 0x03ffff (0x4e6f bytes) 000766 1779203267.702819 read 0x00eee3 thru 0x010ae4 (0x1c02 bytes) 000767 1779203267.704346 read 0x02c754 thru 0x031712 (0x4fbf bytes) 000770 1779203267.709807 mapwrite 0x003827 thru 0x007c59 (0x4433 bytes) 000772 1779203267.825364 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3a364f2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3a364f2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000774 1779203268.278013 read 0x01a6f4 thru 0x01f9d0 (0x52dd bytes) 000775 1779203268.302869 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000776 1779203268.470518 write 0x014e13 thru 0x016782 (0x1970 bytes) 000777 1779203268.486192 mapwrite 0x02b12e thru 0x030a0b (0x58de bytes) 000779 1779203268.576022 trunc from 0x040000 to 0x004c74 (0x3b38b bytes) 000781 1779203268.615391 mapwrite 0x0038b3 thru 0x00ecd1 (0xb41f bytes) 000784 1779203268.805656 read 0x00a0c9 thru 0x00ecd1 (0x4c09 bytes) 000786 1779203268.817956 MIRROR_EXTEND to 2 mirrors 000787 1779203269.861128 trunc from 0x00ecd2 to 0x024b04 (0x15e33 bytes) 000788 1779203269.989356 mapwrite 0x02ef7c thru 0x03ad71 (0xbdf6 bytes) 000791 1779203270.122435 write 0x01b63a thru 0x01edb5 (0x377c bytes) 000795 1779203270.137002 trunc from 0x03ad72 to 0x03c4df (0x176e bytes) 000796 1779203270.164823 mapread 0x014bfe thru 0x0161df (0x15e2 bytes) 000800 1779203270.173569 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000803 1779203270.440825 write 0x00becb thru 0x00d79c (0x18d2 bytes) 000804 1779203270.458384 trunc from 0x03c4e0 to 0x033fd0 (0x850f bytes) 000806 1779203270.482627 MIRROR_EXTEND to 2 mirrors 000807 1779203271.721763 trunc from 0x033fd1 to 0x03a851 (0x6881 bytes) 000815 1779203271.819045 trunc from 0x03a852 to 0x00d2dc (0x2d575 bytes) 000817 1779203271.846992 mapread 0x0028f0 thru 0x00caa8 (0xa1b9 bytes) 000818 1779203271.872971 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000819 1779203272.030000 write 0x00bcd4 thru 0x019c2e (0xdf5b bytes) 000820 1779203272.042454 trunc from 0x019c2f to 0x0211e3 (0x75b5 bytes) 000821 1779203272.067400 read 0x01bd97 thru 0x0211e3 (0x544d bytes) 000822 1779203272.078051 mapread 0x00eb7d thru 0x01de01 (0xf285 bytes) 000824 1779203272.094980 MIRROR_EXTEND to 2 mirrors 000825 1779203272.845193 read 0x01e4ba thru 0x0211e3 (0x2d2a bytes) 000826 1779203272.861757 mapread 0x01b70f thru 0x0211e3 (0x5ad5 bytes) 000828 1779203272.891677 mapread 0x01dbc8 thru 0x0211e3 (0x361c bytes) 000829 1779203272.894248 trunc from 0x0211e4 to 0x01ce09 (0x43da bytes) 000831 1779203273.026179 mapwrite 0x023c77 thru 0x0281d9 (0x4563 bytes) 000832 1779203273.137342 read 0x019ff4 thru 0x01ba61 (0x1a6e bytes) 000836 1779203273.161561 write 0x02366f thru 0x031c6c (0xe5fe bytes) 000838 1779203273.171006 MIRROR_RESYNC 2 mirrors 000839 1779203274.042150 mapread 0x023201 thru 0x026967 (0x3767 bytes) 000841 1779203274.083057 read 0x013928 thru 0x02094a (0xd023 bytes) 000842 1779203274.094958 write 0x00e3bc thru 0x019315 (0xaf5a bytes) 000843 1779203274.254344 read 0x01fefc thru 0x02659a (0x669f bytes) 000844 1779203274.269439 trunc from 0x031c6d to 0x03c3c6 (0xa75a bytes) 000845 1779203274.350532 trunc from 0x03c3c7 to 0x0054be (0x36f08 bytes) 000846 1779203274.395078 trunc from 0x0054bf to 0x0125f7 (0xd139 bytes) 000847 1779203274.499995 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x125f8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x125f8): Mirror 2: 0x2b23a50a (stale) CRC-32 checksum value for chunk [0, 0x125f8): Mirror 1: 0xc96729a2 Mirror 3: 0xc96729a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x125f8: skipped 000849 1779203275.358766 mapread 0x010a2c thru 0x0125f7 (0x1bcc bytes) 000850 1779203275.396258 trunc from 0x0125f8 to 0x03a887 (0x28290 bytes) 000854 1779203275.650356 mapwrite 0x03509e thru 0x03c79e (0x7701 bytes) 000855 1779203275.816657 mapread 0x0235bb thru 0x025b16 (0x255c bytes) 000860 1779203275.837588 read 0x02295f thru 0x0321e3 (0xf885 bytes) 000861 1779203275.844015 write 0x02fd8d thru 0x03850e (0x8782 bytes) 000862 1779203275.845558 write 0x03bc11 thru 0x03ffff (0x43ef bytes) 000863 1779203275.847998 write 0x021db9 thru 0x0317fd (0xfa45 bytes) 000866 1779203275.851387 read 0x00d5c5 thru 0x013499 (0x5ed5 bytes) 000868 1779203275.877120 mapwrite 0x036bea thru 0x03d2c3 (0x66da bytes) 000869 1779203275.952446 read 0x02e258 thru 0x03b29d (0xd046 bytes) 000874 1779203275.957258 write 0x016aec thru 0x0226ab (0xbbc0 bytes) 000876 1779203275.962248 write 0x014122 thru 0x016f63 (0x2e42 bytes) 000878 1779203275.964946 trunc from 0x040000 to 0x0096a2 (0x3695d bytes) 000883 1779203275.994638 write 0x00e297 thru 0x00ee3e (0xba8 bytes) 000884 1779203275.995812 write 0x02a332 thru 0x03966f (0xf33e bytes) 000885 1779203276.010710 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39670): Mirror 3: 0xbee31f0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39670): Mirror 3: 0xbee31f0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39670: skipped 000891 1779203276.423830 trunc from 0x039670 to 0x01cfeb (0x1c684 bytes) 000892 1779203276.463235 mapread 0x0092d8 thru 0x00dbe1 (0x490a bytes) 000893 1779203276.490801 mapread 0x01a847 thru 0x01cfeb (0x27a5 bytes) 000895 1779203276.512847 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000896 1779203276.706250 mapread 0x016fee thru 0x0197da (0x27ed bytes) 000898 1779203276.744236 read 0x004d5d thru 0x009f3a (0x51de bytes) 000900 1779203276.777283 read 0x00b6f0 thru 0x00cab5 (0x13c6 bytes) 000901 1779203276.804316 write 0x015ba3 thru 0x01ac73 (0x50d1 bytes) 000902 1779203276.807654 trunc from 0x01cfec to 0x00f0f8 (0xdef3 bytes) 000903 1779203276.845763 mapwrite 0x0204e5 thru 0x02eb21 (0xe63d bytes) 000907 1779203276.988644 mapwrite 0x00de68 thru 0x017f3c (0xa0d5 bytes) 000908 1779203277.130993 mapwrite 0x0030c5 thru 0x007097 (0x3fd3 bytes) 000910 1779203277.209986 trunc from 0x02eb22 to 0x02d497 (0x168a bytes) 000911 1779203277.239913 mapread 0x026b22 thru 0x029b6d (0x304c bytes) 000912 1779203277.244923 mapwrite 0x014ce9 thru 0x014f1c (0x234 bytes) 000913 1779203277.313434 mapwrite 0x03aa8a thru 0x03fa26 (0x4f9d bytes) 000918 1779203277.498121 mapwrite 0x020085 thru 0x02410d (0x4089 bytes) 000920 1779203277.593552 read 0x01b596 thru 0x020a67 (0x54d2 bytes) 000921 1779203277.619236 mapread 0x025cc7 thru 0x02b0aa (0x53e4 bytes) 000922 1779203277.620398 write 0x011a89 thru 0x0185a9 (0x6b21 bytes) 000923 1779203277.621574 write 0x020b2e thru 0x021004 (0x4d7 bytes) 000925 1779203277.629095 mapread 0x03dcf7 thru 0x03fa26 (0x1d30 bytes) 000926 1779203277.630846 mapwrite 0x0109bc thru 0x0167e3 (0x5e28 bytes) 000927 1779203277.752260 read 0x002ee9 thru 0x006ece (0x3fe6 bytes) 000928 1779203277.752998 trunc from 0x03fa27 to 0x03461c (0xb40a bytes) 000931 1779203277.784541 mapread 0x019511 thru 0x02418a (0xac7a bytes) 000933 1779203277.791951 mapwrite 0x0131ee thru 0x020852 (0xd665 bytes) 000934 1779203277.966613 mapwrite 0x0062d3 thru 0x010cf8 (0xaa26 bytes) 000937 1779203278.161542 trunc from 0x03461d to 0x0040eb (0x30531 bytes) 000939 1779203278.196169 MIRROR_EXTEND to 2 mirrors 000941 1779203279.003829 mapwrite 0x00dd3f thru 0x019d6f (0xc031 bytes) 000942 1779203279.254973 write 0x010f71 thru 0x019caa (0x8d3a bytes) 000943 1779203279.282434 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x19d70): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19d70): Mirror 2: 0x83697fd0 (stale) CRC-32 checksum value for chunk [0, 0x19d70): Mirror 1: 0x7c5b78de Mirror 3: 0x7c5b78de lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19d70: skipped 000944 1779203280.393605 trunc from 0x019d70 to 0x003bb4 (0x161bb bytes) 000946 1779203280.588633 write 0x03fade thru 0x03ffff (0x522 bytes) 000947 1779203280.594064 MIRROR_RESYNC 3 mirrors 000948 1779203282.072256 trunc from 0x040000 to 0x020b6a (0x1f495 bytes) 000949 1779203282.196612 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20b6b): Mirror 2: 0x7a095a2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20b6b): Mirror 2: 0x7a095a2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20b6b: skipped 000950 1779203282.642991 read 0x000a9a thru 0x00b0f1 (0xa658 bytes) 000952 1779203282.670215 mapread 0x00f605 thru 0x011b5c (0x2558 bytes) 000954 1779203282.729475 mapread 0x01dd2c thru 0x020b6a (0x2e3f bytes) 000955 1779203282.759865 mapread 0x01bd6f thru 0x020b6a (0x4dfc bytes) 000956 1779203282.764888 write 0x007292 thru 0x00ff00 (0x8c6f bytes) 000957 1779203282.791153 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20b6b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20b6b): Mirror 2: 0x7a095a2b (stale) CRC-32 checksum value for chunk [0, 0x20b6b): Mirror 1: 0x40987c8c Mirror 3: 0x40987c8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20b6b: skipped 000958 1779203283.761936 read 0x012fe4 thru 0x0193e5 (0x6402 bytes) 000961 1779203283.793343 mapread 0x0067ca thru 0x00bfd6 (0x580d bytes) 000963 1779203283.832223 mapread 0x020b04 thru 0x020b6a (0x67 bytes) 000968 1779203283.847855 write 0x008310 thru 0x010308 (0x7ff9 bytes) 000970 1779203283.948901 read 0x00112f thru 0x00d4b7 (0xc389 bytes) 000971 1779203283.949645 write 0x029a7b thru 0x02e22b (0x47b1 bytes) 000972 1779203283.960241 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e22c): Mirror 3: 0xeb52be7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e22c): Mirror 3: 0xeb52be7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e22c: skipped 000975 1779203284.264159 mapwrite 0x019b60 thru 0x021cdc (0x817d bytes) 000976 1779203284.383614 mapread 0x026884 thru 0x02e22b (0x79a8 bytes) 000978 1779203284.395949 mapwrite 0x01aa25 thru 0x021b93 (0x716f bytes) 000979 1779203284.444358 write 0x037095 thru 0x03db07 (0x6a73 bytes) 000981 1779203284.446325 mapwrite 0x02b24c thru 0x02e223 (0x2fd8 bytes) 000982 1779203284.492938 mapread 0x006dc6 thru 0x007c92 (0xecd bytes) 000987 1779203284.514707 mapwrite 0x014251 thru 0x015333 (0x10e3 bytes) 000988 1779203284.548481 mapwrite 0x00bfec thru 0x0190fb (0xd110 bytes) 000989 1779203284.711708 MIRROR_RESYNC 2 mirrors 000991 1779203285.735950 write 0x03485b thru 0x03dc07 (0x93ad bytes) 000992 1779203285.948365 write 0x013b6f thru 0x01b8ec (0x7d7e bytes) 000993 1779203285.980879 read 0x0153f5 thru 0x020f9f (0xbbab bytes) 000995 1779203286.017518 mapread 0x01f2ff thru 0x02d572 (0xe274 bytes) 000998 1779203286.046247 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3dc08): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc08): Mirror 3: 0x1b13310 (stale) CRC-32 checksum value for chunk [0, 0x3dc08): Mirror 1: 0xea460299 Mirror 4: 0xea460299 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc08: skipped 000999 1779203287.183358 mapread 0x002da9 thru 0x00b62d (0x8885 bytes) 001000 1779203287.228151 mapread 0x0262e8 thru 0x02bf01 (0x5c1a bytes) All operations completed A-OK! PASS 70a (112s) == sanity-flr test 71: check mirror extend parent fid ==== 11:08:13 (1779203293) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 11:08:17 (1779203297) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.358789 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x275:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195014 s, 5.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x275:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x274:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x275:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x276:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 11:08:27 (1779203307) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2783616 .. Extending file size to 6480992 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7739360 .. Extending file size to 7785024 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8531680 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9224000 .. Extending file size to 8667520 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8733152 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56554 56556 56557 56558 56560 failed 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) 192.168.203.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:09:52 (1779203392) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1111808 .. Extending file size to 2715648 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6217696 .. Extending file size to 4414368 .. Extending file size to 7352000 .. Extending file size to 7895584 .. Extending file size to 9015936 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9123008 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9171328 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9179200 .. Waiting 58174 58175 58176 58177 58182 done 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) 192.168.203.135@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:11:10 (1779203470) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0948731 s, 22.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.461262 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1ecd6004 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:11:30 (1779203490) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156538 s, 13.4 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 11:11:39 (1779203499) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:11:47 (1779203507) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203486 s, 10.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:11:54 (1779203514) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.300174 s, 7.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:12:03 (1779203523) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.391484 s, 5.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:12:37 (1779203557) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169878 s, 6.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:13:35 (1779203615) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265456 s, 7.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:14:33 (1779203673) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. Creating new pool test_204f oleg335-server: Pool lustre.test_204f created Adding targets to pool oleg335-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg335-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.320387 s, 6.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg335-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg335-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg335-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:15:56 (1779203756) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:16:05 (1779203765) PASS 205b (12s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:16:17 (1779203777) oleg335-server: Pool lustre.test_206 created Adding targets to pool oleg335-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg335-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg335-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg335-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg335-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:16:47 (1779203807) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873234 s, 12.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626016 s, 16.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9fbf83b8 Mirror 3: 0x9fbf83b8 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:16:59 (1779203819) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:17:03 (1779203823) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:17:07 (1779203827) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.134122 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x9f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1090000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1090000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1090000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1320000 Read this (4 bytes): BBBB 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:17:46 (1779203866) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00772285 s, 1.1 MB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:17:53 (1779203873) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606761 s, 17.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c9 lmm_fid: [0x200000404:0x1c9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c9 lmm_fid: [0x200000404:0x1c9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834377 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-19 11:17:54.000000000 -0400 Modify: 2026-05-19 11:17:55.000000000 -0400 Change: 2026-05-19 11:17:56.000000000 -0400 Birth: 2026-05-19 11:17:56.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:18:03 (1779203883) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:18:06 (1779203886) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.738864 s, 55.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0472854 s, 86.6 kB/s size after second write -rw-r--r-- 1 root root 4096 May 19 11:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 19 11:18 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:18:17 (1779203897) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:18:27 (1779203907) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (14s) == sanity-flr test complete, duration 1811 sec =========== 11:18:40 (1779203920) === sanity-flr: start cleanup 11:18:42 (1779203922) === === sanity-flr: finish cleanup 11:18:49 (1779203929) ===