-----============= acceptance-small: sanity-flr ============----- Mon Jun 1 12:34:16 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg344-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:34:35 (1780331675) === oleg344-client.virtnet: executing check_config_client /mnt/lustre oleg344-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg344-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9547d0b12000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9547d0b12000.idle_timeout=debug disable quota as required oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:35:02 (1780331702) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7622] == sanity-flr test 0a: lfs mirror create with -N option == 12:35:04 (1780331704) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:35:19 (1780331719) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:35:22 (1780331722) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:35:25 (1780331725) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:35:49 (1780331749) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:35:51 (1780331751) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:35:55 (1780331755) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9547d0b12000.stats=clear osc.lustre-OST0001-osc-ffff9547d0b12000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:36:04 (1780331764) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:36:15 (1780331775) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:36:22 (1780331782) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:36:25 (1780331785) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:36:32 (1780331792) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 12:36:41 (1780331801) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:36:48 (1780331808) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:36:51 (1780331811) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:36:57 (1780331817) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:37:07 (1780331827) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:37:13 (1780331833) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 2.01238 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151182 s, 6.9 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:37:24 (1780331844) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.183404 s, 45.7 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:37:31 (1780331851) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:37:40 (1780331860) Stopping /mnt/lustre-ost1 (opts:) on oleg344-server oleg344-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9547d0b12000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9547d0b12000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg344-server: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg344-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg344-server oleg344-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9547d0b12000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9547d0b12000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg344-server: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg344-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (60s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:38:41 (1780331921) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.521683 s, 22.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg344-server oleg344-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9547d0b12000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9547d0b12000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg344-server: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg344-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (30s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:39:11 (1780331951) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg344-server oleg344-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9547d0b12000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9547d0b12000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg344-server: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg344-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg344-server oleg344-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9547d0b12000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9547d0b12000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg344-server: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg344-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (58s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:40:09 (1780332009) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.30506 s, 12.2 MB/s -rw-r--r-- 1 root root 52428800 Jun 1 12:40 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.988s user 0m0.001s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg344-server oleg344-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9547d0b12000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9547d0b12000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.037s user 0m0.000s sys 0m0.201s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg344-server: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg344-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg344-server oleg344-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9547d0b12000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9547d0b12000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.753s user 0m0.001s sys 0m0.181s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg344-server: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg344-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9547d0b12000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9547d0b12000.ost_server_uuid in FULL state after 0 sec PASS 33b (66s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:41:15 (1780332075) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:41:18 (1780332078) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:41:21 (1780332081) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 12:41:26 (1780332086) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 12:41:32 (1780332092) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.597439 s, 21.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:41:42 (1780332102) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431922 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280852 s, 37.3 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:41:49 (1780332109) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:41:56 (1780332116) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00174503 s, 587 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00164056 s, 624 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110525 s, 926 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107587 s, 952 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 12:42:05 (1780332125) create test files with size 15740110 .. b64dd224093606f6f3c39debcc433a52 - 1bdf95227d5a9cd03b5cf972dcec2326 - 34daf9c6adf59d7379c3407f4a054a64 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 12:42:31 (1780332151) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:42:48 (1780332168) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:42:56 (1780332176) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11956 s, 8.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209024 s, 50.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 12:43:05 (1780332185) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150213 s, 14.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.384418 s, 10.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229968 s, 9.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 12:43:21 (1780332201) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:43:24 (1780332204) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:43:27 (1780332207) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289572 s, 36.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 12:43:33 (1780332213) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:43:36 (1780332216) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:43:46 (1780332226) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.24473 s, 42.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:43:59 (1780332239) llite.lustre-ffff9547c557b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.338296 s, 31.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg344-client.virtnet /mnt/lustre (opts:) Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9547c2b3c000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:44:12 (1780332252) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:44:47 (1780332287) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:45:18 (1780332318) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:45:20 (1780332320) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.249878 s, 12.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.158448 s, 19.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x50d3ba88 (nosync,stale) Mirror 2: 0x50d3ba88 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1fd761dff17ae6f5d81c3afc425fbf39 - mirror 1 checksum: 2bd23305b7ed765fc4865460aa26c6ca - mirror 2 checksum: 1fd761dff17ae6f5d81c3afc425fbf39 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 12:45:23 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 12:45:23 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1fd761dff17ae6f5d81c3afc425fbf39 - mirror 1 checksum: 2bd23305b7ed765fc4865460aa26c6ca - mirror 2 checksum: 2bd23305b7ed765fc4865460aa26c6ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 12:45:23 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 12:45:23 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:45:31 (1780332331) Failing mds1 on oleg344-server Stopping /mnt/lustre-mds1 (opts:) on oleg344-server 12:45:37 (1780332337) shut down facet: mds1 facet_host: oleg344-server facet_failover_host: oleg344-server Failover mds1 to oleg344-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0000 12:46:02 (1780332362) targets are mounted 12:46:02 (1780332362) facet_failover done oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686076 s, 15.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:46:19 (1780332379) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:46:22 (1780332382) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:46:26 (1780332386) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0110501 s, 1.5 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:46:33 (1780332393) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:46:37 (1780332397) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:46:40 (1780332400) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:47:02 (1780332422) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:47:29 (1780332449) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:47:52 (1780332472) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:48:16 (1780332496) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 12:48:24 (1780332504) mirror create pid 53547 mirror split pid 53548 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53547 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53548 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 12:49:31 (1780332571) Seed set to 2574 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x170eb 000001 1780332574.615652 trunc from 00000000 to 0x0170ea (0x170eb bytes) 000005 1780332574.655236 mapwrite 0x011501 thru 0x01b852 (0xa352 bytes) 000006 1780332574.796219 write 0x01707d thru 0x025fdc (0xef60 bytes) 000008 1780332574.801075 read 0x00a03b thru 0x017868 (0xd82e bytes) 000009 1780332574.808613 mapwrite 0x00dbbd thru 0x00ea89 (0xecd bytes) truncating to largest ever: 0x28a81 000010 1780332574.898012 trunc from 0x025fdd to 0x028a80 (0x2aa4 bytes) 000013 1780332574.937438 read 0x001ff5 thru 0x00bc36 (0x9c42 bytes) 000014 1780332574.968764 mapread 0x01571d thru 0x024894 (0xf178 bytes) 000015 1780332574.970555 write 0x035a54 thru 0x03ffff (0xa5ac bytes) 000017 1780332574.982552 trunc from 0x040000 to 0x00f075 (0x30f8a bytes) truncating to largest ever: 0x3517f 000019 1780332575.012535 trunc from 0x00f076 to 0x03517e (0x26109 bytes) 000020 1780332575.040087 mapread 0x00a12c thru 0x00d878 (0x374d bytes) 000021 1780332575.045984 read 0x0265d4 thru 0x02b1bd (0x4bea bytes) 000022 1780332575.086042 mapwrite 0x033427 thru 0x0388f9 (0x54d3 bytes) 000023 1780332575.233010 trunc from 0x0388fa to 0x02e63d (0xa2bc bytes) 000027 1780332575.288524 read 0x0168a9 thru 0x017a18 (0x1170 bytes) 000028 1780332575.289235 read 0x0003d7 thru 0x002dc3 (0x29ed bytes) 000029 1780332575.289850 write 0x03b53a thru 0x03ffff (0x4ac6 bytes) 000031 1780332575.295867 mapwrite 0x01acd8 thru 0x01dcc8 (0x2ff1 bytes) 000034 1780332575.419650 write 0x0086a7 thru 0x00d622 (0x4f7c bytes) 000037 1780332575.441500 mapwrite 0x016272 thru 0x0241dc (0xdf6b bytes) 000039 1780332575.644994 write 0x0021f2 thru 0x00dbd6 (0xb9e5 bytes) 000040 1780332575.650270 read 0x01e4a7 thru 0x02c833 (0xe38d bytes) 000042 1780332575.657281 mapread 0x03cee5 thru 0x03ffff (0x311b bytes) 000044 1780332575.665742 trunc from 0x040000 to 0x02779e (0x18861 bytes) 000046 1780332575.707471 write 0x029f7d thru 0x0335c2 (0x9646 bytes) 000047 1780332575.711500 mapwrite 0x0349d0 thru 0x03990c (0x4f3d bytes) 000049 1780332575.845637 mapread 0x002d6b thru 0x010fb9 (0xe24f bytes) 000050 1780332575.847994 write 0x01d065 thru 0x026e60 (0x9dfc bytes) 000051 1780332575.853640 read 0x0397a8 thru 0x03990c (0x165 bytes) 000052 1780332575.854257 write 0x033fa0 thru 0x037571 (0x35d2 bytes) 000054 1780332575.860051 read 0x0051f3 thru 0x00bc48 (0x6a56 bytes) 000055 1780332575.860835 mapwrite 0x013c2e thru 0x019660 (0x5a33 bytes) 000058 1780332575.987736 MIRROR_EXTEND to 2 mirrors 000059 1780332576.994433 MIRROR_RESYNC 2 mirrors 000060 1780332577.697413 read 0x02c933 thru 0x02e7dd (0x1eab bytes) 000063 1780332577.732217 write 0x039f08 thru 0x03f7d1 (0x58ca bytes) 000065 1780332577.843672 write 0x038d08 thru 0x03bbb7 (0x2eb0 bytes) 000066 1780332577.851113 mapread 0x015d79 thru 0x01dbb0 (0x7e38 bytes) 000067 1780332577.863911 trunc from 0x03f7d2 to 0x0106fb (0x2f0d6 bytes) 000068 1780332577.904566 trunc from 0x0106fc to 0x02be6e (0x1b773 bytes) 000069 1780332577.937656 write 0x005036 thru 0x01167b (0xc646 bytes) 000073 1780332577.960238 mapread 0x0188e4 thru 0x01b217 (0x2934 bytes) 000074 1780332577.969948 mapread 0x0260e2 thru 0x02bb6e (0x5a8d bytes) 000075 1780332577.978625 trunc from 0x02be6f to 0x02c1f8 (0x38a bytes) 000077 1780332578.000149 write 0x008dd3 thru 0x00f060 (0x628e bytes) 000078 1780332578.010913 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000080 1780332578.242102 read 0x004117 thru 0x011948 (0xd832 bytes) 000081 1780332578.253931 read 0x010585 thru 0x01a07c (0x9af8 bytes) 000082 1780332578.273675 write 0x0150d5 thru 0x01db4d (0x8a79 bytes) 000083 1780332578.278179 mapread 0x016686 thru 0x01fe5a (0x97d5 bytes) truncating to largest ever: 0x385a9 000087 1780332578.292075 trunc from 0x02c1f9 to 0x0385a8 (0xc3b0 bytes) 000088 1780332578.334925 trunc from 0x0385a9 to 0x0367e5 (0x1dc3 bytes) 000089 1780332578.364379 trunc from 0x0367e6 to 0x006416 (0x303cf bytes) 000090 1780332578.395229 MIRROR_EXTEND to 2 mirrors 000091 1780332579.325846 read 0x0038bf thru 0x006416 (0x2b58 bytes) 000093 1780332579.350381 write 0x03d0ad thru 0x03d35b (0x2af bytes) 000096 1780332579.508823 read 0x030d1a thru 0x03357f (0x2866 bytes) 000098 1780332579.536554 write 0x000ce6 thru 0x002e84 (0x219f bytes) 000099 1780332579.537562 mapwrite 0x01695c thru 0x0221a7 (0xb84c bytes) 000101 1780332579.655709 trunc from 0x03d35c to 0x001467 (0x3bef4 bytes) 000103 1780332579.725298 mapread 0x0005a0 thru 0x001467 (0xec8 bytes) 000104 1780332579.733730 MIRROR_RESYNC 2 mirrors 000105 1780332580.528244 write 0x000cb6 thru 0x00f6f4 (0xea3f bytes) 000106 1780332580.684194 mapread 0x004aa3 thru 0x007b06 (0x3064 bytes) 000107 1780332580.687153 write 0x031ff7 thru 0x03bec4 (0x9ece bytes) 000110 1780332580.692077 mapwrite 0x01b219 thru 0x01cbcc (0x19b4 bytes) 000111 1780332580.800774 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bec5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bec5): Mirror 2: 0x7e380648 (stale) CRC-32 checksum value for chunk [0, 0x3bec5): Mirror 1: 0x92d16ebd Mirror 3: 0x92d16ebd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bec5: skipped 000112 1780332582.006625 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bec5): Mirror 2: 0x7e380648 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bec5): Mirror 2: 0x7e380648 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bec5: skipped 000114 1780332582.330403 read 0x01ef01 thru 0x023b3b (0x4c3b bytes) 000115 1780332582.349087 mapwrite 0x02a7fe thru 0x033f77 (0x977a bytes) 000117 1780332582.540744 mapwrite 0x03aaa1 thru 0x03e248 (0x37a8 bytes) 000118 1780332582.714600 write 0x0185e9 thru 0x02204b (0x9a63 bytes) 000121 1780332582.737862 mapread 0x017950 thru 0x02386a (0xbf1b bytes) 000123 1780332582.764980 MIRROR_RESYNC 2 mirrors 000124 1780332583.921740 mapwrite 0x02b310 thru 0x0373dc (0xc0cd bytes) 000125 1780332584.211409 read 0x020972 thru 0x02ce37 (0xc4c6 bytes) 000126 1780332584.232060 trunc from 0x03e249 to 0x00cf25 (0x31323 bytes) 000127 1780332584.273394 MIRROR_RESYNC 2 mirrors 000128 1780332584.897985 mapwrite 0x02f76c thru 0x03b501 (0xbd96 bytes) 000129 1780332585.257170 read 0x009d75 thru 0x00b643 (0x18cf bytes) 000130 1780332585.267438 mapwrite 0x034927 thru 0x03ffff (0xb6d9 bytes) 000131 1780332585.419144 mapread 0x008a8e thru 0x012e5c (0xa3cf bytes) 000133 1780332585.470693 mapwrite 0x01b20d thru 0x027614 (0xc408 bytes) 000134 1780332585.605708 mapwrite 0x01b49f thru 0x028ffb (0xdb5d bytes) 000135 1780332585.711449 MIRROR_RESYNC 2 mirrors 000136 1780332586.400019 read 0x001418 thru 0x00b4d8 (0xa0c1 bytes) 000138 1780332586.417262 mapread 0x00130d thru 0x00c86c (0xb560 bytes) 000139 1780332586.427687 mapwrite 0x036af0 thru 0x03ffff (0x9510 bytes) 000141 1780332586.670953 mapwrite 0x02e8e2 thru 0x0393b9 (0xaad8 bytes) 000145 1780332586.812574 mapwrite 0x0375df thru 0x03ffff (0x8a21 bytes) 000146 1780332586.852579 read 0x00944c thru 0x00bcf8 (0x28ad bytes) 000147 1780332586.885001 MIRROR_RESYNC 2 mirrors 000148 1780332587.805073 read 0x020187 thru 0x028fc5 (0x8e3f bytes) 000150 1780332587.855603 trunc from 0x040000 to 0x03268f (0xd970 bytes) 000152 1780332588.059842 trunc from 0x032690 to 0x000f1b (0x31774 bytes) 000155 1780332588.124628 MIRROR_RESYNC 2 mirrors 000156 1780332588.767055 trunc from 0x000f1c to 0x025b75 (0x24c5a bytes) 000157 1780332588.929769 read 0x004b74 thru 0x011f39 (0xd3c6 bytes) 000158 1780332588.956154 read 0x00561f thru 0x00c358 (0x6d3a bytes) 000159 1780332588.957160 trunc from 0x025b76 to 0x032a16 (0xcea1 bytes) truncating to largest ever: 0x3ec2a 000160 1780332588.995358 trunc from 0x032a17 to 0x03ec29 (0xc213 bytes) 000161 1780332589.016999 write 0x018922 thru 0x0216dd (0x8dbc bytes) 000162 1780332589.039703 write 0x0074ac thru 0x00f213 (0x7d68 bytes) 000163 1780332589.040883 write 0x019ca4 thru 0x01d936 (0x3c93 bytes) 000164 1780332589.053518 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000167 1780332589.238059 read 0x01acba thru 0x025251 (0xa598 bytes) 000169 1780332589.258551 MIRROR_EXTEND to 2 mirrors 000170 1780332590.179841 read 0x01e00f thru 0x02ab72 (0xcb64 bytes) 000171 1780332590.199920 mapwrite 0x022723 thru 0x02f173 (0xca51 bytes) 000174 1780332590.507340 mapread 0x030648 thru 0x031af5 (0x14ae bytes) 000175 1780332590.519576 mapwrite 0x01fe61 thru 0x02822a (0x83ca bytes) 000178 1780332590.603244 write 0x01002e thru 0x010656 (0x629 bytes) 000180 1780332590.629596 mapread 0x002345 thru 0x00f232 (0xceee bytes) 000181 1780332590.653896 write 0x03a3fd thru 0x03ffff (0x5c03 bytes) 000184 1780332590.686005 write 0x01c2f6 thru 0x01de2a (0x1b35 bytes) 000186 1780332590.698576 mapread 0x017534 thru 0x01c38b (0x4e58 bytes) 000189 1780332590.702381 mapread 0x027cd0 thru 0x031408 (0x9739 bytes) 000192 1780332590.713350 trunc from 0x040000 to 0x033581 (0xca7e bytes) 000194 1780332590.778000 mapread 0x011cf8 thru 0x01e762 (0xca6b bytes) 000196 1780332590.781959 write 0x029b18 thru 0x035e17 (0xc300 bytes) 000198 1780332590.786013 trunc from 0x035e18 to 0x039c3c (0x3e25 bytes) 000199 1780332590.803972 read 0x001491 thru 0x0073f9 (0x5f69 bytes) 000203 1780332590.810882 read 0x0091b4 thru 0x0167bf (0xd60c bytes) 000206 1780332590.820574 mapread 0x00c7cb thru 0x00e69a (0x1ed0 bytes) 000207 1780332590.822105 mapread 0x008dcb thru 0x00c748 (0x397e bytes) 000208 1780332590.824207 read 0x032bbd thru 0x03367b (0xabf bytes) 000210 1780332590.825480 read 0x02b43f thru 0x03340c (0x7fce bytes) 000212 1780332590.830040 trunc from 0x039c3d to 0x01c2b1 (0x1d98b bytes) 000214 1780332590.852701 read 0x010317 thru 0x01c2b1 (0xbf9b bytes) 000215 1780332590.853486 read 0x00dd7a thru 0x01934c (0xb5d3 bytes) 000216 1780332590.854863 write 0x01f9ce thru 0x026ca8 (0x72db bytes) 000217 1780332590.856947 mapwrite 0x00db74 thru 0x00ff49 (0x23d6 bytes) 000219 1780332590.909754 mapread 0x019ec5 thru 0x024f4a (0xb086 bytes) 000220 1780332590.920855 mapread 0x014229 thru 0x017550 (0x3328 bytes) 000221 1780332590.928655 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000222 1780332591.123788 trunc from 0x026ca9 to 0x035eba (0xf212 bytes) 000223 1780332591.149338 write 0x02b929 thru 0x039544 (0xdc1c bytes) 000224 1780332591.159020 write 0x019ef9 thru 0x0245a7 (0xa6af bytes) 000225 1780332591.172510 read 0x006739 thru 0x00bdc5 (0x568d bytes) 000229 1780332591.190807 mapread 0x00af6f thru 0x0172cc (0xc35e bytes) 000231 1780332591.214939 trunc from 0x039545 to 0x027c37 (0x1190d bytes) 000232 1780332591.234708 mapread 0x01e18e thru 0x027c37 (0x9aaa bytes) 000235 1780332591.247721 mapread 0x02537f thru 0x027c37 (0x28b9 bytes) 000237 1780332591.250550 write 0x033813 thru 0x03ffff (0xc7ed bytes) 000238 1780332591.252923 read 0x02ac59 thru 0x02f630 (0x49d8 bytes) 000240 1780332591.266853 read 0x02d86f thru 0x03d21a (0xf9ac bytes) 000241 1780332591.267963 read 0x02c620 thru 0x02ef28 (0x2909 bytes) 000244 1780332591.274678 MIRROR_EXTEND to 2 mirrors 000246 1780332592.021332 write 0x00ceb7 thru 0x01ad54 (0xde9e bytes) 000247 1780332592.212668 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000249 1780332592.520809 trunc from 0x040000 to 0x002399 (0x3dc66 bytes) 000250 1780332592.578749 mapwrite 0x029e70 thru 0x02d2cf (0x3460 bytes) 000252 1780332592.773446 mapread 0x02878c thru 0x02d2cf (0x4b44 bytes) 000253 1780332592.778235 read 0x007d95 thru 0x00d789 (0x59f5 bytes) 000254 1780332592.797685 mapread 0x017ac3 thru 0x0230ab (0xb5e9 bytes) 000255 1780332592.818828 read 0x00f778 thru 0x016e53 (0x76dc bytes) 000258 1780332592.831069 read 0x00f3c5 thru 0x01e4af (0xf0eb bytes) 000260 1780332592.833040 mapread 0x01ab74 thru 0x0209d6 (0x5e63 bytes) 000261 1780332592.835756 trunc from 0x02d2d0 to 0x02e22c (0xf5d bytes) 000262 1780332592.853437 write 0x008852 thru 0x015787 (0xcf36 bytes) 000269 1780332592.868188 MIRROR_EXTEND to 2 mirrors 000270 1780332593.945271 mapread 0x024d92 thru 0x02e22c (0x949b bytes) 000274 1780332594.023119 mapread 0x01162a thru 0x019893 (0x826a bytes) 000275 1780332594.092719 mapwrite 0x024daf thru 0x02cb03 (0x7d55 bytes) 000280 1780332594.411727 trunc from 0x02e22d to 0x00f768 (0x1eac4 bytes) 000282 1780332594.476107 read 0x001e7d thru 0x00f768 (0xd8ec bytes) 000284 1780332594.494524 mapread 0x004f0f thru 0x0083e5 (0x34d7 bytes) 000285 1780332594.496146 trunc from 0x00f769 to 0x022e67 (0x136ff bytes) 000288 1780332594.539477 mapread 0x0150bf thru 0x022d56 (0xdc98 bytes) 000290 1780332594.596838 write 0x01a544 thru 0x01bdfc (0x18b9 bytes) 000291 1780332594.615207 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22e68): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e68): Mirror 2: 0xe8aeb209 (stale) CRC-32 checksum value for chunk [0, 0x22e68): Mirror 1: 0xc525c588 Mirror 3: 0xc525c588 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e68: skipped 000293 1780332595.790030 MIRROR_RESYNC 3 mirrors 000295 1780332596.686281 write 0x007db6 thru 0x013616 (0xb861 bytes) 000299 1780332596.783649 trunc from 0x022e68 to 0x035977 (0x12b10 bytes) 000300 1780332596.817196 mapwrite 0x00e452 thru 0x015d04 (0x78b3 bytes) 000301 1780332596.898636 trunc from 0x035978 to 0x0163da (0x1f59d bytes) 000302 1780332596.950891 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x163db): Mirror 3: 0x76df21f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x163db): Mirror 3: 0x76df21f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x163db: skipped 000305 1780332597.557731 mapread 0x01188d thru 0x014316 (0x2a8a bytes) 000306 1780332597.625008 mapread 0x009b77 thru 0x014129 (0xa5b3 bytes) 000307 1780332597.660799 mapwrite 0x01bd50 thru 0x02597d (0x9c2e bytes) 000308 1780332597.936262 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2597e): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2597e): Mirror 3: 0x37017237 (stale) CRC-32 checksum value for chunk [0, 0x2597e): Mirror 1: 0xf15cd491 Mirror 4: 0xf15cd491 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2597e: skipped 000309 1780332599.159844 write 0x009eba thru 0x012caf (0x8df6 bytes) 000310 1780332599.283743 mapwrite 0x03db87 thru 0x03ffff (0x2479 bytes) 000312 1780332599.438164 MIRROR_RESYNC 3 mirrors 000314 1780332600.825444 write 0x027837 thru 0x029ee4 (0x26ae bytes) 000315 1780332600.895820 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa29198d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa29198d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000316 1780332601.341383 write 0x00050b thru 0x0032f7 (0x2ded bytes) 000317 1780332601.352025 write 0x00aba5 thru 0x019292 (0xe6ee bytes) 000318 1780332601.366844 mapread 0x02aff8 thru 0x02e046 (0x304f bytes) 000319 1780332601.390275 write 0x014edc thru 0x022f0c (0xe031 bytes) 000321 1780332601.401967 trunc from 0x040000 to 0x03b3ea (0x4c15 bytes) 000322 1780332601.434355 read 0x02b60b thru 0x0369f2 (0xb3e8 bytes) 000323 1780332601.442461 read 0x00aa27 thru 0x01401c (0x95f6 bytes) 000325 1780332601.456510 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b3eb): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b3eb): Mirror 4: 0x1d6fd0fb (stale) CRC-32 checksum value for chunk [0, 0x3b3eb): Mirror 1: 0xc2bcf266 Mirror 5: 0xc2bcf266 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b3eb: skipped 000326 1780332602.530061 mapread 0x015a18 thru 0x02308c (0xd675 bytes) 000327 1780332602.648998 mapwrite 0x00dd6f thru 0x01570e (0x79a0 bytes) 000328 1780332603.036232 trunc from 0x03b3eb to 0x01ce48 (0x1e5a2 bytes) 000335 1780332603.134121 mapwrite 0x0136fc thru 0x02183e (0xe143 bytes) 000338 1780332603.323200 mapwrite 0x002c93 thru 0x00e5a8 (0xb916 bytes) 000339 1780332603.462812 mapwrite 0x039983 thru 0x03fec2 (0x6540 bytes) 000342 1780332603.600358 read 0x015504 thru 0x01984f (0x434c bytes) 000343 1780332603.601018 read 0x00dc73 thru 0x012a2b (0x4db9 bytes) 000344 1780332603.603023 mapwrite 0x015d28 thru 0x021624 (0xb8fd bytes) 000345 1780332603.695237 mapread 0x005a00 thru 0x005dfa (0x3fb bytes) 000349 1780332603.706818 MIRROR_RESYNC 3 mirrors 000351 1780332604.993488 trunc from 0x03fec3 to 0x03714a (0x8d78 bytes) 000353 1780332605.148890 write 0x026903 thru 0x030ca3 (0xa3a1 bytes) 000355 1780332605.195561 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3714b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3714b): Mirror 4: 0x5832baa8 (stale) Mirror 5: 0x5832baa8 (stale) CRC-32 checksum value for chunk [0, 0x3714b): Mirror 1: 0x869f00dc Mirror 6: 0x869f00dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3714b: skipped 000356 1780332606.220088 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3714b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3714b): Mirror 4: 0x5832baa8 (stale) Mirror 5: 0x5832baa8 (stale) CRC-32 checksum value for chunk [0, 0x3714b): Mirror 1: 0x869f00dc Mirror 6: 0x869f00dc Mirror 7: 0x869f00dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3714b: skipped 000357 1780332607.206171 trunc from 0x03714b to 0x0249ac (0x1279e bytes) 000359 1780332607.355482 mapread 0x01c18f thru 0x0249ac (0x881e bytes) 000360 1780332607.404189 mapread 0x01d766 thru 0x0249ac (0x7247 bytes) 000361 1780332607.405532 trunc from 0x0249ad to 0x00977f (0x1b22d bytes) 000363 1780332607.439206 read 0x0050ff thru 0x00977f (0x4681 bytes) 000365 1780332607.453062 MIRROR_RESYNC 5 mirrors 000366 1780332608.553270 MIRROR_RESYNC 5 mirrors 000369 1780332609.238077 mapwrite 0x01649d thru 0x01a8e6 (0x444a bytes) 000371 1780332609.462395 trunc from 0x01a8e7 to 0x0098b4 (0x11032 bytes) 000373 1780332609.498071 write 0x03d8d2 thru 0x03ffff (0x272e bytes) 000374 1780332609.506098 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x709c8d1c (stale) Mirror 5: 0x709c8d1c (stale) Mirror 6: 0x709c8d1c (stale) Mirror 7: 0x709c8d1c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x37930620 Mirror 8: 0x37930620 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000375 1780332610.849034 mapwrite 0x02fd74 thru 0x03fbb8 (0xfe45 bytes) 000376 1780332611.111980 trunc from 0x040000 to 0x01c609 (0x239f6 bytes) 000377 1780332611.182271 mapwrite 0x0236f4 thru 0x0244a7 (0xdb4 bytes) 000378 1780332611.313294 write 0x01f6c9 thru 0x022914 (0x324c bytes) 000379 1780332611.331725 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x244a8): Mirror 4: 0xc4e8ca84 (stale) Mirror 5: 0xc4e8ca84 (stale) Mirror 6: 0xc4e8ca84 (stale) Mirror 8: 0xc4e8ca84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x244a8): Mirror 4: 0xc4e8ca84 (stale) Mirror 5: 0xc4e8ca84 (stale) Mirror 6: 0xc4e8ca84 (stale) Mirror 8: 0xc4e8ca84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x244a8: skipped 000381 1780332611.916897 read 0x0202d8 thru 0x0244a7 (0x41d0 bytes) 000382 1780332611.929505 mapwrite 0x030b12 thru 0x03f006 (0xe4f5 bytes) 000383 1780332612.142889 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f007): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f007): Mirror 4: 0xd13311f7 (stale) Mirror 5: 0xd13311f7 (stale) Mirror 6: 0xd13311f7 (stale) Mirror 8: 0xe27a604b (stale) CRC-32 checksum value for chunk [0, 0x3f007): Mirror 1: 0xf60e0bba Mirror 9: 0xf60e0bba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f007: skipped 000384 1780332613.431055 mapread 0x02a038 thru 0x031bc0 (0x7b89 bytes) 000386 1780332613.500875 mapwrite 0x00fcc7 thru 0x01b34c (0xb686 bytes) 000388 1780332613.858136 mapread 0x005953 thru 0x00b3e6 (0x5a94 bytes) 000389 1780332613.870967 mapread 0x035e77 thru 0x03f006 (0x9190 bytes) 000391 1780332613.898077 trunc from 0x03f007 to 0x038a8a (0x657c bytes) 000392 1780332613.938386 mapread 0x0379ad thru 0x038a8a (0x10de bytes) 000393 1780332613.940661 mapread 0x032289 thru 0x038a8a (0x6802 bytes) 000394 1780332613.942689 mapwrite 0x01262e thru 0x01fc88 (0xd65b bytes) 000396 1780332614.026755 mapwrite 0x01c86b thru 0x024b9d (0x8333 bytes) 000397 1780332614.129335 mapwrite 0x0006cc thru 0x009859 (0x918e bytes) 000398 1780332614.210804 write 0x003991 thru 0x00acf7 (0x7367 bytes) 000400 1780332614.214332 trunc from 0x038a8b to 0x014e2a (0x23c60 bytes) 000401 1780332614.240421 read 0x00e51b thru 0x0116d2 (0x31b8 bytes) 000402 1780332614.241108 trunc from 0x014e2b to 0x016316 (0x14ec bytes) 000403 1780332614.269946 trunc from 0x016317 to 0x00d258 (0x90be bytes) 000405 1780332614.304871 mapread 0x00800e thru 0x00d258 (0x524b bytes) 000406 1780332614.311867 trunc from 0x00d259 to 0x015cd3 (0x8a7b bytes) 000407 1780332614.334767 mapwrite 0x022ce6 thru 0x032990 (0xfcab bytes) 000412 1780332614.462074 read 0x00bd94 thru 0x01526c (0x94d9 bytes) 000413 1780332614.474017 mapread 0x0034bb thru 0x008006 (0x4b4c bytes) 000415 1780332614.475729 write 0x02df08 thru 0x038823 (0xa91c bytes) 000416 1780332614.477811 write 0x014b6c thru 0x0213b0 (0xc845 bytes) 000418 1780332614.481122 trunc from 0x038824 to 0x036f09 (0x191a bytes) 000420 1780332614.503118 read 0x001131 thru 0x008485 (0x7355 bytes) 000421 1780332614.513209 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36f0a): Mirror 4: 0xf883f0ba (stale) Mirror 5: 0xf883f0ba (stale) Mirror 8: 0xf883f0ba (stale) Mirror 9: 0xd8c30519 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f0a): Mirror 4: 0xf883f0ba (stale) Mirror 5: 0xf883f0ba (stale) Mirror 8: 0xf883f0ba (stale) Mirror 9: 0xd8c30519 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f0a: skipped 000425 1780332615.114924 mapread 0x02e4a1 thru 0x036f09 (0x8a69 bytes) 000426 1780332615.163979 mapwrite 0x031729 thru 0x036581 (0x4e59 bytes) 000427 1780332615.228934 mapwrite 0x00624b thru 0x0082ab (0x2061 bytes) 000428 1780332615.341746 mapread 0x024dbe thru 0x033d73 (0xefb6 bytes) 000432 1780332615.390029 mapread 0x004e54 thru 0x010332 (0xb4df bytes) 000434 1780332615.391992 mapwrite 0x032207 thru 0x03307c (0xe76 bytes) 000435 1780332615.499332 mapwrite 0x038c54 thru 0x0395e3 (0x990 bytes) 000436 1780332615.594746 mapwrite 0x006300 thru 0x00d59f (0x72a0 bytes) 000438 1780332615.735660 read 0x0038fd thru 0x00e8da (0xafde bytes) 000439 1780332615.736371 trunc from 0x0395e4 to 0x0330fc (0x64e7 bytes) 000442 1780332615.779638 read 0x01e4d6 thru 0x0234d5 (0x5000 bytes) 000445 1780332615.781846 mapread 0x025a95 thru 0x02e81f (0x8d8b bytes) 000449 1780332615.792913 write 0x003534 thru 0x0094af (0x5f7c bytes) 000452 1780332615.802610 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x330fd): Mirror 5: 0x5f4a0a7c (stale) Mirror 8: 0x5f4a0a7c (stale) Mirror 9: 0x93165889 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x330fd): Mirror 5: 0x5f4a0a7c (stale) Mirror 8: 0x5f4a0a7c (stale) Mirror 9: 0x93165889 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x330fd: skipped 000453 1780332616.176370 write 0x0163c2 thru 0x016aaa (0x6e9 bytes) 000454 1780332616.188084 write 0x008ebb thru 0x01088f (0x79d5 bytes) 000456 1780332616.209237 mapread 0x02b54f thru 0x0300e1 (0x4b93 bytes) 000458 1780332616.232245 mapread 0x01add0 thru 0x01c488 (0x16b9 bytes) 000460 1780332616.252751 write 0x00cf1c thru 0x00d11a (0x1ff bytes) 000462 1780332616.254384 mapread 0x00bd76 thru 0x013c2b (0x7eb6 bytes) 000463 1780332616.267254 trunc from 0x0330fd to 0x035fb0 (0x2eb4 bytes) 000464 1780332616.287795 write 0x027f22 thru 0x037c96 (0xfd75 bytes) 000465 1780332616.297148 MIRROR_RESYNC 4 mirrors 000466 1780332617.435908 read 0x0242d9 thru 0x032e68 (0xeb90 bytes) 000468 1780332617.453378 MIRROR_RESYNC 4 mirrors 000469 1780332618.142892 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000470 1780332618.610679 mapwrite 0x006852 thru 0x00e7d7 (0x7f86 bytes) 000471 1780332618.875945 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c97): Mirror 5: 0xeede29d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c97): Mirror 5: 0xeede29d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c97: skipped 000474 1780332619.262905 trunc from 0x037c97 to 0x005423 (0x32873 bytes) 000476 1780332619.292975 trunc from 0x005424 to 0x00d1fb (0x7dd8 bytes) 000479 1780332619.313161 mapread 0x008c52 thru 0x00d1fb (0x45aa bytes) 000480 1780332619.333604 mapread 0x001584 thru 0x002267 (0xce4 bytes) 000481 1780332619.340654 trunc from 0x00d1fc to 0x02256b (0x15370 bytes) 000482 1780332619.365085 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000483 1780332619.492745 write 0x0156e8 thru 0x01f11e (0x9a37 bytes) 000486 1780332619.506220 trunc from 0x02256c to 0x0319ca (0xf45f bytes) 000487 1780332619.528039 write 0x00c2d6 thru 0x00cfec (0xd17 bytes) 000488 1780332619.539724 write 0x03b020 thru 0x03ffff (0x4fe0 bytes) 000489 1780332619.541820 read 0x008b29 thru 0x016b63 (0xe03b bytes) 000490 1780332619.543388 write 0x005df9 thru 0x0083c8 (0x25d0 bytes) 000491 1780332619.545394 trunc from 0x040000 to 0x002de0 (0x3d21f bytes) 000492 1780332619.565010 trunc from 0x002de1 to 0x007a1d (0x4c3d bytes) 000493 1780332619.600622 mapread 0x0064e9 thru 0x007a1d (0x1535 bytes) 000494 1780332619.617073 mapread 0x00633e thru 0x007a1d (0x16e0 bytes) 000495 1780332619.619135 mapread 0x004c3c thru 0x007a1d (0x2de2 bytes) 000497 1780332619.626161 read 0x004195 thru 0x007a1d (0x3889 bytes) 000500 1780332619.635691 read 0x003073 thru 0x007a1d (0x49ab bytes) 000501 1780332619.638622 trunc from 0x007a1e to 0x032ac6 (0x2b0a9 bytes) 000502 1780332619.676808 MIRROR_EXTEND to 2 mirrors 000503 1780332620.433045 trunc from 0x032ac7 to 0x021934 (0x11192 bytes) 000504 1780332620.537645 read 0x01fb2e thru 0x02179d (0x1c70 bytes) 000507 1780332620.550679 read 0x00bb1d thru 0x012522 (0x6a06 bytes) 000508 1780332620.591822 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000510 1780332620.800499 trunc from 0x021935 to 0x03d606 (0x1bcd2 bytes) 000512 1780332620.820713 read 0x02dc82 thru 0x033ccf (0x604e bytes) 000513 1780332620.842850 trunc from 0x03d607 to 0x03087c (0xcd8a bytes) 000515 1780332620.884175 write 0x02030a thru 0x02f49f (0xf196 bytes) 000517 1780332620.900814 mapwrite 0x037985 thru 0x039b95 (0x2211 bytes) 000518 1780332620.986912 write 0x02b053 thru 0x032cd7 (0x7c85 bytes) 000520 1780332620.990535 write 0x03a9ba thru 0x03ffff (0x5646 bytes) 000521 1780332620.992043 trunc from 0x040000 to 0x021fe8 (0x1e017 bytes) 000523 1780332621.009438 write 0x015720 thru 0x01c5ca (0x6eab bytes) 000524 1780332621.025089 mapread 0x00ce30 thru 0x019491 (0xc662 bytes) 000525 1780332621.037671 mapread 0x005b3c thru 0x009298 (0x375d bytes) 000526 1780332621.038952 mapwrite 0x01f507 thru 0x0223b9 (0x2eb3 bytes) 000527 1780332621.132458 MIRROR_EXTEND to 2 mirrors 000528 1780332621.920145 MIRROR_EXTEND to 3 mirrors 000530 1780332622.708014 mapwrite 0x039434 thru 0x03ffff (0x6bcc bytes) 000533 1780332622.868073 mapread 0x010ffb thru 0x01d5f7 (0xc5fd bytes) 000535 1780332622.898468 read 0x0107d8 thru 0x011064 (0x88d bytes) 000536 1780332622.899167 mapread 0x00609e thru 0x0085a1 (0x2504 bytes) 000538 1780332622.916685 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x34c09d88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x34c09d88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000539 1780332623.270359 mapwrite 0x03e3ea thru 0x03faba (0x16d1 bytes) 000541 1780332623.331032 mapread 0x0091d5 thru 0x00e140 (0x4f6c bytes) 000542 1780332623.344757 write 0x028440 thru 0x03368c (0xb24d bytes) 000544 1780332623.361129 mapread 0x015136 thru 0x020304 (0xb1cf bytes) 000545 1780332623.377471 mapread 0x03315d thru 0x036fa9 (0x3e4d bytes) 000547 1780332623.388594 write 0x002047 thru 0x00a0ee (0x80a8 bytes) 000551 1780332623.392668 trunc from 0x040000 to 0x006482 (0x39b7d bytes) 000553 1780332623.412473 read 0x002581 thru 0x006482 (0x3f02 bytes) 000554 1780332623.413235 mapread 0x005110 thru 0x006482 (0x1373 bytes) 000555 1780332623.416474 read 0x003380 thru 0x006482 (0x3103 bytes) 000556 1780332623.422363 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000559 1780332623.558094 mapwrite 0x011243 thru 0x01a3a1 (0x915f bytes) 000560 1780332623.643113 read 0x00bee8 thru 0x018a0f (0xcb28 bytes) 000561 1780332623.650619 mapwrite 0x000536 thru 0x000607 (0xd2 bytes) 000562 1780332623.701116 mapread 0x00fb8c thru 0x01a3a1 (0xa816 bytes) 000565 1780332623.707957 write 0x026e9a thru 0x02f310 (0x8477 bytes) 000566 1780332623.710169 trunc from 0x02f311 to 0x008b54 (0x267bc bytes) 000567 1780332623.727827 read 0x000a66 thru 0x008b54 (0x80ef bytes) 000570 1780332623.730835 trunc from 0x008b55 to 0x03e09b (0x35547 bytes) 000571 1780332623.751080 trunc from 0x03e09c to 0x00ffb6 (0x2e0e5 bytes) 000572 1780332623.772950 mapread 0x00c621 thru 0x00ffb6 (0x3996 bytes) 000573 1780332623.788037 read 0x00472b thru 0x00ffb6 (0xb88c bytes) 000574 1780332623.789068 write 0x0205d5 thru 0x02d138 (0xcb64 bytes) 000575 1780332623.796151 MIRROR_EXTEND to 2 mirrors 000576 1780332624.794409 write 0x016c25 thru 0x021f23 (0xb2ff bytes) 000578 1780332624.935898 write 0x03e8b6 thru 0x03ffff (0x174a bytes) 000580 1780332624.944952 trunc from 0x040000 to 0x01855e (0x27aa1 bytes) 000582 1780332624.991930 mapwrite 0x03574d thru 0x03ffff (0xa8b3 bytes) 000584 1780332625.234915 write 0x002bfb thru 0x0037a7 (0xbad bytes) 000585 1780332625.259132 mapwrite 0x008dee thru 0x011a21 (0x8c34 bytes) 000586 1780332625.345619 MIRROR_RESYNC 2 mirrors 000588 1780332626.234528 mapread 0x02b2a7 thru 0x02f677 (0x43d1 bytes) 000589 1780332626.273277 trunc from 0x040000 to 0x016717 (0x298e8 bytes) 000590 1780332626.430808 trunc from 0x016718 to 0x01b742 (0x502b bytes) 000591 1780332626.463141 mapread 0x005e17 thru 0x0087df (0x29c9 bytes) 000594 1780332626.490793 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000596 1780332626.715047 mapwrite 0x034160 thru 0x03c881 (0x8722 bytes) 000600 1780332626.860798 trunc from 0x03c882 to 0x01260d (0x2a274 bytes) 000603 1780332626.906145 write 0x01c43c thru 0x0236c7 (0x728c bytes) 000605 1780332626.927886 MIRROR_EXTEND to 2 mirrors 000606 1780332628.200664 read 0x000d29 thru 0x00b969 (0xac41 bytes) 000607 1780332628.227178 trunc from 0x0236c8 to 0x02cd82 (0x96bb bytes) 000608 1780332628.364499 write 0x0122cf thru 0x01a707 (0x8439 bytes) 000609 1780332628.385518 mapread 0x0071f3 thru 0x012e29 (0xbc37 bytes) 000610 1780332628.427296 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000611 1780332628.754032 write 0x002d70 thru 0x010c08 (0xde99 bytes) 000614 1780332628.770780 mapread 0x0080bc thru 0x0149be (0xc903 bytes) 000615 1780332628.787480 mapread 0x013033 thru 0x0186c7 (0x5695 bytes) 000616 1780332628.789069 write 0x003a63 thru 0x008727 (0x4cc5 bytes) 000619 1780332628.793021 mapread 0x023d5f thru 0x0292c1 (0x5563 bytes) 000620 1780332628.845063 write 0x02e864 thru 0x030cd9 (0x2476 bytes) 000621 1780332628.846163 trunc from 0x030cda to 0x015fbe (0x1ad1b bytes) 000622 1780332628.870872 mapread 0x008494 thru 0x00ee2e (0x699b bytes) 000625 1780332628.875110 write 0x03310f thru 0x03ffff (0xcef1 bytes) 000626 1780332628.882582 trunc from 0x040000 to 0x02f506 (0x10af9 bytes) 000629 1780332628.905122 mapread 0x0037c2 thru 0x005067 (0x18a6 bytes) 000635 1780332628.910967 mapread 0x029b66 thru 0x02f506 (0x59a1 bytes) 000636 1780332628.928880 mapread 0x022340 thru 0x02f506 (0xd1c7 bytes) 000637 1780332628.931051 mapread 0x022571 thru 0x02f3c0 (0xce50 bytes) 000639 1780332628.935696 mapwrite 0x011969 thru 0x01ca7f (0xb117 bytes) 000641 1780332629.034499 MIRROR_EXTEND to 2 mirrors 000643 1780332629.954354 mapwrite 0x01f304 thru 0x020f5c (0x1c59 bytes) 000647 1780332630.297017 write 0x02a066 thru 0x02f164 (0x50ff bytes) 000650 1780332630.309927 read 0x0060f7 thru 0x01005d (0x9f67 bytes) 000654 1780332630.327127 mapwrite 0x020456 thru 0x02a745 (0xa2f0 bytes) 000655 1780332630.410974 mapread 0x0285e1 thru 0x02f506 (0x6f26 bytes) 000658 1780332630.413069 write 0x018c9a thru 0x027dc5 (0xf12c bytes) 000659 1780332630.432237 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000660 1780332630.638587 read 0x0074ee thru 0x0167ec (0xf2ff bytes) 000661 1780332630.648123 trunc from 0x02f507 to 0x024ee4 (0xa622 bytes) 000662 1780332630.676569 read 0x00db13 thru 0x01a5c8 (0xcab6 bytes) 000663 1780332630.687256 read 0x0203a4 thru 0x024ee4 (0x4b41 bytes) 000664 1780332630.688222 mapread 0x0204e8 thru 0x024ee4 (0x49fd bytes) 000667 1780332630.691792 trunc from 0x024ee5 to 0x0014b8 (0x23a2c bytes) 000669 1780332630.707989 trunc from 0x0014b9 to 0x03b3a5 (0x39eed bytes) 000672 1780332630.728944 write 0x01b756 thru 0x0236ce (0x7f79 bytes) 000673 1780332630.749281 MIRROR_EXTEND to 2 mirrors 000675 1780332631.775966 mapread 0x031c4c thru 0x03b3a5 (0x975a bytes) 000676 1780332631.852095 mapwrite 0x031659 thru 0x033b82 (0x252a bytes) 000677 1780332632.056613 mapread 0x003964 thru 0x011341 (0xd9de bytes) 000678 1780332632.086943 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000679 1780332632.258235 trunc from 0x03b3a6 to 0x009437 (0x31f6e bytes) 000681 1780332632.282016 mapwrite 0x0348e5 thru 0x03ffff (0xb71b bytes) 000682 1780332632.439791 mapwrite 0x01f538 thru 0x025ab4 (0x657d bytes) 000684 1780332632.499839 write 0x004d7a thru 0x011f0d (0xd194 bytes) 000686 1780332632.514083 mapwrite 0x00c556 thru 0x010b9f (0x464a bytes) 000688 1780332632.567143 write 0x0171b4 thru 0x022ae9 (0xb936 bytes) 000689 1780332632.582172 write 0x001246 thru 0x00c857 (0xb612 bytes) 000690 1780332632.583860 mapwrite 0x03aa1f thru 0x03fd2e (0x5310 bytes) 000691 1780332632.667974 write 0x01f028 thru 0x023da0 (0x4d79 bytes) 000693 1780332632.669507 read 0x00006c thru 0x00a345 (0xa2da bytes) 000694 1780332632.675078 mapwrite 0x019f80 thru 0x025b7b (0xbbfc bytes) 000695 1780332632.718748 read 0x011466 thru 0x01d7a9 (0xc344 bytes) 000696 1780332632.720003 write 0x0194c1 thru 0x025f92 (0xcad2 bytes) 000697 1780332632.721800 trunc from 0x040000 to 0x01fb3d (0x204c2 bytes) 000698 1780332632.737866 trunc from 0x01fb3e to 0x01721d (0x8920 bytes) 000700 1780332632.756752 mapwrite 0x03ffdc thru 0x03ffff (0x24 bytes) 000702 1780332632.835616 write 0x035498 thru 0x037ba3 (0x270c bytes) 000703 1780332632.837990 write 0x033af7 thru 0x038e9a (0x53a4 bytes) 000704 1780332632.839920 read 0x0381cd thru 0x03ffff (0x7e33 bytes) 000705 1780332632.845140 MIRROR_EXTEND to 2 mirrors 000710 1780332633.517940 read 0x0242e9 thru 0x027c93 (0x39ab bytes) 000711 1780332633.544397 MIRROR_EXTEND to 3 mirrors 000712 1780332634.572802 read 0x01c5cd thru 0x02c013 (0xfa47 bytes) 000713 1780332634.599528 MIRROR_RESYNC 3 mirrors 000714 1780332635.173112 mapread 0x02a890 thru 0x0336e2 (0x8e53 bytes) 000715 1780332635.201176 trunc from 0x040000 to 0x01ae88 (0x25177 bytes) 000716 1780332635.307451 mapwrite 0x035d2b thru 0x03c385 (0x665b bytes) 000717 1780332635.476203 mapread 0x002df0 thru 0x005d81 (0x2f92 bytes) 000719 1780332635.509144 mapread 0x005c85 thru 0x011ce1 (0xc05d bytes) 000720 1780332635.537343 write 0x031a10 thru 0x0381ed (0x67de bytes) 000724 1780332635.557037 trunc from 0x03c386 to 0x003cee (0x38697 bytes) 000725 1780332635.615759 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cef): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cef): Mirror 2: 0xaa61d63f (stale) Mirror 3: 0xaa61d63f (stale) CRC-32 checksum value for chunk [0, 0x3cef): Mirror 1: 0xaa61d63f Mirror 4: 0xaa61d63f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cef: skipped 000726 1780332636.716778 mapread 0x000955 thru 0x003cee (0x339a bytes) 000728 1780332636.829982 trunc from 0x003cef to 0x0002d8 (0x3a16 bytes) 000729 1780332636.955355 mapwrite 0x00de8c thru 0x0143b1 (0x6526 bytes) 000730 1780332637.051826 mapread 0x00b219 thru 0x00ff22 (0x4d0a bytes) 000735 1780332637.058203 write 0x000f1a thru 0x00545f (0x4546 bytes) 000736 1780332637.068623 read 0x005780 thru 0x0124fb (0xcd7c bytes) 000737 1780332637.069969 read 0x009fca thru 0x013c77 (0x9cae bytes) 000739 1780332637.072078 mapwrite 0x010520 thru 0x019acc (0x95ad bytes) 000740 1780332637.158553 read 0x0182d9 thru 0x019acc (0x17f4 bytes) 000741 1780332637.160504 mapwrite 0x0123f2 thru 0x01e664 (0xc273 bytes) 000743 1780332637.240624 write 0x00f5c7 thru 0x012678 (0x30b2 bytes) 000744 1780332637.249835 MIRROR_RESYNC 4 mirrors 000745 1780332638.676079 MIRROR_EXTEND to 5 mirrors 000747 1780332639.757981 trunc from 0x01e665 to 0x024d3b (0x66d7 bytes) 000751 1780332639.947909 write 0x038fa9 thru 0x03ffff (0x7057 bytes) 000752 1780332639.955081 trunc from 0x040000 to 0x0170aa (0x28f55 bytes) 000753 1780332639.999996 mapwrite 0x02e6e3 thru 0x031f0b (0x3829 bytes) 000754 1780332640.173725 trunc from 0x031f0c to 0x016f05 (0x1b006 bytes) 000757 1780332640.205253 read 0x000c82 thru 0x00ce65 (0xc1e4 bytes) 000758 1780332640.239268 mapwrite 0x032728 thru 0x03ffff (0xd8d8 bytes) 000760 1780332640.341905 mapread 0x009b55 thru 0x01124e (0x76fa bytes) 000763 1780332640.360533 trunc from 0x040000 to 0x020632 (0x1f9cd bytes) 000764 1780332640.380233 mapwrite 0x000786 thru 0x006a26 (0x62a1 bytes) 000767 1780332640.432600 write 0x01d63d thru 0x01ec26 (0x15ea bytes) 000768 1780332640.444154 mapread 0x00550b thru 0x00b257 (0x5d4d bytes) 000770 1780332640.447658 mapwrite 0x01c9cd thru 0x0247c2 (0x7df6 bytes) 000771 1780332640.529542 read 0x007d99 thru 0x017cdb (0xff43 bytes) 000774 1780332640.535238 trunc from 0x0247c3 to 0x02e6b5 (0x9ef3 bytes) 000775 1780332640.560491 read 0x00aa2c thru 0x0131a9 (0x877e bytes) 000777 1780332640.561870 read 0x005f28 thru 0x00efb5 (0x908e bytes) 000779 1780332640.564942 mapread 0x025e1a thru 0x02e6b5 (0x889c bytes) 000780 1780332640.580661 mapread 0x007fc4 thru 0x010eb8 (0x8ef5 bytes) 000782 1780332640.584187 mapwrite 0x00393e thru 0x00e892 (0xaf55 bytes) 000783 1780332640.638870 read 0x01c9b0 thru 0x01e402 (0x1a53 bytes) 000784 1780332640.639671 mapwrite 0x0148f1 thru 0x01b246 (0x6956 bytes) 000785 1780332640.690833 MIRROR_RESYNC 5 mirrors 000786 1780332641.943779 trunc from 0x02e6b6 to 0x028be3 (0x5ad2 bytes) 000787 1780332642.042660 trunc from 0x028be4 to 0x017044 (0x11b9f bytes) 000789 1780332642.079107 trunc from 0x017045 to 0x000620 (0x16a24 bytes) 000791 1780332642.128161 mapwrite 0x03e544 thru 0x03ffff (0x1abc bytes) 000792 1780332642.240047 trunc from 0x040000 to 0x01a37f (0x25c80 bytes) 000793 1780332642.282737 mapwrite 0x028a9c thru 0x032f13 (0xa478 bytes) 000795 1780332642.431039 mapread 0x03129a thru 0x03197d (0x6e4 bytes) 000798 1780332642.443483 write 0x003b7b thru 0x008f17 (0x539d bytes) 000799 1780332642.467446 read 0x02a8ff thru 0x02bdba (0x14bc bytes) 000802 1780332642.469937 read 0x01cbce thru 0x01d0a4 (0x4d7 bytes) 000805 1780332642.492049 read 0x028717 thru 0x030fd3 (0x88bd bytes) 000806 1780332642.495252 mapread 0x0151ef thru 0x01e25f (0x9071 bytes) 000809 1780332642.505411 write 0x039067 thru 0x03cdd9 (0x3d73 bytes) 000810 1780332642.506845 mapread 0x00f1e3 thru 0x01a48e (0xb2ac bytes) 000812 1780332642.517727 mapwrite 0x02c9ca thru 0x033b93 (0x71ca bytes) 000814 1780332642.656740 mapwrite 0x004067 thru 0x012061 (0xdffb bytes) 000815 1780332642.797003 read 0x02c64e thru 0x02d5fa (0xfad bytes) 000817 1780332642.798380 read 0x03b282 thru 0x03cdd9 (0x1b58 bytes) 000818 1780332642.799068 read 0x004ddb thru 0x00a891 (0x5ab7 bytes) 000819 1780332642.805891 write 0x016ba3 thru 0x025745 (0xeba3 bytes) 000820 1780332642.831841 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cdda): Mirror 3: 0x238c28f8 (stale) Mirror 4: 0x238c28f8 (stale) Mirror 5: 0x238c28f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cdda): Mirror 3: 0x238c28f8 (stale) Mirror 4: 0x238c28f8 (stale) Mirror 5: 0x238c28f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cdda: skipped 000821 1780332643.418289 MIRROR_RESYNC 4 mirrors 000822 1780332644.756252 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000828 1780332645.116852 MIRROR_RESYNC 3 mirrors 000829 1780332645.647257 trunc from 0x03cdda to 0x0038f8 (0x394e1 bytes) 000831 1780332645.786893 mapwrite 0x01a900 thru 0x021ae7 (0x71e8 bytes) 000834 1780332646.055641 write 0x0108ad thru 0x01780c (0x6f60 bytes) 000835 1780332646.072456 MIRROR_RESYNC 3 mirrors 000838 1780332647.120000 trunc from 0x021ae8 to 0x030a3f (0xef58 bytes) 000840 1780332647.199750 read 0x01c9a7 thru 0x01f0f0 (0x274a bytes) 000841 1780332647.208902 read 0x00657e thru 0x012e91 (0xc914 bytes) 000843 1780332647.225823 MIRROR_RESYNC 3 mirrors 000845 1780332648.027680 mapwrite 0x0282d6 thru 0x034d12 (0xca3d bytes) 000846 1780332648.334138 write 0x01e9e0 thru 0x02ac0a (0xc22b bytes) 000848 1780332648.367424 MIRROR_RESYNC 3 mirrors 000850 1780332649.684873 mapread 0x008917 thru 0x0177bc (0xeea6 bytes) 000851 1780332649.736720 write 0x019af4 thru 0x01f142 (0x564f bytes) 000852 1780332649.831956 write 0x031938 thru 0x0359c5 (0x408e bytes) 000855 1780332649.843284 write 0x0173b8 thru 0x01da9f (0x66e8 bytes) 000857 1780332649.846665 mapwrite 0x034724 thru 0x03ffff (0xb8dc bytes) 000860 1780332649.942753 read 0x03ab93 thru 0x03af02 (0x370 bytes) 000861 1780332649.944010 mapwrite 0x005ea8 thru 0x00c130 (0x6289 bytes) 000864 1780332650.009157 mapread 0x02b297 thru 0x02e8dd (0x3647 bytes) 000866 1780332650.012622 write 0x02776c thru 0x02c48b (0x4d20 bytes) 000867 1780332650.023557 mapread 0x026b86 thru 0x02e750 (0x7bcb bytes) 000868 1780332650.025029 mapwrite 0x03a0a7 thru 0x03c169 (0x20c3 bytes) 000872 1780332650.067779 trunc from 0x040000 to 0x03bbea (0x4415 bytes) 000873 1780332650.086353 mapwrite 0x035b82 thru 0x03f285 (0x9704 bytes) 000874 1780332650.162253 read 0x005cbd thru 0x00c3a3 (0x66e7 bytes) 000876 1780332650.166989 trunc from 0x03f286 to 0x026a8b (0x187fa bytes) 000877 1780332650.193868 read 0x01422e thru 0x01a866 (0x6639 bytes) 000878 1780332650.199462 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a8c): Mirror 4: 0x8af651c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a8c): Mirror 4: 0x8af651c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a8c: skipped 000879 1780332650.674238 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000880 1780332650.887677 write 0x023ea9 thru 0x031094 (0xd1ec bytes) 000881 1780332650.908433 mapwrite 0x00a789 thru 0x0169a2 (0xc21a bytes) 000883 1780332651.074741 read 0x02ac20 thru 0x031094 (0x6475 bytes) 000886 1780332651.086368 write 0x03fa60 thru 0x03ffff (0x5a0 bytes) 000888 1780332651.090131 read 0x02b4e7 thru 0x02c5a9 (0x10c3 bytes) 000891 1780332651.091930 write 0x02ff65 thru 0x03758f (0x762b bytes) 000892 1780332651.131436 read 0x00ebf6 thru 0x016c33 (0x803e bytes) 000894 1780332651.136637 write 0x02c56f thru 0x036a6a (0xa4fc bytes) 000895 1780332651.138659 trunc from 0x040000 to 0x00f141 (0x30ebe bytes) 000896 1780332651.166146 read 0x007520 thru 0x009994 (0x2475 bytes) 000900 1780332651.170843 write 0x00edce thru 0x010e46 (0x2079 bytes) 000901 1780332651.173141 mapwrite 0x0031fb thru 0x00d2fe (0xa104 bytes) 000902 1780332651.334399 read 0x00da54 thru 0x010e46 (0x33f3 bytes) 000903 1780332651.335858 trunc from 0x010e47 to 0x010c2c (0x21a bytes) 000904 1780332651.360727 mapwrite 0x02163e thru 0x0253c9 (0x3d8c bytes) 000907 1780332651.519760 mapread 0x015891 thru 0x022e64 (0xd5d4 bytes) 000908 1780332651.526956 read 0x007f43 thru 0x00a16d (0x222b bytes) 000911 1780332651.534778 mapread 0x0034d6 thru 0x006b21 (0x364c bytes) 000912 1780332651.538673 write 0x03f98c thru 0x03ffff (0x674 bytes) 000914 1780332651.551633 mapread 0x02161d thru 0x02c6f7 (0xb0db bytes) 000917 1780332651.567487 write 0x0098a7 thru 0x010229 (0x6983 bytes) 000918 1780332651.575840 trunc from 0x040000 to 0x03ae2d (0x51d2 bytes) truncating to largest ever: 0x3effa 000921 1780332651.614822 trunc from 0x03ae2e to 0x03eff9 (0x41cc bytes) 000923 1780332651.646548 mapwrite 0x014348 thru 0x01fd0d (0xb9c6 bytes) 000924 1780332651.733762 mapread 0x01fb4a thru 0x022e80 (0x3337 bytes) 000925 1780332651.736804 write 0x03bdef thru 0x03ffff (0x4211 bytes) 000927 1780332651.746092 read 0x0049b9 thru 0x005389 (0x9d1 bytes) 000928 1780332651.747138 read 0x00ed9e thru 0x01e104 (0xf367 bytes) 000929 1780332651.750237 read 0x03f569 thru 0x03ffff (0xa97 bytes) 000931 1780332651.760841 mapread 0x0301fa thru 0x031be8 (0x19ef bytes) 000932 1780332651.768660 mapwrite 0x03863d thru 0x03ffff (0x79c3 bytes) 000934 1780332651.880377 write 0x035c8a thru 0x0361eb (0x562 bytes) 000936 1780332651.892588 read 0x02cabe thru 0x031d79 (0x52bc bytes) 000940 1780332651.897805 mapread 0x0350d0 thru 0x039f88 (0x4eb9 bytes) 000943 1780332651.901687 mapread 0x03fd42 thru 0x03ffff (0x2be bytes) 000944 1780332651.909977 trunc from 0x040000 to 0x039a71 (0x658e bytes) 000945 1780332651.945868 trunc from 0x039a72 to 0x03db37 (0x40c6 bytes) 000947 1780332652.008868 mapread 0x0067db thru 0x008f57 (0x277d bytes) 000948 1780332652.010757 write 0x02e41c thru 0x035dc7 (0x79ac bytes) 000951 1780332652.021337 mapwrite 0x03413c thru 0x03f65a (0xb51f bytes) 000952 1780332652.224762 read 0x004c7b thru 0x014835 (0xfbbb bytes) 000954 1780332652.229376 trunc from 0x03f65b to 0x03b13b (0x451f bytes) 000955 1780332652.276086 mapread 0x00d0fe thru 0x012676 (0x5579 bytes) 000958 1780332652.288518 MIRROR_EXTEND to 2 mirrors 000959 1780332653.392069 mapread 0x02c794 thru 0x03b13b (0xe9a8 bytes) 000961 1780332653.478366 read 0x0088b3 thru 0x009148 (0x896 bytes) 000962 1780332653.501488 write 0x024b8f thru 0x0255c9 (0xa3b bytes) 000964 1780332653.715001 read 0x032ef6 thru 0x03aced (0x7df8 bytes) 000965 1780332653.731425 write 0x02d713 thru 0x0335ab (0x5e99 bytes) 000966 1780332653.750508 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x183:0x0]: Invalid argument (22) 000967 1780332653.963142 trunc from 0x03b13c to 0x025e34 (0x15307 bytes) 000970 1780332654.025048 write 0x0329ae thru 0x0355ca (0x2c1d bytes) 000975 1780332654.037531 mapwrite 0x0379e0 thru 0x03ffff (0x8620 bytes) 000977 1780332654.194434 MIRROR_EXTEND to 2 mirrors 000980 1780332655.010991 mapwrite 0x01b650 thru 0x029993 (0xe344 bytes) 000981 1780332655.275404 mapread 0x03d195 thru 0x03ee50 (0x1cbc bytes) 000982 1780332655.290356 mapread 0x0305af thru 0x038793 (0x81e5 bytes) 000983 1780332655.320084 MIRROR_RESYNC 2 mirrors 000984 1780332656.401128 mapread 0x008d6d thru 0x00dff5 (0x5289 bytes) 000985 1780332656.436607 mapread 0x00e603 thru 0x0187e4 (0xa1e2 bytes) 000987 1780332656.450514 mapread 0x00bf3e thru 0x017d77 (0xbe3a bytes) 000988 1780332656.451748 write 0x03572b thru 0x03641c (0xcf2 bytes) 000989 1780332656.637653 write 0x03b0b4 thru 0x03ffff (0x4f4c bytes) 000990 1780332656.640895 trunc from 0x040000 to 0x037bac (0x8453 bytes) 000991 1780332656.708559 read 0x015453 thru 0x021d96 (0xc944 bytes) 000995 1780332656.724649 write 0x0326bf thru 0x03b870 (0x91b2 bytes) 000996 1780332656.734712 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b871): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b871): Mirror 2: 0x53825ddf (stale) CRC-32 checksum value for chunk [0, 0x3b871): Mirror 1: 0xd89e9aa2 Mirror 3: 0xd89e9aa2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b871: skipped 000997 1780332657.940039 mapwrite 0x0362f8 thru 0x03ca4e (0x6757 bytes) 001000 1780332658.235223 mapwrite 0x02256e thru 0x023334 (0xdc7 bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 12:51:03 (1780332663) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 12:51:07 (1780332667) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.262028 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235668 s, 4.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 12:51:17 (1780332677) Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6511360 .. Extending file size to 8543072 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7697920 .. Extending file size to 9203616 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8507104 .. Extending file size to 8569568 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8855328 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56618 56620 56621 56622 56625 done 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) 192.168.203.144@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:52:39 (1780332759) Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3637152 .. Extending file size to 6235456 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7905184 .. Extending file size to 6147936 .. Extending file size to 7079200 .. Extending file size to 9083200 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8207648 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8813920 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58080 58081 58083 58085 58087 done 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) 192.168.203.144@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:53:59 (1780332839) Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13024 s, 16.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.398944 s, 5.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x660b49f9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:54:20 (1780332860) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149349 s, 14.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 12:54:27 (1780332867) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:54:36 (1780332876) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156196 s, 13.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:54:45 (1780332885) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233604 s, 9.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:54:54 (1780332894) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313673 s, 6.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:55:27 (1780332927) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg344-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101184 s, 10.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:56:24 (1780332984) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg344-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.305127 s, 6.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:57:22 (1780333042) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg344-server mds-ost sync done. Creating new pool test_204f oleg344-server: Pool lustre.test_204f created Adding targets to pool oleg344-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg344-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.379556 s, 5.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg344-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg344-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg344-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:58:41 (1780333121) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:58:49 (1780333129) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:58:57 (1780333137) oleg344-server: Pool lustre.test_206 created Adding targets to pool oleg344-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg344-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg344-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg344-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg344-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:59:23 (1780333163) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530175 s, 19.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248402 s, 42.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1211c6b9 Mirror 3: 0x1211c6b9 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:59:34 (1780333174) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:59:37 (1780333177) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:59:40 (1780333180) Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.153076 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2600000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2240000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2240000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2240000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1620000 Read this (4 bytes): BBBB 192.168.203.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg344-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:00:16 (1780333216) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0159114 s, 515 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:00:23 (1780333223) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929837 s, 11.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ed lmm_fid: [0x200000404:0x1ed:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ed lmm_fid: [0x200000404:0x1ed:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834413 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 13:00:25.000000000 -0400 Modify: 2026-06-01 13:00:25.000000000 -0400 Change: 2026-06-01 13:00:27.000000000 -0400 Birth: 2026-06-01 13:00:26.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:00:33 (1780333233) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:00:37 (1780333237) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.617494 s, 66.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0380914 s, 108 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 1 13:00 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 1 13:00 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:00:45 (1780333245) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:00:52 (1780333252) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1607 sec =========== 13:01:05 (1780333265) === sanity-flr: start cleanup 13:01:08 (1780333268) === === sanity-flr: finish cleanup 13:01:17 (1780333277) ===