-----============= acceptance-small: sanity-flr ============----- Mon Jun 1 13:45:14 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg347-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:45:32 (1780335932) === oleg347-client.virtnet: executing check_config_client /mnt/lustre oleg347-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg347-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c99d876b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c99d876b800.idle_timeout=debug disable quota as required oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:46:01 (1780335961) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7562] == sanity-flr test 0a: lfs mirror create with -N option == 13:46:03 (1780335963) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:46:17 (1780335977) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:46:19 (1780335979) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:46:22 (1780335982) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:46:45 (1780336005) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:46:48 (1780336008) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 13:46:51 (1780336011) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c99d876b800.stats=clear osc.lustre-OST0001-osc-ffff8c99d876b800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:46:59 (1780336019) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:47:08 (1780336028) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:47:15 (1780336035) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:47:17 (1780336037) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:47:24 (1780336044) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 13:47:32 (1780336052) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:47:37 (1780336057) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:47:40 (1780336060) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:47:46 (1780336066) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:47:54 (1780336074) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:48:00 (1780336080) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.13818 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994154 s, 10.5 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:48:08 (1780336088) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399386 s, 26.3 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:48:14 (1780336094) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:48:23 (1780336103) Stopping /mnt/lustre-ost1 (opts:) on oleg347-server oleg347-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c99d876b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c99d876b800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg347-server: oleg347-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg347-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg347-server oleg347-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c99d876b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c99d876b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg347-server: oleg347-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg347-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (56s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:49:19 (1780336159) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.383009 s, 19.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg347-server oleg347-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c99d876b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c99d876b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg347-server: oleg347-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg347-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (33s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:49:52 (1780336192) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg347-server oleg347-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c99d876b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c99d876b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg347-server: oleg347-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg347-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg347-server oleg347-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c99d876b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c99d876b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg347-server: oleg347-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg347-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (56s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:50:48 (1780336248) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.34248 s, 15.7 MB/s -rw-r--r-- 1 root root 52428800 Jun 1 13:50 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.780s user 0m0.001s sys 0m0.170s Stopping /mnt/lustre-ost1 (opts:) on oleg347-server oleg347-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c99d876b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c99d876b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.956s user 0m0.001s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg347-server: oleg347-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg347-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg347-server oleg347-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c99d876b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c99d876b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.191s user 0m0.000s sys 0m0.167s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg347-server: oleg347-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg347-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg347-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c99d876b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c99d876b800.ost_server_uuid in FULL state after 0 sec PASS 33b (72s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:52:00 (1780336320) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:52:03 (1780336323) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:52:06 (1780336326) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 13:52:08 (1780336328) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 13:52:14 (1780336334) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.608274 s, 20.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:52:26 (1780336346) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022545 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199667 s, 52.5 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:52:31 (1780336351) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:52:38 (1780336358) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0008619 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000868862 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00151885 s, 674 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00410053 s, 250 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 13:52:46 (1780336366) create test files with size 15737996 .. 4da4b2fb7c2701100882507563c77253 - ac594bda0d5727d7e51f563e54c50ecf - 299855b63bc17361dd46fa6b3d7ddc1a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg347-client.virtnet /mnt/lustre (opts:) Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 13:53:05 (1780336385) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:53:16 (1780336396) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:53:24 (1780336404) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139008 s, 7.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645164 s, 16.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 13:53:34 (1780336414) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0970542 s, 21.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.310659 s, 13.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.27362 s, 7.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 13:53:49 (1780336429) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 13:53:52 (1780336432) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:53:56 (1780336436) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574148 s, 18.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 13:54:05 (1780336445) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:54:09 (1780336449) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:54:20 (1780336460) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.287515 s, 36.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg347-client.virtnet /mnt/lustre (opts:) Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:54:31 (1780336471) llite.lustre-ffff8c99c67a7000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.302675 s, 34.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg347-client.virtnet /mnt/lustre (opts:) Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8c99d87bd800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:54:45 (1780336485) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:55:22 (1780336522) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:56:06 (1780336566) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 13:56:10 (1780336570) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140911 s, 22.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.207469 s, 15.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb97ed1df (nosync,stale) Mirror 2: 0xb97ed1df (nosync,stale) ** verify mirror 2 doesn't change original checksum: 51783701df62c81a8a5fb43abdedf88d - mirror 1 checksum: b9b9f9d020349633b170d8007940ba33 - mirror 2 checksum: 51783701df62c81a8a5fb43abdedf88d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 13:56:13 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 13:56:13 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 51783701df62c81a8a5fb43abdedf88d - mirror 1 checksum: b9b9f9d020349633b170d8007940ba33 - mirror 2 checksum: b9b9f9d020349633b170d8007940ba33 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 13:56:13 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 13:56:13 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:56:23 (1780336583) Failing mds1 on oleg347-server Stopping /mnt/lustre-mds1 (opts:) on oleg347-server 13:56:30 (1780336590) shut down facet: mds1 facet_host: oleg347-server facet_failover_host: oleg347-server Failover mds1 to oleg347-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 13:56:53 (1780336613) targets are mounted 13:56:53 (1780336613) facet_failover done oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300196 s, 34.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:57:08 (1780336628) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:57:12 (1780336632) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:57:15 (1780336635) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00395794 s, 4.1 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:57:22 (1780336642) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:57:26 (1780336646) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:57:30 (1780336650) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:57:52 (1780336672) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:58:18 (1780336698) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:58:41 (1780336721) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:59:07 (1780336747) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 13:59:14 (1780336754) mirror create pid 53499 mirror split pid 53501 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53499 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53501 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 14:00:23 (1780336823) Seed set to 6825 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005 1780336825.637507 write 0x0006aa thru 0x00fc7f (0xf5d6 bytes) 000008 1780336825.647239 mapread 0x007427 thru 0x00c112 (0x4cec bytes) 000011 1780336825.653185 mapread 0x006fb3 thru 0x00bc54 (0x4ca2 bytes) 000012 1780336825.655511 mapread 0x00a201 thru 0x00fc7f (0x5a7f bytes) 000013 1780336825.657510 mapwrite 0x009e3a thru 0x010de7 (0x6fae bytes) 000014 1780336825.823408 MIRROR_EXTEND to 2 mirrors 000018 1780336826.589068 mapwrite 0x0234d6 thru 0x02c926 (0x9451 bytes) 000019 1780336826.845604 read 0x006ea2 thru 0x00a333 (0x3492 bytes) 000020 1780336826.872653 mapread 0x029b1a thru 0x02c926 (0x2e0d bytes) 000021 1780336826.880475 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000022 1780336827.149737 mapread 0x0105b7 thru 0x01925a (0x8ca4 bytes) 000023 1780336827.184480 mapwrite 0x02de7b thru 0x037ee8 (0xa06e bytes) 000024 1780336827.295695 MIRROR_EXTEND to 2 mirrors 000026 1780336828.126628 mapwrite 0x00781d thru 0x015873 (0xe057 bytes) 000027 1780336828.530304 write 0x031e57 thru 0x03ffff (0xe1a9 bytes) 000028 1780336828.565012 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) truncating to largest ever: 0x1ddd7 000030 1780336828.779280 trunc from 0x040000 to 0x01ddd6 (0x22229 bytes) truncating to largest ever: 0x1f482 000031 1780336828.824359 trunc from 0x01ddd7 to 0x01f481 (0x16ab bytes) 000032 1780336828.843890 mapread 0x019bf6 thru 0x01ef0d (0x5318 bytes) 000035 1780336828.864269 mapread 0x015f24 thru 0x01f481 (0x955e bytes) 000038 1780336828.867723 mapread 0x00f9a6 thru 0x011567 (0x1bc2 bytes) 000042 1780336828.880037 mapread 0x00fa41 thru 0x012540 (0x2b00 bytes) 000044 1780336828.881614 mapread 0x015ddd thru 0x01867e (0x28a2 bytes) 000046 1780336828.884966 mapread 0x01187a thru 0x01e4cf (0xcc56 bytes) truncating to largest ever: 0x3101d 000050 1780336828.889535 trunc from 0x01f482 to 0x03101c (0x11b9b bytes) 000051 1780336828.918394 read 0x02364e thru 0x02a99a (0x734d bytes) 000054 1780336828.946263 mapread 0x0022c7 thru 0x0081ca (0x5f04 bytes) 000056 1780336828.965397 mapread 0x0119c2 thru 0x020528 (0xeb67 bytes) 000057 1780336828.976266 mapwrite 0x0224da thru 0x02d259 (0xad80 bytes) 000058 1780336829.085182 write 0x03508d thru 0x03ffff (0xaf73 bytes) 000060 1780336829.106616 MIRROR_EXTEND to 2 mirrors 000061 1780336830.021455 mapwrite 0x01c324 thru 0x02bda8 (0xfa85 bytes) 000063 1780336830.249279 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc5b20592 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9858ed48 Mirror 3: 0x9858ed48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000064 1780336831.121119 mapwrite 0x01bfbb thru 0x029b63 (0xdba9 bytes) 000065 1780336831.323755 trunc from 0x040000 to 0x01a115 (0x25eea bytes) 000066 1780336831.344948 write 0x02c482 thru 0x03beb9 (0xfa38 bytes) 000068 1780336831.348118 mapread 0x030e84 thru 0x03bc28 (0xada5 bytes) 000070 1780336831.350674 write 0x01c9b3 thru 0x020f10 (0x455e bytes) 000073 1780336831.362526 mapwrite 0x0383f6 thru 0x03f24d (0x6e58 bytes) 000074 1780336831.424489 mapread 0x00b76f thru 0x00c327 (0xbb9 bytes) 000075 1780336831.435659 write 0x01aaac thru 0x024567 (0x9abc bytes) 000077 1780336831.438966 trunc from 0x03f24e to 0x015976 (0x298d7 bytes) 000078 1780336831.454491 trunc from 0x015977 to 0x02e419 (0x18aa3 bytes) 000079 1780336831.473029 read 0x02d431 thru 0x02e419 (0xfe9 bytes) 000080 1780336831.479384 mapread 0x013ad6 thru 0x021b0f (0xe03a bytes) 000082 1780336831.501548 read 0x0036e9 thru 0x009851 (0x6169 bytes) 000083 1780336831.520654 write 0x03e409 thru 0x03ffff (0x1bf7 bytes) 000084 1780336831.535126 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc5b20592 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc5b20592 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000085 1780336831.987798 mapwrite 0x00df75 thru 0x01ac9c (0xcd28 bytes) truncating to largest ever: 0x370a4 000089 1780336832.166984 trunc from 0x040000 to 0x0370a3 (0x8f5c bytes) 000090 1780336832.213374 read 0x0303fd thru 0x030623 (0x227 bytes) 000091 1780336832.220668 trunc from 0x0370a4 to 0x02079a (0x16909 bytes) 000092 1780336832.239588 trunc from 0x02079b to 0x02c860 (0xc0c6 bytes) 000093 1780336832.267978 write 0x02916f thru 0x03530b (0xc19d bytes) 000094 1780336832.287074 read 0x02f252 thru 0x033a76 (0x4825 bytes) 000098 1780336832.291333 read 0x0059ce thru 0x01468c (0xecbf bytes) 000099 1780336832.310392 mapread 0x000dc7 thru 0x00bc79 (0xaeb3 bytes) truncating to largest ever: 0x3c7ac 000100 1780336832.312935 trunc from 0x03530c to 0x03c7ab (0x74a0 bytes) 000102 1780336832.338365 write 0x00f4dc thru 0x01ad3c (0xb861 bytes) 000105 1780336832.342017 mapwrite 0x011d2c thru 0x01b671 (0x9946 bytes) 000107 1780336832.399439 mapwrite 0x002d9e thru 0x0056c7 (0x292a bytes) 000109 1780336832.466109 mapwrite 0x03b0a2 thru 0x03ffff (0x4f5e bytes) 000110 1780336832.555825 write 0x03a978 thru 0x03ffff (0x5688 bytes) 000111 1780336832.557008 write 0x03b00c thru 0x03ffff (0x4ff4 bytes) 000113 1780336832.559751 mapread 0x038e3d thru 0x03ffff (0x71c3 bytes) 000115 1780336832.562379 mapwrite 0x0032ca thru 0x00d7ae (0xa4e5 bytes) 000117 1780336832.619491 trunc from 0x040000 to 0x03b7ca (0x4835 bytes) 000120 1780336832.640331 mapread 0x02a04f thru 0x0321c4 (0x8176 bytes) 000121 1780336832.642187 mapread 0x009254 thru 0x012842 (0x95ef bytes) 000122 1780336832.644116 mapwrite 0x0212b3 thru 0x02e5e2 (0xd330 bytes) 000124 1780336832.719003 write 0x008688 thru 0x012d93 (0xa70c bytes) 000125 1780336832.720298 mapwrite 0x035ff5 thru 0x03a6ee (0x46fa bytes) 000127 1780336832.772254 trunc from 0x03b7cb to 0x03c18c (0x9c2 bytes) 000128 1780336832.795040 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000130 1780336832.916436 read 0x033530 thru 0x0388ff (0x53d0 bytes) 000131 1780336832.926977 read 0x02a744 thru 0x038c24 (0xe4e1 bytes) 000132 1780336832.941298 write 0x01de48 thru 0x021699 (0x3852 bytes) 000134 1780336832.943024 mapwrite 0x030b19 thru 0x036240 (0x5728 bytes) 000135 1780336832.979965 mapwrite 0x03fa76 thru 0x03ffff (0x58a bytes) 000136 1780336833.046672 write 0x02f2ca thru 0x039f3c (0xac73 bytes) 000137 1780336833.048000 trunc from 0x040000 to 0x02a47e (0x15b81 bytes) 000138 1780336833.078640 read 0x00f9cb thru 0x019d20 (0xa356 bytes) 000141 1780336833.085265 mapwrite 0x0299ac thru 0x02cddf (0x3434 bytes) 000142 1780336833.194404 read 0x0119e9 thru 0x018ee9 (0x7501 bytes) 000143 1780336833.197623 mapread 0x016696 thru 0x01fb46 (0x94b1 bytes) 000144 1780336833.210102 MIRROR_EXTEND to 2 mirrors 000145 1780336834.049224 mapread 0x0158e7 thru 0x022af8 (0xd212 bytes) 000147 1780336834.095730 trunc from 0x02cde0 to 0x02a65c (0x2783 bytes) 000149 1780336834.248984 read 0x01211f thru 0x019809 (0x76eb bytes) 000151 1780336834.270450 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000152 1780336834.517539 write 0x014b5d thru 0x01b770 (0x6c14 bytes) 000157 1780336834.536245 read 0x0165eb thru 0x019487 (0x2e9d bytes) 000159 1780336834.538422 mapwrite 0x031c86 thru 0x035d19 (0x4094 bytes) 000163 1780336834.632551 mapwrite 0x01513f thru 0x02173a (0xc5fc bytes) 000164 1780336834.764924 mapwrite 0x028276 thru 0x02c70e (0x4499 bytes) 000165 1780336834.826384 mapread 0x015105 thru 0x015dbe (0xcba bytes) 000166 1780336834.828644 mapwrite 0x037795 thru 0x03ffff (0x886b bytes) 000167 1780336834.943596 mapread 0x00a92a thru 0x017c9a (0xd371 bytes) 000168 1780336834.967992 write 0x01b881 thru 0x0214d6 (0x5c56 bytes) 000169 1780336834.977438 mapread 0x01daf3 thru 0x02da86 (0xff94 bytes) 000172 1780336834.995746 mapwrite 0x01afd2 thru 0x029b97 (0xebc6 bytes) 000175 1780336835.145026 read 0x010146 thru 0x011ce0 (0x1b9b bytes) 000176 1780336835.146437 read 0x01a166 thru 0x0239f3 (0x988e bytes) 000177 1780336835.147555 write 0x031ee9 thru 0x03ffff (0xe117 bytes) 000179 1780336835.149787 read 0x01e0b4 thru 0x02ae61 (0xcdae bytes) 000180 1780336835.175092 mapread 0x00940c thru 0x015c64 (0xc859 bytes) 000181 1780336835.178633 trunc from 0x040000 to 0x035d71 (0xa28e bytes) 000182 1780336835.200188 mapread 0x034ca4 thru 0x035d71 (0x10ce bytes) 000183 1780336835.254400 MIRROR_EXTEND to 2 mirrors 000184 1780336835.978222 MIRROR_EXTEND to 3 mirrors 000185 1780336836.478955 mapread 0x02e69f thru 0x0337fb (0x515d bytes) 000189 1780336836.507966 MIRROR_EXTEND to 4 mirrors 000192 1780336837.522310 MIRROR_RESYNC 4 mirrors 000195 1780336838.008825 MIRROR_EXTEND to 5 mirrors 000197 1780336838.777111 mapread 0x034704 thru 0x035d71 (0x166e bytes) 000199 1780336838.788538 write 0x031316 thru 0x03ffff (0xecea bytes) 000200 1780336838.858070 read 0x0245d0 thru 0x031540 (0xcf71 bytes) 000201 1780336838.865363 mapwrite 0x030b8c thru 0x039115 (0x858a bytes) 000203 1780336838.915495 mapwrite 0x01a5db thru 0x01fa47 (0x546d bytes) 000204 1780336838.986973 trunc from 0x040000 to 0x016ff8 (0x29007 bytes) 000205 1780336839.043767 read 0x00474b thru 0x0105fb (0xbeb1 bytes) 000206 1780336839.054599 mapwrite 0x028b3c thru 0x033d49 (0xb20e bytes) 000207 1780336839.156203 mapread 0x02b36f thru 0x02f5cd (0x425f bytes) 000208 1780336839.157309 trunc from 0x033d4a to 0x033d36 (0x13 bytes) 000209 1780336839.171553 write 0x024037 thru 0x026b22 (0x2aec bytes) 000210 1780336839.172659 write 0x017189 thru 0x022726 (0xb59e bytes) 000212 1780336839.179647 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33d37): Mirror 2: 0x3a87f456 (stale) Mirror 4: 0x3a87f456 (stale) Mirror 5: 0x3a87f456 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d37): Mirror 2: 0x3a87f456 (stale) Mirror 4: 0x3a87f456 (stale) Mirror 5: 0x3a87f456 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d37: skipped 000217 1780336839.485628 mapread 0x02dc79 thru 0x033371 (0x56f9 bytes) 000218 1780336839.513005 mapwrite 0x00612e thru 0x01383b (0xd70e bytes) 000220 1780336839.590282 write 0x02064a thru 0x02cd8a (0xc741 bytes) 000222 1780336839.608199 read 0x01240d thru 0x0222ef (0xfee3 bytes) 000226 1780336839.616012 write 0x03d002 thru 0x03ffff (0x2ffe bytes) 000227 1780336839.617858 trunc from 0x040000 to 0x005ec1 (0x3a13e bytes) 000228 1780336839.645959 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5ec2): Mirror 4: 0xad8298fa (stale) Mirror 5: 0xad8298fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5ec2): Mirror 4: 0xad8298fa (stale) Mirror 5: 0xad8298fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5ec2: skipped 000230 1780336839.923863 mapread 0x00480a thru 0x005ec1 (0x16b8 bytes) 000233 1780336839.940172 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5ec2): Mirror 4: 0xad8298fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5ec2): Mirror 4: 0xad8298fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5ec2: skipped 000234 1780336840.316028 trunc from 0x005ec2 to 0x01bb95 (0x15cd4 bytes) 000235 1780336840.343094 MIRROR_RESYNC 2 mirrors 000236 1780336841.204309 trunc from 0x01bb96 to 0x00fde2 (0xbdb3 bytes) 000237 1780336841.279098 MIRROR_RESYNC 2 mirrors 000238 1780336841.918430 mapread 0x002f7c thru 0x00fde2 (0xce67 bytes) 000240 1780336841.962238 mapwrite 0x029f32 thru 0x033d60 (0x9e2f bytes) 000242 1780336842.156454 write 0x03c67f thru 0x03e72d (0x20af bytes) 000243 1780336842.158439 mapwrite 0x0120e0 thru 0x016db2 (0x4cd3 bytes) 000244 1780336842.212658 read 0x02149c thru 0x0217dc (0x341 bytes) 000246 1780336842.225202 write 0x00c610 thru 0x011cd0 (0x56c1 bytes) 000247 1780336842.227984 read 0x0289ac thru 0x02e61f (0x5c74 bytes) 000249 1780336842.230531 mapwrite 0x01bc7a thru 0x01d738 (0x1abf bytes) 000250 1780336842.299436 write 0x01d147 thru 0x02a92b (0xd7e5 bytes) 000251 1780336842.300795 mapread 0x03b548 thru 0x03e72d (0x31e6 bytes) 000252 1780336842.301785 write 0x032274 thru 0x03f4fd (0xd28a bytes) 000253 1780336842.308274 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f4fe): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f4fe): Mirror 4: 0x595e4edd (stale) CRC-32 checksum value for chunk [0, 0x3f4fe): Mirror 1: 0xab2d1a4e Mirror 5: 0xab2d1a4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f4fe: skipped 000254 1780336843.002023 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f4fe): Mirror 4: 0x595e4edd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f4fe): Mirror 4: 0x595e4edd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f4fe: skipped 000255 1780336843.298504 read 0x02539e thru 0x02c708 (0x736b bytes) 000256 1780336843.308432 mapwrite 0x01291a thru 0x01b24f (0x8936 bytes) 000258 1780336843.410351 write 0x022a27 thru 0x026902 (0x3edc bytes) 000259 1780336843.420136 read 0x014863 thru 0x02243c (0xdbda bytes) 000260 1780336843.421294 trunc from 0x03f4fe to 0x014d57 (0x2a7a6 bytes) 000262 1780336843.491125 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000263 1780336843.689418 read 0x006380 thru 0x007370 (0xff1 bytes) 000265 1780336843.709151 write 0x0038d6 thru 0x00f8cc (0xbff7 bytes) 000266 1780336843.750248 write 0x01ce79 thru 0x02050c (0x3694 bytes) 000267 1780336843.751420 mapread 0x007122 thru 0x00b4c5 (0x43a4 bytes) 000268 1780336843.760906 mapread 0x00f44f thru 0x011839 (0x23eb bytes) 000273 1780336843.783508 write 0x001c46 thru 0x00796c (0x5d27 bytes) 000275 1780336843.793021 trunc from 0x02050d to 0x024ed4 (0x49c8 bytes) 000276 1780336843.811655 trunc from 0x024ed5 to 0x02372b (0x17a9 bytes) 000279 1780336843.860853 trunc from 0x02372c to 0x03434d (0x10c22 bytes) 000280 1780336843.896649 trunc from 0x03434e to 0x03c480 (0x8133 bytes) 000281 1780336843.937306 read 0x01cab8 thru 0x01fc48 (0x3191 bytes) 000282 1780336843.945260 MIRROR_EXTEND to 2 mirrors 000285 1780336844.658708 trunc from 0x03c481 to 0x008a2e (0x33a52 bytes) 000286 1780336844.778554 trunc from 0x008a2f to 0x02d892 (0x24e64 bytes) 000287 1780336844.813492 write 0x03e934 thru 0x03ffff (0x16cc bytes) 000289 1780336844.815526 trunc from 0x040000 to 0x026151 (0x19eae bytes) truncating to largest ever: 0x3d7e1 000290 1780336844.844685 trunc from 0x026152 to 0x03d7e0 (0x1768f bytes) 000291 1780336844.877125 mapwrite 0x00ca8e thru 0x00d13f (0x6b2 bytes) 000292 1780336844.943525 read 0x00e644 thru 0x0172e2 (0x8c9f bytes) 000293 1780336844.951881 read 0x00b121 thru 0x01775d (0xc63d bytes) 000294 1780336844.972817 mapread 0x01886e thru 0x0265fa (0xdd8d bytes) 000297 1780336844.978069 write 0x03f019 thru 0x03ffff (0xfe7 bytes) 000300 1780336844.981480 trunc from 0x040000 to 0x02c8d4 (0x1372b bytes) 000301 1780336845.000515 read 0x027de2 thru 0x02c8d4 (0x4af3 bytes) 000304 1780336845.006508 mapread 0x003118 thru 0x0125dc (0xf4c5 bytes) 000306 1780336845.010370 trunc from 0x02c8d5 to 0x01fcf2 (0xcbe2 bytes) 000307 1780336845.038847 mapwrite 0x0166cd thru 0x016ff6 (0x92a bytes) 000308 1780336845.138660 write 0x0102cb thru 0x01434d (0x4083 bytes) 000309 1780336845.140414 mapread 0x006fb6 thru 0x0072e5 (0x330 bytes) 000310 1780336845.141548 mapwrite 0x036276 thru 0x0364a3 (0x22e bytes) 000311 1780336845.263761 write 0x01e8a4 thru 0x02813f (0x989c bytes) 000312 1780336845.279443 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x364a4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364a4): Mirror 2: 0xff37eca6 (stale) CRC-32 checksum value for chunk [0, 0x364a4): Mirror 1: 0xf79b4d1b Mirror 3: 0xf79b4d1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364a4: skipped 000314 1780336846.119023 read 0x01a728 thru 0x027a83 (0xd35c bytes) 000315 1780336846.128152 mapwrite 0x03eece thru 0x03ffff (0x1132 bytes) 000316 1780336846.290344 trunc from 0x040000 to 0x030b5f (0xf4a0 bytes) 000317 1780336846.315833 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30b60): Mirror 3: 0xdaf2cb5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b60): Mirror 3: 0xdaf2cb5b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b60: skipped 000318 1780336846.603410 read 0x0014d5 thru 0x003973 (0x249f bytes) 000319 1780336846.611063 trunc from 0x030b60 to 0x00b282 (0x258dd bytes) 000320 1780336846.628433 mapwrite 0x015324 thru 0x024afc (0xf7d9 bytes) 000321 1780336846.738254 mapwrite 0x0238c5 thru 0x02a828 (0x6f64 bytes) 000322 1780336846.805490 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000323 1780336846.937639 mapwrite 0x011682 thru 0x012238 (0xbb7 bytes) 000325 1780336847.028325 write 0x02ce2e thru 0x031276 (0x4449 bytes) 000327 1780336847.037379 mapwrite 0x03b64c thru 0x03ffff (0x49b4 bytes) 000328 1780336847.130352 read 0x0098c3 thru 0x017a95 (0xe1d3 bytes) 000330 1780336847.141098 trunc from 0x040000 to 0x02d5a5 (0x12a5a bytes) 000331 1780336847.187147 MIRROR_EXTEND to 2 mirrors 000332 1780336847.924453 read 0x004716 thru 0x00f988 (0xb273 bytes) 000333 1780336847.939049 MIRROR_EXTEND to 3 mirrors 000334 1780336848.502307 mapwrite 0x02288b thru 0x0308f7 (0xe06d bytes) 000335 1780336848.810427 mapwrite 0x0302ff thru 0x03b98f (0xb691 bytes) 000337 1780336848.956040 trunc from 0x03b990 to 0x00429e (0x376f1 bytes) 000338 1780336848.979948 trunc from 0x00429f to 0x006252 (0x1fb4 bytes) 000339 1780336849.000752 mapread 0x0040d7 thru 0x006252 (0x217c bytes) 000340 1780336849.014450 MIRROR_RESYNC 3 mirrors 000342 1780336849.720999 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000344 1780336850.157819 mapwrite 0x0205b6 thru 0x0235d4 (0x301f bytes) 000345 1780336850.521923 write 0x009164 thru 0x00cf70 (0x3e0d bytes) 000347 1780336850.532407 mapread 0x01c984 thru 0x0235d4 (0x6c51 bytes) 000348 1780336850.533705 mapread 0x00b51d thru 0x01b49d (0xff81 bytes) 000350 1780336850.545197 read 0x01c650 thru 0x0235d4 (0x6f85 bytes) 000352 1780336850.548340 mapwrite 0x038f4b thru 0x03ffff (0x70b5 bytes) 000353 1780336850.638975 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x88e94fe7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x94a3da6d Mirror 3: 0x94a3da6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000354 1780336851.564379 read 0x0055c9 thru 0x0066e2 (0x111a bytes) 000356 1780336851.573841 trunc from 0x040000 to 0x00e7b8 (0x31847 bytes) 000357 1780336851.666394 write 0x022abf thru 0x03241a (0xf95c bytes) 000358 1780336851.674086 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3241b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3241b): Mirror 2: 0x37295dac (stale) Mirror 3: 0x9ebef291 (stale) CRC-32 checksum value for chunk [0, 0x3241b): Mirror 1: 0x292dc1f9 Mirror 4: 0x292dc1f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3241b: skipped 000360 1780336852.665199 trunc from 0x03241b to 0x02f3e8 (0x3032 bytes) 000362 1780336852.748297 trunc from 0x02f3e9 to 0x02e5fc (0xdec bytes) 000364 1780336852.777041 MIRROR_RESYNC 4 mirrors 000367 1780336853.803140 mapread 0x024c1f thru 0x02de83 (0x9265 bytes) 000368 1780336853.842410 mapwrite 0x00c78d thru 0x01aa90 (0xe304 bytes) 000370 1780336854.057472 mapread 0x019cd4 thru 0x01edbf (0x50ec bytes) 000371 1780336854.083100 mapread 0x024163 thru 0x02e5fc (0xa49a bytes) 000372 1780336854.108167 mapread 0x01b9cb thru 0x02b88c (0xfec2 bytes) 000373 1780336854.111984 trunc from 0x02e5fd to 0x02ad0e (0x38ee bytes) 000375 1780336854.180112 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ad0f): Mirror 3: 0x62cc3fee (stale) Mirror 4: 0x62cc3fee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad0f): Mirror 3: 0x62cc3fee (stale) Mirror 4: 0x62cc3fee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad0f: skipped 000380 1780336854.510123 trunc from 0x02ad0f to 0x03ba6e (0x10d60 bytes) 000385 1780336854.532240 trunc from 0x03ba6f to 0x00e700 (0x2d36e bytes) 000389 1780336854.547923 write 0x021d83 thru 0x02362a (0x18a8 bytes) 000392 1780336854.554791 trunc from 0x02362b to 0x0256cc (0x20a2 bytes) 000393 1780336854.591004 read 0x00f0c1 thru 0x0172d0 (0x8210 bytes) 000395 1780336854.601358 write 0x0337b9 thru 0x03ab23 (0x736b bytes) 000397 1780336854.603472 trunc from 0x03ab24 to 0x01ec8d (0x1be96 bytes) 000398 1780336854.643308 trunc from 0x01ec8e to 0x01c9a6 (0x22e7 bytes) 000400 1780336854.672148 read 0x00f9d6 thru 0x019fb1 (0xa5dc bytes) 000403 1780336854.690490 trunc from 0x01c9a7 to 0x007dc4 (0x14be2 bytes) 000404 1780336854.722114 write 0x0338ca thru 0x03db70 (0xa2a7 bytes) 000405 1780336854.725215 trunc from 0x03db71 to 0x01b98a (0x221e6 bytes) 000408 1780336854.754235 write 0x01130c thru 0x01cbb8 (0xb8ad bytes) 000410 1780336854.763096 write 0x01860d thru 0x01c925 (0x4319 bytes) 000415 1780336854.767718 mapread 0x009f54 thru 0x015f9d (0xc04a bytes) 000416 1780336854.833051 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1cbb9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cbb9): Mirror 3: 0xa4fcbdfa (stale) Mirror 4: 0xa4fcbdfa (stale) CRC-32 checksum value for chunk [0, 0x1cbb9): Mirror 1: 0x215cb0dd Mirror 5: 0x215cb0dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cbb9: skipped 000418 1780336855.945110 mapwrite 0x022686 thru 0x0299bc (0x7337 bytes) 000419 1780336856.090029 read 0x007130 thru 0x00d4c0 (0x6391 bytes) 000423 1780336856.099340 write 0x02d9c9 thru 0x038d3b (0xb373 bytes) 000425 1780336856.101493 mapread 0x018c3e thru 0x020816 (0x7bd9 bytes) 000426 1780336856.113745 mapwrite 0x01805c thru 0x025f29 (0xdece bytes) 000430 1780336856.201475 read 0x01fa71 thru 0x022023 (0x25b3 bytes) 000431 1780336856.202391 read 0x011a8e thru 0x01438a (0x28fd bytes) 000437 1780336856.211918 read 0x00b547 thru 0x015d79 (0xa833 bytes) 000438 1780336856.233869 mapwrite 0x01563a thru 0x020cb3 (0xb67a bytes) 000440 1780336856.280911 mapread 0x02c591 thru 0x031931 (0x53a1 bytes) 000441 1780336856.281987 read 0x01988b thru 0x01bb31 (0x22a7 bytes) 000443 1780336856.284098 mapwrite 0x031509 thru 0x03437a (0x2e72 bytes) 000444 1780336856.326296 mapwrite 0x0249f1 thru 0x0341ba (0xf7ca bytes) 000447 1780336856.402937 write 0x038043 thru 0x03ffff (0x7fbd bytes) 000450 1780336856.421767 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xefa2db19 (stale) Mirror 4: 0xefa2db19 (stale) Mirror 5: 0xee8362b3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7250a3f2 Mirror 6: 0x7250a3f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000451 1780336857.593226 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xefa2db19 (stale) Mirror 4: 0xefa2db19 (stale) Mirror 5: 0xee8362b3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7250a3f2 Mirror 6: 0x7250a3f2 Mirror 7: 0x7250a3f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000452 1780336858.611879 mapwrite 0x01e55d thru 0x024166 (0x5c0a bytes) 000453 1780336858.904439 read 0x02b604 thru 0x031231 (0x5c2e bytes) 000454 1780336858.925541 mapread 0x02f784 thru 0x03982a (0xa0a7 bytes) 000456 1780336858.944856 read 0x004275 thru 0x004a27 (0x7b3 bytes) 000457 1780336858.979883 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xefa2db19 (stale) Mirror 5: 0xee8362b3 (stale) Mirror 6: 0x7250a3f2 (stale) Mirror 7: 0x7250a3f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xefa2db19 (stale) Mirror 5: 0xee8362b3 (stale) Mirror 6: 0x7250a3f2 (stale) Mirror 7: 0x7250a3f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000458 1780336859.528029 read 0x03843d thru 0x03ffff (0x7bc3 bytes) 000461 1780336859.547906 write 0x03b121 thru 0x03ffff (0x4edf bytes) 000463 1780336859.551825 read 0x03c960 thru 0x03d35c (0x9fd bytes) 000466 1780336859.555349 write 0x00351f thru 0x010707 (0xd1e9 bytes) 000467 1780336859.572924 write 0x026ed2 thru 0x02f24e (0x837d bytes) 000468 1780336859.574302 mapwrite 0x02a8d2 thru 0x036ad0 (0xc1ff bytes) 000469 1780336859.669216 read 0x0092f7 thru 0x00bddb (0x2ae5 bytes) 000470 1780336859.676641 MIRROR_RESYNC 5 mirrors 000472 1780336860.726110 write 0x032ed2 thru 0x03beaa (0x8fd9 bytes) 000477 1780336860.839926 write 0x02a12d thru 0x039eff (0xfdd3 bytes) 000480 1780336860.872907 mapread 0x006965 thru 0x013a4a (0xd0e6 bytes) 000483 1780336860.906793 write 0x000070 thru 0x009123 (0x90b4 bytes) 000484 1780336860.915942 write 0x0099a1 thru 0x00a090 (0x6f0 bytes) 000486 1780336860.918369 trunc from 0x040000 to 0x01c014 (0x23feb bytes) 000488 1780336860.975246 read 0x01bd16 thru 0x01c014 (0x2ff bytes) 000489 1780336860.981293 write 0x011185 thru 0x013d53 (0x2bcf bytes) 000490 1780336860.984153 write 0x03e360 thru 0x03ffff (0x1ca0 bytes) 000491 1780336860.986851 mapread 0x007b51 thru 0x00cbb2 (0x5062 bytes) 000492 1780336860.988969 mapread 0x012557 thru 0x0175ac (0x5056 bytes) 000495 1780336860.993221 write 0x0335c4 thru 0x03ffff (0xca3c bytes) 000497 1780336861.013220 MIRROR_RESYNC 5 mirrors 000499 1780336862.298440 trunc from 0x040000 to 0x00343a (0x3cbc5 bytes) 000500 1780336862.366194 mapwrite 0x038496 thru 0x03ffff (0x7b6a bytes) 000501 1780336862.453364 mapread 0x00c86f thru 0x00ded4 (0x1666 bytes) 000502 1780336862.463610 read 0x026e22 thru 0x0271e7 (0x3c6 bytes) 000504 1780336862.470255 write 0x006253 thru 0x00d6e1 (0x748f bytes) 000505 1780336862.471486 mapread 0x000b48 thru 0x007421 (0x68da bytes) 000506 1780336862.479495 read 0x017b1f thru 0x0264b6 (0xe998 bytes) 000508 1780336862.490101 read 0x035132 thru 0x03ffff (0xaece bytes) 000510 1780336862.491453 write 0x036d3e thru 0x03ffff (0x92c2 bytes) 000513 1780336862.496047 mapwrite 0x025e6a thru 0x02df94 (0x812b bytes) 000515 1780336862.547301 write 0x00d18a thru 0x0193c0 (0xc237 bytes) 000516 1780336862.549235 mapwrite 0x01cff0 thru 0x023f05 (0x6f16 bytes) 000518 1780336862.602801 mapread 0x01b5d2 thru 0x01eb8f (0x35be bytes) 000521 1780336862.606293 write 0x039a8b thru 0x03ffff (0x6575 bytes) 000522 1780336862.607533 mapwrite 0x02f9c4 thru 0x032886 (0x2ec3 bytes) 000524 1780336862.659095 mapwrite 0x02e1ea thru 0x031a2a (0x3841 bytes) 000525 1780336862.714692 mapwrite 0x017586 thru 0x0244f6 (0xcf71 bytes) 000526 1780336862.760944 read 0x003db8 thru 0x00e552 (0xa79b bytes) 000527 1780336862.762017 trunc from 0x040000 to 0x013622 (0x2c9dd bytes) 000528 1780336862.779795 read 0x00ee58 thru 0x013622 (0x47cb bytes) 000529 1780336862.780703 mapwrite 0x021d5d thru 0x030fa8 (0xf24c bytes) 000530 1780336862.862647 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30fa9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fa9): Mirror 4: 0x6e1878b0 (stale) Mirror 5: 0x6e1878b0 (stale) Mirror 6: 0x6e1878b0 (stale) Mirror 7: 0x6e1878b0 (stale) CRC-32 checksum value for chunk [0, 0x30fa9): Mirror 1: 0x4259d02d Mirror 8: 0x4259d02d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fa9: skipped 000532 1780336863.843744 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30fa9): Mirror 4: 0x6e1878b0 (stale) Mirror 5: 0x6e1878b0 (stale) Mirror 6: 0x6e1878b0 (stale) Mirror 7: 0x6e1878b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fa9): Mirror 4: 0x6e1878b0 (stale) Mirror 5: 0x6e1878b0 (stale) Mirror 6: 0x6e1878b0 (stale) Mirror 7: 0x6e1878b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fa9: skipped 000535 1780336864.180482 trunc from 0x030fa9 to 0x03784f (0x68a7 bytes) 000536 1780336864.221290 read 0x02dffd thru 0x02f94f (0x1953 bytes) 000538 1780336864.238426 trunc from 0x037850 to 0x00f54b (0x28304 bytes) 000539 1780336864.273841 trunc from 0x00f54c to 0x001781 (0xddca bytes) 000540 1780336864.310404 write 0x02a66b thru 0x032e0a (0x87a0 bytes) 000541 1780336864.315824 write 0x03548b thru 0x03c1cc (0x6d42 bytes) 000542 1780336864.329438 MIRROR_RESYNC 5 mirrors 000543 1780336865.358161 trunc from 0x03c1cd to 0x02b7e7 (0x109e5 bytes) 000546 1780336865.452913 write 0x021ac9 thru 0x027640 (0x5b78 bytes) 000549 1780336865.468398 mapwrite 0x02a1ea thru 0x039661 (0xf478 bytes) 000550 1780336865.569907 mapwrite 0x02f2e3 thru 0x033fdc (0x4cfa bytes) 000551 1780336865.619446 trunc from 0x039662 to 0x023d7a (0x158e7 bytes) 000552 1780336865.637854 trunc from 0x023d7b to 0x00c48d (0x178ed bytes) 000554 1780336865.679542 read 0x0087ae thru 0x00c48d (0x3ce0 bytes) 000555 1780336865.704801 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc48e): Mirror 4: 0xb002eb3f (stale) Mirror 5: 0xb002eb3f (stale) Mirror 7: 0xb002eb3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc48e): Mirror 4: 0xb002eb3f (stale) Mirror 5: 0xb002eb3f (stale) Mirror 7: 0xb002eb3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc48e: skipped 000557 1780336866.197023 read 0x0050ab thru 0x00674d (0x16a3 bytes) 000558 1780336866.202082 mapread 0x005d19 thru 0x00ba54 (0x5d3c bytes) 000559 1780336866.244329 read 0x0069a7 thru 0x00c48d (0x5ae7 bytes) 000560 1780336866.254705 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc48e): Mirror 5: 0xb002eb3f (stale) Mirror 7: 0xb002eb3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc48e): Mirror 5: 0xb002eb3f (stale) Mirror 7: 0xb002eb3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc48e: skipped 000561 1780336866.647846 write 0x005134 thru 0x00b149 (0x6016 bytes) 000562 1780336866.657253 read 0x009a61 thru 0x00c48d (0x2a2d bytes) 000563 1780336866.662204 mapread 0x006507 thru 0x00c48d (0x5f87 bytes) 000566 1780336866.666935 trunc from 0x00c48e to 0x0307a0 (0x24313 bytes) 000569 1780336866.682939 write 0x011072 thru 0x018541 (0x74d0 bytes) 000571 1780336866.693808 mapread 0x020fc9 thru 0x021b86 (0xbbe bytes) 000572 1780336866.703175 write 0x01f0ec thru 0x02dea2 (0xedb7 bytes) 000576 1780336866.711728 mapwrite 0x03948a thru 0x03ffff (0x6b76 bytes) 000578 1780336866.818049 write 0x03c01e thru 0x03ffff (0x3fe2 bytes) 000579 1780336866.820914 mapwrite 0x0213b7 thru 0x029e09 (0x8a53 bytes) 000580 1780336866.875285 trunc from 0x040000 to 0x00d1b5 (0x32e4a bytes) 000582 1780336866.893050 mapwrite 0x023c5e thru 0x02f8ca (0xbc6d bytes) 000583 1780336867.035122 mapread 0x013544 thru 0x01a897 (0x7354 bytes) 000585 1780336867.060260 mapwrite 0x00a322 thru 0x0182eb (0xdfca bytes) 000586 1780336867.105915 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f8cb): Mirror 5: 0x6061c49e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f8cb): Mirror 5: 0x6061c49e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f8cb: skipped 000587 1780336867.401757 mapwrite 0x03f515 thru 0x03ffff (0xaeb bytes) 000588 1780336867.471053 MIRROR_RESYNC 2 mirrors 000589 1780336868.185018 write 0x015c9d thru 0x01caee (0x6e52 bytes) 000590 1780336868.333360 trunc from 0x040000 to 0x0265ab (0x19a54 bytes) 000591 1780336868.388462 write 0x01e5d1 thru 0x023d12 (0x5742 bytes) 000593 1780336868.416656 mapwrite 0x018a40 thru 0x01b59c (0x2b5d bytes) 000595 1780336868.467044 mapread 0x023416 thru 0x0265ab (0x3196 bytes) 000596 1780336868.471887 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x265ac): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x265ac): Mirror 5: 0x8fd89d2e (stale) CRC-32 checksum value for chunk [0, 0x265ac): Mirror 1: 0x1940e323 Mirror 6: 0x1940e323 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x265ac: skipped 000597 1780336868.956421 trunc from 0x0265ac to 0x033a2a (0xd47f bytes) 000598 1780336869.078536 mapread 0x020740 thru 0x02b40c (0xaccd bytes) 000599 1780336869.125977 mapread 0x02a861 thru 0x02f859 (0x4ff9 bytes) 000604 1780336869.132098 write 0x025621 thru 0x0288f9 (0x32d9 bytes) 000606 1780336869.134278 mapread 0x030795 thru 0x0338cd (0x3139 bytes) 000607 1780336869.135894 mapwrite 0x004a96 thru 0x0126b1 (0xdc1c bytes) 000608 1780336869.197123 trunc from 0x033a2b to 0x02bca6 (0x7d84 bytes) 000609 1780336869.218071 MIRROR_RESYNC 3 mirrors 000611 1780336869.849096 trunc from 0x02bca7 to 0x001f8d (0x29d19 bytes) 000617 1780336869.943825 read 0x000c24 thru 0x001f8d (0x136a bytes) 000618 1780336869.949061 mapread 0x000039 thru 0x001f8d (0x1f55 bytes) 000619 1780336869.953837 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f8e): Mirror 6: 0x43e05582 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f8e): Mirror 6: 0x43e05582 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f8e: skipped 000620 1780336870.161822 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000621 1780336870.261457 mapread 0x001238 thru 0x001f8d (0xd56 bytes) 000623 1780336870.272739 write 0x03e3ae thru 0x03ffff (0x1c52 bytes) 000628 1780336870.278204 mapread 0x00d687 thru 0x0190a0 (0xba1a bytes) 000630 1780336870.298594 trunc from 0x040000 to 0x006f3b (0x390c4 bytes) 000632 1780336870.319113 write 0x025136 thru 0x02d80b (0x86d6 bytes) 000633 1780336870.321085 mapwrite 0x02f1dd thru 0x031562 (0x2386 bytes) 000634 1780336870.394411 mapread 0x01018a thru 0x015a68 (0x58df bytes) 000635 1780336870.409448 trunc from 0x031563 to 0x035141 (0x3bdf bytes) 000637 1780336870.435074 trunc from 0x035142 to 0x020a68 (0x146d9 bytes) 000638 1780336870.457171 MIRROR_EXTEND to 2 mirrors 000640 1780336871.216209 trunc from 0x020a69 to 0x005d84 (0x1ace4 bytes) 000642 1780336871.305297 trunc from 0x005d85 to 0x00b6dd (0x5959 bytes) 000644 1780336871.336060 mapwrite 0x009b3e thru 0x00c658 (0x2b1b bytes) 000645 1780336871.415392 trunc from 0x00c659 to 0x01209e (0x5a46 bytes) 000646 1780336871.436062 write 0x02b962 thru 0x0344b1 (0x8b50 bytes) 000647 1780336871.437456 mapwrite 0x02dc91 thru 0x03c77d (0xeaed bytes) 000650 1780336871.518836 mapread 0x0099e4 thru 0x01724a (0xd867 bytes) 000653 1780336871.539964 write 0x03b41c thru 0x03ffff (0x4be4 bytes) 000654 1780336871.541819 write 0x02dcba thru 0x03aa30 (0xcd77 bytes) 000655 1780336871.545710 read 0x008f50 thru 0x011eae (0x8f5f bytes) 000656 1780336871.547715 mapwrite 0x00c171 thru 0x01bed6 (0xfd66 bytes) 000660 1780336871.644286 mapwrite 0x02a063 thru 0x02add0 (0xd6e bytes) 000663 1780336871.750607 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8cc4b85d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9288a13 Mirror 3: 0xc9288a13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000664 1780336872.744401 write 0x00c874 thru 0x010c80 (0x440d bytes) 000665 1780336872.917073 mapread 0x03967c thru 0x03ffff (0x6984 bytes) 000666 1780336872.935657 write 0x028061 thru 0x034044 (0xbfe4 bytes) 000667 1780336872.957400 mapread 0x02c5cc thru 0x02f602 (0x3037 bytes) 000668 1780336872.959125 trunc from 0x040000 to 0x01990a (0x266f5 bytes) 000669 1780336872.980367 mapwrite 0x02d0ef thru 0x031f4d (0x4e5f bytes) 000670 1780336873.049201 mapread 0x02c3dc thru 0x031f4d (0x5b72 bytes) 000671 1780336873.050282 write 0x01ae7b thru 0x027150 (0xc2d6 bytes) 000673 1780336873.064035 MIRROR_RESYNC 3 mirrors 000674 1780336873.636079 mapwrite 0x02c718 thru 0x03190b (0x51f4 bytes) 000675 1780336873.752731 write 0x0058c2 thru 0x012882 (0xcfc1 bytes) 000678 1780336873.767233 trunc from 0x031f4e to 0x024050 (0xdefd bytes) 000679 1780336873.808219 read 0x007bcb thru 0x00d5a7 (0x59dd bytes) 000680 1780336873.812506 MIRROR_RESYNC 3 mirrors 000683 1780336874.359243 trunc from 0x024051 to 0x0276d3 (0x3683 bytes) 000685 1780336874.426909 write 0x017cb9 thru 0x023403 (0xb74b bytes) 000688 1780336874.437917 read 0x006ae5 thru 0x015646 (0xeb62 bytes) 000690 1780336874.452307 read 0x018a82 thru 0x023b25 (0xb0a4 bytes) 000691 1780336874.453005 read 0x009666 thru 0x01165f (0x7ffa bytes) 000692 1780336874.458413 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x276d4): Mirror 3: 0xb660cbfa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x276d4): Mirror 3: 0xb660cbfa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x276d4: skipped 000693 1780336874.764464 mapwrite 0x03c7a8 thru 0x03ffff (0x3858 bytes) 000696 1780336874.863243 mapwrite 0x023201 thru 0x02ebef (0xb9ef bytes) 000697 1780336874.990231 mapwrite 0x017f15 thru 0x01f56c (0x7658 bytes) 000698 1780336875.101983 write 0x03714e thru 0x0376c3 (0x576 bytes) 000701 1780336875.105403 mapwrite 0x038cce thru 0x03ffff (0x7332 bytes) 000702 1780336875.185380 write 0x03347c thru 0x03ffff (0xcb84 bytes) 000704 1780336875.191698 mapread 0x01dd35 thru 0x021f3e (0x420a bytes) 000705 1780336875.197020 trunc from 0x040000 to 0x00893a (0x376c5 bytes) 000706 1780336875.215271 mapwrite 0x0069fb thru 0x00f6ba (0x8cc0 bytes) 000708 1780336875.334242 mapwrite 0x00db31 thru 0x0159ea (0x7eba bytes) 000712 1780336875.460254 mapwrite 0x001dd9 thru 0x0038fc (0x1b24 bytes) 000713 1780336875.510513 trunc from 0x0159eb to 0x00dc6f (0x7d7b bytes) 000715 1780336875.533545 mapwrite 0x03810c thru 0x03ffff (0x7ef4 bytes) 000716 1780336875.633718 mapread 0x037809 thru 0x03ee71 (0x7669 bytes) 000717 1780336875.641841 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000720 1780336875.848484 MIRROR_EXTEND to 2 mirrors 000721 1780336876.616912 mapwrite 0x019674 thru 0x01fc8f (0x661c bytes) 000722 1780336876.863681 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000723 1780336877.120436 read 0x027954 thru 0x029e2b (0x24d8 bytes) 000727 1780336877.138950 read 0x032b6f thru 0x03865a (0x5aec bytes) 000730 1780336877.153104 read 0x02cdcc thru 0x03a64e (0xd883 bytes) 000731 1780336877.154199 mapread 0x0338ea thru 0x036578 (0x2c8f bytes) 000732 1780336877.156818 mapread 0x02619e thru 0x0304f3 (0xa356 bytes) 000733 1780336877.158918 write 0x02f7cd thru 0x03912b (0x995f bytes) 000735 1780336877.162176 read 0x01b3cf thru 0x01bbdb (0x80d bytes) 000739 1780336877.166089 write 0x003d5e thru 0x009c99 (0x5f3c bytes) 000740 1780336877.168715 mapread 0x01187d thru 0x01666c (0x4df0 bytes) 000742 1780336877.170463 write 0x006bac thru 0x009534 (0x2989 bytes) 000743 1780336877.174786 mapwrite 0x00c14b thru 0x018c2d (0xcae3 bytes) 000744 1780336877.311151 write 0x03af8b thru 0x03afad (0x23 bytes) 000745 1780336877.313710 trunc from 0x040000 to 0x0356e4 (0xa91b bytes) 000750 1780336877.345084 read 0x0330bf thru 0x0356e4 (0x2626 bytes) 000751 1780336877.346093 mapread 0x02b604 thru 0x02b9e9 (0x3e6 bytes) 000753 1780336877.348171 read 0x031447 thru 0x0356e4 (0x429e bytes) 000754 1780336877.352195 read 0x034bfb thru 0x0356e4 (0xaea bytes) 000755 1780336877.358384 read 0x018173 thru 0x020fc9 (0x8e57 bytes) 000756 1780336877.361047 write 0x03e7c8 thru 0x03ffff (0x1838 bytes) 000758 1780336877.373607 MIRROR_EXTEND to 2 mirrors 000759 1780336877.984377 read 0x03e187 thru 0x03ffff (0x1e79 bytes) 000760 1780336877.998013 mapwrite 0x00d919 thru 0x01b9cd (0xe0b5 bytes) 000761 1780336878.213938 mapread 0x001a6b thru 0x008fbd (0x7553 bytes) 000762 1780336878.222663 mapread 0x01f147 thru 0x020b27 (0x19e1 bytes) 000764 1780336878.233007 mapwrite 0x00a6ba thru 0x01a618 (0xff5f bytes) 000765 1780336878.280812 trunc from 0x040000 to 0x017c6f (0x28390 bytes) 000766 1780336878.323670 mapwrite 0x01d9b8 thru 0x02874b (0xad94 bytes) 000768 1780336878.402878 write 0x017124 thru 0x017629 (0x506 bytes) 000770 1780336878.408683 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2874c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2874c): Mirror 2: 0x75a4dcce (stale) CRC-32 checksum value for chunk [0, 0x2874c): Mirror 1: 0xd2b1c46c Mirror 3: 0xd2b1c46c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2874c: skipped 000773 1780336879.326264 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000774 1780336879.784415 MIRROR_RESYNC 2 mirrors 000776 1780336880.233754 read 0x010ba8 thru 0x01ff2b (0xf384 bytes) 000777 1780336880.253698 mapread 0x008f4d thru 0x00e7d3 (0x5887 bytes) 000778 1780336880.295267 read 0x021bee thru 0x02874b (0x6b5e bytes) 000779 1780336880.315739 trunc from 0x02874c to 0x013a5a (0x14cf1 bytes) 000780 1780336880.447280 mapwrite 0x01f54e thru 0x024ff1 (0x5aa4 bytes) 000781 1780336880.606913 MIRROR_RESYNC 2 mirrors 000782 1780336881.286524 read 0x001b82 thru 0x00cda0 (0xb21f bytes) 000783 1780336881.294808 mapread 0x01c7bc thru 0x0200f6 (0x393b bytes) 000784 1780336881.314370 write 0x0174da thru 0x0223a9 (0xaed0 bytes) 000785 1780336881.398686 mapwrite 0x019e58 thru 0x01b329 (0x14d2 bytes) 000786 1780336881.451368 write 0x02f623 thru 0x033a17 (0x43f5 bytes) 000787 1780336881.452901 trunc from 0x033a18 to 0x01c06e (0x179a9 bytes) 000789 1780336881.484775 MIRROR_RESYNC 2 mirrors 000790 1780336882.261585 trunc from 0x01c06f to 0x009dec (0x12282 bytes) 000791 1780336882.356178 read 0x00549a thru 0x009dec (0x4953 bytes) 000792 1780336882.362379 mapwrite 0x01fc49 thru 0x027ce1 (0x8099 bytes) 000793 1780336882.461865 trunc from 0x027ce2 to 0x00dddb (0x19f06 bytes) 000794 1780336882.483313 trunc from 0x00dddc to 0x01da37 (0xfc5c bytes) 000795 1780336882.502534 mapread 0x013460 thru 0x01ac44 (0x77e5 bytes) 000796 1780336882.518608 read 0x019bd8 thru 0x01da37 (0x3e60 bytes) 000798 1780336882.563108 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1da38): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1da38): Mirror 3: 0x7261560a (stale) CRC-32 checksum value for chunk [0, 0x1da38): Mirror 1: 0xffb7c108 Mirror 4: 0xffb7c108 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1da38: skipped 000803 1780336883.839283 trunc from 0x01da38 to 0x01a0cf (0x3968 bytes) 000804 1780336884.063530 mapread 0x00a7b2 thru 0x0117dd (0x702c bytes) 000806 1780336884.105010 trunc from 0x01a0d0 to 0x0151ac (0x4f23 bytes) 000807 1780336884.139351 mapwrite 0x0084a0 thru 0x016635 (0xe196 bytes) 000808 1780336884.221761 MIRROR_RESYNC 3 mirrors 000809 1780336884.889031 mapwrite 0x005ef3 thru 0x0138a6 (0xd9b4 bytes) 000810 1780336885.018923 mapwrite 0x021fe8 thru 0x02bb29 (0x9b42 bytes) 000811 1780336885.110426 trunc from 0x02bb2a to 0x00915c (0x229cd bytes) 000812 1780336885.123986 mapread 0x008ff3 thru 0x00915c (0x16a bytes) 000813 1780336885.125927 mapread 0x004d65 thru 0x00915c (0x43f8 bytes) 000814 1780336885.132306 MIRROR_RESYNC 3 mirrors 000815 1780336885.596251 read 0x0019df thru 0x00915c (0x777e bytes) 000818 1780336885.610255 trunc from 0x00915d to 0x03cf90 (0x33e34 bytes) 000819 1780336885.689752 write 0x02122f thru 0x02c8c2 (0xb694 bytes) 000820 1780336885.699361 trunc from 0x03cf91 to 0x003a09 (0x39587 bytes) 000822 1780336885.729018 read 0x00006b thru 0x003a09 (0x399f bytes) 000823 1780336885.744888 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a0a): Mirror 4: 0xcaf1b390 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0a): Mirror 4: 0xcaf1b390 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0a: skipped 000824 1780336886.015131 mapread 0x0001c0 thru 0x0036ca (0x350b bytes) 000827 1780336886.057182 mapread 0x0005c5 thru 0x003a09 (0x3445 bytes) 000829 1780336886.064234 MIRROR_RESYNC 2 mirrors 000830 1780336886.829216 MIRROR_RESYNC 2 mirrors 000832 1780336887.164842 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000834 1780336887.337141 read 0x000d87 thru 0x003a09 (0x2c83 bytes) 000835 1780336887.343017 trunc from 0x003a0a to 0x03343e (0x2fa35 bytes) 000839 1780336887.370296 mapwrite 0x038d9b thru 0x03ffff (0x7265 bytes) 000842 1780336887.441168 mapwrite 0x01417c thru 0x01ffd2 (0xbe57 bytes) 000843 1780336887.502614 trunc from 0x040000 to 0x037a12 (0x85ed bytes) 000844 1780336887.519362 trunc from 0x037a13 to 0x031af1 (0x5f21 bytes) 000845 1780336887.536895 read 0x026dbc thru 0x031af1 (0xad36 bytes) 000846 1780336887.547341 read 0x008d0f thru 0x00c0cb (0x33bd bytes) 000847 1780336887.558825 MIRROR_EXTEND to 2 mirrors 000848 1780336888.572517 MIRROR_RESYNC 2 mirrors 000851 1780336888.987454 write 0x0327b2 thru 0x03bb74 (0x93c3 bytes) 000852 1780336889.075443 trunc from 0x03bb75 to 0x036e5c (0x4d18 bytes) 000855 1780336889.108855 mapwrite 0x02d81b thru 0x03c33d (0xeb23 bytes) 000858 1780336889.234810 write 0x02dce2 thru 0x030d68 (0x3087 bytes) 000861 1780336889.239026 write 0x011782 thru 0x020c16 (0xf495 bytes) 000862 1780336889.256905 trunc from 0x03c33e to 0x00aa1a (0x31923 bytes) 000864 1780336889.287647 read 0x008d33 thru 0x00aa1a (0x1ce8 bytes) 000865 1780336889.294794 mapwrite 0x028e59 thru 0x02d773 (0x491b bytes) 000866 1780336889.387505 trunc from 0x02d774 to 0x0202d3 (0xd4a0 bytes) 000868 1780336889.408002 trunc from 0x0202d4 to 0x027a04 (0x7731 bytes) 000871 1780336889.426483 write 0x00fd49 thru 0x01fae0 (0xfd98 bytes) 000873 1780336889.445969 MIRROR_RESYNC 2 mirrors 000876 1780336890.148107 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000878 1780336890.294111 trunc from 0x027a05 to 0x00d291 (0x1a773 bytes) 000881 1780336890.357149 mapwrite 0x00df94 thru 0x01d9c1 (0xfa2e bytes) 000882 1780336890.464978 mapwrite 0x03e6db thru 0x03ffff (0x1925 bytes) 000884 1780336890.629023 trunc from 0x040000 to 0x00fceb (0x30314 bytes) 000886 1780336890.661711 mapwrite 0x00800a thru 0x016525 (0xe51c bytes) 000887 1780336890.804833 read 0x01338b thru 0x016525 (0x319b bytes) 000888 1780336890.810479 MIRROR_EXTEND to 2 mirrors 000902 1780336891.880454 write 0x002d3f thru 0x00f657 (0xc919 bytes) 000903 1780336891.975537 MIRROR_RESYNC 2 mirrors 000904 1780336892.769117 mapwrite 0x03df0f thru 0x03ffff (0x20f1 bytes) 000905 1780336892.906056 trunc from 0x040000 to 0x03b105 (0x4efa bytes) 000907 1780336892.931397 mapwrite 0x00781e thru 0x015657 (0xde3a bytes) 000909 1780336892.993513 read 0x012526 thru 0x01f791 (0xd26c bytes) 000910 1780336893.002031 write 0x03faf0 thru 0x03ffff (0x510 bytes) 000912 1780336893.006236 read 0x02e6fd thru 0x032284 (0x3b88 bytes) 000913 1780336893.013912 write 0x0313a4 thru 0x03eba1 (0xd7fe bytes) 000914 1780336893.018620 trunc from 0x040000 to 0x00c467 (0x33b98 bytes) 000915 1780336893.051226 read 0x007946 thru 0x00c467 (0x4b22 bytes) 000917 1780336893.056075 trunc from 0x00c468 to 0x03cbba (0x30753 bytes) 000921 1780336893.078181 trunc from 0x03cbbb to 0x017465 (0x25755 bytes) 000923 1780336893.096802 write 0x0263fb thru 0x032ef5 (0xcafb bytes) 000924 1780336893.101259 write 0x00953c thru 0x00be91 (0x2956 bytes) 000927 1780336893.111233 MIRROR_RESYNC 2 mirrors 000931 1780336893.624788 read 0x001cd2 thru 0x0066db (0x4a0a bytes) 000932 1780336893.635734 trunc from 0x032ef6 to 0x01dcb7 (0x1523e bytes) 000933 1780336893.738507 write 0x027991 thru 0x029ec1 (0x2531 bytes) 000934 1780336893.739995 mapwrite 0x03f75d thru 0x03ffff (0x8a3 bytes) 000936 1780336893.850377 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000937 1780336894.003770 write 0x0269ed thru 0x02c1af (0x57c3 bytes) 000938 1780336894.020888 mapwrite 0x00a5eb thru 0x00ca54 (0x246a bytes) 000940 1780336894.103185 mapread 0x0347cc thru 0x037b1f (0x3354 bytes) 000941 1780336894.121421 read 0x03041e thru 0x03426e (0x3e51 bytes) 000942 1780336894.122252 mapwrite 0x021cbb thru 0x024eeb (0x3231 bytes) 000944 1780336894.189070 trunc from 0x040000 to 0x015e31 (0x2a1ce bytes) 000945 1780336894.204845 mapread 0x00423f thru 0x010061 (0xbe23 bytes) 000948 1780336894.214040 mapwrite 0x035d7f thru 0x03a6b1 (0x4933 bytes) 000950 1780336894.289043 mapread 0x019c69 thru 0x01c288 (0x2620 bytes) 000954 1780336894.302851 read 0x002943 thru 0x009a08 (0x70c6 bytes) 000955 1780336894.313548 write 0x03aa2a thru 0x03ffff (0x55d6 bytes) 000956 1780336894.315846 trunc from 0x040000 to 0x034d1c (0xb2e3 bytes) 000957 1780336894.342942 mapread 0x0207e1 thru 0x02f7e3 (0xf003 bytes) 000958 1780336894.345527 mapread 0x00449b thru 0x00d738 (0x929e bytes) 000961 1780336894.350770 read 0x01cb78 thru 0x025e85 (0x930e bytes) 000963 1780336894.352635 read 0x0079cf thru 0x013fde (0xc610 bytes) 000968 1780336894.357457 mapread 0x00b1d0 thru 0x00debe (0x2cef bytes) 000971 1780336894.360486 read 0x01c906 thru 0x0233bb (0x6ab6 bytes) 000975 1780336894.365149 trunc from 0x034d1d to 0x026093 (0xec89 bytes) 000976 1780336894.402070 mapwrite 0x0364bc thru 0x03f0f7 (0x8c3c bytes) 000978 1780336894.463420 read 0x036d7b thru 0x03f0f7 (0x837d bytes) 000979 1780336894.464131 read 0x02dbba thru 0x03ca23 (0xee6a bytes) 000980 1780336894.464776 trunc from 0x03f0f8 to 0x02299b (0x1c75c bytes) 000981 1780336894.485479 write 0x0322c0 thru 0x03ffff (0xdd40 bytes) 000983 1780336894.492388 MIRROR_EXTEND to 2 mirrors 000984 1780336895.238800 trunc from 0x040000 to 0x018947 (0x276b8 bytes) 000986 1780336895.417812 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x180:0x0]: Invalid argument (22) 000987 1780336895.609236 write 0x00b535 thru 0x0104b5 (0x4f81 bytes) 000988 1780336895.623210 mapwrite 0x03d16a thru 0x03ffff (0x2e96 bytes) 000993 1780336895.769257 trunc from 0x040000 to 0x01eae8 (0x21517 bytes) 000994 1780336895.787697 mapwrite 0x008e26 thru 0x01092a (0x7b05 bytes) 000995 1780336895.839252 mapread 0x015d60 thru 0x01bbb8 (0x5e59 bytes) 000996 1780336895.848675 mapread 0x002072 thru 0x00640a (0x4399 bytes) All operations completed A-OK! PASS 70a (78s) == sanity-flr test 71: check mirror extend parent fid ==== 14:01:41 (1780336901) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 14:01:44 (1780336904) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.348753 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212294 s, 4.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 14:01:53 (1780336913) Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre2 Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3506944 .. Extending file size to 4747584 .. Extending file size to 5352704 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7383936 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7747232 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5492480 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8970272 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7912928 .. Extending file size to 8427168 .. Extending file size to 9062272 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56552 56553 56554 56557 56560 done 192.168.203.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg347-client.virtnet /mnt/lustre2 (opts:) 192.168.203.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg347-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:03:16 (1780336996) Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre2 Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4622464 .. Extending file size to 6821856 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8464704 .. Extending file size to 8082048 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8697824 .. Extending file size to 8307552 .. Extending file size to 8916992 .. Extending file size to 9126400 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58125 58126 58127 58131 58132 done 192.168.203.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg347-client.virtnet /mnt/lustre2 (opts:) 192.168.203.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg347-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:04:35 (1780337075) Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110929 s, 18.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.528047 s, 4.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8de4f086 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg347-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:04:57 (1780337097) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16528 s, 12.7 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 14:05:05 (1780337105) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cd:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:05:13 (1780337113) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191253 s, 11.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:05:21 (1780337121) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191581 s, 10.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:05:29 (1780337129) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2851 s, 7.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:06:03 (1780337163) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg347-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107679 s, 9.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:07:02 (1780337222) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg347-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219124 s, 9.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:08:08 (1780337288) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg347-server mds-ost sync done. Creating new pool test_204f oleg347-server: Pool lustre.test_204f created Adding targets to pool oleg347-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg347-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.445906 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg347-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg347-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg347-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:09:29 (1780337369) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:09:40 (1780337380) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:09:48 (1780337388) oleg347-server: Pool lustre.test_206 created Adding targets to pool oleg347-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg347-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg347-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg347-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg347-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:10:16 (1780337416) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866453 s, 12.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116462 s, 9.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf3269d41 Mirror 3: 0xf3269d41 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:10:29 (1780337429) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:10:32 (1780337432) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:10:36 (1780337436) Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.119674 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1120000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1120000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1120000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xa60000 Read this (4 bytes): BBBB 192.168.203.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg347-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:11:17 (1780337477) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0238877 s, 343 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:11:23 (1780337483) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373565 s, 28.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f2 lmm_fid: [0x200000404:0x1f2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f2 lmm_fid: [0x200000404:0x1f2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834418 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 14:11:24.000000000 -0400 Modify: 2026-06-01 14:11:25.000000000 -0400 Change: 2026-06-01 14:11:26.000000000 -0400 Birth: 2026-06-01 14:11:25.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:11:31 (1780337491) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:11:34 (1780337494) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.511098 s, 80.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0593107 s, 69.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 1 14:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 1 14:11 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:11:43 (1780337503) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:11:50 (1780337510) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1603 sec =========== 14:11:59 (1780337519) === sanity-flr: start cleanup 14:12:01 (1780337521) === === sanity-flr: finish cleanup 14:12:07 (1780337527) ===