-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 07:42:44 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg301-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:42:56 (1787830976) === oleg301-client.virtnet: executing check_config_client /mnt/lustre oleg301-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg301-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9838926f8000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9838926f8000.idle_timeout=debug disable quota as required oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:43:23 (1787831003) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7612] == sanity-flr test 0a: lfs mirror create with -N option == 07:43:26 (1787831006) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:43:40 (1787831020) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:43:44 (1787831024) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:43:46 (1787831026) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:44:05 (1787831045) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:44:07 (1787831047) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 07:44:09 (1787831049) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9838926f8000.stats=clear osc.lustre-OST0001-osc-ffff9838926f8000.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:44:15 (1787831055) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:44:24 (1787831064) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:44:30 (1787831070) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:44:33 (1787831073) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:44:38 (1787831078) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 07:44:45 (1787831085) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:44:50 (1787831090) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:44:52 (1787831092) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:44:57 (1787831097) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:45:05 (1787831105) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:45:10 (1787831110) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16389 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962381 s, 10.9 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:45:17 (1787831117) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.185421 s, 56.6 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:45:23 (1787831123) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:45:30 (1787831130) Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9838926f8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9838926f8000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9838926f8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9838926f8000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (57s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:46:27 (1787831187) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.388167 s, 10.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9838926f8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9838926f8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:47:09 (1787831229) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9838926f8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9838926f8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9838926f8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9838926f8000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (71s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:48:21 (1787831301) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.53016 s, 11.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 07:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.981s user 0m0.002s sys 0m0.177s Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9838926f8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9838926f8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.828s user 0m0.001s sys 0m0.187s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9838926f8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9838926f8000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.126s user 0m0.000s sys 0m0.170s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9838926f8000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9838926f8000.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:49:51 (1787831391) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:49:55 (1787831395) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:49:58 (1787831398) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 07:50:01 (1787831401) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 07:50:08 (1787831408) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.523978 s, 24.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:50:22 (1787831422) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481259 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816587 s, 12.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:50:29 (1787831429) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:50:37 (1787831437) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00131904 s, 776 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000729307 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00164291 s, 623 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00339032 s, 302 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 07:50:48 (1787831448) create test files with size 15758524 .. 5b3472b8b77c0cb054acaa81c2fcf420 - c3a0d4d0d7ae41d42a180489eeb5440b - b7b6db229064d5d78437549db9b49577 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 07:51:10 (1787831470) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:51:25 (1787831485) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:51:32 (1787831492) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201699 s, 5.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611895 s, 17.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 07:51:43 (1787831503) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0981791 s, 21.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.498862 s, 8.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.315581 s, 6.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 07:52:01 (1787831521) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 07:52:04 (1787831524) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:52:08 (1787831528) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670864 s, 15.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 07:52:16 (1787831536) llite.lustre-ffff9838b0615800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 07:52:23 (1787831543) llite.lustre-ffff9838b0615800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490371 s, 21.4 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289248 s, 36.3 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 07:52:31 (1787831551) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:52:35 (1787831555) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:52:45 (1787831565) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.268269 s, 39.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:52:57 (1787831577) llite.lustre-ffff983884c57800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.180949 s, 57.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff9838b0612800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:53:11 (1787831591) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:53:53 (1787831633) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:54:25 (1787831665) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 07:54:28 (1787831668) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.282493 s, 11.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.246922 s, 12.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x16d05ccc (nosync,stale) Mirror 2: 0x16d05ccc (nosync,stale) ** verify mirror 2 doesn't change original checksum: 69f8682a6a2f5af03b2a9ec5decf49f9 - mirror 1 checksum: 07a47ce50d39a54190b22b0f07ab6dda - mirror 2 checksum: 69f8682a6a2f5af03b2a9ec5decf49f9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T07:54:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T07:54:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 69f8682a6a2f5af03b2a9ec5decf49f9 - mirror 1 checksum: 07a47ce50d39a54190b22b0f07ab6dda - mirror 2 checksum: 07a47ce50d39a54190b22b0f07ab6dda - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T07:54:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T07:54:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:54:45 (1787831685) Failing mds1 on oleg301-server Stopping /mnt/lustre-mds1 (opts:) on oleg301-server 07:54:53 (1787831693) shut down facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server Failover mds1 to oleg301-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 07:55:28 (1787831728) targets are mounted 07:55:28 (1787831728) facet_failover done oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274545 s, 38.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:55:46 (1787831746) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:55:49 (1787831749) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:55:52 (1787831752) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.026448 s, 619 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:56:01 (1787831761) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:56:05 (1787831765) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0130649 s, 314 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.102182 s, 40.1 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:56:13 (1787831773) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:56:37 (1787831797) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:57:06 (1787831826) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:57:30 (1787831850) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:57:55 (1787831875) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 07:58:04 (1787831884) mirror create pid 55541 mirror split pid 55543 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6991: 55543 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6991: 55543 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55541 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55543 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 07:59:12 (1787831952) Seed set to 1954 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1ab5 000002 1787831954.444823 trunc from 00000000 to 0x001ab4 (0x1ab5 bytes) 000003 1787831954.467956 mapwrite 0x01ccb6 thru 0x025bfa (0x8f45 bytes) 000005 1787831954.737160 write 0x018188 thru 0x018d80 (0xbf9 bytes) 000007 1787831954.742197 mapwrite 0x02b747 thru 0x033ea1 (0x875b bytes) 000008 1787831954.877577 write 0x00821e thru 0x00d275 (0x5058 bytes) 000009 1787831954.896135 write 0x02863c thru 0x038248 (0xfc0d bytes) 000010 1787831954.897542 read 0x036707 thru 0x038248 (0x1b42 bytes) truncating to largest ever: 0x24c43 000014 1787831954.909632 trunc from 0x038249 to 0x024c42 (0x13606 bytes) truncating to largest ever: 0x3aef8 000016 1787831954.940089 trunc from 0x024c43 to 0x03aef7 (0x162b5 bytes) 000019 1787831954.961959 trunc from 0x03aef8 to 0x0039bb (0x3753c bytes) 000020 1787831955.001743 trunc from 0x0039bc to 0x036a55 (0x3309a bytes) 000022 1787831955.031803 mapwrite 0x00b0c0 thru 0x00f368 (0x42a9 bytes) 000024 1787831955.103329 mapread 0x01f8db thru 0x0254cb (0x5bf1 bytes) 000025 1787831955.119937 write 0x028cd0 thru 0x02edec (0x611d bytes) 000026 1787831955.144927 read 0x0246fb thru 0x0271e9 (0x2aef bytes) 000027 1787831955.145640 read 0x007d21 thru 0x012aca (0xadaa bytes) 000033 1787831955.150905 mapread 0x01572c thru 0x016f21 (0x17f6 bytes) 000035 1787831955.184709 MIRROR_EXTEND to 2 mirrors 000039 1787831956.807807 read 0x02a9a8 thru 0x036452 (0xbaab bytes) 000040 1787831956.830632 write 0x03b81c thru 0x03c661 (0xe46 bytes) 000041 1787831956.998368 mapwrite 0x032d0f thru 0x03baad (0x8d9f bytes) 000042 1787831957.148985 write 0x03abab thru 0x03ffff (0x5455 bytes) 000043 1787831957.150405 mapwrite 0x0304b0 thru 0x03f67a (0xf1cb bytes) 000046 1787831957.214277 write 0x032ee1 thru 0x03638b (0x34ab bytes) 000047 1787831957.227031 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe579afea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb134dcb6 Mirror 3: 0xb134dcb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000048 1787831958.890903 mapread 0x03f042 thru 0x03ffff (0xfbe bytes) 000050 1787831958.904571 write 0x037c67 thru 0x0396a9 (0x1a43 bytes) 000052 1787831959.023948 write 0x036dbf thru 0x03d987 (0x6bc9 bytes) 000053 1787831959.036352 mapwrite 0x030b70 thru 0x035183 (0x4614 bytes) 000054 1787831959.102967 write 0x03cfd6 thru 0x03ffff (0x302a bytes) 000057 1787831959.107806 write 0x02a18e thru 0x0378bb (0xd72e bytes) 000060 1787831959.112133 write 0x035e54 thru 0x0374b3 (0x1660 bytes) 000061 1787831959.114323 mapread 0x004ae2 thru 0x00ffad (0xb4cc bytes) 000066 1787831959.138158 write 0x02c5be thru 0x036c33 (0xa676 bytes) 000068 1787831959.142017 read 0x0319df thru 0x038cb6 (0x72d8 bytes) 000069 1787831959.143098 mapwrite 0x0229f1 thru 0x0322a0 (0xf8b0 bytes) 000070 1787831959.248387 MIRROR_RESYNC 3 mirrors 000074 1787831960.692880 mapwrite 0x00c76a thru 0x017fe4 (0xb87b bytes) 000075 1787831960.910216 MIRROR_RESYNC 3 mirrors 000076 1787831962.147107 mapwrite 0x02668a thru 0x02c3b4 (0x5d2b bytes) 000077 1787831962.359191 mapread 0x017091 thru 0x017d00 (0xc70 bytes) 000078 1787831962.372698 mapread 0x033f24 thru 0x036406 (0x24e3 bytes) 000083 1787831962.394991 mapread 0x0346e0 thru 0x037d92 (0x36b3 bytes) 000084 1787831962.397248 mapread 0x0236cd thru 0x02f7a6 (0xc0da bytes) 000087 1787831962.400723 read 0x005304 thru 0x01057a (0xb277 bytes) truncating to largest ever: 0x3da00 000090 1787831962.413922 trunc from 0x040000 to 0x03d9ff (0x2600 bytes) 000091 1787831962.449545 read 0x02f8ec thru 0x039786 (0x9e9b bytes) 000093 1787831962.459169 write 0x015e99 thru 0x01ed06 (0x8e6e bytes) 000094 1787831962.463214 write 0x02c04a thru 0x033ade (0x7a95 bytes) 000095 1787831962.469814 trunc from 0x03da00 to 0x00c418 (0x315e7 bytes) 000096 1787831962.517839 read 0x003e33 thru 0x00c418 (0x85e6 bytes) 000098 1787831962.543743 write 0x01661d thru 0x01dead (0x7891 bytes) 000100 1787831962.561938 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1deae): Mirror 3: 0xc3aeb4eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1deae): Mirror 3: 0xc3aeb4eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1deae: skipped 000103 1787831962.972821 mapwrite 0x00ed43 thru 0x01301f (0x42dd bytes) 000104 1787831963.056458 trunc from 0x01deae to 0x036185 (0x182d8 bytes) 000106 1787831963.077830 write 0x0270b1 thru 0x02dc08 (0x6b58 bytes) 000107 1787831963.098946 read 0x02ab10 thru 0x0321ce (0x76bf bytes) 000109 1787831963.109061 write 0x0217b7 thru 0x02b86e (0xa0b8 bytes) 000110 1787831963.120259 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000111 1787831963.322088 read 0x0213ed thru 0x02d9ed (0xc601 bytes) 000113 1787831963.348370 write 0x024f41 thru 0x02c9f5 (0x7ab5 bytes) 000116 1787831963.352228 mapwrite 0x00245d thru 0x00e06c (0xbc10 bytes) 000118 1787831963.462326 read 0x033c7e thru 0x034b97 (0xf1a bytes) 000119 1787831963.488938 write 0x03d2bc thru 0x03ffff (0x2d44 bytes) 000120 1787831963.489982 mapwrite 0x02606d thru 0x02ea14 (0x89a8 bytes) 000122 1787831963.627883 write 0x01b5a2 thru 0x0218f8 (0x6357 bytes) 000123 1787831963.628891 mapread 0x01f8e7 thru 0x02623b (0x6955 bytes) 000125 1787831963.639911 read 0x028578 thru 0x03823e (0xfcc7 bytes) 000126 1787831963.653796 trunc from 0x040000 to 0x006e53 (0x391ac bytes) 000127 1787831963.732875 trunc from 0x006e54 to 0x0142e3 (0xd490 bytes) 000130 1787831963.763080 write 0x021ce7 thru 0x023e38 (0x2152 bytes) 000131 1787831963.765175 write 0x03ed80 thru 0x03ffff (0x1280 bytes) 000136 1787831963.773996 write 0x02faf8 thru 0x030b0a (0x1013 bytes) 000138 1787831963.793991 read 0x024ae0 thru 0x02be17 (0x7338 bytes) 000139 1787831963.808676 MIRROR_EXTEND to 2 mirrors 000141 1787831965.293713 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000142 1787831965.536623 trunc from 0x040000 to 0x001e71 (0x3e18e bytes) 000146 1787831965.609845 mapread 0x00066b thru 0x001e71 (0x1807 bytes) 000147 1787831965.646746 trunc from 0x001e72 to 0x01891e (0x16aad bytes) 000148 1787831965.665801 read 0x0033ff thru 0x0098ff (0x6501 bytes) 000152 1787831965.683402 mapwrite 0x00e7bd thru 0x0140fb (0x593f bytes) 000153 1787831965.778873 write 0x00631d thru 0x00b324 (0x5008 bytes) 000154 1787831965.780640 mapread 0x0127f6 thru 0x01891e (0x6129 bytes) 000155 1787831965.798144 write 0x021f4d thru 0x023a3a (0x1aee bytes) 000156 1787831965.801790 mapread 0x01894d thru 0x01a5b8 (0x1c6c bytes) 000157 1787831965.813125 mapwrite 0x02521b thru 0x02fbd0 (0xa9b6 bytes) 000158 1787831965.943708 MIRROR_EXTEND to 2 mirrors 000163 1787831966.763964 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000164 1787831966.952840 read 0x02d18d thru 0x02fbd0 (0x2a44 bytes) 000168 1787831966.969941 read 0x00b79f thru 0x0142be (0x8b20 bytes) 000170 1787831966.992617 mapread 0x00ae12 thru 0x00cd4a (0x1f39 bytes) 000171 1787831966.993751 mapread 0x019353 thru 0x02751d (0xe1cb bytes) 000172 1787831966.997894 read 0x021ae7 thru 0x02fbd0 (0xe0ea bytes) 000173 1787831967.006959 MIRROR_EXTEND to 2 mirrors 000176 1787831968.023581 mapread 0x0008b2 thru 0x006e91 (0x65e0 bytes) 000182 1787831968.057933 write 0x022215 thru 0x023a15 (0x1801 bytes) 000183 1787831968.158678 read 0x0109ae thru 0x0202a4 (0xf8f7 bytes) 000184 1787831968.168839 trunc from 0x02fbd1 to 0x02ef7d (0xc53 bytes) 000186 1787831968.213804 trunc from 0x02ef7e to 0x00ddc4 (0x211b9 bytes) 000189 1787831968.265904 mapread 0x009860 thru 0x00ddc4 (0x4565 bytes) 000191 1787831968.295820 trunc from 0x00ddc5 to 0x01deee (0x1012a bytes) 000192 1787831968.334430 trunc from 0x01deef to 0x029ea5 (0xbfb7 bytes) 000194 1787831968.379931 mapread 0x008c96 thru 0x017c65 (0xefd0 bytes) 000195 1787831968.402687 mapread 0x00f4b8 thru 0x018b5a (0x96a3 bytes) 000196 1787831968.403758 mapwrite 0x02d92a thru 0x03858a (0xac61 bytes) 000199 1787831968.563846 mapread 0x01f24d thru 0x026081 (0x6e35 bytes) 000200 1787831968.577386 write 0x0009b5 thru 0x01035a (0xf9a6 bytes) 000201 1787831968.582644 mapwrite 0x00626a thru 0x0095ab (0x3342 bytes) 000202 1787831968.651297 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000203 1787831968.820893 MIRROR_EXTEND to 2 mirrors 000205 1787831969.800375 mapread 0x0166fe thru 0x025417 (0xed1a bytes) 000206 1787831969.859511 mapwrite 0x038a25 thru 0x03ffff (0x75db bytes) 000207 1787831970.087719 read 0x026d39 thru 0x0300a3 (0x936b bytes) 000208 1787831970.097044 read 0x031224 thru 0x034f09 (0x3ce6 bytes) 000211 1787831970.098870 write 0x0220b4 thru 0x02323e (0x118b bytes) 000212 1787831970.107733 write 0x0136f2 thru 0x01dd32 (0xa641 bytes) 000213 1787831970.108781 write 0x0387bb thru 0x03ffff (0x7845 bytes) 000217 1787831970.111496 mapread 0x02a2c7 thru 0x038723 (0xe45d bytes) 000218 1787831970.112569 mapread 0x03bf06 thru 0x03ffff (0x40fa bytes) 000219 1787831970.117202 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000220 1787831970.264141 write 0x005a18 thru 0x00ef3d (0x9526 bytes) 000223 1787831970.277532 trunc from 0x040000 to 0x025c0e (0x1a3f1 bytes) 000225 1787831970.299610 read 0x012b8a thru 0x01312d (0x5a4 bytes) 000230 1787831970.308929 read 0x002477 thru 0x005466 (0x2ff0 bytes) 000232 1787831970.331710 mapread 0x00e882 thru 0x01baca (0xd249 bytes) 000236 1787831970.335399 read 0x009ae1 thru 0x0112a8 (0x77c8 bytes) 000240 1787831970.337834 trunc from 0x025c0f to 0x01fabe (0x6150 bytes) 000241 1787831970.356127 trunc from 0x01fabf to 0x03380d (0x13d4f bytes) 000242 1787831970.377011 trunc from 0x03380e to 0x0214a8 (0x12365 bytes) 000243 1787831970.392728 mapread 0x015853 thru 0x018870 (0x301e bytes) 000245 1787831970.394000 write 0x0375fa thru 0x03ffff (0x8a06 bytes) 000253 1787831970.402680 mapread 0x0335b1 thru 0x03610a (0x2b5a bytes) 000254 1787831970.409633 mapwrite 0x004b97 thru 0x006410 (0x187a bytes) 000255 1787831970.464761 trunc from 0x040000 to 0x00065c (0x3f9a3 bytes) 000256 1787831970.484379 write 0x021e27 thru 0x026040 (0x421a bytes) 000259 1787831970.486665 mapread 0x01c73f thru 0x0258ab (0x916d bytes) 000260 1787831970.512023 trunc from 0x026041 to 0x013a50 (0x125f0 bytes) 000261 1787831970.525457 trunc from 0x013a51 to 0x005255 (0xe7fb bytes) 000262 1787831970.538155 read 0x0029b3 thru 0x005255 (0x28a3 bytes) 000265 1787831970.551665 write 0x02a12a thru 0x033709 (0x95e0 bytes) 000270 1787831970.557161 mapwrite 0x02a0fb thru 0x03456a (0xa470 bytes) 000274 1787831970.674623 read 0x011914 thru 0x020768 (0xee55 bytes) 000275 1787831970.675603 write 0x02e61b thru 0x02f9ff (0x13e5 bytes) 000276 1787831970.676994 mapread 0x00f247 thru 0x01036f (0x1129 bytes) 000278 1787831970.679083 read 0x02f777 thru 0x030fae (0x1838 bytes) 000279 1787831970.680079 mapread 0x012273 thru 0x01bbbc (0x994a bytes) 000280 1787831970.681905 mapwrite 0x03b5b1 thru 0x03ffff (0x4a4f bytes) 000281 1787831970.802108 trunc from 0x040000 to 0x008971 (0x3768e bytes) 000282 1787831970.827868 write 0x03da45 thru 0x03ffff (0x25bb bytes) 000283 1787831970.829532 mapread 0x02844c thru 0x032b4e (0xa703 bytes) 000287 1787831970.882448 trunc from 0x040000 to 0x0360cb (0x9f34 bytes) 000290 1787831970.902505 mapread 0x02c91f thru 0x034f69 (0x864b bytes) 000291 1787831970.912124 mapread 0x015e3d thru 0x01ece4 (0x8ea8 bytes) 000292 1787831970.960700 mapread 0x01abfc thru 0x01e7c6 (0x3bcb bytes) 000293 1787831970.962071 mapread 0x021f7a thru 0x02e672 (0xc6f9 bytes) 000296 1787831970.992116 write 0x037c18 thru 0x03c593 (0x497c bytes) 000298 1787831970.994871 mapread 0x029f9d thru 0x039bc1 (0xfc25 bytes) 000299 1787831971.005144 trunc from 0x03c594 to 0x01f8c2 (0x1ccd1 bytes) 000300 1787831971.032679 mapwrite 0x034c50 thru 0x035dab (0x115c bytes) 000302 1787831971.119717 mapwrite 0x039c14 thru 0x03c7e5 (0x2bd2 bytes) 000303 1787831971.208530 MIRROR_EXTEND to 2 mirrors 000304 1787831972.223911 read 0x0029cb thru 0x00cf0e (0xa544 bytes) 000306 1787831972.255877 mapwrite 0x00caae thru 0x0147f3 (0x7d46 bytes) 000308 1787831972.442867 write 0x033518 thru 0x03a0b7 (0x6ba0 bytes) 000309 1787831972.467147 read 0x00995b thru 0x016f96 (0xd63c bytes) 000310 1787831972.468891 mapwrite 0x024625 thru 0x026a0d (0x23e9 bytes) 000311 1787831972.660432 mapread 0x00eb36 thru 0x0150d4 (0x659f bytes) 000312 1787831972.661358 mapwrite 0x0359fb thru 0x03ffff (0xa605 bytes) 000313 1787831972.848753 trunc from 0x040000 to 0x016e06 (0x291f9 bytes) 000314 1787831972.888735 write 0x01d011 thru 0x027fb6 (0xafa6 bytes) 000315 1787831972.893573 mapwrite 0x0354a7 thru 0x03ffff (0xab59 bytes) 000317 1787831973.157151 mapwrite 0x021256 thru 0x0228ee (0x1699 bytes) 000318 1787831973.243160 trunc from 0x040000 to 0x010eb3 (0x2f14c bytes) 000319 1787831973.271420 trunc from 0x010eb4 to 0x003849 (0xd66a bytes) 000320 1787831973.312000 trunc from 0x00384a to 0x005704 (0x1ebb bytes) 000322 1787831973.332416 write 0x03c038 thru 0x03ec45 (0x2c0e bytes) 000324 1787831973.336321 trunc from 0x03ec46 to 0x001ca2 (0x3cfa3 bytes) 000325 1787831973.360169 mapread 0x000614 thru 0x001b4b (0x1538 bytes) 000327 1787831973.362451 mapwrite 0x00e222 thru 0x017cc4 (0x9aa3 bytes) 000332 1787831973.624324 MIRROR_RESYNC 2 mirrors 000333 1787831974.325408 write 0x029814 thru 0x0382b2 (0xea9f bytes) 000335 1787831974.426304 write 0x005b3a thru 0x00c6c8 (0x6b8f bytes) 000336 1787831974.454197 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x382b3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x382b3): Mirror 2: 0x96d4e5f9 (stale) CRC-32 checksum value for chunk [0, 0x382b3): Mirror 1: 0x4d91f606 Mirror 3: 0x4d91f606 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x382b3: skipped 000337 1787831975.640333 read 0x010e6b thru 0x018e64 (0x7ffa bytes) 000338 1787831975.656351 write 0x003e81 thru 0x00e367 (0xa4e7 bytes) 000340 1787831975.798712 trunc from 0x0382b3 to 0x008e77 (0x2f43b bytes) 000342 1787831975.836330 read 0x004dc2 thru 0x007a7d (0x2cbc bytes) 000343 1787831975.837879 write 0x022995 thru 0x02c6cb (0x9d37 bytes) 000345 1787831975.859905 mapwrite 0x006d4d thru 0x00e525 (0x77d9 bytes) 000347 1787831975.962219 trunc from 0x02c6cc to 0x019d0e (0x129bd bytes) 000348 1787831975.999105 read 0x0127d1 thru 0x01405b (0x188b bytes) 000349 1787831976.005134 MIRROR_RESYNC 3 mirrors 000350 1787831977.189804 trunc from 0x019d0f to 0x036c07 (0x1cef9 bytes) 000351 1787831977.290075 mapread 0x02304f thru 0x024786 (0x1738 bytes) 000352 1787831977.315997 trunc from 0x036c08 to 0x00ca29 (0x2a1de bytes) 000355 1787831977.378698 mapread 0x006175 thru 0x00bf23 (0x5daf bytes) 000356 1787831977.419347 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xca2a): Mirror 2: 0xb7d191d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xca2a): Mirror 2: 0xb7d191d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xca2a: skipped 000357 1787831977.937854 trunc from 0x00ca2a to 0x010b3a (0x4111 bytes) 000359 1787831977.979557 read 0x005f9f thru 0x010b3a (0xab9c bytes) 000360 1787831978.002229 read 0x00b2f6 thru 0x00b826 (0x531 bytes) 000361 1787831978.003822 mapwrite 0x03b1b2 thru 0x03ffff (0x4e4e bytes) 000362 1787831978.151744 mapwrite 0x02405e thru 0x02e627 (0xa5ca bytes) 000366 1787831978.286191 mapread 0x012c6d thru 0x01e73b (0xbacf bytes) 000367 1787831978.303842 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4417d2bf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5234bb2 Mirror 3: 0xf5234bb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000368 1787831979.712031 read 0x034475 thru 0x034c26 (0x7b2 bytes) 000369 1787831979.719884 mapwrite 0x010230 thru 0x01734c (0x711d bytes) 000372 1787831979.919280 read 0x032c36 thru 0x037040 (0x440b bytes) 000373 1787831979.942271 read 0x02750d thru 0x02d349 (0x5e3d bytes) 000374 1787831979.943316 mapwrite 0x00376e thru 0x004ebc (0x174f bytes) 000376 1787831980.069221 read 0x03768b thru 0x039451 (0x1dc7 bytes) 000380 1787831980.087737 MIRROR_RESYNC 3 mirrors 000383 1787831981.305161 mapwrite 0x03032e thru 0x039dce (0x9aa1 bytes) 000385 1787831981.556325 mapread 0x036b6f thru 0x03ffff (0x9491 bytes) 000386 1787831981.570127 write 0x0057ed thru 0x0061b1 (0x9c5 bytes) 000387 1787831981.582460 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1ad35b68 (stale) Mirror 3: 0x1ad35b68 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x471a81c8 Mirror 4: 0x471a81c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1787831982.692150 write 0x03ddac thru 0x03ffff (0x2254 bytes) 000392 1787831982.795051 mapread 0x00201c thru 0x010cd7 (0xecbc bytes) 000394 1787831982.823670 mapread 0x036610 thru 0x03db32 (0x7523 bytes) 000396 1787831982.834140 trunc from 0x040000 to 0x01806d (0x27f92 bytes) 000397 1787831982.868424 write 0x001e28 thru 0x008d33 (0x6f0c bytes) 000398 1787831982.878274 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1806e): Mirror 3: 0x359db48b (stale) Mirror 4: 0x2edc62cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1806e): Mirror 3: 0x359db48b (stale) Mirror 4: 0x2edc62cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1806e: skipped 000399 1787831983.383768 read 0x0146fc thru 0x01806d (0x3972 bytes) 000400 1787831983.400407 mapwrite 0x02c521 thru 0x0358d0 (0x93b0 bytes) 000401 1787831983.499763 read 0x000a6d thru 0x004139 (0x36cd bytes) 000402 1787831983.512098 read 0x00227d thru 0x003759 (0x14dd bytes) 000403 1787831983.512831 trunc from 0x0358d1 to 0x00b565 (0x2a36b bytes) 000404 1787831983.539113 MIRROR_RESYNC 3 mirrors 000405 1787831984.534516 mapwrite 0x030cf6 thru 0x034f26 (0x4231 bytes) 000407 1787831984.787738 mapread 0x026911 thru 0x030449 (0x9b39 bytes) 000408 1787831984.807265 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34f27): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f27): Mirror 3: 0xe29b4699 (stale) Mirror 4: 0xe29b4699 (stale) CRC-32 checksum value for chunk [0, 0x34f27): Mirror 1: 0x7e9edcaa Mirror 5: 0x7e9edcaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f27: skipped 000409 1787831985.947736 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34f27): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f27): Mirror 3: 0xe29b4699 (stale) Mirror 4: 0xe29b4699 (stale) CRC-32 checksum value for chunk [0, 0x34f27): Mirror 1: 0x7e9edcaa Mirror 5: 0x7e9edcaa Mirror 6: 0x7e9edcaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f27: skipped 000410 1787831987.192527 write 0x03e611 thru 0x03ffff (0x19ef bytes) 000411 1787831987.349688 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe7020144 (stale) Mirror 4: 0xe7020144 (stale) Mirror 5: 0x75624a7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe7020144 (stale) Mirror 4: 0xe7020144 (stale) Mirror 5: 0x75624a7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000412 1787831987.845261 write 0x020759 thru 0x0255d3 (0x4e7b bytes) 000413 1787831987.859591 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe7020144 (stale) Mirror 5: 0x75624a7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe7020144 (stale) Mirror 5: 0x75624a7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000415 1787831988.139610 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x75624a7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x75624a7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000416 1787831988.476653 write 0x017f2d thru 0x0241b3 (0xc287 bytes) 000418 1787831988.502715 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000420 1787831988.675884 trunc from 0x040000 to 0x0000d3 (0x3ff2c bytes) 000421 1787831988.697616 mapwrite 0x002d4e thru 0x00d7ba (0xaa6d bytes) 000422 1787831988.860471 trunc from 0x00d7bb to 0x02f3e1 (0x21c27 bytes) 000424 1787831988.906545 mapwrite 0x006567 thru 0x008427 (0x1ec1 bytes) 000425 1787831989.029348 mapwrite 0x02ab0e thru 0x039fbe (0xf4b1 bytes) 000426 1787831989.240079 mapread 0x00d273 thru 0x0158eb (0x8679 bytes) 000427 1787831989.262261 write 0x03b3b2 thru 0x03ffff (0x4c4e bytes) 000430 1787831989.272244 MIRROR_EXTEND to 2 mirrors 000431 1787831990.574875 write 0x039cd7 thru 0x03eef2 (0x521c bytes) 000432 1787831990.652010 trunc from 0x040000 to 0x00ec28 (0x313d7 bytes) 000433 1787831990.682090 trunc from 0x00ec29 to 0x00013f (0xeae9 bytes) 000434 1787831990.697902 mapwrite 0x00562b thru 0x00ecfd (0x96d3 bytes) 000435 1787831990.806010 read 0x00d92c thru 0x00ecfd (0x13d2 bytes) 000436 1787831990.806750 write 0x033cda thru 0x038d85 (0x50ac bytes) 000437 1787831990.809215 mapwrite 0x033a67 thru 0x03e3a3 (0xa93d bytes) 000438 1787831990.906511 mapwrite 0x036893 thru 0x03ffff (0x976d bytes) 000439 1787831991.022450 read 0x003e97 thru 0x00533f (0x14a9 bytes) 000441 1787831991.032952 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x90ea9158 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe802e0c2 Mirror 3: 0xe802e0c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000443 1787831992.474192 read 0x026669 thru 0x02f5f7 (0x8f8f bytes) 000444 1787831992.489301 read 0x02b8c1 thru 0x02e2d6 (0x2a16 bytes) 000445 1787831992.489734 mapread 0x025d03 thru 0x02c836 (0x6b34 bytes) 000446 1787831992.516980 read 0x023c4b thru 0x02b07d (0x7433 bytes) 000447 1787831992.521612 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x90ea9158 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe802e0c2 Mirror 3: 0xe802e0c2 Mirror 4: 0xe802e0c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000449 1787831994.153253 mapread 0x029c77 thru 0x02d7da (0x3b64 bytes) 000450 1787831994.189343 write 0x038562 thru 0x03fe88 (0x7927 bytes) 000452 1787831994.339196 mapread 0x03e3ce thru 0x03ffff (0x1c32 bytes) 000455 1787831994.341292 trunc from 0x040000 to 0x03873d (0x78c2 bytes) 000459 1787831994.360408 trunc from 0x03873e to 0x00e791 (0x29fac bytes) 000460 1787831994.383103 mapread 0x00334e thru 0x00c912 (0x95c5 bytes) 000462 1787831994.398084 mapread 0x0099cc thru 0x00e791 (0x4dc6 bytes) 000463 1787831994.403763 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xe792): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe792): Mirror 2: 0xafac44a6 (stale) Mirror 3: 0x94b150dd (stale) Mirror 4: 0x94b150dd (stale) CRC-32 checksum value for chunk [0, 0xe792): Mirror 1: 0x94b150dd Mirror 5: 0x94b150dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe792: skipped 000464 1787831995.516797 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe792): Mirror 2: 0xafac44a6 (stale) Mirror 3: 0x94b150dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe792): Mirror 2: 0xafac44a6 (stale) Mirror 3: 0x94b150dd (stale) CRC-32 checksum value for chunk [0, 0xe792): Mirror 1: 0x94b150dd Mirror 5: 0x94b150dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe792: skipped 000465 1787831996.144379 write 0x02833c thru 0x037a29 (0xf6ee bytes) 000466 1787831996.343719 read 0x0160ad thru 0x024b4f (0xeaa3 bytes) 000468 1787831996.374938 mapread 0x0240d8 thru 0x02a876 (0x679f bytes) 000469 1787831996.397852 trunc from 0x037a2a to 0x03cdc0 (0x5397 bytes) 000470 1787831996.454602 read 0x026a7e thru 0x02d8a1 (0x6e24 bytes) 000471 1787831996.455219 mapwrite 0x0231b6 thru 0x0300f8 (0xcf43 bytes) 000473 1787831996.510012 read 0x021d63 thru 0x023f02 (0x21a0 bytes) 000477 1787831996.517689 read 0x030045 thru 0x037554 (0x7510 bytes) 000478 1787831996.518339 trunc from 0x03cdc1 to 0x007917 (0x354a9 bytes) 000480 1787831996.545727 write 0x01e538 thru 0x02cfb4 (0xea7d bytes) 000481 1787831996.553027 read 0x0105d5 thru 0x014209 (0x3c35 bytes) 000484 1787831996.573559 mapread 0x023418 thru 0x02cfb4 (0x9b9d bytes) 000485 1787831996.577640 mapread 0x01cf8c thru 0x024914 (0x7989 bytes) 000486 1787831996.579758 mapread 0x01b49e thru 0x0259b1 (0xa514 bytes) 000487 1787831996.591960 MIRROR_RESYNC 4 mirrors 000488 1787831997.941812 write 0x039151 thru 0x03e4b4 (0x5364 bytes) 000489 1787831998.073139 read 0x013e99 thru 0x016711 (0x2879 bytes) 000490 1787831998.087461 mapread 0x030f25 thru 0x034701 (0x37dd bytes) 000491 1787831998.131229 trunc from 0x03e4b5 to 0x037a3a (0x6a7a bytes) 000492 1787831998.200798 mapread 0x0237bc thru 0x0277be (0x4003 bytes) 000494 1787831998.223981 mapread 0x002655 thru 0x00fd12 (0xd6be bytes) 000495 1787831998.237659 mapread 0x00def3 thru 0x014386 (0x6494 bytes) 000496 1787831998.260684 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a3b): Mirror 3: 0x2519cc82 (stale) Mirror 5: 0x2519cc82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a3b): Mirror 3: 0x2519cc82 (stale) Mirror 5: 0x2519cc82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a3b: skipped 000497 1787831998.758129 MIRROR_RESYNC 3 mirrors 000498 1787831999.765689 trunc from 0x037a3b to 0x0363b3 (0x1687 bytes) 000499 1787831999.924019 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x363b4): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x363b4): Mirror 3: 0xfece488c (stale) Mirror 5: 0xfece488c (stale) CRC-32 checksum value for chunk [0, 0x363b4): Mirror 1: 0xfece488c Mirror 6: 0xfece488c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x363b4: skipped 000500 1787832001.416461 write 0x03edeb thru 0x03ffff (0x1215 bytes) 000502 1787832001.534189 mapwrite 0x01b128 thru 0x0273f1 (0xc2ca bytes) 000503 1787832001.725625 write 0x033978 thru 0x03463e (0xcc7 bytes) 000504 1787832001.739841 write 0x02d1c1 thru 0x03bf54 (0xed94 bytes) 000505 1787832001.754403 mapread 0x0302f9 thru 0x0307c8 (0x4d0 bytes) 000506 1787832001.757869 mapread 0x033985 thru 0x03b2c2 (0x793e bytes) 000508 1787832001.763747 mapwrite 0x03a4d7 thru 0x03deef (0x3a19 bytes) 000512 1787832001.860635 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x563cabba (stale) Mirror 6: 0x563cabba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x563cabba (stale) Mirror 6: 0x563cabba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1787832002.442089 mapread 0x03db45 thru 0x03ffff (0x24bb bytes) 000516 1787832002.488229 read 0x02937f thru 0x02f199 (0x5e1b bytes) 000518 1787832002.515300 write 0x003467 thru 0x0090a2 (0x5c3c bytes) 000519 1787832002.537204 mapread 0x037c06 thru 0x03c812 (0x4c0d bytes) 000520 1787832002.553085 write 0x012f76 thru 0x021fb3 (0xf03e bytes) 000522 1787832002.568759 mapwrite 0x014a58 thru 0x020e91 (0xc43a bytes) 000523 1787832002.630185 mapread 0x01f24f thru 0x02af15 (0xbcc7 bytes) 000525 1787832002.646804 trunc from 0x040000 to 0x038b41 (0x74be bytes) 000526 1787832002.673298 trunc from 0x038b42 to 0x01aa27 (0x1e11a bytes) 000527 1787832002.689150 trunc from 0x01aa28 to 0x0189ad (0x207a bytes) 000531 1787832002.732038 trunc from 0x0189ae to 0x0262e0 (0xd933 bytes) 000533 1787832002.761260 mapwrite 0x005428 thru 0x00f2f4 (0x9ecd bytes) 000535 1787832002.877171 write 0x024f1f thru 0x02cd6e (0x7e50 bytes) 000538 1787832002.886591 mapwrite 0x019515 thru 0x01a827 (0x1313 bytes) 000539 1787832002.941621 mapwrite 0x0206f1 thru 0x0258ae (0x51be bytes) 000542 1787832003.010294 mapwrite 0x015413 thru 0x018b86 (0x3774 bytes) 000544 1787832003.068164 write 0x029f8f thru 0x0353fc (0xb46e bytes) 000545 1787832003.074118 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x353fd): Mirror 6: 0x22d32e95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x353fd): Mirror 6: 0x22d32e95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353fd: skipped 000547 1787832003.531309 read 0x012ab8 thru 0x01bea8 (0x93f1 bytes) 000548 1787832003.541862 mapwrite 0x0325b2 thru 0x0393c6 (0x6e15 bytes) 000553 1787832003.637261 write 0x02dd32 thru 0x02f760 (0x1a2f bytes) 000554 1787832003.639104 mapwrite 0x016afc thru 0x025ad9 (0xefde bytes) 000558 1787832003.768340 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x393c7): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x393c7): Mirror 6: 0xbe3fb5b0 (stale) CRC-32 checksum value for chunk [0, 0x393c7): Mirror 1: 0xacf0745f Mirror 7: 0xacf0745f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x393c7: skipped 000559 1787832005.029260 trunc from 0x0393c7 to 0x00b67b (0x2dd4b bytes) 000560 1787832005.179446 mapread 0x007619 thru 0x00b67b (0x4063 bytes) 000563 1787832005.220858 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb67c): Mirror 7: 0x19ae501e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb67c): Mirror 7: 0x19ae501e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb67c: skipped 000564 1787832005.566432 mapread 0x0085e5 thru 0x00b67b (0x3097 bytes) 000566 1787832005.594387 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xb67c): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb67c): Mirror 7: 0x19ae501e (stale) CRC-32 checksum value for chunk [0, 0xb67c): Mirror 1: 0x19ae501e Mirror 8: 0x19ae501e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb67c: skipped 000567 1787832006.738363 trunc from 0x00b67c to 0x02f36b (0x23cf0 bytes) 000568 1787832006.854994 mapwrite 0x003257 thru 0x00c703 (0x94ad bytes) 000569 1787832006.983904 mapwrite 0x020c41 thru 0x02e1e3 (0xd5a3 bytes) 000570 1787832007.106238 mapwrite 0x01e2a1 thru 0x02020b (0x1f6b bytes) 000571 1787832007.178367 mapread 0x025de5 thru 0x02cfa4 (0x71c0 bytes) 000574 1787832007.191750 mapread 0x004480 thru 0x00dbba (0x973b bytes) 000575 1787832007.195199 mapread 0x0165a7 thru 0x021877 (0xb2d1 bytes) 000576 1787832007.209518 read 0x023ae4 thru 0x02f36b (0xb888 bytes) 000577 1787832007.210747 mapread 0x00137b thru 0x0043c6 (0x304c bytes) 000579 1787832007.213407 write 0x00f2f7 thru 0x017653 (0x835d bytes) 000580 1787832007.217530 write 0x009e66 thru 0x00e88e (0x4a29 bytes) 000582 1787832007.223050 mapread 0x00dde7 thru 0x013f11 (0x612b bytes) 000584 1787832007.226057 read 0x01ad32 thru 0x023b0c (0x8ddb bytes) 000589 1787832007.229714 trunc from 0x02f36c to 0x01f206 (0x10165 bytes) 000590 1787832007.271202 read 0x005823 thru 0x011d66 (0xc544 bytes) 000591 1787832007.271939 trunc from 0x01f207 to 0x01f3ae (0x1a8 bytes) 000593 1787832007.309981 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f3af): Mirror 8: 0x41266e3c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f3af): Mirror 8: 0x41266e3c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f3af: skipped 000594 1787832007.609405 read 0x016ce0 thru 0x01e9ac (0x7ccd bytes) 000597 1787832007.635219 write 0x005518 thru 0x014f8e (0xfa77 bytes) 000601 1787832007.657429 mapread 0x00603e thru 0x00a844 (0x4807 bytes) 000602 1787832007.660509 trunc from 0x01f3af to 0x0135dd (0xbdd1 bytes) 000604 1787832007.698214 mapwrite 0x02b61c thru 0x02f208 (0x3bed bytes) 000606 1787832007.853853 mapwrite 0x0321a3 thru 0x03ffff (0xde5d bytes) 000608 1787832007.947407 write 0x028915 thru 0x0313b3 (0x8a9f bytes) 000609 1787832007.949101 mapread 0x01374d thru 0x0157d4 (0x2088 bytes) 000610 1787832007.965947 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x61e1fb88 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf7a9e7b4 Mirror 9: 0xf7a9e7b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000611 1787832008.894184 read 0x01b0fa thru 0x029caf (0xebb6 bytes) 000612 1787832008.923110 read 0x03e679 thru 0x03ffff (0x1987 bytes) 000613 1787832008.938198 trunc from 0x040000 to 0x017abe (0x28541 bytes) 000615 1787832009.115967 trunc from 0x017abf to 0x00ccfd (0xadc1 bytes) truncating to largest ever: 0x3ffa0 000617 1787832009.158008 trunc from 0x00ccfe to 0x03ff9f (0x332a2 bytes) 000618 1787832009.189470 trunc from 0x03ffa0 to 0x025974 (0x1a62b bytes) 000620 1787832009.214052 read 0x0129f3 thru 0x014057 (0x1665 bytes) 000621 1787832009.225063 mapwrite 0x006e16 thru 0x0158b0 (0xea9b bytes) 000623 1787832009.331782 read 0x00c96b thru 0x019c7b (0xd311 bytes) 000624 1787832009.332711 trunc from 0x025975 to 0x015ce7 (0xfc8d bytes) 000625 1787832009.361170 mapwrite 0x033e61 thru 0x0364db (0x267b bytes) 000626 1787832009.501540 mapwrite 0x017908 thru 0x01f1b4 (0x78ad bytes) 000628 1787832009.587332 read 0x02e9e1 thru 0x032282 (0x38a2 bytes) 000629 1787832009.590842 write 0x008215 thru 0x013f95 (0xbd81 bytes) 000630 1787832009.595849 mapwrite 0x02a82e thru 0x02da49 (0x321c bytes) 000631 1787832009.671356 mapwrite 0x0342e7 thru 0x03f251 (0xaf6b bytes) 000632 1787832009.772990 MIRROR_RESYNC 3 mirrors 000633 1787832010.701595 mapwrite 0x03a0c2 thru 0x03ffff (0x5f3e bytes) 000634 1787832010.982603 write 0x03ef63 thru 0x03ffff (0x109d bytes) 000635 1787832010.988657 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x4a00e5f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x4a00e5f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000637 1787832011.334438 trunc from 0x040000 to 0x003411 (0x3cbee bytes) 000639 1787832011.359399 mapwrite 0x00074b thru 0x005419 (0x4ccf bytes) 000640 1787832011.502691 write 0x010388 thru 0x016824 (0x649d bytes) 000643 1787832011.505377 write 0x0381a7 thru 0x0385e4 (0x43e bytes) 000644 1787832011.508231 trunc from 0x0385e5 to 0x00be24 (0x2c7c0 bytes) 000645 1787832011.540148 read 0x0060cc thru 0x00be24 (0x5d59 bytes) 000646 1787832011.552085 write 0x008db3 thru 0x0177f7 (0xea45 bytes) 000647 1787832011.556292 read 0x0143bc thru 0x0177f7 (0x343c bytes) 000651 1787832011.561406 read 0x0046ca thru 0x00a1f4 (0x5b2b bytes) 000653 1787832011.570056 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x177f8): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x177f8): Mirror 9: 0x303565b6 (stale) CRC-32 checksum value for chunk [0, 0x177f8): Mirror 1: 0x2c25caff Mirror 10: 0x2c25caff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x177f8: skipped 000654 1787832012.820194 mapwrite 0x03f00f thru 0x03ffff (0xff1 bytes) 000655 1787832012.952372 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xd5509f30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xd5509f30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000657 1787832013.247802 mapread 0x02c6f0 thru 0x03b9a3 (0xf2b4 bytes) 000659 1787832013.286346 write 0x01e948 thru 0x027a4e (0x9107 bytes) 000661 1787832013.293343 trunc from 0x040000 to 0x00404d (0x3bfb2 bytes) 000662 1787832013.312944 read 0x003c3a thru 0x00404d (0x414 bytes) 000664 1787832013.323463 mapwrite 0x029612 thru 0x02c979 (0x3368 bytes) 000673 1787832013.416654 trunc from 0x02c97a to 0x01aeb9 (0x11ac0 bytes) 000674 1787832013.448822 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000676 1787832013.664749 mapwrite 0x035a87 thru 0x03ffff (0xa579 bytes) 000677 1787832013.832670 mapwrite 0x0208ce thru 0x02a51c (0x9c4f bytes) 000679 1787832013.908978 mapwrite 0x005297 thru 0x010457 (0xb1c1 bytes) 000680 1787832013.981315 read 0x036a94 thru 0x03ffff (0x956c bytes) 000684 1787832013.984590 mapread 0x0303bc thru 0x03669c (0x62e1 bytes) 000685 1787832013.986395 trunc from 0x040000 to 0x00b21c (0x34de3 bytes) 000686 1787832014.051445 MIRROR_EXTEND to 2 mirrors 000687 1787832014.848105 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000688 1787832014.974392 mapwrite 0x02bea5 thru 0x035f22 (0xa07e bytes) 000689 1787832015.104521 MIRROR_EXTEND to 2 mirrors 000690 1787832016.192618 mapwrite 0x005404 thru 0x009181 (0x3d7e bytes) 000696 1787832016.468725 trunc from 0x035f23 to 0x033bcf (0x2353 bytes) 000697 1787832016.517315 write 0x008f88 thru 0x00a670 (0x16e9 bytes) 000700 1787832016.524219 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000703 1787832016.807030 mapread 0x002e89 thru 0x00ff05 (0xd07d bytes) 000704 1787832016.844134 mapread 0x02d731 thru 0x0319f3 (0x42c3 bytes) 000706 1787832016.855592 write 0x01d6fd thru 0x02cb1f (0xf423 bytes) 000711 1787832016.867282 trunc from 0x033bd0 to 0x00a969 (0x29266 bytes) 000714 1787832016.889462 trunc from 0x00a96a to 0x038709 (0x2dda0 bytes) 000715 1787832016.911109 read 0x027c46 thru 0x034eae (0xd269 bytes) 000717 1787832016.929784 write 0x02a950 thru 0x034d62 (0xa413 bytes) 000718 1787832016.931035 trunc from 0x03870a to 0x00fbbc (0x28b4d bytes) 000719 1787832016.951541 mapwrite 0x00a6de thru 0x019bcf (0xf4f2 bytes) 000721 1787832017.103972 mapwrite 0x03eabd thru 0x03f837 (0xd7b bytes) 000723 1787832017.179782 read 0x0210aa thru 0x02743d (0x6394 bytes) 000724 1787832017.207388 mapwrite 0x010a29 thru 0x013ae6 (0x30be bytes) 000727 1787832017.268026 write 0x01c91a thru 0x02a3aa (0xda91 bytes) 000729 1787832017.282749 read 0x031c4c thru 0x038c82 (0x7037 bytes) 000731 1787832017.283831 mapwrite 0x032408 thru 0x032fed (0xbe6 bytes) 000735 1787832017.379098 mapwrite 0x03d9cd thru 0x03ffff (0x2633 bytes) 000737 1787832017.487322 trunc from 0x040000 to 0x00e42c (0x31bd3 bytes) 000739 1787832017.530815 write 0x015f8b thru 0x01d439 (0x74af bytes) 000742 1787832017.534116 mapread 0x008991 thru 0x012c19 (0xa289 bytes) 000743 1787832017.547658 read 0x00d8da thru 0x015e8e (0x85b5 bytes) 000744 1787832017.548324 read 0x00a305 thru 0x014f55 (0xac51 bytes) 000745 1787832017.553018 mapread 0x00c1ba thru 0x016349 (0xa190 bytes) 000746 1787832017.555187 mapwrite 0x020f22 thru 0x027222 (0x6301 bytes) 000749 1787832017.653794 write 0x01dbc0 thru 0x02ba06 (0xde47 bytes) 000750 1787832017.657651 mapwrite 0x031d3b thru 0x03c49e (0xa764 bytes) 000751 1787832017.763688 mapwrite 0x02caae thru 0x036293 (0x97e6 bytes) 000755 1787832017.866704 trunc from 0x03c49f to 0x02f138 (0xd366 bytes) 000756 1787832017.903380 mapwrite 0x038701 thru 0x03ffff (0x78ff bytes) 000758 1787832018.152701 read 0x03a651 thru 0x03ffff (0x59af bytes) 000759 1787832018.154262 mapwrite 0x007f15 thru 0x0175c6 (0xf6b2 bytes) 000761 1787832018.254985 write 0x02539d thru 0x02e6be (0x9322 bytes) 000762 1787832018.256183 mapread 0x022d65 thru 0x02c468 (0x9704 bytes) 000763 1787832018.257591 mapread 0x02bcd4 thru 0x037eac (0xc1d9 bytes) 000767 1787832018.281086 mapwrite 0x0112b3 thru 0x0174d1 (0x621f bytes) 000768 1787832018.372582 mapwrite 0x00ef08 thru 0x013ad8 (0x4bd1 bytes) 000770 1787832018.442737 mapwrite 0x02f244 thru 0x034f41 (0x5cfe bytes) 000771 1787832018.500172 write 0x00e6d6 thru 0x016b58 (0x8483 bytes) 000773 1787832018.502922 read 0x00c4a7 thru 0x01a690 (0xe1ea bytes) 000776 1787832018.504843 mapread 0x03b0fd thru 0x03ffff (0x4f03 bytes) 000777 1787832018.505798 mapread 0x00cabd thru 0x00ece1 (0x2225 bytes) 000779 1787832018.506929 trunc from 0x040000 to 0x02b5b6 (0x14a49 bytes) 000781 1787832018.530380 mapwrite 0x003daa thru 0x011b59 (0xddb0 bytes) 000783 1787832018.634201 read 0x01ae9b thru 0x028e37 (0xdf9d bytes) 000786 1787832018.636174 write 0x03b927 thru 0x03c582 (0xc5c bytes) 000788 1787832018.639907 mapread 0x016f28 thru 0x01981a (0x28f3 bytes) 000789 1787832018.644632 read 0x0193f1 thru 0x01c782 (0x3392 bytes) 000791 1787832018.647754 read 0x01ccd5 thru 0x022cb3 (0x5fdf bytes) 000792 1787832018.648450 trunc from 0x03c583 to 0x02a03c (0x12546 bytes) 000793 1787832018.695120 trunc from 0x02a03d to 0x005316 (0x24d26 bytes) 000794 1787832018.730004 write 0x018e3c thru 0x0233f2 (0xa5b7 bytes) 000796 1787832018.732451 write 0x03cb33 thru 0x03ffff (0x34cd bytes) 000799 1787832018.737406 mapread 0x0310e4 thru 0x0361ac (0x50c9 bytes) 000802 1787832018.754828 mapread 0x024e08 thru 0x03178c (0xc985 bytes) 000803 1787832018.757130 trunc from 0x040000 to 0x02fbd1 (0x1042e bytes) 000805 1787832018.808632 mapread 0x01aa5c thru 0x01d475 (0x2a1a bytes) 000807 1787832018.815269 read 0x01002d thru 0x018c3c (0x8c10 bytes) 000808 1787832018.817317 read 0x01b6cc thru 0x027e67 (0xc79c bytes) 000812 1787832018.823888 read 0x012c4b thru 0x01431b (0x16d1 bytes) 000813 1787832018.824927 write 0x01918b thru 0x0200a4 (0x6f1a bytes) 000815 1787832018.828407 mapread 0x01a991 thru 0x0252d9 (0xa949 bytes) 000816 1787832018.829844 read 0x002093 thru 0x00d40a (0xb378 bytes) 000817 1787832018.831386 mapread 0x013012 thru 0x019178 (0x6167 bytes) 000818 1787832018.834552 mapread 0x018273 thru 0x027bb9 (0xf947 bytes) 000820 1787832018.841947 write 0x00b6c2 thru 0x0168c1 (0xb200 bytes) 000821 1787832018.848926 read 0x0298e1 thru 0x02fbd1 (0x62f1 bytes) 000823 1787832018.850698 mapread 0x018518 thru 0x0263a7 (0xde90 bytes) 000824 1787832018.851747 write 0x01aaec thru 0x026a56 (0xbf6b bytes) 000826 1787832018.854201 trunc from 0x02fbd2 to 0x0111b2 (0x1ea1f bytes) 000829 1787832018.876978 read 0x00d6de thru 0x0111b2 (0x3ad5 bytes) 000830 1787832018.877688 write 0x0146e7 thru 0x0235da (0xeef4 bytes) 000831 1787832018.879405 mapread 0x01ff33 thru 0x0235da (0x36a8 bytes) 000832 1787832018.880995 trunc from 0x0235db to 0x0395b7 (0x15fdd bytes) 000834 1787832018.903320 trunc from 0x0395b8 to 0x01a281 (0x1f336 bytes) 000835 1787832018.927554 write 0x02243b thru 0x0300de (0xdca4 bytes) 000836 1787832018.930010 trunc from 0x0300df to 0x038b10 (0x8a32 bytes) 000837 1787832018.945358 mapread 0x0105f2 thru 0x01f6f6 (0xf105 bytes) 000838 1787832018.960161 trunc from 0x038b11 to 0x027f54 (0x10bbc bytes) 000839 1787832018.976980 mapwrite 0x0319ce thru 0x03c376 (0xa9a9 bytes) 000840 1787832019.189267 trunc from 0x03c377 to 0x02c933 (0xfa43 bytes) 000842 1787832019.213422 trunc from 0x02c934 to 0x01e779 (0xe1ba bytes) 000843 1787832019.247390 read 0x01092a thru 0x014fea (0x46c1 bytes) 000844 1787832019.250438 mapread 0x00dfa9 thru 0x010b47 (0x2b9f bytes) 000845 1787832019.251369 mapwrite 0x0272e6 thru 0x02da13 (0x672e bytes) 000847 1787832019.403650 read 0x01c509 thru 0x01ff78 (0x3a70 bytes) 000848 1787832019.416941 MIRROR_EXTEND to 2 mirrors 000849 1787832020.248327 read 0x01e9fd thru 0x0215dd (0x2be1 bytes) 000850 1787832020.267846 mapread 0x01f689 thru 0x02294b (0x32c3 bytes) 000851 1787832020.273564 read 0x005506 thru 0x009376 (0x3e71 bytes) 000852 1787832020.302707 mapwrite 0x00b83c thru 0x01322e (0x79f3 bytes) 000853 1787832020.494341 mapwrite 0x029f2a thru 0x02da53 (0x3b2a bytes) 000854 1787832020.608745 read 0x013ab3 thru 0x01dd06 (0xa254 bytes) 000855 1787832020.618549 trunc from 0x02da54 to 0x024e54 (0x8bff bytes) 000857 1787832020.641470 mapwrite 0x014da3 thru 0x021527 (0xc785 bytes) 000858 1787832020.714438 read 0x004e0a thru 0x00e516 (0x970d bytes) 000859 1787832020.715306 trunc from 0x024e55 to 0x00e56f (0x168e5 bytes) 000860 1787832020.756099 mapwrite 0x00a2cb thru 0x011421 (0x7157 bytes) 000861 1787832020.912040 write 0x00c5fe thru 0x0100a4 (0x3aa7 bytes) 000862 1787832020.918117 MIRROR_RESYNC 2 mirrors 000866 1787832021.778960 trunc from 0x011422 to 0x01c125 (0xad04 bytes) 000868 1787832021.890651 mapwrite 0x00aa9e thru 0x00dede (0x3441 bytes) 000871 1787832021.987829 write 0x021712 thru 0x024ebe (0x37ad bytes) 000872 1787832021.989570 write 0x022469 thru 0x02a2ce (0x7e66 bytes) 000876 1787832021.993737 mapwrite 0x022b86 thru 0x026dd2 (0x424d bytes) 000878 1787832022.051228 write 0x026a20 thru 0x0347bb (0xdd9c bytes) 000879 1787832022.063423 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x347bc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347bc): Mirror 2: 0x3efd5bb5 (stale) CRC-32 checksum value for chunk [0, 0x347bc): Mirror 1: 0xb4434c65 Mirror 3: 0xb4434c65 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347bc: skipped 000880 1787832023.174696 MIRROR_RESYNC 3 mirrors 000882 1787832024.014459 mapread 0x0272e7 thru 0x0347bb (0xd4d5 bytes) 000883 1787832024.050377 read 0x00d940 thru 0x019034 (0xb6f5 bytes) 000884 1787832024.085832 MIRROR_RESYNC 3 mirrors 000886 1787832024.687467 read 0x017c1f thru 0x01ae21 (0x3203 bytes) 000888 1787832024.694456 mapwrite 0x028631 thru 0x0344b4 (0xbe84 bytes) 000889 1787832024.879362 mapwrite 0x014601 thru 0x0214b0 (0xceb0 bytes) 000890 1787832024.995087 mapwrite 0x006189 thru 0x00c270 (0x60e8 bytes) 000892 1787832025.073983 write 0x012bf4 thru 0x017e78 (0x5285 bytes) 000894 1787832025.080555 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x347bc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347bc): Mirror 2: 0xb4434c65 (stale) Mirror 3: 0xb4434c65 (stale) CRC-32 checksum value for chunk [0, 0x347bc): Mirror 1: 0x429e491a Mirror 4: 0x429e491a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347bc: skipped 000895 1787832026.202769 trunc from 0x0347bc to 0x01dfb3 (0x16808 bytes) 000896 1787832026.285258 write 0x001481 thru 0x00a3ed (0x8f6d bytes) 000897 1787832026.297695 write 0x03ef48 thru 0x03ffff (0x10b8 bytes) 000898 1787832026.301612 write 0x032353 thru 0x03ffff (0xdcad bytes) 000900 1787832026.319674 read 0x010147 thru 0x014d9e (0x4c58 bytes) 000901 1787832026.325327 mapread 0x02dfb5 thru 0x033f4d (0x5f99 bytes) 000903 1787832026.367942 mapread 0x02c1a0 thru 0x036625 (0xa486 bytes) 000904 1787832026.369191 write 0x015ede thru 0x024e8d (0xefb0 bytes) 000905 1787832026.387095 mapwrite 0x033cc1 thru 0x03a20d (0x654d bytes) 000906 1787832026.473280 mapread 0x02106c thru 0x02afe8 (0x9f7d bytes) 000907 1787832026.480833 mapread 0x017120 thru 0x025f01 (0xede2 bytes) 000908 1787832026.482703 mapwrite 0x02fea2 thru 0x035ee3 (0x6042 bytes) 000911 1787832026.528571 mapread 0x01bd10 thru 0x027791 (0xba82 bytes) 000912 1787832026.530067 read 0x03f27d thru 0x03ffff (0xd83 bytes) 000916 1787832026.533068 write 0x0063a6 thru 0x013066 (0xccc1 bytes) 000918 1787832026.538064 mapread 0x025dc8 thru 0x02b423 (0x565c bytes) 000920 1787832026.539424 mapread 0x01fbb6 thru 0x022aff (0x2f4a bytes) 000922 1787832026.546232 MIRROR_RESYNC 4 mirrors 000926 1787832027.534110 MIRROR_EXTEND to 5 mirrors 000928 1787832028.275959 trunc from 0x040000 to 0x0009b3 (0x3f64c bytes) 000929 1787832028.428406 trunc from 0x0009b4 to 0x02832b (0x27978 bytes) 000930 1787832028.463929 mapwrite 0x017e43 thru 0x025254 (0xd412 bytes) 000932 1787832028.604679 mapwrite 0x02c866 thru 0x034a3a (0x81d5 bytes) 000934 1787832028.700096 trunc from 0x034a3b to 0x03ab21 (0x60e7 bytes) 000936 1787832028.726412 trunc from 0x03ab22 to 0x02bb74 (0xefad bytes) 000937 1787832028.747847 write 0x0387bd thru 0x03ffff (0x7843 bytes) 000938 1787832028.749292 trunc from 0x040000 to 0x03dc70 (0x238f bytes) 000939 1787832028.782445 mapwrite 0x0176f9 thru 0x01d293 (0x5b9b bytes) 000941 1787832028.859002 mapread 0x003aec thru 0x00fee3 (0xc3f8 bytes) 000942 1787832028.876406 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3dc71): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc71): Mirror 2: 0xeb2f4e26 (stale) Mirror 3: 0xeb2f4e26 (stale) Mirror 4: 0xeb2f4e26 (stale) Mirror 5: 0xeb2f4e26 (stale) CRC-32 checksum value for chunk [0, 0x3dc71): Mirror 1: 0xc5b44afe Mirror 6: 0xc5b44afe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc71: skipped 000944 1787832029.900392 read 0x00a151 thru 0x00b875 (0x1725 bytes) 000945 1787832029.915757 mapwrite 0x028499 thru 0x02ec2d (0x6795 bytes) 000947 1787832030.190874 mapwrite 0x039c6c thru 0x03ffff (0x6394 bytes) 000949 1787832030.280850 trunc from 0x040000 to 0x0197ac (0x26853 bytes) 000950 1787832030.305065 mapread 0x007343 thru 0x011426 (0xa0e4 bytes) 000953 1787832030.338890 MIRROR_RESYNC 6 mirrors 000954 1787832031.608808 mapwrite 0x02032e thru 0x026e09 (0x6adc bytes) 000955 1787832031.842086 trunc from 0x026e0a to 0x00c720 (0x1a6e9 bytes) 000956 1787832031.866307 read 0x00a94d thru 0x00c720 (0x1dd4 bytes) 000957 1787832031.879101 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xc721): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc721): Mirror 2: 0xf2d75baa (stale) Mirror 3: 0xf2d75baa (stale) Mirror 4: 0xf2d75baa (stale) Mirror 5: 0xf2d75baa (stale) Mirror 6: 0xf2d75baa (stale) CRC-32 checksum value for chunk [0, 0xc721): Mirror 1: 0xf2d75baa Mirror 7: 0xf2d75baa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc721: skipped 000961 1787832033.152893 write 0x007707 thru 0x00a1e2 (0x2adc bytes) 000963 1787832033.271238 read 0x00814b thru 0x00c720 (0x45d6 bytes) 000965 1787832033.278659 read 0x00194f thru 0x007105 (0x57b7 bytes) 000967 1787832033.290240 trunc from 0x00c721 to 0x008e12 (0x390e bytes) 000969 1787832033.315517 read 0x004206 thru 0x005bef (0x19ea bytes) 000970 1787832033.316134 trunc from 0x008e13 to 0x010e79 (0x8067 bytes) 000971 1787832033.339938 MIRROR_RESYNC 7 mirrors 000972 1787832034.938371 trunc from 0x010e7a to 0x02efd4 (0x1e15b bytes) 000975 1787832035.035082 mapread 0x01d6f3 thru 0x023ed7 (0x67e5 bytes) 000976 1787832035.060012 trunc from 0x02efd5 to 0x00aecf (0x24105 bytes) 000977 1787832035.083469 trunc from 0x00aed0 to 0x03e983 (0x33ab4 bytes) 000978 1787832035.106467 mapwrite 0x038bc7 thru 0x03909c (0x4d6 bytes) 000979 1787832035.182113 mapwrite 0x008bf1 thru 0x00b5bd (0x29cd bytes) 000981 1787832035.263567 trunc from 0x03e984 to 0x03ee43 (0x4c0 bytes) 000982 1787832035.284886 read 0x020ac6 thru 0x0217a7 (0xce2 bytes) 000983 1787832035.300389 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ee44): Mirror 2: 0xd16952be (stale) Mirror 3: 0xd16952be (stale) Mirror 4: 0xd16952be (stale) Mirror 5: 0xd16952be (stale) Mirror 7: 0xd16952be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee44): Mirror 2: 0xd16952be (stale) Mirror 3: 0xd16952be (stale) Mirror 4: 0xd16952be (stale) Mirror 5: 0xd16952be (stale) Mirror 7: 0xd16952be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee44: skipped 000984 1787832035.738298 trunc from 0x03ee44 to 0x0249f4 (0x1a44f bytes) 000985 1787832035.757867 mapread 0x014b03 thru 0x0225bc (0xdaba bytes) 000987 1787832035.786338 write 0x01a082 thru 0x0262b8 (0xc237 bytes) 000989 1787832035.792837 MIRROR_RESYNC 6 mirrors 000991 1787832037.084697 mapread 0x022436 thru 0x0262b8 (0x3e83 bytes) 000992 1787832037.109789 mapread 0x00ba29 thru 0x011efb (0x64d3 bytes) 000997 1787832037.137006 trunc from 0x0262b9 to 0x010be4 (0x156d4 bytes) 000998 1787832037.237300 trunc from 0x010be5 to 0x00d356 (0x388e bytes) 000999 1787832037.272338 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd357): Mirror 2: 0xd899184b (stale) Mirror 3: 0xd899184b (stale) Mirror 4: 0xd899184b (stale) Mirror 7: 0xd899184b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd357): Mirror 2: 0xd899184b (stale) Mirror 3: 0xd899184b (stale) Mirror 4: 0xd899184b (stale) Mirror 7: 0xd899184b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd357: skipped 001000 1787832037.665177 mapread 0x00c96a thru 0x00d356 (0x9ed bytes) All operations completed A-OK! PASS 70a (90s) == sanity-flr test 71: check mirror extend parent fid ==== 08:00:42 (1787832042) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 08:00:45 (1787832045) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.281287 s, 7.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176202 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 08:00:54 (1787832054) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3406336 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1285376 .. done Extending file size to 4642688 .. Extending file size to 4825856 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6802784 .. Extending file size to 4851520 .. Extending file size to 8709056 .. Extending file size to 8663360 .. Extending file size to 8856064 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9138272 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8879296 .. Extending file size to 9367840 .. Waiting 58612 58614 58616 58617 58620 failed 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) 192.168.203.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:02:11 (1787832131) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1228096 .. Extending file size to 7601824 .. Extending file size to 6832672 .. Extending file size to 7832480 .. Extending file size to 8636096 .. Extending file size to 8237344 .. Extending file size to 8271328 .. Extending file size to 8877376 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9010176 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8453920 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8797184 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8817280 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9038976 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60212 60214 60215 60217 60218 done 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) 192.168.203.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:03:29 (1787832209) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0805795 s, 26.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.330205 s, 6.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xbd1c3f90 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:03:46 (1787832226) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0940909 s, 22.3 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 08:03:52 (1787832232) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:03:58 (1787832238) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206288 s, 10.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:04:05 (1787832245) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166732 s, 12.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:04:12 (1787832252) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180819 s, 11.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:04:45 (1787832285) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145498 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:05:42 (1787832342) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239345 s, 8.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:06:37 (1787832397) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. Creating new pool test_204f oleg301-server: Pool lustre.test_204f created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg301-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.342331 s, 6.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg301-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:07:58 (1787832478) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:08:06 (1787832486) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:08:13 (1787832493) oleg301-server: Pool lustre.test_206 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg301-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:08:38 (1787832518) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724075 s, 14.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489985 s, 21.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6980b5b6 Mirror 3: 0x6980b5b6 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:08:50 (1787832530) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:08:54 (1787832534) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:08:59 (1787832539) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.114698 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1380000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1380000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1380000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2460000 Read this (4 bytes): BBBB 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:09:33 (1787832573) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0216039 s, 379 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:09:40 (1787832580) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518613 s, 20.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f5 lmm_fid: [0x200000404:0x1f5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f5 lmm_fid: [0x200000404:0x1f5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834421 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 08:09:41.000000000 -0400 Modify: 2026-08-27 08:09:41.000000000 -0400 Change: 2026-08-27 08:09:43.000000000 -0400 Birth: 2026-08-27 08:09:40.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:09:47 (1787832587) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:09:50 (1787832590) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.603688 s, 67.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0490322 s, 83.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 08:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f7:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 08:09 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:09:59 (1787832599) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:10:07 (1787832607) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1651 sec =========== 08:10:16 (1787832616) === sanity-flr: start cleanup 08:10:18 (1787832618) === === sanity-flr: finish cleanup 08:10:25 (1787832625) ===