-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 10:16:33 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg456-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:16:54 (1787840214) === oleg456-client.virtnet: executing check_config_client /mnt/lustre oleg456-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg456-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff997947d71000.idle_timeout=debug osc.lustre-OST0001-osc-ffff997947d71000.idle_timeout=debug disable quota as required oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:17:30 (1787840250) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7636] == sanity-flr test 0a: lfs mirror create with -N option == 10:17:33 (1787840253) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:17:52 (1787840272) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:17:55 (1787840275) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:17:58 (1787840278) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:18:25 (1787840305) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:18:28 (1787840308) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 10:18:32 (1787840312) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff997947d71000.stats=clear osc.lustre-OST0001-osc-ffff997947d71000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:18:40 (1787840320) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:18:51 (1787840331) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:19:00 (1787840340) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:19:04 (1787840344) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:19:12 (1787840352) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 10:19:25 (1787840365) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:19:32 (1787840372) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:19:35 (1787840375) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:19:43 (1787840383) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:19:54 (1787840394) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:20:01 (1787840401) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.78795 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100015 s, 10.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:20:12 (1787840412) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.375916 s, 39.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:20:20 (1787840420) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:20:29 (1787840429) Stopping /mnt/lustre-ost1 (opts:) on oleg456-server oleg456-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff997947d71000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff997947d71000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg456-server: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg456-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg456-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg456-server oleg456-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff997947d71000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff997947d71000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg456-server: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg456-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (141s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:22:50 (1787840570) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.629114 s, 16.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg456-server oleg456-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff997947d71000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff997947d71000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg456-server: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg456-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:23:34 (1787840614) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg456-server oleg456-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff997947d71000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff997947d71000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg456-server: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg456-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg456-server oleg456-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff997947d71000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff997947d71000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg456-server: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg456-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:24:54 (1787840694) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.89266 s, 10.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 10:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.542s user 0m0.002s sys 0m0.204s Stopping /mnt/lustre-ost1 (opts:) on oleg456-server oleg456-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff997947d71000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff997947d71000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.400s user 0m0.002s sys 0m0.167s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg456-server: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg456-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg456-server oleg456-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff997947d71000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff997947d71000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.649s user 0m0.000s sys 0m0.193s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg456-server: oleg456-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg456-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg456-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff997947d71000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff997947d71000.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:26:39 (1787840799) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:26:43 (1787840803) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:26:49 (1787840809) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (6s) == sanity-flr test 35: allow to write to mirrored files == 10:26:54 (1787840814) PASS 35 (11s) == sanity-flr test 36a: write to mirrored files ========== 10:27:05 (1787840825) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.593537 s, 21.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:27:18 (1787840838) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307099 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261494 s, 40.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:27:25 (1787840845) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:27:33 (1787840853) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00137334 s, 746 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101003 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000940233 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00281637 s, 364 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 10:27:45 (1787840865) create test files with size 15743902 .. fd9ce4e6654f09adcf3b5705deb3fdf4 - 0b3b5df9136c4629a90090063d2cfdd1 - b58fbc12a77368d39ca887ed50b1c1d3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg456-client.virtnet /mnt/lustre (opts:) Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 10:28:13 (1787840893) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:28:30 (1787840910) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:28:38 (1787840918) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115692 s, 9.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281433 s, 37.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 10:28:50 (1787840930) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143572 s, 14.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.337183 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.31899 s, 6.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 10:29:10 (1787840950) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 10:29:14 (1787840954) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:29:17 (1787840957) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607708 s, 17.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 10:29:26 (1787840966) llite.lustre-ffff9979455ff000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 10:29:35 (1787840975) llite.lustre-ffff9979455ff000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037026 s, 28.3 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397921 s, 26.4 MB/s **verify components PASS 43d (12s) == sanity-flr test 44a: lfs mirror split check =========== 10:29:47 (1787840987) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:29:51 (1787840991) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:30:02 (1787841002) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.33342 s, 31.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg456-client.virtnet /mnt/lustre (opts:) Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:30:14 (1787841014) llite.lustre-ffff99795147d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.286623 s, 36.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg456-client.virtnet /mnt/lustre (opts:) Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff997944e20800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:30:28 (1787841028) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:31:14 (1787841074) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (45s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:31:59 (1787841119) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:32:02 (1787841122) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.286743 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.161303 s, 19.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5509d7d3 (nosync,stale) Mirror 2: 0x5509d7d3 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 328cfbee4bd4ea0896138d171c8c23c0 - mirror 1 checksum: 639450672ff22ce40fc22137be7ab2d6 - mirror 2 checksum: 328cfbee4bd4ea0896138d171c8c23c0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T10:32:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T10:32:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 328cfbee4bd4ea0896138d171c8c23c0 - mirror 1 checksum: 639450672ff22ce40fc22137be7ab2d6 - mirror 2 checksum: 639450672ff22ce40fc22137be7ab2d6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T10:32:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T10:32:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe2:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:32:15 (1787841135) Failing mds1 on oleg456-server Stopping /mnt/lustre-mds1 (opts:) on oleg456-server 10:32:22 (1787841142) shut down facet: mds1 facet_host: oleg456-server facet_failover_host: oleg456-server Failover mds1 to oleg456-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 10:32:59 (1787841179) targets are mounted 10:32:59 (1787841179) facet_failover done oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452997 s, 23.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:33:20 (1787841200) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:33:24 (1787841204) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:33:28 (1787841208) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0225727 s, 726 kB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:33:40 (1787841220) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:33:46 (1787841226) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0142458 s, 288 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.068966 s, 59.4 kB/s PASS 60a (12s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:33:57 (1787841237) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:34:23 (1787841263) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:34:56 (1787841296) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:35:20 (1787841320) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:35:48 (1787841348) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 10:35:58 (1787841358) mirror create pid 55608 mirror split pid 55609 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55608 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55609 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 10:37:06 (1787841426) Seed set to 1428 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1787841428.905698 write 0x01b872 thru 0x02107e (0x580d bytes) 000002 1787841428.939194 read 0x001122 thru 0x00c022 (0xaf01 bytes) 000007 1787841428.954983 mapread 0x019b92 thru 0x02107e (0x74ed bytes) truncating to largest ever: 0x33e37 000008 1787841428.969779 trunc from 0x02107f to 0x033e36 (0x12db8 bytes) 000009 1787841428.990910 mapwrite 0x029d9e thru 0x02b280 (0x14e3 bytes) 000012 1787841429.080055 mapwrite 0x00c92a thru 0x00dbe4 (0x12bb bytes) 000015 1787841429.252993 trunc from 0x033e37 to 0x0309fe (0x3438 bytes) 000016 1787841429.271615 mapwrite 0x0030c6 thru 0x006f36 (0x3e71 bytes) 000017 1787841429.327998 write 0x016805 thru 0x017d5f (0x155b bytes) 000019 1787841429.336689 trunc from 0x0309ff to 0x00c564 (0x2449a bytes) 000022 1787841429.371264 write 0x036d90 thru 0x03ffff (0x9270 bytes) 000031 1787841429.385836 read 0x010312 thru 0x01b1ec (0xaedb bytes) 000034 1787841429.405673 write 0x015a54 thru 0x016d1c (0x12c9 bytes) 000038 1787841429.419350 write 0x027244 thru 0x02aaed (0x38aa bytes) 000040 1787841429.421056 mapread 0x010a80 thru 0x020429 (0xf9aa bytes) 000044 1787841429.429813 write 0x02308b thru 0x029efe (0x6e74 bytes) 000046 1787841429.434910 read 0x0271dc thru 0x03062e (0x9453 bytes) 000047 1787841429.435991 mapwrite 0x01a695 thru 0x01b752 (0x10be bytes) 000048 1787841429.570326 mapwrite 0x029f8f thru 0x0308fe (0x6970 bytes) 000049 1787841429.661834 mapwrite 0x034b7b thru 0x035917 (0xd9d bytes) 000052 1787841429.826112 trunc from 0x040000 to 0x0201b4 (0x1fe4b bytes) 000053 1787841429.872208 mapwrite 0x03b2b7 thru 0x03ffff (0x4d49 bytes) 000056 1787841430.117990 read 0x0137fb thru 0x01a323 (0x6b29 bytes) 000057 1787841430.119743 read 0x0069ab thru 0x01141c (0xaa72 bytes) 000058 1787841430.120390 write 0x0333f4 thru 0x037d71 (0x497e bytes) 000059 1787841430.124536 trunc from 0x040000 to 0x013583 (0x2ca7c bytes) 000060 1787841430.166571 read 0x00c046 thru 0x013583 (0x753e bytes) truncating to largest ever: 0x381e6 000061 1787841430.167268 trunc from 0x013584 to 0x0381e5 (0x24c62 bytes) 000063 1787841430.201536 mapwrite 0x02989c thru 0x02cda1 (0x3506 bytes) 000064 1787841430.276672 read 0x02c4e2 thru 0x0381e5 (0xbd04 bytes) 000065 1787841430.277660 mapread 0x002394 thru 0x006242 (0x3eaf bytes) 000066 1787841430.279455 mapwrite 0x00b49a thru 0x01b09f (0xfc06 bytes) 000067 1787841430.332171 mapread 0x0330b9 thru 0x0381e5 (0x512d bytes) 000068 1787841430.333896 mapread 0x00192d thru 0x00ab64 (0x9238 bytes) truncating to largest ever: 0x39fbf 000071 1787841430.337872 trunc from 0x0381e6 to 0x039fbe (0x1dd9 bytes) 000075 1787841430.359992 trunc from 0x039fbf to 0x0001a8 (0x39e16 bytes) 000078 1787841430.386384 mapread 0x000180 thru 0x0001a8 (0x29 bytes) 000079 1787841430.388890 read 0x00008f thru 0x0001a8 (0x11a bytes) 000080 1787841430.390378 mapread 0x000182 thru 0x0001a8 (0x27 bytes) 000081 1787841430.391490 mapwrite 0x010825 thru 0x01cd82 (0xc55e bytes) 000085 1787841430.511471 mapwrite 0x03e59c thru 0x03ffff (0x1a64 bytes) 000086 1787841430.642344 mapread 0x01bb08 thru 0x02170c (0x5c05 bytes) 000087 1787841430.643817 mapwrite 0x01b527 thru 0x02b419 (0xfef3 bytes) 000089 1787841430.702891 MIRROR_EXTEND to 2 mirrors 000091 1787841431.554334 mapread 0x03f469 thru 0x03ffff (0xb97 bytes) 000092 1787841431.560296 mapwrite 0x0372f9 thru 0x03dafb (0x6803 bytes) 000093 1787841431.727175 trunc from 0x040000 to 0x02d0e9 (0x12f16 bytes) 000095 1787841431.780770 mapread 0x0202dc thru 0x028e47 (0x8b6c bytes) 000097 1787841431.794973 write 0x020d86 thru 0x025260 (0x44db bytes) 000099 1787841431.797784 read 0x0254af thru 0x02d0e9 (0x7c3b bytes) 000100 1787841431.810210 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000101 1787841432.120449 MIRROR_EXTEND to 2 mirrors 000102 1787841433.264447 mapwrite 0x036005 thru 0x039b14 (0x3b10 bytes) 000103 1787841433.453830 MIRROR_RESYNC 2 mirrors 000105 1787841434.405175 mapwrite 0x03cf23 thru 0x03ffff (0x30dd bytes) 000106 1787841434.531198 trunc from 0x040000 to 0x00a985 (0x3567a bytes) 000108 1787841434.554834 mapwrite 0x02d2f6 thru 0x039a5b (0xc766 bytes) 000110 1787841434.631892 mapread 0x01d0b5 thru 0x0209d5 (0x3921 bytes) 000112 1787841434.644085 mapread 0x004e22 thru 0x011179 (0xc358 bytes) 000113 1787841434.663064 mapread 0x038ba8 thru 0x039a5b (0xeb4 bytes) 000115 1787841434.664697 mapwrite 0x01bd31 thru 0x02b801 (0xfad1 bytes) 000116 1787841434.714834 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000118 1787841434.861034 mapwrite 0x01d52c thru 0x0219ce (0x44a3 bytes) 000122 1787841435.001071 mapread 0x020e44 thru 0x02fdb3 (0xef70 bytes) 000124 1787841435.017327 trunc from 0x039a5c to 0x01cc7a (0x1cde1 bytes) 000125 1787841435.040921 read 0x01aa9d thru 0x01cc7a (0x21de bytes) 000126 1787841435.042051 trunc from 0x01cc7b to 0x0328bd (0x15c43 bytes) 000129 1787841435.083432 trunc from 0x0328be to 0x003510 (0x2f3ad bytes) 000130 1787841435.107178 mapwrite 0x03054f thru 0x03e6af (0xe161 bytes) 000132 1787841435.289375 trunc from 0x03e6b0 to 0x026985 (0x17d2a bytes) 000138 1787841435.316105 mapread 0x00519b thru 0x009830 (0x4696 bytes) 000139 1787841435.326492 mapread 0x00e57a thru 0x01d403 (0xee8a bytes) 000141 1787841435.344894 write 0x006f18 thru 0x013055 (0xc13e bytes) 000144 1787841435.348449 write 0x00409f thru 0x012d2f (0xec91 bytes) 000148 1787841435.352751 mapwrite 0x02e40d thru 0x0301eb (0x1ddf bytes) 000151 1787841435.454545 mapwrite 0x03f4f1 thru 0x03ffff (0xb0f bytes) 000152 1787841435.545723 trunc from 0x040000 to 0x00bb23 (0x344dc bytes) 000153 1787841435.561978 mapwrite 0x01ec31 thru 0x01ed4a (0x11a bytes) 000154 1787841435.629296 read 0x01000e thru 0x01726a (0x725d bytes) 000156 1787841435.641209 mapread 0x014ff4 thru 0x01cba5 (0x7bb2 bytes) 000159 1787841435.644003 read 0x0059a6 thru 0x00d58e (0x7be9 bytes) 000161 1787841435.646003 write 0x03d5c8 thru 0x03ffff (0x2a38 bytes) 000162 1787841435.647455 write 0x008315 thru 0x00a2f0 (0x1fdc bytes) 000164 1787841435.649874 read 0x002843 thru 0x00a8bc (0x807a bytes) 000165 1787841435.651184 mapwrite 0x02fe3b thru 0x032cd0 (0x2e96 bytes) 000168 1787841435.707556 write 0x01d7d6 thru 0x01eb59 (0x1384 bytes) 000169 1787841435.709421 mapwrite 0x02b9a0 thru 0x032a9e (0x70ff bytes) 000170 1787841435.800190 MIRROR_EXTEND to 2 mirrors 000172 1787841436.958938 MIRROR_EXTEND to 3 mirrors 000173 1787841438.239515 mapwrite 0x03e1b6 thru 0x03ffff (0x1e4a bytes) 000176 1787841438.452057 mapread 0x01d63a thru 0x02c3e2 (0xeda9 bytes) 000177 1787841438.515012 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8b55bc28 (stale) Mirror 3: 0x8b55bc28 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe3b02558 Mirror 4: 0xe3b02558 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000180 1787841440.271733 trunc from 0x040000 to 0x000079 (0x3ff86 bytes) 000182 1787841440.470537 mapwrite 0x014c0c thru 0x01c419 (0x780e bytes) 000183 1787841440.595780 write 0x0003eb thru 0x000f1d (0xb33 bytes) 000185 1787841440.614901 trunc from 0x01c41a to 0x032161 (0x15d48 bytes) 000187 1787841440.646855 write 0x02e302 thru 0x032dd5 (0x4ad4 bytes) 000188 1787841440.663761 read 0x031646 thru 0x032dd5 (0x1790 bytes) 000190 1787841440.667724 write 0x009191 thru 0x00f43f (0x62af bytes) 000191 1787841440.674774 write 0x022096 thru 0x028621 (0x658c bytes) 000192 1787841440.687690 mapread 0x02eb87 thru 0x032dd5 (0x424f bytes) 000193 1787841440.689778 mapwrite 0x016990 thru 0x01847f (0x1af0 bytes) 000197 1787841440.757559 mapwrite 0x01b67c thru 0x01ee1f (0x37a4 bytes) 000201 1787841440.818952 mapwrite 0x016ec5 thru 0x026ab3 (0xfbef bytes) 000202 1787841440.875851 trunc from 0x032dd6 to 0x00e5a2 (0x24833 bytes) 000203 1787841440.894739 read 0x00bf66 thru 0x00e5a2 (0x263d bytes) 000205 1787841440.896523 mapread 0x0053b3 thru 0x009439 (0x4087 bytes) 000207 1787841440.908304 mapread 0x000770 thru 0x002a18 (0x22a9 bytes) 000208 1787841440.909412 read 0x00393c thru 0x00e2a3 (0xa968 bytes) 000210 1787841440.915327 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xe5a3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe5a3): Mirror 2: 0x8c7170e2 (stale) Mirror 3: 0x8c7170e2 (stale) Mirror 4: 0x8c7170e2 (stale) CRC-32 checksum value for chunk [0, 0xe5a3): Mirror 1: 0x982dbd50 Mirror 5: 0x982dbd50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe5a3: skipped 000211 1787841442.413383 trunc from 0x00e5a3 to 0x005992 (0x8c10 bytes) 000213 1787841442.625268 mapwrite 0x006eb7 thru 0x010872 (0x99bc bytes) 000214 1787841442.909187 read 0x00a933 thru 0x010872 (0x5f40 bytes) 000215 1787841442.911010 mapwrite 0x039147 thru 0x03e25d (0x5117 bytes) 000217 1787841443.078434 trunc from 0x03e25e to 0x023f0b (0x1a352 bytes) 000220 1787841443.122757 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23f0c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23f0c): Mirror 2: 0x940c5b30 (stale) Mirror 3: 0x940c5b30 (stale) Mirror 4: 0x940c5b30 (stale) Mirror 5: 0x1b729336 (stale) CRC-32 checksum value for chunk [0, 0x23f0c): Mirror 1: 0xf1df6529 Mirror 6: 0xf1df6529 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23f0c: skipped 000221 1787841444.753410 read 0x01c458 thru 0x023f0b (0x7ab4 bytes) 000223 1787841444.770591 MIRROR_RESYNC 6 mirrors 000224 1787841446.089985 MIRROR_RESYNC 6 mirrors 000226 1787841446.519565 mapread 0x020817 thru 0x023f0b (0x36f5 bytes) 000227 1787841446.540823 write 0x02f46e thru 0x03313c (0x3ccf bytes) 000228 1787841446.620538 read 0x00ef1f thru 0x01c9c1 (0xdaa3 bytes) 000229 1787841446.634986 read 0x007b65 thru 0x013b51 (0xbfed bytes) 000232 1787841446.652677 read 0x027670 thru 0x03313c (0xbacd bytes) 000233 1787841446.653904 mapwrite 0x034fb2 thru 0x036f6c (0x1fbb bytes) 000234 1787841446.777824 mapwrite 0x03cdf4 thru 0x03eb0e (0x1d1b bytes) 000236 1787841446.862211 mapwrite 0x03bd09 thru 0x03ffff (0x42f7 bytes) 000237 1787841446.929745 read 0x018747 thru 0x01d4fa (0x4db4 bytes) 000238 1787841446.930419 mapwrite 0x0240e5 thru 0x02acd0 (0x6bec bytes) 000239 1787841446.982819 mapread 0x0129f3 thru 0x01f4f5 (0xcb03 bytes) 000241 1787841447.006044 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x349ed683 (stale) Mirror 3: 0x349ed683 (stale) Mirror 5: 0x349ed683 (stale) Mirror 6: 0x349ed683 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x349ed683 (stale) Mirror 3: 0x349ed683 (stale) Mirror 5: 0x349ed683 (stale) Mirror 6: 0x349ed683 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000242 1787841447.584835 read 0x01cb47 thru 0x0203bd (0x3877 bytes) 000244 1787841447.593060 mapread 0x01e1f7 thru 0x01e748 (0x552 bytes) 000248 1787841447.604177 MIRROR_RESYNC 5 mirrors 000249 1787841450.383241 read 0x00de11 thru 0x013851 (0x5a41 bytes) 000250 1787841450.420948 trunc from 0x040000 to 0x025b0f (0x1a4f0 bytes) 000251 1787841450.594318 mapwrite 0x005324 thru 0x010c9b (0xb978 bytes) 000252 1787841450.803684 write 0x027beb thru 0x032657 (0xaa6d bytes) 000253 1787841450.815246 mapwrite 0x01deb4 thru 0x0223bf (0x450c bytes) 000255 1787841450.940902 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32658): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32658): Mirror 2: 0x6f51b56a (stale) Mirror 3: 0x6f51b56a (stale) Mirror 5: 0x6f51b56a (stale) Mirror 6: 0x6f51b56a (stale) CRC-32 checksum value for chunk [0, 0x32658): Mirror 1: 0xd7cc6bc6 Mirror 7: 0xd7cc6bc6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32658: skipped 000256 1787841452.493349 mapwrite 0x016feb thru 0x020b83 (0x9b99 bytes) 000257 1787841452.943278 read 0x00eb3d thru 0x0166e0 (0x7ba4 bytes) 000258 1787841452.956964 write 0x000746 thru 0x00d836 (0xd0f1 bytes) 000259 1787841453.014479 write 0x006b65 thru 0x00bbd6 (0x5072 bytes) 000261 1787841453.025251 read 0x02ac16 thru 0x02e1ee (0x35d9 bytes) 000263 1787841453.049009 mapwrite 0x03ca64 thru 0x03ffff (0x359c bytes) 000266 1787841453.260012 write 0x028693 thru 0x02984c (0x11ba bytes) 000267 1787841453.262539 trunc from 0x040000 to 0x009ce3 (0x3631c bytes) 000268 1787841453.284987 mapread 0x000d78 thru 0x009785 (0x8a0e bytes) 000269 1787841453.288683 mapwrite 0x00da12 thru 0x0181db (0xa7ca bytes) 000271 1787841453.607190 write 0x030219 thru 0x03f98f (0xf777 bytes) 000272 1787841453.610771 mapwrite 0x01a1d6 thru 0x026c6f (0xca9a bytes) 000273 1787841453.814110 read 0x0021d1 thru 0x009030 (0x6e60 bytes) 000274 1787841453.814695 read 0x00b0f3 thru 0x00cc30 (0x1b3e bytes) 000275 1787841453.824389 trunc from 0x03f990 to 0x038249 (0x7746 bytes) 000277 1787841453.838851 read 0x036907 thru 0x038249 (0x1943 bytes) 000279 1787841453.839879 write 0x01302b thru 0x015171 (0x2147 bytes) 000281 1787841453.845207 trunc from 0x03824a to 0x030b1e (0x772b bytes) 000282 1787841453.866333 mapread 0x00e302 thru 0x019bba (0xb8b9 bytes) 000284 1787841453.873197 write 0x006cd9 thru 0x009ab8 (0x2de0 bytes) 000285 1787841453.878687 read 0x009fbe thru 0x0139d3 (0x9a16 bytes) 000289 1787841453.883041 trunc from 0x030b1f to 0x004a41 (0x2c0dd bytes) 000290 1787841453.912869 MIRROR_RESYNC 6 mirrors 000292 1787841455.453926 read 0x002753 thru 0x002be2 (0x490 bytes) truncating to largest ever: 0x3d65e 000293 1787841455.469300 trunc from 0x004a42 to 0x03d65d (0x38c1c bytes) 000294 1787841455.638813 mapread 0x01f84a thru 0x02e3a1 (0xeb58 bytes) 000296 1787841455.715461 write 0x029c52 thru 0x02f5d4 (0x5983 bytes) 000297 1787841455.716516 read 0x00b256 thru 0x016a47 (0xb7f2 bytes) 000300 1787841455.750211 read 0x0300e9 thru 0x032fe6 (0x2efe bytes) 000301 1787841455.751587 mapwrite 0x011d70 thru 0x01e87b (0xcb0c bytes) 000304 1787841455.887464 write 0x02bec4 thru 0x036c2a (0xad67 bytes) 000305 1787841455.894641 write 0x000b81 thru 0x00ac4b (0xa0cb bytes) 000306 1787841455.901398 mapwrite 0x037f14 thru 0x03f2cb (0x73b8 bytes) 000307 1787841456.115441 mapwrite 0x036c95 thru 0x03ffff (0x936b bytes) 000308 1787841456.268708 mapwrite 0x01aab1 thru 0x01bf48 (0x1498 bytes) 000309 1787841456.339303 mapwrite 0x011851 thru 0x017d88 (0x6538 bytes) 000311 1787841456.395731 mapwrite 0x01bd6d thru 0x01ccf9 (0xf8d bytes) 000313 1787841456.446655 write 0x01cedc thru 0x02a524 (0xd649 bytes) 000314 1787841456.448779 read 0x018296 thru 0x01d6c8 (0x5433 bytes) 000317 1787841456.451820 read 0x00fc58 thru 0x01b055 (0xb3fe bytes) 000318 1787841456.456896 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3d27713 (stale) Mirror 5: 0x3d27713 (stale) Mirror 6: 0x3d27713 (stale) Mirror 7: 0x3d27713 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3d27713 (stale) Mirror 5: 0x3d27713 (stale) Mirror 6: 0x3d27713 (stale) Mirror 7: 0x3d27713 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000319 1787841456.831342 trunc from 0x040000 to 0x033921 (0xc6de bytes) 000320 1787841456.853631 read 0x0282dd thru 0x0287f9 (0x51d bytes) 000321 1787841456.859235 read 0x02ae20 thru 0x033921 (0x8b02 bytes) 000322 1787841456.875594 mapwrite 0x02a970 thru 0x02c408 (0x1a99 bytes) 000325 1787841456.976721 read 0x02b7c3 thru 0x033921 (0x815f bytes) 000332 1787841456.996642 mapread 0x019803 thru 0x01ffee (0x67ec bytes) 000333 1787841456.997566 read 0x01978a thru 0x0230f2 (0x9969 bytes) 000335 1787841457.008995 MIRROR_RESYNC 5 mirrors 000336 1787841458.132835 MIRROR_EXTEND to 6 mirrors 000339 1787841458.962376 trunc from 0x033922 to 0x01e267 (0x156ba bytes) 000342 1787841459.090368 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1e268): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e268): Mirror 3: 0xa1a9f4d7 (stale) Mirror 5: 0xa1a9f4d7 (stale) Mirror 6: 0xa1a9f4d7 (stale) Mirror 7: 0xa1a9f4d7 (stale) Mirror 8: 0xa1a9f4d7 (stale) CRC-32 checksum value for chunk [0, 0x1e268): Mirror 1: 0xa1a9f4d7 Mirror 9: 0xa1a9f4d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e268: skipped 000345 1787841460.066282 write 0x003257 thru 0x004e93 (0x1c3d bytes) 000346 1787841460.145937 read 0x005989 thru 0x0154d9 (0xfb51 bytes) 000347 1787841460.153870 mapread 0x00e363 thru 0x012305 (0x3fa3 bytes) 000351 1787841460.157268 trunc from 0x01e268 to 0x000007 (0x1e260 bytes) 000352 1787841460.198101 write 0x01749b thru 0x025729 (0xe28f bytes) 000353 1787841460.212674 write 0x02591f thru 0x02913d (0x381f bytes) 000354 1787841460.217160 mapwrite 0x03b5d4 thru 0x03d4cd (0x1efa bytes) 000358 1787841460.422743 mapread 0x008405 thru 0x016a6b (0xe667 bytes) 000362 1787841460.476954 mapwrite 0x01a1b4 thru 0x02073e (0x658b bytes) 000365 1787841460.566930 write 0x0325a0 thru 0x036ff4 (0x4a55 bytes) 000368 1787841460.580202 trunc from 0x03d4ce to 0x01961d (0x23eb0 bytes) 000371 1787841460.648641 mapwrite 0x03cf92 thru 0x03ffff (0x306e bytes) 000373 1787841460.806315 write 0x00b8e4 thru 0x00d2da (0x19f7 bytes) 000374 1787841460.807387 read 0x0146d7 thru 0x01dd5d (0x9687 bytes) 000375 1787841460.822643 mapwrite 0x011a2a thru 0x011b51 (0x128 bytes) 000376 1787841460.880820 MIRROR_RESYNC 7 mirrors 000379 1787841463.385928 read 0x03552c thru 0x035fb9 (0xa8e bytes) 000380 1787841463.395711 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000382 1787841463.762766 trunc from 0x040000 to 0x02dd7d (0x12282 bytes) 000383 1787841463.955425 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd7e): Mirror 3: 0x192a68ed (stale) Mirror 6: 0x192a68ed (stale) Mirror 8: 0x192a68ed (stale) Mirror 9: 0x192a68ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd7e): Mirror 3: 0x192a68ed (stale) Mirror 6: 0x192a68ed (stale) Mirror 8: 0x192a68ed (stale) Mirror 9: 0x192a68ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd7e: skipped 000384 1787841464.540431 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd7e): Mirror 3: 0x192a68ed (stale) Mirror 6: 0x192a68ed (stale) Mirror 8: 0x192a68ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd7e): Mirror 3: 0x192a68ed (stale) Mirror 6: 0x192a68ed (stale) Mirror 8: 0x192a68ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd7e: skipped 000386 1787841465.185589 mapwrite 0x038a82 thru 0x039baa (0x1129 bytes) 000387 1787841465.339888 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39bab): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39bab): Mirror 3: 0x39b5cafc (stale) Mirror 6: 0x39b5cafc (stale) Mirror 8: 0x39b5cafc (stale) CRC-32 checksum value for chunk [0, 0x39bab): Mirror 1: 0xa1665055 Mirror 9: 0xa1665055 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39bab: skipped 000389 1787841466.683739 mapread 0x01271b thru 0x01b3e2 (0x8cc8 bytes) 000390 1787841466.712886 trunc from 0x039bab to 0x02cf0f (0xcc9b bytes) 000391 1787841466.820561 mapread 0x01ee8a thru 0x02a085 (0xb1fc bytes) 000393 1787841466.859435 trunc from 0x02cf10 to 0x011905 (0x1b60a bytes) 000396 1787841466.887841 MIRROR_RESYNC 5 mirrors 000397 1787841468.636714 write 0x033482 thru 0x039137 (0x5cb6 bytes) 000401 1787841468.800905 mapread 0x0096f8 thru 0x0151f2 (0xbafb bytes) 000402 1787841468.890382 write 0x02a6ca thru 0x0311ce (0x6b05 bytes) 000403 1787841468.916597 read 0x023518 thru 0x0320e3 (0xebcc bytes) 000404 1787841468.937294 write 0x005f2e thru 0x00e7fd (0x88d0 bytes) 000405 1787841468.946469 MIRROR_RESYNC 5 mirrors 000406 1787841471.441470 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000408 1787841471.890165 mapwrite 0x0373b0 thru 0x03ecf7 (0x7948 bytes) 000409 1787841472.098120 write 0x03398a thru 0x035784 (0x1dfb bytes) 000410 1787841472.099093 mapwrite 0x02cf87 thru 0x03625b (0x92d5 bytes) 000411 1787841472.162700 mapread 0x01577e thru 0x01bbcf (0x6452 bytes) 000413 1787841472.176042 read 0x004d6c thru 0x0079ba (0x2c4f bytes) 000414 1787841472.188898 read 0x023538 thru 0x02b713 (0x81dc bytes) 000415 1787841472.200625 trunc from 0x03ecf8 to 0x032c9b (0xc05c bytes) 000416 1787841472.231064 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32c9c): Mirror 6: 0x6c7fcbf5 (stale) Mirror 8: 0x6c7fcbf5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c9c): Mirror 6: 0x6c7fcbf5 (stale) Mirror 8: 0x6c7fcbf5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c9c: skipped 000417 1787841472.671622 mapwrite 0x01abd2 thru 0x025176 (0xa5a5 bytes) 000418 1787841472.886864 write 0x00d1af thru 0x014496 (0x72e8 bytes) 000419 1787841472.922542 trunc from 0x032c9c to 0x005f1a (0x2cd81 bytes) 000420 1787841472.952404 write 0x009771 thru 0x011c67 (0x84f7 bytes) 000422 1787841472.954218 mapread 0x00bb9b thru 0x011c67 (0x60cd bytes) 000423 1787841472.961418 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x11c68): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c68): Mirror 6: 0x3e73a56d (stale) Mirror 8: 0x3e73a56d (stale) CRC-32 checksum value for chunk [0, 0x11c68): Mirror 1: 0x5dfa930a Mirror 9: 0x5dfa930a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c68: skipped 000424 1787841474.831018 trunc from 0x011c68 to 0x00c8d8 (0x538f bytes) 000425 1787841475.061645 mapread 0x004f51 thru 0x00b4e8 (0x6598 bytes) 000426 1787841475.116981 trunc from 0x00c8d9 to 0x017040 (0xa768 bytes) 000427 1787841475.175051 mapwrite 0x0062b8 thru 0x009bfe (0x3947 bytes) 000429 1787841475.313293 trunc from 0x017041 to 0x0176f3 (0x6b3 bytes) 000430 1787841475.363703 mapwrite 0x029ca2 thru 0x037fcd (0xe32c bytes) 000431 1787841475.609244 write 0x033fc1 thru 0x03583b (0x187b bytes) 000434 1787841475.630786 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37fce): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fce): Mirror 6: 0x17a16893 (stale) Mirror 8: 0x17a16893 (stale) Mirror 9: 0x67f83d4 (stale) CRC-32 checksum value for chunk [0, 0x37fce): Mirror 1: 0xdc6293ad Mirror 10: 0xdc6293ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fce: skipped 000435 1787841477.365592 trunc from 0x037fce to 0x0014c6 (0x36b07 bytes) 000436 1787841477.498087 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14c7): Mirror 8: 0x21d9a9ff (stale) Mirror 9: 0x21d9a9ff (stale) Mirror 10: 0x21d9a9ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14c7): Mirror 8: 0x21d9a9ff (stale) Mirror 9: 0x21d9a9ff (stale) Mirror 10: 0x21d9a9ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14c7: skipped 000439 1787841477.785721 write 0x01e935 thru 0x021bfa (0x32c6 bytes) 000440 1787841477.787673 trunc from 0x021bfb to 0x02d6ae (0xbab4 bytes) 000441 1787841477.801795 trunc from 0x02d6af to 0x005326 (0x28388 bytes) 000442 1787841477.827929 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x5327): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5327): Mirror 8: 0x7fab80af (stale) Mirror 9: 0x7fab80af (stale) Mirror 10: 0x7fab80af (stale) CRC-32 checksum value for chunk [0, 0x5327): Mirror 1: 0x7fab80af Mirror 11: 0x7fab80af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5327: skipped 000443 1787841478.930078 write 0x035fdc thru 0x03dff8 (0x801d bytes) 000444 1787841479.011438 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3dff9): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dff9): Mirror 8: 0x476c2222 (stale) Mirror 9: 0x64759645 (stale) Mirror 10: 0x9105c2b8 (stale) Mirror 11: 0xf37fe98d (stale) CRC-32 checksum value for chunk [0, 0x3dff9): Mirror 1: 0xff6fef50 Mirror 12: 0xff6fef50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dff9: skipped 000445 1787841480.726350 write 0x0127f1 thru 0x01c683 (0x9e93 bytes) 000446 1787841480.911531 write 0x037dd7 thru 0x03ef2e (0x7158 bytes) 000447 1787841480.919590 read 0x004f01 thru 0x00887d (0x397d bytes) 000448 1787841480.941664 mapread 0x032a3d thru 0x03630c (0x38d0 bytes) 000451 1787841480.984420 mapread 0x002dcf thru 0x0088be (0x5af0 bytes) 000452 1787841481.010589 trunc from 0x03ef2f to 0x038109 (0x6e25 bytes) 000456 1787841481.064273 mapread 0x016106 thru 0x023b91 (0xda8c bytes) 000457 1787841481.092197 mapwrite 0x018c0f thru 0x02256b (0x995d bytes) 000458 1787841481.327654 trunc from 0x03810a to 0x00d6c3 (0x2aa46 bytes) 000461 1787841481.377695 trunc from 0x00d6c4 to 0x0153c1 (0x7cfe bytes) 000465 1787841481.434731 write 0x03779e thru 0x03ffff (0x8862 bytes) 000466 1787841481.445364 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa82cca90 (stale) Mirror 10: 0x75388007 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xd683d957 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa82cca90 (stale) Mirror 10: 0x75388007 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xd683d957 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000469 1787841482.226780 trunc from 0x040000 to 0x013052 (0x2cfad bytes) 000470 1787841482.275651 mapwrite 0x01b824 thru 0x0208bd (0x509a bytes) 000473 1787841482.493903 MIRROR_RESYNC 5 mirrors 000474 1787841484.885263 mapwrite 0x0158a2 thru 0x022efa (0xd659 bytes) 000475 1787841485.080672 mapread 0x01cdbb thru 0x022efa (0x6140 bytes) 000476 1787841485.082277 write 0x037865 thru 0x03ffff (0x879b bytes) 000477 1787841485.084232 write 0x000cbc thru 0x003c79 (0x2fbe bytes) 000479 1787841485.095099 mapread 0x020a16 thru 0x027947 (0x6f32 bytes) 000480 1787841485.103757 read 0x028330 thru 0x034f92 (0xcc63 bytes) 000481 1787841485.128583 mapread 0x03cebe thru 0x03ffff (0x3142 bytes) 000482 1787841485.129849 write 0x015def thru 0x01bc00 (0x5e12 bytes) 000484 1787841485.132934 read 0x0066d8 thru 0x00d556 (0x6e7f bytes) 000486 1787841485.144351 mapread 0x02746c thru 0x0289b4 (0x1549 bytes) 000487 1787841485.145305 mapread 0x00e4ab thru 0x017838 (0x938e bytes) 000488 1787841485.148303 write 0x029e38 thru 0x039a3a (0xfc03 bytes) 000490 1787841485.156481 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2f3cba69 (stale) Mirror 10: 0x2f3cba69 (stale) Mirror 11: 0x2f3cba69 (stale) Mirror 12: 0x2f3cba69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcdcfe82f Mirror 13: 0xcdcfe82f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000491 1787841486.851838 mapread 0x038998 thru 0x03a2d7 (0x1940 bytes) 000492 1787841486.878310 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2f3cba69 (stale) Mirror 10: 0x2f3cba69 (stale) Mirror 11: 0x2f3cba69 (stale) Mirror 12: 0x2f3cba69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcdcfe82f Mirror 13: 0xcdcfe82f Mirror 14: 0xcdcfe82f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000495 1787841488.128365 MIRROR_RESYNC 7 mirrors 000497 1787841490.188850 write 0x019da5 thru 0x01bf22 (0x217e bytes) 000498 1787841490.363956 trunc from 0x040000 to 0x032022 (0xdfdd bytes) 000500 1787841490.450992 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32023): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32023): Mirror 9: 0x4f19935d (stale) Mirror 10: 0x4f19935d (stale) Mirror 11: 0x4f19935d (stale) Mirror 12: 0x4f19935d (stale) Mirror 13: 0x4f19935d (stale) Mirror 14: 0x4f19935d (stale) CRC-32 checksum value for chunk [0, 0x32023): Mirror 1: 0xe5185353 Mirror 15: 0xe5185353 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32023: skipped 000501 1787841491.973667 MIRROR_RESYNC 8 mirrors 000503 1787841493.991725 trunc from 0x032023 to 0x01bdfd (0x16225 bytes) 000504 1787841494.068988 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bdfe): Mirror 9: 0x17f91562 (stale) Mirror 10: 0x17f91562 (stale) Mirror 11: 0x17f91562 (stale) Mirror 13: 0x17f91562 (stale) Mirror 14: 0x17f91562 (stale) Mirror 15: 0x17f91562 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bdfe): Mirror 9: 0x17f91562 (stale) Mirror 10: 0x17f91562 (stale) Mirror 11: 0x17f91562 (stale) Mirror 13: 0x17f91562 (stale) Mirror 14: 0x17f91562 (stale) Mirror 15: 0x17f91562 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bdfe: skipped 000505 1787841494.630577 read 0x003499 thru 0x00c44b (0x8fb3 bytes) 000507 1787841494.652005 write 0x036ac7 thru 0x03ffff (0x9539 bytes) 000508 1787841494.653418 trunc from 0x040000 to 0x008121 (0x37ede bytes) truncating to largest ever: 0x3eb5a 000510 1787841494.682689 trunc from 0x008122 to 0x03eb59 (0x36a38 bytes) 000511 1787841494.701895 write 0x03c2bf thru 0x03c648 (0x38a bytes) 000513 1787841494.713744 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3eb5a): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb5a): Mirror 9: 0xbd44e30d (stale) Mirror 10: 0xbd44e30d (stale) Mirror 11: 0xbd44e30d (stale) Mirror 13: 0xbd44e30d (stale) Mirror 14: 0xbd44e30d (stale) Mirror 15: 0xbd44e30d (stale) CRC-32 checksum value for chunk [0, 0x3eb5a): Mirror 1: 0x6840ef93 Mirror 16: 0x6840ef93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb5a: skipped 000514 1787841495.889572 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb5a): Mirror 9: 0xbd44e30d (stale) Mirror 10: 0xbd44e30d (stale) Mirror 11: 0xbd44e30d (stale) Mirror 13: 0xbd44e30d (stale) Mirror 14: 0xbd44e30d (stale) Mirror 15: 0xbd44e30d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb5a): Mirror 9: 0xbd44e30d (stale) Mirror 10: 0xbd44e30d (stale) Mirror 11: 0xbd44e30d (stale) Mirror 13: 0xbd44e30d (stale) Mirror 14: 0xbd44e30d (stale) Mirror 15: 0xbd44e30d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb5a: skipped 000517 1787841496.527311 write 0x01e9ba thru 0x02651a (0x7b61 bytes) 000519 1787841496.575140 read 0x02d523 thru 0x033aa4 (0x6582 bytes) 000522 1787841496.614781 write 0x01dd05 thru 0x02804f (0xa34b bytes) 000523 1787841496.631959 write 0x03e907 thru 0x03ffff (0x16f9 bytes) 000526 1787841496.645953 trunc from 0x040000 to 0x0131f0 (0x2ce0f bytes) 000527 1787841496.691368 read 0x003843 thru 0x006ef1 (0x36af bytes) 000528 1787841496.720113 write 0x03072d thru 0x039a52 (0x9326 bytes) 000529 1787841496.721496 trunc from 0x039a53 to 0x032064 (0x79ee bytes) 000531 1787841496.757168 trunc from 0x032065 to 0x00ea2e (0x23636 bytes) 000532 1787841496.791637 mapread 0x00689a thru 0x0095bd (0x2d24 bytes) 000534 1787841496.795929 mapwrite 0x03b0de thru 0x03ffff (0x4f22 bytes) 000535 1787841496.922816 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6c0c006 (stale) Mirror 10: 0x6c0c006 (stale) Mirror 11: 0x6c0c006 (stale) Mirror 13: 0x6c0c006 (stale) Mirror 15: 0x6c0c006 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x6c0c006 (stale) Mirror 10: 0x6c0c006 (stale) Mirror 11: 0x6c0c006 (stale) Mirror 13: 0x6c0c006 (stale) Mirror 15: 0x6c0c006 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000536 1787841497.378696 write 0x01b0f2 thru 0x0224d6 (0x73e5 bytes) 000537 1787841497.401456 mapwrite 0x025f9f thru 0x03067f (0xa6e1 bytes) 000538 1787841497.535772 read 0x022a20 thru 0x031f5e (0xf53f bytes) 000539 1787841497.538336 read 0x01dd92 thru 0x02bb01 (0xdd70 bytes) 000543 1787841497.542740 write 0x035f13 thru 0x03e4f1 (0x85df bytes) 000544 1787841497.573318 trunc from 0x040000 to 0x019580 (0x26a7f bytes) 000545 1787841497.618865 read 0x00459d thru 0x01293b (0xe39f bytes) 000548 1787841497.632854 MIRROR_RESYNC 6 mirrors 000549 1787841499.181575 mapwrite 0x01d3f8 thru 0x027a11 (0xa61a bytes) 000550 1787841499.443311 trunc from 0x027a12 to 0x03ca0f (0x14ffe bytes) 000551 1787841499.484245 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 9: 0xafe9039f (stale) Mirror 10: 0xafe9039f (stale) Mirror 11: 0xafe9039f (stale) Mirror 13: 0xafe9039f (stale) Mirror 15: 0xafe9039f (stale) CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 1: 0x66e56911 Mirror 16: 0x66e56911 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca10: skipped 000552 1787841500.862928 mapwrite 0x02cab1 thru 0x03b6c6 (0xec16 bytes) 000554 1787841501.099215 mapread 0x025846 thru 0x02e5ad (0x8d68 bytes) 000555 1787841501.115170 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 9: 0xafe9039f (stale) Mirror 10: 0xafe9039f (stale) Mirror 11: 0xafe9039f (stale) Mirror 13: 0xafe9039f (stale) Mirror 15: 0xafe9039f (stale) Mirror 16: 0x66e56911 (stale) CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 1: 0xa4cbbacc Mirror 17: 0xa4cbbacc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca10: skipped 000557 1787841502.390747 mapread 0x026cfc thru 0x0296d7 (0x29dc bytes) 000558 1787841502.424881 mapread 0x032898 thru 0x038fd0 (0x6739 bytes) 000563 1787841502.437606 mapread 0x030524 thru 0x038f7d (0x8a5a bytes) 000564 1787841502.438935 write 0x00318f thru 0x00d00f (0x9e81 bytes) 000566 1787841502.553603 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 9: 0xafe9039f (stale) Mirror 10: 0xafe9039f (stale) Mirror 11: 0xafe9039f (stale) Mirror 13: 0xafe9039f (stale) Mirror 15: 0xafe9039f (stale) Mirror 16: 0x66e56911 (stale) Mirror 17: 0xa4cbbacc (stale) CRC-32 checksum value for chunk [0, 0x3ca10): Mirror 1: 0x507a8d0a Mirror 18: 0x507a8d0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca10: skipped 000567 1787841504.073299 trunc from 0x03ca10 to 0x0076e6 (0x35329 bytes) 000568 1787841504.197833 read 0x0006c7 thru 0x00426e (0x3ba8 bytes) 000569 1787841504.205798 write 0x02c251 thru 0x034135 (0x7ee5 bytes) 000570 1787841504.207527 mapwrite 0x03ecaf thru 0x03ffff (0x1351 bytes) 000571 1787841504.288091 write 0x002b5f thru 0x006024 (0x34c6 bytes) 000572 1787841504.298041 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2253ac01 (stale) Mirror 10: 0x2253ac01 (stale) Mirror 11: 0x2253ac01 (stale) Mirror 13: 0x2253ac01 (stale) Mirror 15: 0x2253ac01 (stale) Mirror 16: 0xf1338168 (stale) Mirror 17: 0x309650d2 (stale) Mirror 18: 0xb166321a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9ae81b93 Mirror 19: 0x9ae81b93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000573 1787841505.395556 mapwrite 0x00644a thru 0x008910 (0x24c7 bytes) 000574 1787841505.662259 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2253ac01 (stale) Mirror 10: 0x2253ac01 (stale) Mirror 11: 0x2253ac01 (stale) Mirror 13: 0x2253ac01 (stale) Mirror 15: 0x2253ac01 (stale) Mirror 16: 0xf1338168 (stale) Mirror 18: 0xb166321a (stale) Mirror 19: 0x9ae81b93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2253ac01 (stale) Mirror 10: 0x2253ac01 (stale) Mirror 11: 0x2253ac01 (stale) Mirror 13: 0x2253ac01 (stale) Mirror 15: 0x2253ac01 (stale) Mirror 16: 0xf1338168 (stale) Mirror 18: 0xb166321a (stale) Mirror 19: 0x9ae81b93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000575 1787841506.211397 mapwrite 0x01e4e6 thru 0x029664 (0xb17f bytes) 000577 1787841506.310905 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2253ac01 (stale) Mirror 10: 0x2253ac01 (stale) Mirror 11: 0x2253ac01 (stale) Mirror 13: 0x2253ac01 (stale) Mirror 15: 0x2253ac01 (stale) Mirror 16: 0xf1338168 (stale) Mirror 18: 0xb166321a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2253ac01 (stale) Mirror 10: 0x2253ac01 (stale) Mirror 11: 0x2253ac01 (stale) Mirror 13: 0x2253ac01 (stale) Mirror 15: 0x2253ac01 (stale) Mirror 16: 0xf1338168 (stale) Mirror 18: 0xb166321a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000579 1787841506.909219 trunc from 0x040000 to 0x0345b2 (0xba4d bytes) 000580 1787841506.982945 mapread 0x033b05 thru 0x0345b2 (0xaae bytes) 000583 1787841506.999424 trunc from 0x0345b3 to 0x023554 (0x1105e bytes) 000584 1787841507.017461 mapread 0x011bb5 thru 0x016ba9 (0x4ff5 bytes) 000585 1787841507.026611 mapread 0x0124e0 thru 0x018794 (0x62b5 bytes) 000587 1787841507.028481 mapread 0x01f915 thru 0x023554 (0x3c40 bytes) 000593 1787841507.041909 mapread 0x0073d5 thru 0x00facc (0x86f8 bytes) 000595 1787841507.056802 write 0x03462d thru 0x03b1bb (0x6b8f bytes) 000596 1787841507.058053 mapwrite 0x0167c8 thru 0x0230e8 (0xc921 bytes) 000599 1787841507.118266 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b1bc): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b1bc): Mirror 9: 0x68d9d63 (stale) Mirror 10: 0x68d9d63 (stale) Mirror 11: 0x68d9d63 (stale) Mirror 13: 0x68d9d63 (stale) Mirror 15: 0x68d9d63 (stale) Mirror 16: 0xf9dbd976 (stale) Mirror 18: 0x5be0bea (stale) CRC-32 checksum value for chunk [0, 0x3b1bc): Mirror 1: 0xe0ff202c Mirror 19: 0xe0ff202c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b1bc: skipped 000600 1787841508.184736 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b1bc): Mirror 10: 0x68d9d63 (stale) Mirror 11: 0x68d9d63 (stale) Mirror 13: 0x68d9d63 (stale) Mirror 15: 0x68d9d63 (stale) Mirror 16: 0xf9dbd976 (stale) Mirror 18: 0x5be0bea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b1bc): Mirror 10: 0x68d9d63 (stale) Mirror 11: 0x68d9d63 (stale) Mirror 13: 0x68d9d63 (stale) Mirror 15: 0x68d9d63 (stale) Mirror 16: 0xf9dbd976 (stale) Mirror 18: 0x5be0bea (stale) CRC-32 checksum value for chunk [0, 0x3b1bc): Mirror 1: 0xe0ff202c Mirror 19: 0xe0ff202c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b1bc: skipped 000601 1787841508.854314 trunc from 0x03b1bc to 0x02b4f9 (0xfcc2 bytes) 000603 1787841508.982163 read 0x022461 thru 0x02b4f9 (0x9099 bytes) 000604 1787841508.999198 trunc from 0x02b4fa to 0x01c8c8 (0xec31 bytes) 000606 1787841509.056878 MIRROR_RESYNC 8 mirrors 000608 1787841510.587847 trunc from 0x01c8c9 to 0x02c889 (0xffc1 bytes) 000609 1787841510.670066 write 0x02a7d2 thru 0x02af2f (0x75e bytes) 000610 1787841510.683896 mapwrite 0x030781 thru 0x03cac2 (0xc342 bytes) 000611 1787841510.772547 mapwrite 0x02ea58 thru 0x03a52f (0xbad8 bytes) 000612 1787841510.818164 trunc from 0x03cac3 to 0x010e22 (0x2bca0 bytes) 000613 1787841510.837986 read 0x00f3e7 thru 0x010e22 (0x1a3c bytes) 000614 1787841510.843406 trunc from 0x010e23 to 0x016601 (0x57df bytes) 000619 1787841510.867388 trunc from 0x016602 to 0x013582 (0x307f bytes) 000620 1787841510.888253 mapread 0x00a73e thru 0x00e050 (0x3913 bytes) 000622 1787841510.902010 read 0x00b9e5 thru 0x013582 (0x7b9e bytes) 000623 1787841510.910003 read 0x00a196 thru 0x013573 (0x93de bytes) 000626 1787841510.915337 MIRROR_RESYNC 8 mirrors 000628 1787841512.396584 mapread 0x012715 thru 0x013582 (0xe6e bytes) 000630 1787841512.431047 MIRROR_EXTEND to 9 mirrors 000631 1787841513.598913 write 0x036579 thru 0x03807b (0x1b03 bytes) 000632 1787841513.681606 write 0x03dabe thru 0x03fb51 (0x2094 bytes) 000634 1787841513.684550 mapwrite 0x017cd1 thru 0x01e7d5 (0x6b05 bytes) 000636 1787841513.750893 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3fb52): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb52): Mirror 10: 0xdf073e8b (stale) Mirror 11: 0xdf073e8b (stale) Mirror 13: 0xdf073e8b (stale) Mirror 15: 0xdf073e8b (stale) Mirror 16: 0xdf073e8b (stale) Mirror 18: 0xdf073e8b (stale) Mirror 19: 0xdf073e8b (stale) Mirror 20: 0xdf073e8b (stale) CRC-32 checksum value for chunk [0, 0x3fb52): Mirror 1: 0xa136832d Mirror 21: 0xa136832d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb52: skipped 000637 1787841515.147374 mapwrite 0x00678f thru 0x01614a (0xf9bc bytes) 000639 1787841515.397852 read 0x038d77 thru 0x03fb51 (0x6ddb bytes) 000640 1787841515.411527 trunc from 0x03fb52 to 0x02d7c0 (0x12391 bytes) 000642 1787841515.437088 mapread 0x01d81a thru 0x02cac1 (0xf2a8 bytes) 000643 1787841515.463932 read 0x003e25 thru 0x0134a2 (0xf67e bytes) 000644 1787841515.464534 mapread 0x0091b2 thru 0x019010 (0xfe5f bytes) 000645 1787841515.465431 mapread 0x003563 thru 0x00f400 (0xbe9e bytes) 000647 1787841515.472095 trunc from 0x02d7c1 to 0x02f8ee (0x212e bytes) 000648 1787841515.499976 trunc from 0x02f8ef to 0x02a67a (0x5274 bytes) 000649 1787841515.516714 mapwrite 0x01ea38 thru 0x028f01 (0xa4ca bytes) 000650 1787841515.557761 write 0x03a48c thru 0x03c128 (0x1c9d bytes) 000653 1787841515.559874 write 0x026867 thru 0x027882 (0x101c bytes) 000655 1787841515.562053 write 0x0291c1 thru 0x02f41e (0x625e bytes) 000656 1787841515.569511 mapread 0x03c0ca thru 0x03c128 (0x5f bytes) 000657 1787841515.570848 write 0x00873a thru 0x0101ff (0x7ac6 bytes) 000660 1787841515.573797 write 0x018db0 thru 0x01ac08 (0x1e59 bytes) 000662 1787841515.575183 write 0x006aeb thru 0x01104c (0xa562 bytes) 000663 1787841515.577027 mapread 0x013d09 thru 0x016e32 (0x312a bytes) 000664 1787841515.581758 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3c129): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c129): Mirror 10: 0x1469a09b (stale) Mirror 11: 0x1469a09b (stale) Mirror 13: 0x1469a09b (stale) Mirror 15: 0x1469a09b (stale) Mirror 16: 0x1469a09b (stale) Mirror 18: 0x1469a09b (stale) Mirror 19: 0x1469a09b (stale) Mirror 20: 0x1469a09b (stale) Mirror 21: 0x160e64ad (stale) CRC-32 checksum value for chunk [0, 0x3c129): Mirror 1: 0xcabc2632 Mirror 22: 0xcabc2632 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c129: skipped 000665 1787841516.795523 mapwrite 0x01e6fa thru 0x026703 (0x800a bytes) 000666 1787841517.028587 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3c129): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c129): Mirror 10: 0x1469a09b (stale) Mirror 11: 0x1469a09b (stale) Mirror 13: 0x1469a09b (stale) Mirror 15: 0x1469a09b (stale) Mirror 16: 0x1469a09b (stale) Mirror 18: 0x1469a09b (stale) Mirror 19: 0x1469a09b (stale) Mirror 20: 0x1469a09b (stale) Mirror 21: 0x160e64ad (stale) Mirror 22: 0xcabc2632 (stale) CRC-32 checksum value for chunk [0, 0x3c129): Mirror 1: 0x2b9c8a63 Mirror 23: 0x2b9c8a63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c129: skipped 000670 1787841518.413920 trunc from 0x03c129 to 0x037a1a (0x470e bytes) 000671 1787841518.544350 trunc from 0x037a1b to 0x009b90 (0x2de8a bytes) 000672 1787841518.586353 read 0x0088aa thru 0x009b90 (0x12e7 bytes) 000673 1787841518.608101 mapwrite 0x015619 thru 0x021f81 (0xc969 bytes) 000675 1787841518.834771 read 0x006836 thru 0x007b36 (0x1301 bytes) 000676 1787841518.854453 trunc from 0x021f82 to 0x01ec25 (0x335c bytes) 000678 1787841518.885072 trunc from 0x01ec26 to 0x02c7d3 (0xdbae bytes) 000680 1787841518.919691 MIRROR_RESYNC 12 mirrors 000683 1787841522.324209 read 0x004f75 thru 0x013d11 (0xed9d bytes) 000684 1787841522.333728 read 0x02bd74 thru 0x02c7d3 (0xa60 bytes) 000685 1787841522.347558 mapwrite 0x032ae6 thru 0x039112 (0x662d bytes) 000686 1787841522.486867 read 0x01d5a4 thru 0x029f8b (0xc9e8 bytes) 000688 1787841522.501513 trunc from 0x039113 to 0x02940c (0xfd06 bytes) 000689 1787841522.519803 write 0x009a49 thru 0x016148 (0xc700 bytes) 000690 1787841522.521334 write 0x026e15 thru 0x028149 (0x1335 bytes) 000691 1787841522.524536 mapread 0x01750f thru 0x023f67 (0xca59 bytes) 000692 1787841522.525581 mapwrite 0x00d2b1 thru 0x015125 (0x7e75 bytes) 000693 1787841522.568534 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2940d): Mirror 10: 0x10b18b99 (stale) Mirror 11: 0x10b18b99 (stale) Mirror 13: 0x10b18b99 (stale) Mirror 15: 0x10b18b99 (stale) Mirror 16: 0x10b18b99 (stale) Mirror 18: 0x10b18b99 (stale) Mirror 20: 0x10b18b99 (stale) Mirror 21: 0x10b18b99 (stale) Mirror 22: 0x10b18b99 (stale) Mirror 23: 0x10b18b99 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2940d): Mirror 10: 0x10b18b99 (stale) Mirror 11: 0x10b18b99 (stale) Mirror 13: 0x10b18b99 (stale) Mirror 15: 0x10b18b99 (stale) Mirror 16: 0x10b18b99 (stale) Mirror 18: 0x10b18b99 (stale) Mirror 20: 0x10b18b99 (stale) Mirror 21: 0x10b18b99 (stale) Mirror 22: 0x10b18b99 (stale) Mirror 23: 0x10b18b99 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2940d: skipped 000694 1787841523.247772 mapwrite 0x00644f thru 0x0098ca (0x347c bytes) 000695 1787841523.325865 MIRROR_RESYNC 11 mirrors 000696 1787841525.235690 mapwrite 0x01605b thru 0x01e77f (0x8725 bytes) 000697 1787841525.424364 read 0x010cef thru 0x010eba (0x1cc bytes) 000698 1787841525.425619 trunc from 0x02940d to 0x008c69 (0x207a3 bytes) 000699 1787841525.455814 mapwrite 0x0253d1 thru 0x025578 (0x1a8 bytes) 000701 1787841525.531447 mapwrite 0x02af49 thru 0x034574 (0x962c bytes) 000703 1787841525.641137 write 0x03cd67 thru 0x03ffff (0x3299 bytes) 000707 1787841525.644965 mapread 0x026e37 thru 0x02844e (0x1618 bytes) 000708 1787841525.645933 mapread 0x0321d5 thru 0x03ffff (0xde2b bytes) 000709 1787841525.654429 mapread 0x00953e thru 0x01777b (0xe23e bytes) 000710 1787841525.672863 mapread 0x029673 thru 0x033bc5 (0xa553 bytes) 000711 1787841525.675377 mapread 0x0041a8 thru 0x00a108 (0x5f61 bytes) 000713 1787841525.677440 read 0x02c872 thru 0x033c8d (0x741c bytes) 000714 1787841525.679014 mapwrite 0x0215ac thru 0x02527c (0x3cd1 bytes) 000715 1787841525.744143 write 0x01e632 thru 0x02c9f1 (0xe3c0 bytes) 000716 1787841525.746692 read 0x03adb3 thru 0x03ffff (0x524d bytes) 000720 1787841525.750679 trunc from 0x040000 to 0x038356 (0x7ca9 bytes) 000723 1787841525.777705 write 0x009e16 thru 0x018143 (0xe32e bytes) 000724 1787841525.779758 mapwrite 0x01ed47 thru 0x02162d (0x28e7 bytes) 000725 1787841525.868314 MIRROR_RESYNC 11 mirrors 000729 1787841527.329488 mapread 0x000049 thru 0x000d1d (0xcd5 bytes) 000730 1787841527.341018 read 0x0095e2 thru 0x00a828 (0x1247 bytes) 000731 1787841527.350535 mapread 0x00aa92 thru 0x019a6b (0xefda bytes) 000732 1787841527.419689 write 0x012063 thru 0x01ef5a (0xcef8 bytes) 000733 1787841527.532970 trunc from 0x038357 to 0x02c1af (0xc1a7 bytes) 000735 1787841527.573934 mapread 0x025bf5 thru 0x02c1af (0x65bb bytes) 000736 1787841527.588527 mapread 0x01bf9b thru 0x02817a (0xc1e0 bytes) 000738 1787841527.590016 mapread 0x01924c thru 0x02782f (0xe5e4 bytes) 000739 1787841527.591764 trunc from 0x02c1b0 to 0x0029e8 (0x297c7 bytes) 000741 1787841527.607768 read 0x00154a thru 0x0029e8 (0x149f bytes) 000743 1787841527.615452 mapwrite 0x038701 thru 0x03e71f (0x601f bytes) 000745 1787841527.732273 mapread 0x0380ff thru 0x03e71f (0x6621 bytes) 000748 1787841527.736238 trunc from 0x03e720 to 0x00e87c (0x2fea3 bytes) 000749 1787841527.781812 trunc from 0x00e87d to 0x03964f (0x2add3 bytes) 000750 1787841527.837374 read 0x023810 thru 0x02702d (0x381e bytes) 000751 1787841527.856589 mapread 0x00906a thru 0x00f626 (0x65bd bytes) 000753 1787841527.859609 write 0x03dea4 thru 0x03ffff (0x215c bytes) 000754 1787841527.862403 mapread 0x02ebbc thru 0x037122 (0x8567 bytes) 000755 1787841527.866002 mapread 0x001cb9 thru 0x00945c (0x77a4 bytes) 000756 1787841527.869987 read 0x025f19 thru 0x02c676 (0x675e bytes) 000757 1787841527.871707 read 0x02bb71 thru 0x032f31 (0x73c1 bytes) 000758 1787841527.873703 trunc from 0x040000 to 0x02b200 (0x14dff bytes) 000760 1787841527.915379 MIRROR_RESYNC 11 mirrors 000762 1787841529.803496 trunc from 0x02b201 to 0x0329e3 (0x77e3 bytes) 000763 1787841529.883169 mapread 0x008143 thru 0x015c88 (0xdb46 bytes) 000766 1787841529.920740 MIRROR_RESYNC 11 mirrors 000767 1787841532.968016 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000768 1787841533.735495 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000770 1787841534.762650 write 0x007a58 thru 0x00b8c4 (0x3e6d bytes) 000771 1787841534.924410 mapwrite 0x012732 thru 0x0213ec (0xecbb bytes) 000772 1787841535.101341 mapwrite 0x01615d thru 0x019209 (0x30ad bytes) 000773 1787841535.167922 mapwrite 0x0149bf thru 0x018355 (0x3997 bytes) 000774 1787841535.289338 trunc from 0x0329e4 to 0x01d504 (0x154df bytes) 000776 1787841535.324512 trunc from 0x01d505 to 0x0028f6 (0x1ac0e bytes) 000778 1787841535.347542 read 0x001c67 thru 0x0028f6 (0xc90 bytes) 000780 1787841535.375376 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28f7): Mirror 11: 0xa46f8544 (stale) Mirror 13: 0xa46f8544 (stale) Mirror 15: 0xa46f8544 (stale) Mirror 16: 0xa46f8544 (stale) Mirror 18: 0xa46f8544 (stale) Mirror 21: 0xa46f8544 (stale) Mirror 23: 0xa46f8544 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28f7): Mirror 11: 0xa46f8544 (stale) Mirror 13: 0xa46f8544 (stale) Mirror 15: 0xa46f8544 (stale) Mirror 16: 0xa46f8544 (stale) Mirror 18: 0xa46f8544 (stale) Mirror 21: 0xa46f8544 (stale) Mirror 23: 0xa46f8544 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28f7: skipped 000781 1787841536.071702 mapwrite 0x01131e thru 0x01dcbc (0xc99f bytes) 000782 1787841536.169938 mapwrite 0x023b34 thru 0x025c11 (0x20de bytes) 000783 1787841536.359461 mapwrite 0x038be9 thru 0x03ffff (0x7417 bytes) 000784 1787841536.503879 mapread 0x00f7e6 thru 0x013aad (0x42c8 bytes) 000786 1787841536.511352 mapread 0x016f09 thru 0x01fd7b (0x8e73 bytes) 000788 1787841536.515110 read 0x01e70e thru 0x02e1a8 (0xfa9b bytes) 000789 1787841536.555578 mapwrite 0x025149 thru 0x02f6a8 (0xa560 bytes) 000790 1787841536.620551 read 0x0202d6 thru 0x021d4b (0x1a76 bytes) 000791 1787841536.621101 read 0x0174c3 thru 0x018907 (0x1445 bytes) 000792 1787841536.621633 read 0x02c965 thru 0x0374b3 (0xab4f bytes) 000797 1787841536.631427 mapread 0x02c24a thru 0x03a3ba (0xe171 bytes) 000798 1787841536.632371 read 0x02ed1e thru 0x03166c (0x294f bytes) 000799 1787841536.634554 trunc from 0x040000 to 0x011ffc (0x2e003 bytes) 000800 1787841536.666363 mapread 0x002853 thru 0x011893 (0xf041 bytes) 000801 1787841536.701282 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11ffd): Mirror 11: 0x19456e19 (stale) Mirror 13: 0x19456e19 (stale) Mirror 15: 0x19456e19 (stale) Mirror 18: 0x19456e19 (stale) Mirror 21: 0x19456e19 (stale) Mirror 23: 0x19456e19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ffd): Mirror 11: 0x19456e19 (stale) Mirror 13: 0x19456e19 (stale) Mirror 15: 0x19456e19 (stale) Mirror 18: 0x19456e19 (stale) Mirror 21: 0x19456e19 (stale) Mirror 23: 0x19456e19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ffd: skipped 000802 1787841537.671785 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11ffd): Mirror 11: 0x19456e19 (stale) Mirror 13: 0x19456e19 (stale) Mirror 15: 0x19456e19 (stale) Mirror 18: 0x19456e19 (stale) Mirror 23: 0x19456e19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ffd): Mirror 11: 0x19456e19 (stale) Mirror 13: 0x19456e19 (stale) Mirror 15: 0x19456e19 (stale) Mirror 18: 0x19456e19 (stale) Mirror 23: 0x19456e19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ffd: skipped 000804 1787841538.050030 trunc from 0x011ffd to 0x003ecf (0xe12d bytes) 000807 1787841538.072389 read 0x00288d thru 0x00378a (0xefe bytes) 000808 1787841538.077424 write 0x00e340 thru 0x013b57 (0x5818 bytes) 000809 1787841538.078839 mapwrite 0x01447b thru 0x01b3eb (0x6f71 bytes) 000811 1787841538.210705 mapread 0x00f6a4 thru 0x012b19 (0x3476 bytes) 000812 1787841538.211609 mapread 0x00c047 thru 0x01b06e (0xf028 bytes) 000813 1787841538.224193 mapread 0x000862 thru 0x007f7d (0x771c bytes) 000814 1787841538.230174 mapread 0x0074bb thru 0x00d351 (0x5e97 bytes) 000816 1787841538.231540 mapwrite 0x0098b3 thru 0x010bb6 (0x7304 bytes) 000818 1787841538.301110 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b3ec): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b3ec): Mirror 11: 0xa13c202a (stale) Mirror 13: 0xa13c202a (stale) Mirror 15: 0xa13c202a (stale) Mirror 18: 0xa13c202a (stale) Mirror 23: 0xa13c202a (stale) CRC-32 checksum value for chunk [0, 0x1b3ec): Mirror 1: 0x9dd13806 Mirror 24: 0x9dd13806 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b3ec: skipped 000819 1787841539.577699 read 0x000a42 thru 0x007a0b (0x6fca bytes) 000820 1787841539.596857 read 0x00c1bd thru 0x00fcf1 (0x3b35 bytes) 000823 1787841539.624626 mapread 0x0151b7 thru 0x01b3eb (0x6235 bytes) 000824 1787841539.625376 mapread 0x0041fb thru 0x012608 (0xe40e bytes) 000827 1787841539.628901 mapwrite 0x001081 thru 0x00c1f2 (0xb172 bytes) 000831 1787841540.096315 write 0x01296e thru 0x0153b2 (0x2a45 bytes) 000834 1787841540.106660 trunc from 0x01b3ec to 0x02d880 (0x12495 bytes) 000835 1787841540.161996 read 0x01ee91 thru 0x027702 (0x8872 bytes) 000836 1787841540.197853 trunc from 0x02d881 to 0x0237f4 (0xa08c bytes) 000839 1787841540.247189 write 0x02da39 thru 0x03bf00 (0xe4c8 bytes) 000840 1787841540.257281 trunc from 0x03bf01 to 0x03284f (0x96b1 bytes) truncating to largest ever: 0x3ff71 000843 1787841540.308214 trunc from 0x032850 to 0x03ff70 (0xd721 bytes) 000844 1787841540.405491 read 0x03a7f9 thru 0x03ff70 (0x5778 bytes) 000845 1787841540.428591 trunc from 0x03ff71 to 0x01f93f (0x20631 bytes) 000847 1787841540.467532 mapwrite 0x03ad80 thru 0x03b3ff (0x680 bytes) 000848 1787841540.641974 trunc from 0x03b400 to 0x00bd19 (0x2f6e6 bytes) 000850 1787841540.719714 MIRROR_RESYNC 7 mirrors 000851 1787841542.606904 mapwrite 0x0030de thru 0x01051f (0xd442 bytes) 000852 1787841542.914378 mapwrite 0x03d373 thru 0x03ffff (0x2c8d bytes) 000853 1787841543.030266 mapwrite 0x0176ee thru 0x026de9 (0xf6fc bytes) 000854 1787841543.185469 write 0x0393ac thru 0x03ffff (0x6c54 bytes) 000855 1787841543.201890 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5832bf7a (stale) Mirror 13: 0x5832bf7a (stale) Mirror 15: 0x5832bf7a (stale) Mirror 18: 0x5832bf7a (stale) Mirror 23: 0x5832bf7a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x5832bf7a (stale) Mirror 13: 0x5832bf7a (stale) Mirror 15: 0x5832bf7a (stale) Mirror 18: 0x5832bf7a (stale) Mirror 23: 0x5832bf7a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000858 1787841543.888862 mapread 0x019c3e thru 0x026347 (0xc70a bytes) 000860 1787841543.941454 read 0x03470b thru 0x03fd22 (0xb618 bytes) 000861 1787841543.975330 read 0x016227 thru 0x019032 (0x2e0c bytes) 000862 1787841543.975898 read 0x01e3f6 thru 0x029ce7 (0xb8f2 bytes) 000863 1787841543.976479 mapwrite 0x030233 thru 0x0308f1 (0x6bf bytes) 000864 1787841544.111372 write 0x00a56d thru 0x018536 (0xdfca bytes) 000866 1787841544.113224 read 0x03e869 thru 0x03ffff (0x1797 bytes) 000867 1787841544.120340 mapread 0x01191e thru 0x01d30b (0xb9ee bytes) 000872 1787841544.128347 trunc from 0x040000 to 0x01bc3c (0x243c3 bytes) 000873 1787841544.180439 trunc from 0x01bc3d to 0x021bf0 (0x5fb4 bytes) 000874 1787841544.222174 mapread 0x00809d thru 0x0080f7 (0x5b bytes) 000876 1787841544.223708 trunc from 0x021bf1 to 0x00599b (0x1c255 bytes) 000877 1787841544.263553 write 0x02c57d thru 0x03067e (0x4102 bytes) 000878 1787841544.267972 write 0x00e33f thru 0x016b73 (0x8835 bytes) 000879 1787841544.307505 MIRROR_RESYNC 6 mirrors 000881 1787841546.572460 mapwrite 0x021589 thru 0x024d48 (0x37c0 bytes) 000882 1787841546.924897 trunc from 0x03067f to 0x034126 (0x3aa8 bytes) 000886 1787841546.979338 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34127): Mirror 11: 0x378c6c70 (stale) Mirror 13: 0x378c6c70 (stale) Mirror 18: 0x378c6c70 (stale) Mirror 23: 0x378c6c70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34127): Mirror 11: 0x378c6c70 (stale) Mirror 13: 0x378c6c70 (stale) Mirror 18: 0x378c6c70 (stale) Mirror 23: 0x378c6c70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34127: skipped 000887 1787841547.701669 mapread 0x00ba23 thru 0x015080 (0x965e bytes) 000888 1787841547.740684 mapwrite 0x020501 thru 0x0221a5 (0x1ca5 bytes) 000889 1787841547.806443 mapwrite 0x035f54 thru 0x03ffff (0xa0ac bytes) 000890 1787841547.959109 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 18: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x1e77a32a (stale) Mirror 13: 0x1e77a32a (stale) Mirror 18: 0x1e77a32a (stale) Mirror 23: 0x1e77a32a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6bdce361 Mirror 24: 0x6bdce361 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000892 1787841549.617037 mapwrite 0x0101cd thru 0x01f644 (0xf478 bytes) 000894 1787841549.887151 mapread 0x014fa7 thru 0x01b689 (0x66e3 bytes) 000895 1787841549.890024 mapread 0x0388b4 thru 0x03f85d (0x6faa bytes) 000896 1787841549.906996 mapread 0x036b49 thru 0x03ffff (0x94b7 bytes) 000897 1787841549.921169 trunc from 0x040000 to 0x01dc70 (0x2238f bytes) 000899 1787841549.963733 trunc from 0x01dc71 to 0x00d202 (0x10a6e bytes) 000900 1787841549.985685 read 0x008d41 thru 0x00d202 (0x44c2 bytes) 000907 1787841550.000618 mapwrite 0x015999 thru 0x023e3e (0xe4a6 bytes) 000909 1787841550.150343 mapread 0x0132d5 thru 0x015778 (0x24a4 bytes) 000911 1787841550.161094 write 0x01718b thru 0x01fea8 (0x8d1e bytes) 000912 1787841550.167328 mapread 0x023a1d thru 0x023e3e (0x422 bytes) 000914 1787841550.168987 read 0x01277e thru 0x01da59 (0xb2dc bytes) 000915 1787841550.196344 MIRROR_RESYNC 6 mirrors 000916 1787841552.134814 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) 000917 1787841552.594181 read 0x006009 thru 0x00f37a (0x9372 bytes) 000919 1787841552.613092 mapread 0x0091a5 thru 0x0124d3 (0x932f bytes) 000920 1787841552.647480 mapread 0x0173c2 thru 0x01c683 (0x52c2 bytes) 000921 1787841552.663640 write 0x00a00f thru 0x015a76 (0xba68 bytes) 000922 1787841552.877865 read 0x01e6b9 thru 0x023c49 (0x5591 bytes) 000924 1787841552.899447 trunc from 0x023e3f to 0x017c83 (0xc1bb bytes) 000925 1787841552.948603 trunc from 0x017c84 to 0x01aa56 (0x2dd3 bytes) 000928 1787841552.985617 read 0x00253f thru 0x00951c (0x6fde bytes) 000931 1787841552.988150 mapread 0x00cdd4 thru 0x00f38e (0x25bb bytes) 000932 1787841552.989265 mapwrite 0x03d10d thru 0x03ffff (0x2ef3 bytes) 000933 1787841553.134065 mapwrite 0x02b075 thru 0x02f2d5 (0x4261 bytes) 000934 1787841553.227898 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x663e5b91 (stale) Mirror 13: 0x663e5b91 (stale) Mirror 23: 0x663e5b91 (stale) Mirror 24: 0x663e5b91 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7143fb91 Mirror 25: 0x7143fb91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000935 1787841554.155773 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x663e5b91 (stale) Mirror 13: 0x663e5b91 (stale) Mirror 23: 0x663e5b91 (stale) Mirror 24: 0x663e5b91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x663e5b91 (stale) Mirror 13: 0x663e5b91 (stale) Mirror 23: 0x663e5b91 (stale) Mirror 24: 0x663e5b91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000936 1787841554.748111 read 0x0129e8 thru 0x019f45 (0x755e bytes) 000937 1787841554.764568 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x663e5b91 (stale) Mirror 13: 0x663e5b91 (stale) Mirror 23: 0x663e5b91 (stale) Mirror 24: 0x663e5b91 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7143fb91 Mirror 25: 0x7143fb91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000940 1787841555.953904 MIRROR_RESYNC 6 mirrors 000941 1787841558.351827 write 0x02f85e thru 0x032372 (0x2b15 bytes) 000942 1787841558.422922 mapwrite 0x03e75f thru 0x03ffff (0x18a1 bytes) 000943 1787841558.556793 mapread 0x01e5a8 thru 0x02dc7d (0xf6d6 bytes) 000944 1787841558.599447 write 0x0349a3 thru 0x03c6e5 (0x7d43 bytes) 000946 1787841558.621222 trunc from 0x040000 to 0x039814 (0x67eb bytes) 000948 1787841558.661240 write 0x01c91c thru 0x024bfa (0x82df bytes) 000949 1787841558.662401 write 0x0004f8 thru 0x006559 (0x6062 bytes) 000950 1787841558.696324 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39815): Mirror 13: 0x487a265a (stale) Mirror 23: 0x487a265a (stale) Mirror 24: 0x487a265a (stale) Mirror 25: 0x487a265a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39815): Mirror 13: 0x487a265a (stale) Mirror 23: 0x487a265a (stale) Mirror 24: 0x487a265a (stale) Mirror 25: 0x487a265a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39815: skipped 000951 1787841559.207498 mapwrite 0x02bbe8 thru 0x03159c (0x59b5 bytes) 000952 1787841559.266909 mapread 0x025205 thru 0x02f24f (0xa04b bytes) 000954 1787841559.276828 mapread 0x024aca thru 0x026820 (0x1d57 bytes) 000955 1787841559.278025 read 0x00e358 thru 0x00e601 (0x2aa bytes) 000956 1787841559.282775 mapwrite 0x00db50 thru 0x0173bf (0x9870 bytes) 000957 1787841559.351065 trunc from 0x039815 to 0x01cb74 (0x1cca0 bytes) 000958 1787841559.368385 mapwrite 0x01e8ba thru 0x02a513 (0xbc5a bytes) 000959 1787841559.445572 mapread 0x028b07 thru 0x02a513 (0x1a0d bytes) 000960 1787841559.447722 mapread 0x0147fc thru 0x02239f (0xdba4 bytes) 000961 1787841559.449485 write 0x035176 thru 0x035df9 (0xc84 bytes) 000963 1787841559.451645 read 0x009283 thru 0x00e80d (0x558b bytes) 000964 1787841559.452295 write 0x02a2ff thru 0x02bea2 (0x1ba4 bytes) 000965 1787841559.465302 mapread 0x01b74d thru 0x01c2a6 (0xb5a bytes) 000966 1787841559.466590 mapwrite 0x0228bf thru 0x02dd45 (0xb487 bytes) 000970 1787841559.531381 mapread 0x033613 thru 0x035df9 (0x27e7 bytes) 000971 1787841559.532887 mapread 0x0195e0 thru 0x024fca (0xb9eb bytes) 000972 1787841559.534484 mapread 0x0307f0 thru 0x035df9 (0x560a bytes) 000973 1787841559.535824 write 0x029aec thru 0x02b013 (0x1528 bytes) 000974 1787841559.537483 mapread 0x009c5d thru 0x01722e (0xd5d2 bytes) 000976 1787841559.540165 mapwrite 0x0150e0 thru 0x023693 (0xe5b4 bytes) 000978 1787841559.616091 trunc from 0x035dfa to 0x0139f4 (0x22405 bytes) 000979 1787841559.655828 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x139f5): Mirror 13: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x139f5): Mirror 13: 0x7f1deeee (stale) Mirror 23: 0x7f1deeee (stale) Mirror 24: 0x7f1deeee (stale) Mirror 25: 0x7f1deeee (stale) CRC-32 checksum value for chunk [0, 0x139f5): Mirror 1: 0xd054bcf4 Mirror 26: 0xd054bcf4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x139f5: skipped 000980 1787841560.328027 MIRROR_RESYNC 6 mirrors 000981 1787841561.537639 MIRROR_RESYNC 6 mirrors 000984 1787841562.025245 mapread 0x00a929 thru 0x01066e (0x5d46 bytes) 000987 1787841562.052003 mapread 0x012411 thru 0x0139f4 (0x15e4 bytes) 000990 1787841562.053926 trunc from 0x0139f5 to 0x02e04c (0x1a658 bytes) 000991 1787841562.259596 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e04d): Mirror 13: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e04d): Mirror 13: 0x5fe5f7d8 (stale) Mirror 23: 0x5fe5f7d8 (stale) Mirror 24: 0x5fe5f7d8 (stale) Mirror 25: 0x5fe5f7d8 (stale) Mirror 26: 0x5fe5f7d8 (stale) CRC-32 checksum value for chunk [0, 0x2e04d): Mirror 1: 0x5fe5f7d8 Mirror 27: 0x5fe5f7d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e04d: skipped 000992 1787841564.026410 read 0x00f5eb thru 0x015c1e (0x6634 bytes) 000993 1787841564.059651 trunc from 0x02e04d to 0x00f9f0 (0x1e65c bytes) 000994 1787841564.167698 trunc from 0x00f9f1 to 0x01dd15 (0xe325 bytes) 000995 1787841564.199701 mapwrite 0x032e5d thru 0x03ffff (0xd1a3 bytes) 000998 1787841564.333994 mapwrite 0x02f12a thru 0x039f8e (0xae65 bytes) 000999 1787841564.419681 mapwrite 0x02fc81 thru 0x03e7d5 (0xeb55 bytes) 001000 1787841564.517679 write 0x03d25e thru 0x03ffff (0x2da2 bytes) All operations completed A-OK! PASS 70a (145s) == sanity-flr test 71: check mirror extend parent fid ==== 10:39:31 (1787841571) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 10:39:36 (1787841576) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265942 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364724 s, 2.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 10:39:47 (1787841587) Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre2 Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2860416 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7901536 .. done Extending file size to 3507488 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8176448 .. Extending file size to 8570464 .. Extending file size to 5590528 .. Extending file size to 8757664 .. Extending file size to 7117024 .. Extending file size to 8912864 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8853056 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8946592 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 58734 58735 58737 58739 58742 192.168.204.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg456-client.virtnet /mnt/lustre2 (opts:) 192.168.204.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg456-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:41:06 (1787841666) Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre2 Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9161376 .. Extending file size to 1039040 .. failed Extending file size to 5942208 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6185792 .. done Extending file size to 7259168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7442816 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7968000 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8893216 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9231328 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60300 60301 60302 60305 60307 done 192.168.204.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg456-client.virtnet /mnt/lustre2 (opts:) 192.168.204.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg456-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:42:28 (1787841748) Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154341 s, 13.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.538537 s, 3.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xad0168c2 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg456-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:42:51 (1787841771) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100843 s, 20.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 10:42:59 (1787841779) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:43:06 (1787841786) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.250734 s, 8.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:43:14 (1787841794) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.26498 s, 7.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:43:22 (1787841802) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257115 s, 8.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:43:57 (1787841837) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg456-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113883 s, 9.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:44:58 (1787841898) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg456-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.399981 s, 5.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:45:55 (1787841955) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg456-server mds-ost sync done. Creating new pool test_204f oleg456-server: Pool lustre.test_204f created Adding targets to pool oleg456-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg456-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.329493 s, 6.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg456-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg456-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg456-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:47:14 (1787842034) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:47:22 (1787842042) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:47:29 (1787842049) oleg456-server: Pool lustre.test_206 created Adding targets to pool oleg456-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg456-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg456-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg456-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg456-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:47:54 (1787842074) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992907 s, 10.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585248 s, 17.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x23859b30 Mirror 3: 0x23859b30 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:48:05 (1787842085) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:48:08 (1787842088) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:48:11 (1787842091) Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0898905 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x11f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xe80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xe80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xe80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb10000 Read this (4 bytes): BBBB 192.168.204.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg456-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:48:48 (1787842128) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0140747 s, 582 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:48:56 (1787842136) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507552 s, 20.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834453 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 10:48:57.000000000 -0400 Modify: 2026-08-27 10:48:57.000000000 -0400 Change: 2026-08-27 10:48:58.000000000 -0400 Birth: 2026-08-27 10:48:57.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:49:03 (1787842143) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:49:06 (1787842146) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.446914 s, 91.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.055243 s, 74.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 10:49 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x217:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 10:49 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:49:14 (1787842154) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:49:23 (1787842163) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1978 sec =========== 10:49:33 (1787842173) === sanity-flr: start cleanup 10:49:35 (1787842175) === === sanity-flr: finish cleanup 10:49:42 (1787842182) ===