-----============= acceptance-small: sanity-flr ============----- Mon May 4 11:55:54 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg460-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:56:10 (1777910170) === oleg460-client.virtnet: executing check_config_client /mnt/lustre oleg460-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg460-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97ab9057e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff97ab9057e000.idle_timeout=debug disable quota as required oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:56:39 (1777910199) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7573] == sanity-flr test 0a: lfs mirror create with -N option == 11:56:41 (1777910201) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:56:55 (1777910215) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:56:57 (1777910217) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:57:00 (1777910220) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:57:25 (1777910245) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:57:28 (1777910248) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 11:57:31 (1777910251) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff97ab9057e000.stats=clear osc.lustre-OST0001-osc-ffff97ab9057e000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:57:38 (1777910258) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:57:47 (1777910267) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:57:55 (1777910275) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:57:59 (1777910279) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:58:07 (1777910287) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 11:58:16 (1777910296) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:58:22 (1777910302) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:58:26 (1777910306) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:58:33 (1777910313) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:58:42 (1777910322) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:58:48 (1777910328) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.55354 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186616 s, 5.6 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:58:59 (1777910339) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.41524 s, 32.8 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:59:07 (1777910347) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:59:17 (1777910357) Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97ab9057e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97ab9057e000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg460-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97ab9057e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97ab9057e000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (127s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:01:23 (1777910483) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.229761 s, 18.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97ab9057e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97ab9057e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (32s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:01:55 (1777910515) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97ab9057e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97ab9057e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97ab9057e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97ab9057e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (66s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:03:01 (1777910581) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.86041 s, 10.8 MB/s -rw-r--r-- 1 root root 52428800 May 4 12:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.905s user 0m0.003s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97ab9057e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97ab9057e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.089s user 0m0.000s sys 0m0.171s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg460-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg460-server oleg460-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97ab9057e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97ab9057e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.904s user 0m0.003s sys 0m0.178s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg460-server: oleg460-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg460-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg460-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ab9057e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97ab9057e000.ost_server_uuid in FULL state after 0 sec PASS 33b (66s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:04:07 (1777910647) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:04:10 (1777910650) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:04:12 (1777910652) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:04:15 (1777910655) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 12:04:21 (1777910661) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.70538 s, 17.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:04:32 (1777910672) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025645 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429665 s, 24.4 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:04:37 (1777910677) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:04:43 (1777910683) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127477 s, 803 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000703293 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000870793 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000784342 s, 1.3 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 12:04:51 (1777910691) create test files with size 15741709 .. df43374c5222aa036dbb6977f79ab7a2 - 2b057f7a3cbfa2940b73b6765d6625d6 - 8ca41a56bac223691769f5b233bca9e5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre PASS 37 (17s) == sanity-flr test 38: resync ============================ 12:05:08 (1777910708) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:05:19 (1777910719) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:05:25 (1777910725) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116497 s, 9.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522663 s, 20.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 12:05:32 (1777910732) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109344 s, 19.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.336026 s, 12.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187155 s, 11.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 12:05:45 (1777910745) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 12:05:48 (1777910748) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:05:50 (1777910750) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251099 s, 41.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 12:05:56 (1777910756) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:05:58 (1777910758) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe0:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe0:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:06:06 (1777910766) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.21807 s, 48.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:06:14 (1777910774) llite.lustre-ffff97ab92394800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.209174 s, 50.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg460-client.virtnet /mnt/lustre (opts:) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff97ab89964000.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:06:24 (1777910784) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (28s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:06:52 (1777910812) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (29s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:07:21 (1777910841) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 12:07:23 (1777910843) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.252362 s, 12.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.10533 s, 29.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x2de732d (stale) Mirror 2: 0x2de732d (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: c31f1fdb442155716b2b86fb89dcdfe8 - mirror 1 checksum: 625318a5e3a543b07391a2826db60d47 - mirror 2 checksum: c31f1fdb442155716b2b86fb89dcdfe8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 4 12:07:26 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 4 12:07:26 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c31f1fdb442155716b2b86fb89dcdfe8 - mirror 1 checksum: 625318a5e3a543b07391a2826db60d47 - mirror 2 checksum: 625318a5e3a543b07391a2826db60d47 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 4 12:07:26 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 4 12:07:26 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:07:33 (1777910853) Failing mds1 on oleg460-server Stopping /mnt/lustre-mds1 (opts:) on oleg460-server 12:07:38 (1777910858) shut down facet: mds1 facet_host: oleg460-server facet_failover_host: oleg460-server Failover mds1 to oleg460-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 12:07:57 (1777910877) targets are mounted 12:07:57 (1777910877) facet_failover done oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239173 s, 43.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (36s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:08:09 (1777910889) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:08:12 (1777910892) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:08:14 (1777910894) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00868671 s, 1.9 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:08:19 (1777910899) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:08:22 (1777910902) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:08:25 (1777910905) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:08:45 (1777910925) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:09:09 (1777910949) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:09:30 (1777910970) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:09:52 (1777910992) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 12:09:57 (1777910997) mirror create pid 53520 mirror split pid 53522 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53520 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53522 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 12:11:02 (1777911062) Seed set to 1063 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1ff09 000005 1777911063.570837 trunc from 00000000 to 0x01ff08 (0x1ff09 bytes) 000006 1777911063.593258 mapwrite 0x008df1 thru 0x014d88 (0xbf98 bytes) truncating to largest ever: 0x2ab8f 000007 1777911063.692586 trunc from 0x01ff09 to 0x02ab8e (0xac86 bytes) 000009 1777911063.718544 mapread 0x01465f thru 0x0239d1 (0xf373 bytes) 000011 1777911063.732654 trunc from 0x02ab8f to 0x01df36 (0xcc58 bytes) 000013 1777911063.762397 read 0x014b1d thru 0x019fa9 (0x548d bytes) 000014 1777911063.767789 write 0x01a848 thru 0x01c176 (0x192f bytes) 000015 1777911063.768896 mapwrite 0x03f19a thru 0x03ffff (0xe66 bytes) 000016 1777911063.876276 mapwrite 0x00e335 thru 0x014d24 (0x69f0 bytes) 000018 1777911063.980560 mapread 0x0007db thru 0x00ef3a (0xe760 bytes) 000020 1777911063.998695 read 0x038cd1 thru 0x03db38 (0x4e68 bytes) 000022 1777911064.002181 mapwrite 0x00d0b6 thru 0x017a77 (0xa9c2 bytes) 000024 1777911064.087168 trunc from 0x040000 to 0x02a499 (0x15b66 bytes) 000025 1777911064.113048 MIRROR_EXTEND to 2 mirrors 000026 1777911064.821253 read 0x010ea6 thru 0x01e403 (0xd55e bytes) 000027 1777911064.838816 mapread 0x007ad7 thru 0x00ebc2 (0x70ec bytes) 000028 1777911064.888596 mapread 0x00cc12 thru 0x00cdc2 (0x1b1 bytes) 000030 1777911064.911719 write 0x020e05 thru 0x023dc4 (0x2fc0 bytes) truncating to largest ever: 0x36f43 000032 1777911065.002405 trunc from 0x02a49a to 0x036f42 (0xcaa9 bytes) 000033 1777911065.021904 mapwrite 0x013835 thru 0x016f64 (0x3730 bytes) 000034 1777911065.097909 mapread 0x01ff8d thru 0x028240 (0x82b4 bytes) 000037 1777911065.103936 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36f43): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f43): Mirror 2: 0xa12c5cbd (stale) CRC-32 checksum value for chunk [0, 0x36f43): Mirror 1: 0x9a3c402 Mirror 3: 0x9a3c402 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f43: skipped 000038 1777911065.790029 mapread 0x036a61 thru 0x036f42 (0x4e2 bytes) 000039 1777911065.814328 write 0x02e6bb thru 0x035bbb (0x7501 bytes) 000040 1777911065.924700 read 0x032728 thru 0x036030 (0x3909 bytes) 000042 1777911065.931864 mapwrite 0x030d0f thru 0x03f41b (0xe70d bytes) 000045 1777911066.054353 mapwrite 0x01750b thru 0x0230fb (0xbbf1 bytes) 000047 1777911066.160779 trunc from 0x03f41c to 0x008d19 (0x36702 bytes) 000049 1777911066.195844 mapread 0x008979 thru 0x008d19 (0x3a1 bytes) 000050 1777911066.218989 read 0x005728 thru 0x0072b8 (0x1b91 bytes) 000052 1777911066.220578 trunc from 0x008d1a to 0x033912 (0x2abf9 bytes) 000053 1777911066.246146 MIRROR_RESYNC 3 mirrors 000054 1777911067.065598 read 0x00f4ad thru 0x019895 (0xa3e9 bytes) 000056 1777911067.077712 read 0x0067a2 thru 0x00832c (0x1b8b bytes) 000057 1777911067.093108 read 0x0143dd thru 0x018290 (0x3eb4 bytes) 000058 1777911067.093542 read 0x0273e5 thru 0x033912 (0xc52e bytes) 000060 1777911067.094154 write 0x014a94 thru 0x022b70 (0xe0dd bytes) 000063 1777911067.174392 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33913): Mirror 2: 0xa86bdd55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33913): Mirror 2: 0xa86bdd55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33913: skipped 000065 1777911067.502039 mapwrite 0x01b2e0 thru 0x020bec (0x590d bytes) 000068 1777911067.621762 MIRROR_RESYNC 2 mirrors 000071 1777911068.258338 mapread 0x0188f2 thru 0x0274a6 (0xebb5 bytes) 000072 1777911068.289642 write 0x023e2f thru 0x0319cb (0xdb9d bytes) 000074 1777911068.364901 write 0x00f6c9 thru 0x01b492 (0xbdca bytes) 000075 1777911068.384589 write 0x03275f thru 0x035a4d (0x32ef bytes) 000076 1777911068.393702 mapread 0x00cd80 thru 0x013083 (0x6304 bytes) 000077 1777911068.399200 mapwrite 0x02d63f thru 0x02e0fe (0xac0 bytes) 000078 1777911068.450657 trunc from 0x035a4e to 0x02d5e6 (0x8467 bytes) 000080 1777911068.508788 write 0x013ccc thru 0x01e4b1 (0xa7e6 bytes) 000081 1777911068.510998 read 0x021e42 thru 0x021f18 (0xd7 bytes) 000082 1777911068.513265 write 0x02366a thru 0x024db6 (0x174d bytes) 000083 1777911068.514370 mapwrite 0x008bcc thru 0x01699d (0xddd2 bytes) 000086 1777911068.573918 mapwrite 0x023b99 thru 0x02c17d (0x85e5 bytes) 000089 1777911068.619057 trunc from 0x02d5e7 to 0x02dc8b (0x6a5 bytes) 000090 1777911068.640959 mapwrite 0x00a424 thru 0x011f19 (0x7af6 bytes) 000092 1777911068.690093 write 0x003ab0 thru 0x005b20 (0x2071 bytes) 000093 1777911068.706647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000097 1777911068.879846 read 0x0108b8 thru 0x017e9e (0x75e7 bytes) 000099 1777911068.908251 MIRROR_EXTEND to 2 mirrors 000100 1777911069.503888 read 0x010be0 thru 0x01c966 (0xbd87 bytes) 000102 1777911069.530819 mapwrite 0x00c8b8 thru 0x016a8f (0xa1d8 bytes) 000103 1777911069.735356 mapwrite 0x008398 thru 0x010184 (0x7ded bytes) 000106 1777911069.792937 write 0x03344f thru 0x03b529 (0x80db bytes) 000110 1777911069.803355 mapread 0x01401b thru 0x01c03e (0x8024 bytes) 000112 1777911069.808703 mapread 0x02d473 thru 0x035a22 (0x85b0 bytes) 000113 1777911069.824033 mapread 0x028685 thru 0x030a99 (0x8415 bytes) 000115 1777911069.832907 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000116 1777911069.989036 mapwrite 0x03a3f9 thru 0x03bbd0 (0x17d8 bytes) 000117 1777911070.076996 read 0x031ff1 thru 0x03bbd0 (0x9be0 bytes) 000118 1777911070.088110 mapwrite 0x0344ed thru 0x03a82e (0x6342 bytes) 000120 1777911070.128881 write 0x01b25f thru 0x025b96 (0xa938 bytes) 000121 1777911070.151700 write 0x02acdb thru 0x039094 (0xe3ba bytes) 000123 1777911070.172830 mapread 0x0048dd thru 0x00c100 (0x7824 bytes) 000124 1777911070.186589 trunc from 0x03bbd1 to 0x009f3b (0x31c95 bytes) 000128 1777911070.207468 write 0x03783c thru 0x039551 (0x1d16 bytes) 000131 1777911070.210775 mapwrite 0x0067af thru 0x00dcdc (0x752e bytes) 000133 1777911070.273264 mapwrite 0x0242d5 thru 0x02f40d (0xb139 bytes) 000135 1777911070.341056 read 0x029a06 thru 0x02dd2d (0x4328 bytes) 000136 1777911070.355402 MIRROR_EXTEND to 2 mirrors 000140 1777911070.903920 mapread 0x033189 thru 0x039551 (0x63c9 bytes) 000143 1777911070.949034 trunc from 0x039552 to 0x017e0a (0x21747 bytes) 000144 1777911071.047591 trunc from 0x017e0b to 0x029ac5 (0x11cbb bytes) 000147 1777911071.078560 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000148 1777911071.239114 read 0x0271eb thru 0x029ac5 (0x28db bytes) 000150 1777911071.252316 read 0x01d5e1 thru 0x022fa1 (0x59c1 bytes) 000151 1777911071.283693 MIRROR_EXTEND to 2 mirrors 000153 1777911071.803146 read 0x00f6b0 thru 0x01d818 (0xe169 bytes) 000154 1777911071.816858 mapread 0x019cfa thru 0x024134 (0xa43b bytes) 000155 1777911071.841898 trunc from 0x029ac6 to 0x017d61 (0x11d64 bytes) 000156 1777911071.958400 MIRROR_RESYNC 2 mirrors 000158 1777911072.561083 mapread 0x005661 thru 0x007898 (0x2238 bytes) 000159 1777911072.582205 trunc from 0x017d62 to 0x00b3b5 (0xc9ac bytes) 000160 1777911072.656062 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xb3b6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb3b6): Mirror 2: 0x458a6088 (stale) CRC-32 checksum value for chunk [0, 0xb3b6): Mirror 1: 0x458a6088 Mirror 3: 0x458a6088 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb3b6: skipped 000161 1777911073.413730 mapread 0x006d7b thru 0x00b3b5 (0x463b bytes) 000163 1777911073.475652 trunc from 0x00b3b6 to 0x016c4b (0xb896 bytes) 000164 1777911073.553523 read 0x013fe2 thru 0x016c4b (0x2c6a bytes) 000166 1777911073.559932 mapread 0x00d5ff thru 0x013bbc (0x65be bytes) 000168 1777911073.577387 read 0x0045a0 thru 0x00edbe (0xa81f bytes) 000169 1777911073.592215 mapwrite 0x01ef4e thru 0x022a62 (0x3b15 bytes) 000170 1777911073.696632 write 0x03dc95 thru 0x03ffff (0x236b bytes) 000171 1777911073.699332 read 0x02ca8a thru 0x02eba1 (0x2118 bytes) 000173 1777911073.712869 trunc from 0x040000 to 0x021910 (0x1e6ef bytes) 000174 1777911073.733786 write 0x017575 thru 0x01c235 (0x4cc1 bytes) 000176 1777911073.737661 mapread 0x009701 thru 0x0115ff (0x7eff bytes) 000178 1777911073.740770 write 0x02d6b0 thru 0x03cb40 (0xf491 bytes) 000180 1777911073.744283 mapread 0x0263b4 thru 0x02a91c (0x4569 bytes) truncating to largest ever: 0x3cf6e 000181 1777911073.785020 trunc from 0x03cb41 to 0x03cf6d (0x42d bytes) 000183 1777911073.814691 trunc from 0x03cf6e to 0x004592 (0x389db bytes) 000184 1777911073.844030 write 0x0177be thru 0x01b1f1 (0x3a34 bytes) 000187 1777911073.848756 mapwrite 0x013656 thru 0x021670 (0xe01b bytes) 000188 1777911073.990038 MIRROR_RESYNC 3 mirrors 000189 1777911074.740697 mapwrite 0x024540 thru 0x025887 (0x1348 bytes) 000191 1777911074.888282 mapwrite 0x0199a2 thru 0x027418 (0xda77 bytes) 000192 1777911074.979024 mapwrite 0x024c56 thru 0x02d6b6 (0x8a61 bytes) 000193 1777911075.072669 write 0x00011a thru 0x00181b (0x1702 bytes) 000194 1777911075.082256 write 0x03eb15 thru 0x03ffff (0x14eb bytes) 000195 1777911075.090823 MIRROR_RESYNC 3 mirrors 000196 1777911075.787473 mapwrite 0x03e71c thru 0x03ffff (0x18e4 bytes) 000197 1777911075.940896 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x984a035 (stale) Mirror 3: 0x984a035 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9d12457 Mirror 4: 0xc9d12457 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000200 1777911076.687916 MIRROR_RESYNC 4 mirrors 000202 1777911077.347150 trunc from 0x040000 to 0x03bf20 (0x40df bytes) 000203 1777911077.414951 trunc from 0x03bf21 to 0x01bdbf (0x20161 bytes) 000204 1777911077.436452 read 0x00c468 thru 0x0163ef (0x9f88 bytes) 000205 1777911077.445892 write 0x02a7d3 thru 0x02c803 (0x2031 bytes) 000208 1777911077.456510 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c804): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c804): Mirror 2: 0x4ae33183 (stale) Mirror 3: 0x4ae33183 (stale) Mirror 4: 0x4ae33183 (stale) CRC-32 checksum value for chunk [0, 0x2c804): Mirror 1: 0xa3498673 Mirror 5: 0xa3498673 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c804: skipped 000209 1777911078.072684 mapwrite 0x03cb52 thru 0x03ffff (0x34ae bytes) 000210 1777911078.264912 mapwrite 0x0139ed thru 0x015d83 (0x2397 bytes) 000211 1777911078.324234 write 0x005ddc thru 0x015564 (0xf789 bytes) 000213 1777911078.340509 mapread 0x017571 thru 0x01fe86 (0x8916 bytes) 000214 1777911078.350082 write 0x02de3a thru 0x037ad5 (0x9c9c bytes) 000215 1777911078.361046 write 0x03f0cf thru 0x03ffff (0xf31 bytes) 000216 1777911078.362157 read 0x00b347 thru 0x00fa26 (0x46e0 bytes) 000219 1777911078.365541 write 0x0399b5 thru 0x03ffff (0x664b bytes) 000221 1777911078.368395 read 0x03130b thru 0x03b42e (0xa124 bytes) 000227 1777911078.372779 mapwrite 0x036d33 thru 0x03f7f2 (0x8ac0 bytes) 000228 1777911078.451724 mapwrite 0x03b69c thru 0x03ffff (0x4964 bytes) 000229 1777911078.507826 read 0x03fe35 thru 0x03ffff (0x1cb bytes) 000231 1777911078.509225 write 0x018775 thru 0x02459f (0xbe2b bytes) 000233 1777911078.513920 trunc from 0x040000 to 0x01054b (0x2fab4 bytes) 000235 1777911078.544048 trunc from 0x01054c to 0x0122bc (0x1d71 bytes) 000237 1777911078.570771 mapwrite 0x02c6a7 thru 0x039062 (0xc9bc bytes) 000241 1777911078.662769 read 0x01b773 thru 0x024568 (0x8df6 bytes) 000242 1777911078.667627 mapread 0x026a31 thru 0x029b55 (0x3125 bytes) 000247 1777911078.672081 write 0x0073da thru 0x0120fc (0xad23 bytes) 000248 1777911078.675240 mapwrite 0x00d770 thru 0x011a9b (0x432c bytes) 000252 1777911078.727732 mapwrite 0x002a35 thru 0x00470f (0x1cdb bytes) 000253 1777911078.772771 MIRROR_RESYNC 5 mirrors 000255 1777911079.615220 MIRROR_RESYNC 5 mirrors 000256 1777911080.089867 mapwrite 0x02c88d thru 0x02cc7b (0x3ef bytes) 000258 1777911080.281474 write 0x03ae42 thru 0x03b7d4 (0x993 bytes) 000259 1777911080.287853 mapread 0x024bef thru 0x02d0b8 (0x84ca bytes) 000261 1777911080.335538 mapread 0x0342fe thru 0x034dd5 (0xad8 bytes) 000264 1777911080.346393 trunc from 0x03b7d5 to 0x035ef2 (0x58e2 bytes) 000266 1777911080.413299 MIRROR_RESYNC 5 mirrors 000269 1777911081.334659 MIRROR_EXTEND to 6 mirrors 000270 1777911081.911968 mapread 0x025e7e thru 0x035a5b (0xfbde bytes) 000271 1777911081.948896 mapwrite 0x021ee8 thru 0x022752 (0x86b bytes) 000272 1777911082.070129 mapwrite 0x039f31 thru 0x03ffff (0x60cf bytes) 000273 1777911082.149389 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8259a561 (stale) Mirror 3: 0x8259a561 (stale) Mirror 4: 0x8259a561 (stale) Mirror 5: 0x8259a561 (stale) Mirror 6: 0x8259a561 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5f0a092e Mirror 7: 0x5f0a092e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000274 1777911082.895158 write 0x01e4b0 thru 0x025c19 (0x776a bytes) 000275 1777911082.995250 mapread 0x02dcae thru 0x034430 (0x6783 bytes) 000277 1777911083.012107 read 0x02002c thru 0x029e4f (0x9e24 bytes) 000278 1777911083.017599 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8259a561 (stale) Mirror 3: 0x8259a561 (stale) Mirror 4: 0x8259a561 (stale) Mirror 5: 0x8259a561 (stale) Mirror 6: 0x8259a561 (stale) Mirror 7: 0x5f0a092e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb25906cc Mirror 8: 0xb25906cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1777911083.901540 mapread 0x016f50 thru 0x020b91 (0x9c42 bytes) 000281 1777911083.954090 MIRROR_RESYNC 8 mirrors 000282 1777911085.224894 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000288 1777911085.684129 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000290 1777911086.077676 mapwrite 0x029361 thru 0x02bd50 (0x29f0 bytes) 000291 1777911086.275954 write 0x00dc96 thru 0x013ea3 (0x620e bytes) 000292 1777911086.288314 trunc from 0x040000 to 0x00fb28 (0x304d7 bytes) 000295 1777911086.316206 read 0x00e4c2 thru 0x00fb28 (0x1667 bytes) 000296 1777911086.322080 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfb29): Mirror 4: 0x2403d26a (stale) Mirror 5: 0x2403d26a (stale) Mirror 6: 0x2403d26a (stale) Mirror 8: 0x2403d26a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfb29): Mirror 4: 0x2403d26a (stale) Mirror 5: 0x2403d26a (stale) Mirror 6: 0x2403d26a (stale) Mirror 8: 0x2403d26a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfb29: skipped 000297 1777911086.691060 read 0x006ce2 thru 0x00fb28 (0x8e47 bytes) 000300 1777911086.709122 mapwrite 0x023a68 thru 0x0313a4 (0xd93d bytes) 000301 1777911086.841871 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x313a5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x313a5): Mirror 4: 0x356636b9 (stale) Mirror 5: 0x356636b9 (stale) Mirror 6: 0x356636b9 (stale) Mirror 8: 0x356636b9 (stale) CRC-32 checksum value for chunk [0, 0x313a5): Mirror 1: 0x78834cb2 Mirror 9: 0x78834cb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x313a5: skipped 000302 1777911087.855230 write 0x026bbe thru 0x033444 (0xc887 bytes) 000303 1777911087.948738 trunc from 0x033445 to 0x00e45e (0x24fe6 bytes) 000305 1777911087.997657 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe45f): Mirror 4: 0x9628878a (stale) Mirror 5: 0x9628878a (stale) Mirror 6: 0x9628878a (stale) Mirror 9: 0x7aec0757 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe45f): Mirror 4: 0x9628878a (stale) Mirror 5: 0x9628878a (stale) Mirror 6: 0x9628878a (stale) Mirror 9: 0x7aec0757 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe45f: skipped 000306 1777911088.371208 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xe45f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe45f): Mirror 4: 0x9628878a (stale) Mirror 5: 0x9628878a (stale) Mirror 6: 0x9628878a (stale) Mirror 9: 0x7aec0757 (stale) CRC-32 checksum value for chunk [0, 0xe45f): Mirror 1: 0x7aec0757 Mirror 10: 0x7aec0757 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe45f: skipped 000307 1777911089.399179 read 0x00864f thru 0x00987f (0x1231 bytes) 000310 1777911089.447017 mapwrite 0x001b6b thru 0x009793 (0x7c29 bytes) 000312 1777911089.647006 read 0x002410 thru 0x005f1c (0x3b0d bytes) 000313 1777911089.648226 mapwrite 0x01970b thru 0x025f0f (0xc805 bytes) 000315 1777911089.747325 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x25f10): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25f10): Mirror 4: 0xd2fca20c (stale) Mirror 5: 0xd2fca20c (stale) Mirror 6: 0xd2fca20c (stale) Mirror 9: 0xbfe288dc (stale) Mirror 10: 0xa47cebfc (stale) CRC-32 checksum value for chunk [0, 0x25f10): Mirror 1: 0x44e01066 Mirror 11: 0x44e01066 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25f10: skipped 000317 1777911090.785194 write 0x01feec thru 0x020e69 (0xf7e bytes) 000318 1777911090.885757 write 0x039071 thru 0x03ffff (0x6f8f bytes) 000319 1777911090.887300 trunc from 0x040000 to 0x0096b0 (0x3694f bytes) 000322 1777911090.921159 trunc from 0x0096b1 to 0x03401c (0x2a96c bytes) 000323 1777911090.938708 write 0x028a2d thru 0x036ec0 (0xe494 bytes) 000330 1777911090.959893 mapread 0x027df0 thru 0x03080b (0x8a1c bytes) 000331 1777911090.961042 write 0x032eaa thru 0x03af95 (0x80ec bytes) 000333 1777911090.971232 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af96): Mirror 4: 0x7e982872 (stale) Mirror 5: 0x7e982872 (stale) Mirror 9: 0x47254175 (stale) Mirror 10: 0xc15c526f (stale) Mirror 11: 0x14728650 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af96): Mirror 4: 0x7e982872 (stale) Mirror 5: 0x7e982872 (stale) Mirror 9: 0x47254175 (stale) Mirror 10: 0xc15c526f (stale) Mirror 11: 0x14728650 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af96: skipped 000335 1777911091.507655 mapread 0x0059f4 thru 0x00dbfb (0x8208 bytes) 000336 1777911091.533124 trunc from 0x03af96 to 0x0049ed (0x365a8 bytes) 000337 1777911091.567082 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x49ee): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x49ee): Mirror 4: 0xaec458d3 (stale) Mirror 5: 0xaec458d3 (stale) Mirror 9: 0xaec458d3 (stale) Mirror 10: 0xaec458d3 (stale) Mirror 11: 0x1bc88cad (stale) CRC-32 checksum value for chunk [0, 0x49ee): Mirror 1: 0x1bc88cad Mirror 12: 0x1bc88cad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x49ee: skipped 000340 1777911092.574327 mapwrite 0x00ac99 thru 0x01970f (0xea77 bytes) 000341 1777911092.841477 mapwrite 0x009f3d thru 0x01680d (0xc8d1 bytes) 000344 1777911092.923026 read 0x00b57e thru 0x00dbd1 (0x2654 bytes) 000345 1777911092.934652 MIRROR_RESYNC 7 mirrors 000347 1777911094.173599 mapwrite 0x00a61b thru 0x011fb0 (0x7996 bytes) 000348 1777911094.365939 write 0x0257a5 thru 0x02f1df (0x9a3b bytes) 000350 1777911094.379582 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f1e0): Mirror 4: 0x477b5e05 (stale) Mirror 5: 0x477b5e05 (stale) Mirror 9: 0x477b5e05 (stale) Mirror 10: 0x477b5e05 (stale) Mirror 12: 0x477b5e05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f1e0): Mirror 4: 0x477b5e05 (stale) Mirror 5: 0x477b5e05 (stale) Mirror 9: 0x477b5e05 (stale) Mirror 10: 0x477b5e05 (stale) Mirror 12: 0x477b5e05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f1e0: skipped 000353 1777911094.864107 trunc from 0x02f1e0 to 0x02cd79 (0x2466 bytes) 000354 1777911094.883914 read 0x016b01 thru 0x01eaa3 (0x7fa3 bytes) 000355 1777911094.898944 MIRROR_RESYNC 6 mirrors 000358 1777911095.979348 mapwrite 0x03a761 thru 0x03ffff (0x589f bytes) 000360 1777911096.180272 write 0x00c850 thru 0x00d3e7 (0xb98 bytes) 000362 1777911096.196724 write 0x029940 thru 0x030bda (0x729b bytes) 000363 1777911096.207263 mapwrite 0x0073da thru 0x01035f (0x8f86 bytes) 000365 1777911096.296548 read 0x026032 thru 0x02773d (0x170c bytes) 000366 1777911096.299189 mapread 0x02ef8d thru 0x03b831 (0xc8a5 bytes) 000368 1777911096.307676 mapwrite 0x01e578 thru 0x02a2b4 (0xbd3d bytes) 000369 1777911096.379444 mapread 0x00dfc5 thru 0x018f90 (0xafcc bytes) 000370 1777911096.390881 mapread 0x035cf5 thru 0x03ffff (0xa30b bytes) 000371 1777911096.392151 mapread 0x02d145 thru 0x0335d6 (0x6492 bytes) 000372 1777911096.393976 mapread 0x01cf11 thru 0x02237e (0x546e bytes) 000374 1777911096.395840 trunc from 0x040000 to 0x0233ca (0x1cc35 bytes) 000375 1777911096.434997 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x233cb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233cb): Mirror 4: 0x16e35e8e (stale) Mirror 5: 0x16e35e8e (stale) Mirror 9: 0x16e35e8e (stale) Mirror 10: 0x16e35e8e (stale) Mirror 12: 0x16e35e8e (stale) CRC-32 checksum value for chunk [0, 0x233cb): Mirror 1: 0xd44da418 Mirror 13: 0xd44da418 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233cb: skipped 000376 1777911097.373115 MIRROR_RESYNC 7 mirrors 000377 1777911098.434765 write 0x016ac0 thru 0x01d67c (0x6bbd bytes) 000379 1777911098.561280 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x233cb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x233cb): Mirror 4: 0xd44da418 (stale) Mirror 5: 0xd44da418 (stale) Mirror 9: 0xd44da418 (stale) Mirror 10: 0xd44da418 (stale) Mirror 12: 0xd44da418 (stale) Mirror 13: 0xd44da418 (stale) CRC-32 checksum value for chunk [0, 0x233cb): Mirror 1: 0x62833a32 Mirror 14: 0x62833a32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x233cb: skipped 000380 1777911099.515876 mapwrite 0x02af7c thru 0x0350e2 (0xa167 bytes) 000381 1777911099.721465 mapwrite 0x0084b5 thru 0x016e11 (0xe95d bytes) 000382 1777911099.783911 trunc from 0x0350e3 to 0x008ee1 (0x2c201 bytes) 000383 1777911099.811884 write 0x01f8e6 thru 0x029466 (0x9b81 bytes) 000384 1777911099.814540 write 0x0281ff thru 0x0358d0 (0xd6d2 bytes) 000386 1777911099.817257 write 0x00ca97 thru 0x01983e (0xcda8 bytes) 000388 1777911099.836143 mapread 0x01b23f thru 0x028f31 (0xdcf3 bytes) 000389 1777911099.844503 trunc from 0x0358d1 to 0x01c0c6 (0x1980a bytes) 000390 1777911099.870303 write 0x009a25 thru 0x018d6f (0xf34b bytes) 000392 1777911099.874382 write 0x023adc thru 0x026dee (0x3313 bytes) 000397 1777911099.879014 mapread 0x006ba3 thru 0x0149f0 (0xde4e bytes) 000398 1777911099.880540 mapwrite 0x028112 thru 0x02d363 (0x5252 bytes) 000399 1777911100.010367 mapread 0x01114e thru 0x017767 (0x661a bytes) 000402 1777911100.020149 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d364): Mirror 4: 0x812ddbe0 (stale) Mirror 5: 0x812ddbe0 (stale) Mirror 10: 0x812ddbe0 (stale) Mirror 12: 0x812ddbe0 (stale) Mirror 13: 0x812ddbe0 (stale) Mirror 14: 0x64543458 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d364): Mirror 4: 0x812ddbe0 (stale) Mirror 5: 0x812ddbe0 (stale) Mirror 10: 0x812ddbe0 (stale) Mirror 12: 0x812ddbe0 (stale) Mirror 13: 0x812ddbe0 (stale) Mirror 14: 0x64543458 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d364: skipped 000403 1777911100.567139 mapread 0x026bd1 thru 0x02aee6 (0x4316 bytes) 000404 1777911100.624642 read 0x00f4b5 thru 0x01c0b4 (0xcc00 bytes) 000405 1777911100.635189 write 0x02c418 thru 0x037375 (0xaf5e bytes) 000406 1777911100.641244 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37376): Mirror 4: 0xe8ecb8ac (stale) Mirror 5: 0xe8ecb8ac (stale) Mirror 10: 0xe8ecb8ac (stale) Mirror 13: 0xe8ecb8ac (stale) Mirror 14: 0x9f3bffd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37376): Mirror 4: 0xe8ecb8ac (stale) Mirror 5: 0xe8ecb8ac (stale) Mirror 10: 0xe8ecb8ac (stale) Mirror 13: 0xe8ecb8ac (stale) Mirror 14: 0x9f3bffd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37376: skipped 000407 1777911101.077211 MIRROR_RESYNC 6 mirrors 000408 1777911102.200732 trunc from 0x037376 to 0x0032d3 (0x340a2 bytes) 000409 1777911102.299487 mapwrite 0x0295d8 thru 0x03234f (0x8d78 bytes) 000411 1777911102.443886 write 0x038f49 thru 0x03fc51 (0x6d09 bytes) 000413 1777911102.452099 MIRROR_RESYNC 6 mirrors 000416 1777911103.592739 mapwrite 0x008785 thru 0x0103c7 (0x7c43 bytes) 000418 1777911103.807864 mapwrite 0x007857 thru 0x00ed1d (0x74c7 bytes) 000419 1777911103.858980 mapread 0x016a5e thru 0x0171da (0x77d bytes) 000420 1777911103.871729 mapread 0x0278f7 thru 0x02d22e (0x5938 bytes) truncating to largest ever: 0x3f14f 000421 1777911103.889779 trunc from 0x03fc52 to 0x03f14e (0xb03 bytes) 000422 1777911103.914202 trunc from 0x03f14f to 0x000630 (0x3eb1e bytes) 000423 1777911103.934633 mapwrite 0x03a8ce thru 0x03ffff (0x5732 bytes) 000424 1777911104.011754 mapread 0x02e4a4 thru 0x03d97e (0xf4db bytes) 000425 1777911104.022550 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x35c09162 (stale) Mirror 5: 0x35c09162 (stale) Mirror 10: 0x35c09162 (stale) Mirror 13: 0x35c09162 (stale) Mirror 14: 0x35c09162 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5de3f74 Mirror 15: 0x5de3f74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000428 1777911104.956252 read 0x028888 thru 0x02b9bb (0x3134 bytes) 000430 1777911104.977271 write 0x0085e6 thru 0x011903 (0x931e bytes) 000433 1777911105.105974 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x35c09162 (stale) Mirror 5: 0x35c09162 (stale) Mirror 10: 0x35c09162 (stale) Mirror 13: 0x35c09162 (stale) Mirror 14: 0x35c09162 (stale) Mirror 15: 0x5de3f74 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb02eb3f4 Mirror 16: 0xb02eb3f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000434 1777911106.148662 mapread 0x03f96d thru 0x03ffff (0x693 bytes) 000435 1777911106.187019 read 0x03a0d4 thru 0x03ffff (0x5f2c bytes) 000437 1777911106.217265 mapread 0x01ba41 thru 0x01c815 (0xdd5 bytes) 000438 1777911106.279213 MIRROR_RESYNC 8 mirrors 000439 1777911107.826627 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000440 1777911108.232772 write 0x007d51 thru 0x00e504 (0x67b4 bytes) 000444 1777911108.353183 mapwrite 0x023bc4 thru 0x0269a9 (0x2de6 bytes) 000446 1777911108.437610 read 0x0164b0 thru 0x02292d (0xc47e bytes) 000449 1777911108.457600 read 0x0130a1 thru 0x01eff6 (0xbf56 bytes) 000450 1777911108.475152 write 0x008894 thru 0x01234f (0x9abc bytes) 000451 1777911108.478753 mapwrite 0x03955e thru 0x03b495 (0x1f38 bytes) 000452 1777911108.525359 mapwrite 0x02d767 thru 0x03cbf7 (0xf491 bytes) 000453 1777911108.573706 mapread 0x037a3e thru 0x039ed6 (0x2499 bytes) 000454 1777911108.576685 mapread 0x0176ab thru 0x023547 (0xbe9d bytes) 000455 1777911108.584180 MIRROR_RESYNC 7 mirrors 000456 1777911109.800922 read 0x025ad1 thru 0x027b4a (0x207a bytes) 000457 1777911109.805720 mapread 0x0090ba thru 0x00c0ca (0x3011 bytes) 000459 1777911109.824045 write 0x016396 thru 0x0179a0 (0x160b bytes) 000460 1777911109.894607 mapread 0x02b216 thru 0x02bbf7 (0x9e2 bytes) 000461 1777911109.908157 mapread 0x0218cd thru 0x02ca2c (0xb160 bytes) 000462 1777911109.923312 mapwrite 0x001f5a thru 0x009aa8 (0x7b4f bytes) 000463 1777911109.997304 trunc from 0x040000 to 0x00f783 (0x3087c bytes) 000464 1777911110.021196 read 0x0081c7 thru 0x00bfd8 (0x3e12 bytes) 000467 1777911110.024994 write 0x02a6ef thru 0x035380 (0xac92 bytes) 000469 1777911110.027827 read 0x021ce3 thru 0x025289 (0x35a7 bytes) 000471 1777911110.045995 trunc from 0x035381 to 0x034175 (0x120b bytes) 000473 1777911110.066899 mapread 0x004de0 thru 0x01184e (0xca6f bytes) 000475 1777911110.070152 write 0x0149cb thru 0x0152d4 (0x90a bytes) 000476 1777911110.071695 trunc from 0x034176 to 0x02901b (0xb15a bytes) 000477 1777911110.097636 mapread 0x011df4 thru 0x019a84 (0x7c91 bytes) 000478 1777911110.099052 mapwrite 0x00da26 thru 0x016937 (0x8f12 bytes) 000479 1777911110.157670 trunc from 0x02901c to 0x033a93 (0xaa78 bytes) 000483 1777911110.198485 MIRROR_RESYNC 7 mirrors 000488 1777911111.200069 mapread 0x003a8d thru 0x005f64 (0x24d8 bytes) 000490 1777911111.223201 write 0x01d7dc thru 0x025e0f (0x8634 bytes) 000494 1777911111.306656 read 0x030f99 thru 0x033a93 (0x2afb bytes) 000496 1777911111.322743 read 0x00c9ce thru 0x00f8f1 (0x2f24 bytes) 000497 1777911111.334067 mapread 0x01ca19 thru 0x02510d (0x86f5 bytes) 000498 1777911111.335246 write 0x0245ba thru 0x02531f (0xd66 bytes) 000500 1777911111.343169 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33a94): Mirror 4: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a94): Mirror 4: 0x673e4d42 (stale) Mirror 10: 0x673e4d42 (stale) Mirror 13: 0x673e4d42 (stale) Mirror 14: 0x673e4d42 (stale) Mirror 15: 0x673e4d42 (stale) Mirror 16: 0x673e4d42 (stale) CRC-32 checksum value for chunk [0, 0x33a94): Mirror 1: 0x588ea766 Mirror 17: 0x588ea766 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a94: skipped 000501 1777911112.061203 read 0x00dfb6 thru 0x01764d (0x9698 bytes) 000503 1777911112.086586 mapwrite 0x003ecc thru 0x00ab95 (0x6cca bytes) 000504 1777911112.280561 read 0x025af9 thru 0x033a93 (0xdf9b bytes) 000505 1777911112.291929 mapread 0x02b166 thru 0x032282 (0x711d bytes) 000506 1777911112.293605 mapwrite 0x0275ca thru 0x03343c (0xbe73 bytes) 000508 1777911112.347156 read 0x006b69 thru 0x00a9c2 (0x3e5a bytes) truncating to largest ever: 0x3f16b 000509 1777911112.348236 trunc from 0x033a94 to 0x03f16a (0xb6d7 bytes) 000510 1777911112.406567 mapwrite 0x031e76 thru 0x03effb (0xd186 bytes) 000511 1777911112.467890 trunc from 0x03f16b to 0x03c87d (0x28ed bytes) 000512 1777911112.484710 trunc from 0x03c87e to 0x01545e (0x2741f bytes) 000513 1777911112.509284 mapread 0x004dab thru 0x0092b7 (0x450d bytes) 000515 1777911112.513389 read 0x0124e7 thru 0x01545e (0x2f78 bytes) 000516 1777911112.514510 mapread 0x0039c7 thru 0x005e1e (0x2458 bytes) 000517 1777911112.515660 trunc from 0x01545f to 0x01f144 (0x9ce6 bytes) 000519 1777911112.537724 read 0x01393c thru 0x01ecbc (0xb381 bytes) 000522 1777911112.557985 read 0x007f29 thru 0x012864 (0xa93c bytes) 000524 1777911112.567489 MIRROR_RESYNC 8 mirrors 000525 1777911113.624570 trunc from 0x01f145 to 0x02f6c5 (0x10581 bytes) 000526 1777911113.707985 write 0x020276 thru 0x02df65 (0xdcf0 bytes) 000527 1777911113.718969 read 0x005d7b thru 0x0098d6 (0x3b5c bytes) 000528 1777911113.727641 trunc from 0x02f6c6 to 0x005b6a (0x29b5b bytes) 000529 1777911113.756482 trunc from 0x005b6b to 0x007eaf (0x2345 bytes) 000530 1777911113.774764 write 0x00f7ed thru 0x015ecf (0x66e3 bytes) 000533 1777911113.777148 read 0x010423 thru 0x015ecf (0x5aad bytes) 000537 1777911113.779667 trunc from 0x015ed0 to 0x0205d6 (0xa707 bytes) 000542 1777911113.806107 mapread 0x0179b0 thru 0x0199ad (0x1ffe bytes) 000544 1777911113.824353 mapwrite 0x02ed31 thru 0x03d618 (0xe8e8 bytes) 000547 1777911113.925940 mapwrite 0x0398a4 thru 0x03ffff (0x675c bytes) 000550 1777911114.039043 read 0x0208cc thru 0x02461e (0x3d53 bytes) 000551 1777911114.040025 mapwrite 0x030e50 thru 0x0316e2 (0x893 bytes) 000552 1777911114.108390 mapwrite 0x02a13f thru 0x02a435 (0x2f7 bytes) 000554 1777911114.152081 mapwrite 0x02399d thru 0x024ca2 (0x1306 bytes) 000556 1777911114.201761 write 0x00e0e3 thru 0x00ead9 (0x9f7 bytes) 000558 1777911114.203900 trunc from 0x040000 to 0x03c38e (0x3c71 bytes) 000559 1777911114.234229 mapwrite 0x01e784 thru 0x021c03 (0x3480 bytes) 000560 1777911114.306256 write 0x034246 thru 0x0373db (0x3196 bytes) 000562 1777911114.308051 mapwrite 0x01b22b thru 0x01f589 (0x435f bytes) 000564 1777911114.368918 MIRROR_RESYNC 8 mirrors 000567 1777911115.524133 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000568 1777911115.872999 read 0x023a74 thru 0x03299f (0xef2c bytes) 000569 1777911115.881120 read 0x010384 thru 0x0143f8 (0x4075 bytes) 000571 1777911115.900335 MIRROR_RESYNC 7 mirrors 000575 1777911116.311058 read 0x00f6c2 thru 0x0104cc (0xe0b bytes) 000577 1777911116.312227 mapread 0x035ecb thru 0x03c38e (0x64c4 bytes) 000583 1777911116.317115 write 0x037efe thru 0x03ffff (0x8102 bytes) 000584 1777911116.407144 trunc from 0x040000 to 0x038a1e (0x75e1 bytes) 000585 1777911116.439267 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a1f): Mirror 10: 0x1d5bb081 (stale) Mirror 13: 0x1d5bb081 (stale) Mirror 14: 0x1d5bb081 (stale) Mirror 15: 0x1d5bb081 (stale) Mirror 17: 0x1d5bb081 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a1f): Mirror 10: 0x1d5bb081 (stale) Mirror 13: 0x1d5bb081 (stale) Mirror 14: 0x1d5bb081 (stale) Mirror 15: 0x1d5bb081 (stale) Mirror 17: 0x1d5bb081 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a1f: skipped 000586 1777911116.808742 write 0x00cc46 thru 0x00d105 (0x4c0 bytes) 000587 1777911116.826409 read 0x00a41c thru 0x015d1b (0xb900 bytes) 000588 1777911116.836335 mapread 0x0098bc thru 0x00d63d (0x3d82 bytes) 000591 1777911116.853797 trunc from 0x038a1f to 0x030790 (0x828e bytes) 000593 1777911116.881892 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30791): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30791): Mirror 10: 0xe043954 (stale) Mirror 13: 0xe043954 (stale) Mirror 14: 0xe043954 (stale) Mirror 15: 0xe043954 (stale) Mirror 17: 0xe043954 (stale) CRC-32 checksum value for chunk [0, 0x30791): Mirror 1: 0xfa3a7568 Mirror 18: 0xfa3a7568 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30791: skipped 000594 1777911117.701087 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30791): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30791): Mirror 10: 0xe043954 (stale) Mirror 13: 0xe043954 (stale) Mirror 14: 0xe043954 (stale) Mirror 15: 0xe043954 (stale) Mirror 17: 0xe043954 (stale) CRC-32 checksum value for chunk [0, 0x30791): Mirror 1: 0xfa3a7568 Mirror 18: 0xfa3a7568 Mirror 19: 0xfa3a7568 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30791: skipped 000595 1777911118.665929 mapwrite 0x007343 thru 0x00cce0 (0x599e bytes) 000596 1777911118.888301 mapread 0x029662 thru 0x02b563 (0x1f02 bytes) 000598 1777911118.912344 trunc from 0x030791 to 0x00f2d4 (0x214bc bytes) 000600 1777911118.938109 trunc from 0x00f2d5 to 0x01763d (0x8369 bytes) 000602 1777911118.960895 trunc from 0x01763e to 0x012182 (0x54bb bytes) 000603 1777911118.976258 write 0x02496a thru 0x02a5d2 (0x5c69 bytes) 000604 1777911118.981628 mapwrite 0x0302e8 thru 0x033441 (0x315a bytes) 000605 1777911119.058079 read 0x00c4d0 thru 0x013a4a (0x757b bytes) 000606 1777911119.074472 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33442): Mirror 10: 0x1781659a (stale) Mirror 13: 0x1781659a (stale) Mirror 14: 0x1781659a (stale) Mirror 15: 0x1781659a (stale) Mirror 17: 0x1781659a (stale) Mirror 19: 0x4d6c97a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33442): Mirror 10: 0x1781659a (stale) Mirror 13: 0x1781659a (stale) Mirror 14: 0x1781659a (stale) Mirror 15: 0x1781659a (stale) Mirror 17: 0x1781659a (stale) Mirror 19: 0x4d6c97a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33442: skipped 000607 1777911119.502939 trunc from 0x033442 to 0x00d975 (0x25acc bytes) 000608 1777911119.524305 mapread 0x003274 thru 0x00969f (0x642c bytes) 000614 1777911119.550262 trunc from 0x00d976 to 0x01d3e2 (0xfa6d bytes) 000616 1777911119.579235 MIRROR_RESYNC 7 mirrors 000618 1777911120.613343 mapread 0x00d266 thru 0x017822 (0xa5bd bytes) 000619 1777911120.635863 mapwrite 0x0151bf thru 0x020b01 (0xb943 bytes) 000622 1777911120.833340 read 0x01246f thru 0x01964e (0x71e0 bytes) 000624 1777911120.838649 MIRROR_RESYNC 7 mirrors 000626 1777911121.876900 trunc from 0x020b02 to 0x033fea (0x134e9 bytes) 000627 1777911121.950545 read 0x027836 thru 0x03046b (0x8c36 bytes) 000628 1777911121.963245 MIRROR_RESYNC 7 mirrors 000630 1777911122.991285 MIRROR_EXTEND to 8 mirrors 000631 1777911123.733730 MIRROR_EXTEND to 9 mirrors 000632 1777911124.564079 mapread 0x01205f thru 0x0126a4 (0x646 bytes) 000633 1777911124.579640 mapread 0x031730 thru 0x033fea (0x28bb bytes) 000634 1777911124.618578 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) 000635 1777911125.099575 mapwrite 0x02a201 thru 0x031f1b (0x7d1b bytes) 000637 1777911125.330903 write 0x00e78b thru 0x01a131 (0xb9a7 bytes) 000638 1777911125.368383 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33feb): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33feb): Mirror 10: 0x27c8291d (stale) Mirror 13: 0x27c8291d (stale) Mirror 14: 0x27c8291d (stale) Mirror 17: 0x27c8291d (stale) Mirror 19: 0x27c8291d (stale) Mirror 20: 0x27c8291d (stale) Mirror 21: 0x27c8291d (stale) CRC-32 checksum value for chunk [0, 0x33feb): Mirror 1: 0x83d41e97 Mirror 22: 0x83d41e97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33feb: skipped 000645 1777911126.547223 mapread 0x0307b4 thru 0x033fea (0x3837 bytes) 000646 1777911126.588140 mapread 0x01186c thru 0x01f2a1 (0xda36 bytes) 000647 1777911126.629902 mapread 0x01b4a0 thru 0x01d900 (0x2461 bytes) 000649 1777911126.664480 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33feb): Mirror 10: 0x27c8291d (stale) Mirror 13: 0x27c8291d (stale) Mirror 14: 0x27c8291d (stale) Mirror 17: 0x27c8291d (stale) Mirror 20: 0x27c8291d (stale) Mirror 21: 0x27c8291d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33feb): Mirror 10: 0x27c8291d (stale) Mirror 13: 0x27c8291d (stale) Mirror 14: 0x27c8291d (stale) Mirror 17: 0x27c8291d (stale) Mirror 20: 0x27c8291d (stale) Mirror 21: 0x27c8291d (stale) CRC-32 checksum value for chunk [0, 0x33feb): Mirror 1: 0x83d41e97 Mirror 22: 0x83d41e97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33feb: skipped 000650 1777911127.626006 MIRROR_RESYNC 8 mirrors 000651 1777911128.849592 MIRROR_EXTEND to 9 mirrors 000654 1777911129.496695 write 0x0351fe thru 0x03ffff (0xae02 bytes) 000655 1777911129.565392 mapread 0x000cbe thru 0x00ddc9 (0xd10c bytes) 000656 1777911129.588264 mapread 0x02ec79 thru 0x03bc9d (0xd025 bytes) 000657 1777911129.599810 mapwrite 0x00bc88 thru 0x00e4de (0x2857 bytes) 000659 1777911129.664591 mapread 0x02e285 thru 0x032014 (0x3d90 bytes) 000660 1777911129.666106 read 0x03356a thru 0x03a14b (0x6be2 bytes) 000663 1777911129.676571 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xecaf3ecf (stale) Mirror 13: 0xecaf3ecf (stale) Mirror 14: 0xecaf3ecf (stale) Mirror 17: 0xecaf3ecf (stale) Mirror 20: 0xecaf3ecf (stale) Mirror 21: 0xecaf3ecf (stale) Mirror 22: 0xecaf3ecf (stale) Mirror 23: 0xecaf3ecf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb606892 Mirror 24: 0xb606892 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000664 1777911130.449134 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xecaf3ecf (stale) Mirror 13: 0xecaf3ecf (stale) Mirror 14: 0xecaf3ecf (stale) Mirror 17: 0xecaf3ecf (stale) Mirror 20: 0xecaf3ecf (stale) Mirror 21: 0xecaf3ecf (stale) Mirror 22: 0xecaf3ecf (stale) Mirror 23: 0xecaf3ecf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xecaf3ecf (stale) Mirror 13: 0xecaf3ecf (stale) Mirror 14: 0xecaf3ecf (stale) Mirror 17: 0xecaf3ecf (stale) Mirror 20: 0xecaf3ecf (stale) Mirror 21: 0xecaf3ecf (stale) Mirror 22: 0xecaf3ecf (stale) Mirror 23: 0xecaf3ecf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000665 1777911130.880019 mapread 0x02abbf thru 0x037b6f (0xcfb1 bytes) 000667 1777911130.912061 mapread 0x00b499 thru 0x00f05c (0x3bc4 bytes) 000669 1777911130.926770 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xecaf3ecf (stale) Mirror 13: 0xecaf3ecf (stale) Mirror 14: 0xecaf3ecf (stale) Mirror 17: 0xecaf3ecf (stale) Mirror 20: 0xecaf3ecf (stale) Mirror 21: 0xecaf3ecf (stale) Mirror 22: 0xecaf3ecf (stale) Mirror 23: 0xecaf3ecf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb606892 Mirror 24: 0xb606892 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000671 1777911131.890880 read 0x0330da thru 0x033e64 (0xd8b bytes) 000672 1777911131.907712 mapwrite 0x00108c thru 0x00a470 (0x93e5 bytes) 000673 1777911132.131501 mapread 0x02172c thru 0x02753c (0x5e11 bytes) 000674 1777911132.154357 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xecaf3ecf (stale) Mirror 13: 0xecaf3ecf (stale) Mirror 14: 0xecaf3ecf (stale) Mirror 17: 0xecaf3ecf (stale) Mirror 20: 0xecaf3ecf (stale) Mirror 21: 0xecaf3ecf (stale) Mirror 22: 0xecaf3ecf (stale) Mirror 23: 0xecaf3ecf (stale) Mirror 24: 0xb606892 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa08bbe64 Mirror 25: 0xa08bbe64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000675 1777911133.082610 mapwrite 0x029631 thru 0x02d59a (0x3f6a bytes) 000677 1777911133.292451 trunc from 0x040000 to 0x01ffbe (0x20041 bytes) 000679 1777911133.335389 trunc from 0x01ffbf to 0x02834b (0x838d bytes) 000680 1777911133.355349 read 0x010956 thru 0x01fb15 (0xf1c0 bytes) 000681 1777911133.367116 trunc from 0x02834c to 0x0193cc (0xef7f bytes) 000684 1777911133.390703 mapwrite 0x02dd07 thru 0x0358f7 (0x7bf1 bytes) 000685 1777911133.477725 trunc from 0x0358f8 to 0x01cf4a (0x189ad bytes) 000686 1777911133.492683 write 0x00f2b5 thru 0x0123d8 (0x3124 bytes) 000687 1777911133.502026 trunc from 0x01cf4b to 0x00fa91 (0xd4b9 bytes) 000689 1777911133.519543 write 0x011465 thru 0x01df87 (0xcb23 bytes) 000690 1777911133.522779 mapread 0x000651 thru 0x004013 (0x39c3 bytes) 000691 1777911133.536635 mapread 0x01a6bb thru 0x01df87 (0x38cd bytes) 000695 1777911133.543177 mapwrite 0x012ecd thru 0x021566 (0xe69a bytes) 000696 1777911133.639988 mapread 0x00ab5e thru 0x011cbe (0x7161 bytes) 000697 1777911133.646662 trunc from 0x021567 to 0x03bf2c (0x1a9c6 bytes) 000700 1777911133.665963 trunc from 0x03bf2d to 0x014b46 (0x273e6 bytes) 000702 1777911133.683261 mapread 0x009ceb thru 0x00c921 (0x2c37 bytes) 000703 1777911133.684304 mapread 0x011ee0 thru 0x014b46 (0x2c67 bytes) 000705 1777911133.690849 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x14b47): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14b47): Mirror 10: 0x51dc8eea (stale) Mirror 13: 0x51dc8eea (stale) Mirror 14: 0x51dc8eea (stale) Mirror 17: 0x51dc8eea (stale) Mirror 20: 0x51dc8eea (stale) Mirror 21: 0x51dc8eea (stale) Mirror 22: 0x51dc8eea (stale) Mirror 23: 0x51dc8eea (stale) Mirror 24: 0x948e015a (stale) Mirror 25: 0x22c3a6cc (stale) CRC-32 checksum value for chunk [0, 0x14b47): Mirror 1: 0x9240003c Mirror 26: 0x9240003c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14b47: skipped 000706 1777911134.625784 mapread 0x008da0 thru 0x01169e (0x88ff bytes) 000708 1777911134.679238 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14b47): Mirror 10: 0x51dc8eea (stale) Mirror 13: 0x51dc8eea (stale) Mirror 14: 0x51dc8eea (stale) Mirror 20: 0x51dc8eea (stale) Mirror 21: 0x51dc8eea (stale) Mirror 22: 0x51dc8eea (stale) Mirror 23: 0x51dc8eea (stale) Mirror 24: 0x948e015a (stale) Mirror 25: 0x22c3a6cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14b47): Mirror 10: 0x51dc8eea (stale) Mirror 13: 0x51dc8eea (stale) Mirror 14: 0x51dc8eea (stale) Mirror 20: 0x51dc8eea (stale) Mirror 21: 0x51dc8eea (stale) Mirror 22: 0x51dc8eea (stale) Mirror 23: 0x51dc8eea (stale) Mirror 24: 0x948e015a (stale) Mirror 25: 0x22c3a6cc (stale) CRC-32 checksum value for chunk [0, 0x14b47): Mirror 1: 0x9240003c Mirror 26: 0x9240003c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14b47: skipped 000712 1777911135.551504 mapwrite 0x0215ce thru 0x030e2a (0xf85d bytes) 000714 1777911135.701328 trunc from 0x030e2b to 0x022b91 (0xe299 bytes) 000715 1777911135.720443 mapread 0x00b36b thru 0x010d40 (0x59d6 bytes) 000717 1777911135.731801 trunc from 0x022b92 to 0x022b2d (0x64 bytes) 000719 1777911135.755482 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x22b2e): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b2e): Mirror 10: 0xb3f634a7 (stale) Mirror 13: 0xb3f634a7 (stale) Mirror 14: 0xb3f634a7 (stale) Mirror 20: 0xb3f634a7 (stale) Mirror 21: 0xb3f634a7 (stale) Mirror 22: 0xb3f634a7 (stale) Mirror 23: 0xb3f634a7 (stale) Mirror 24: 0x2bdb6a66 (stale) Mirror 25: 0xdf902804 (stale) Mirror 26: 0x1d8a6b37 (stale) CRC-32 checksum value for chunk [0, 0x22b2e): Mirror 1: 0x2b32e254 Mirror 27: 0x2b32e254 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b2e: skipped 000720 1777911136.569093 write 0x03a81e thru 0x03ffff (0x57e2 bytes) 000721 1777911136.633359 write 0x038a07 thru 0x03ffff (0x75f9 bytes) 000722 1777911136.638167 read 0x00fc14 thru 0x01cbb1 (0xcf9e bytes) 000723 1777911136.650000 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xecaf3ecf (stale) Mirror 13: 0xecaf3ecf (stale) Mirror 14: 0xecaf3ecf (stale) Mirror 20: 0xecaf3ecf (stale) Mirror 21: 0xecaf3ecf (stale) Mirror 22: 0xecaf3ecf (stale) Mirror 23: 0xecaf3ecf (stale) Mirror 24: 0xb606892 (stale) Mirror 25: 0xa08bbe64 (stale) Mirror 26: 0x21590d7b (stale) Mirror 27: 0x4cbf7aee (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8700e289 Mirror 28: 0x8700e289 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000724 1777911137.362287 MIRROR_RESYNC 13 mirrors 000726 1777911138.812476 trunc from 0x040000 to 0x006854 (0x397ab bytes) 000729 1777911138.916183 mapwrite 0x009709 thru 0x01746b (0xdd63 bytes) 000732 1777911139.024201 mapread 0x00aca5 thru 0x010879 (0x5bd5 bytes) 000734 1777911139.026760 trunc from 0x01746c to 0x038f97 (0x21b2c bytes) 000735 1777911139.047633 mapwrite 0x023f24 thru 0x02f9c0 (0xba9d bytes) 000736 1777911139.114486 write 0x021e9c thru 0x0258c9 (0x3a2e bytes) 000738 1777911139.116757 write 0x03e39a thru 0x03ffff (0x1c66 bytes) 000739 1777911139.117950 read 0x038d35 thru 0x038e1d (0xe9 bytes) 000741 1777911139.124260 mapread 0x03eb30 thru 0x03ffff (0x14d0 bytes) 000743 1777911139.125948 mapread 0x01a1b1 thru 0x029747 (0xf597 bytes) 000745 1777911139.132037 trunc from 0x040000 to 0x009b68 (0x36497 bytes) 000746 1777911139.153196 mapread 0x00439f thru 0x009b68 (0x57ca bytes) 000749 1777911139.155593 read 0x00960e thru 0x009b68 (0x55b bytes) 000750 1777911139.156563 mapread 0x001d26 thru 0x009b68 (0x7e43 bytes) 000751 1777911139.164816 write 0x024aae thru 0x026b5d (0x20b0 bytes) 000752 1777911139.166033 trunc from 0x026b5e to 0x03e415 (0x178b8 bytes) 000756 1777911139.188004 MIRROR_RESYNC 13 mirrors 000757 1777911140.500793 mapwrite 0x02413b thru 0x028440 (0x4306 bytes) 000758 1777911140.665732 read 0x032a70 thru 0x03e415 (0xb9a6 bytes) 000759 1777911140.676801 trunc from 0x03e416 to 0x014014 (0x2a401 bytes) 000760 1777911140.698085 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x14015): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14015): Mirror 10: 0x395f41f9 (stale) Mirror 13: 0x395f41f9 (stale) Mirror 14: 0x395f41f9 (stale) Mirror 20: 0x395f41f9 (stale) Mirror 21: 0x395f41f9 (stale) Mirror 22: 0x395f41f9 (stale) Mirror 23: 0x395f41f9 (stale) Mirror 24: 0x395f41f9 (stale) Mirror 25: 0x395f41f9 (stale) Mirror 26: 0x395f41f9 (stale) Mirror 27: 0x395f41f9 (stale) Mirror 28: 0x395f41f9 (stale) CRC-32 checksum value for chunk [0, 0x14015): Mirror 1: 0x395f41f9 Mirror 29: 0x395f41f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14015: skipped 000761 1777911141.394704 trunc from 0x014015 to 0x0317e8 (0x1d7d4 bytes) 000763 1777911141.486006 read 0x02ec3c thru 0x0317e8 (0x2bad bytes) 000764 1777911141.494292 read 0x024698 thru 0x02e48e (0x9df7 bytes) 000765 1777911141.503871 mapread 0x01aaf9 thru 0x01ea62 (0x3f6a bytes) 000768 1777911141.506628 mapwrite 0x019a6d thru 0x020de0 (0x7374 bytes) 000769 1777911141.556337 write 0x0301e7 thru 0x03ffaa (0xfdc4 bytes) 000770 1777911141.558163 write 0x030534 thru 0x0346f8 (0x41c5 bytes) 000773 1777911141.564082 mapread 0x029368 thru 0x036cf5 (0xd98e bytes) 000774 1777911141.569801 mapwrite 0x039f58 thru 0x03ce3f (0x2ee8 bytes) 000776 1777911141.620561 read 0x021844 thru 0x025dbf (0x457c bytes) 000777 1777911141.621295 read 0x00c918 thru 0x0146b9 (0x7da2 bytes) 000778 1777911141.622126 mapwrite 0x02f110 thru 0x03c7fd (0xd6ee bytes) 000780 1777911141.673931 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ffab): Mirror 10: 0x87c8007f (stale) Mirror 13: 0x87c8007f (stale) Mirror 14: 0x87c8007f (stale) Mirror 20: 0x87c8007f (stale) Mirror 21: 0x87c8007f (stale) Mirror 23: 0x87c8007f (stale) Mirror 24: 0x87c8007f (stale) Mirror 25: 0x87c8007f (stale) Mirror 26: 0x87c8007f (stale) Mirror 27: 0x87c8007f (stale) Mirror 28: 0x87c8007f (stale) Mirror 29: 0x65b2f716 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffab): Mirror 10: 0x87c8007f (stale) Mirror 13: 0x87c8007f (stale) Mirror 14: 0x87c8007f (stale) Mirror 20: 0x87c8007f (stale) Mirror 21: 0x87c8007f (stale) Mirror 23: 0x87c8007f (stale) Mirror 24: 0x87c8007f (stale) Mirror 25: 0x87c8007f (stale) Mirror 26: 0x87c8007f (stale) Mirror 27: 0x87c8007f (stale) Mirror 28: 0x87c8007f (stale) Mirror 29: 0x65b2f716 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffab: skipped 000783 1777911142.216829 trunc from 0x03ffab to 0x03b409 (0x4ba1 bytes) 000786 1777911142.236258 mapread 0x02a135 thru 0x02d43e (0x330a bytes) 000789 1777911142.257628 mapwrite 0x00708e thru 0x010426 (0x9399 bytes) 000790 1777911142.322859 mapwrite 0x03784c thru 0x03ffff (0x87b4 bytes) 000791 1777911142.402856 mapwrite 0x019cf2 thru 0x01a875 (0xb84 bytes) 000794 1777911142.446462 mapread 0x034dbc thru 0x03c618 (0x785d bytes) 000795 1777911142.447449 mapread 0x01bc80 thru 0x0222ab (0x662c bytes) 000796 1777911142.452361 mapread 0x001fb8 thru 0x00b613 (0x965c bytes) 000797 1777911142.455314 trunc from 0x040000 to 0x013215 (0x2cdea bytes) 000798 1777911142.476452 trunc from 0x013216 to 0x027b64 (0x1494f bytes) 000799 1777911142.495941 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x27b65): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b65): Mirror 10: 0xe6009cff (stale) Mirror 13: 0xe6009cff (stale) Mirror 14: 0xe6009cff (stale) Mirror 20: 0xe6009cff (stale) Mirror 21: 0xe6009cff (stale) Mirror 23: 0xe6009cff (stale) Mirror 24: 0xe6009cff (stale) Mirror 25: 0xe6009cff (stale) Mirror 26: 0xe6009cff (stale) Mirror 27: 0xe6009cff (stale) Mirror 28: 0xe6009cff (stale) Mirror 29: 0x1b402a49 (stale) CRC-32 checksum value for chunk [0, 0x27b65): Mirror 1: 0x66ce7424 Mirror 30: 0x66ce7424 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b65: skipped 000800 1777911143.283873 mapread 0x011586 thru 0x012e91 (0x190c bytes) 000802 1777911143.332327 read 0x001aae thru 0x002d5b (0x12ae bytes) 000803 1777911143.350454 read 0x02247a thru 0x024fac (0x2b33 bytes) 000806 1777911143.386442 MIRROR_RESYNC 14 mirrors 000807 1777911144.742016 write 0x015905 thru 0x01aa2d (0x5129 bytes) 000808 1777911144.823609 read 0x004d79 thru 0x008abc (0x3d44 bytes) 000812 1777911144.831868 write 0x000868 thru 0x00b89f (0xb038 bytes) 000813 1777911144.840644 mapread 0x00b1bb thru 0x00e4c2 (0x3308 bytes) 000814 1777911144.841837 mapwrite 0x019afe thru 0x01d843 (0x3d46 bytes) 000815 1777911144.901584 mapwrite 0x0231d1 thru 0x032d9e (0xfbce bytes) 000816 1777911144.977390 trunc from 0x032d9f to 0x00ba20 (0x2737e bytes) 000817 1777911144.997459 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0xba21): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xba21): Mirror 10: 0x6903354b (stale) Mirror 13: 0x6903354b (stale) Mirror 14: 0x6903354b (stale) Mirror 20: 0x6903354b (stale) Mirror 21: 0x6903354b (stale) Mirror 23: 0x6903354b (stale) Mirror 24: 0x6903354b (stale) Mirror 25: 0x6903354b (stale) Mirror 26: 0x6903354b (stale) Mirror 27: 0x6903354b (stale) Mirror 28: 0x6903354b (stale) Mirror 29: 0x6903354b (stale) Mirror 30: 0x6903354b (stale) CRC-32 checksum value for chunk [0, 0xba21): Mirror 1: 0x3ddf4e0b Mirror 31: 0x3ddf4e0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xba21: skipped 000818 1777911145.595898 mapread 0x0031b9 thru 0x00ba20 (0x8868 bytes) 000819 1777911145.618902 mapread 0x009496 thru 0x00ba20 (0x258b bytes) 000820 1777911145.632544 mapread 0x0095f0 thru 0x00ba20 (0x2431 bytes) 000821 1777911145.644914 read 0x00138e thru 0x005174 (0x3de7 bytes) 000823 1777911145.656958 mapread 0x00b529 thru 0x00ba20 (0x4f8 bytes) 000825 1777911145.675823 mapread 0x001892 thru 0x0021c1 (0x930 bytes) 000826 1777911145.687662 write 0x01dd9d thru 0x01ed35 (0xf99 bytes) 000827 1777911145.758348 mapread 0x007e47 thru 0x00db6f (0x5d29 bytes) 000829 1777911145.766741 write 0x020504 thru 0x0280d8 (0x7bd5 bytes) 000830 1777911145.769550 mapread 0x002bdf thru 0x00bf3d (0x935f bytes) 000831 1777911145.771627 mapwrite 0x0257df thru 0x03566b (0xfe8d bytes) 000832 1777911145.862260 mapread 0x034181 thru 0x03566b (0x14eb bytes) 000833 1777911145.870267 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3566c): Mirror 10: 0xfe74079e (stale) Mirror 13: 0xfe74079e (stale) Mirror 20: 0xfe74079e (stale) Mirror 21: 0xfe74079e (stale) Mirror 23: 0xfe74079e (stale) Mirror 24: 0xfe74079e (stale) Mirror 25: 0xfe74079e (stale) Mirror 26: 0xfe74079e (stale) Mirror 27: 0xfe74079e (stale) Mirror 28: 0xfe74079e (stale) Mirror 29: 0xfe74079e (stale) Mirror 30: 0xfe74079e (stale) Mirror 31: 0xbc08daec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3566c): Mirror 10: 0xfe74079e (stale) Mirror 13: 0xfe74079e (stale) Mirror 20: 0xfe74079e (stale) Mirror 21: 0xfe74079e (stale) Mirror 23: 0xfe74079e (stale) Mirror 24: 0xfe74079e (stale) Mirror 25: 0xfe74079e (stale) Mirror 26: 0xfe74079e (stale) Mirror 27: 0xfe74079e (stale) Mirror 28: 0xfe74079e (stale) Mirror 29: 0xfe74079e (stale) Mirror 30: 0xfe74079e (stale) Mirror 31: 0xbc08daec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3566c: skipped 000834 1777911146.402862 MIRROR_RESYNC 14 mirrors 000835 1777911147.802984 mapwrite 0x025f01 thru 0x02d5f7 (0x76f7 bytes) 000836 1777911147.920782 read 0x016845 thru 0x019705 (0x2ec1 bytes) 000839 1777911147.930592 MIRROR_RESYNC 14 mirrors 000841 1777911149.181889 trunc from 0x03566c to 0x00b50b (0x2a160 bytes) 000844 1777911149.262200 MIRROR_RESYNC 14 mirrors 000845 1777911150.637813 mapread 0x000f42 thru 0x004579 (0x3638 bytes) 000846 1777911150.657124 mapwrite 0x032693 thru 0x039ac3 (0x7431 bytes) 000848 1777911150.807307 mapwrite 0x039bba thru 0x03ffff (0x6446 bytes) 000849 1777911150.878549 mapread 0x01dd1e thru 0x01f348 (0x162b bytes) 000850 1777911150.886638 read 0x0095d1 thru 0x010be9 (0x7619 bytes) 000852 1777911150.897349 mapread 0x009d30 thru 0x017c5b (0xdf2c bytes) 000854 1777911150.906666 trunc from 0x040000 to 0x02dae1 (0x1251e bytes) 000855 1777911150.919408 write 0x00b45d thru 0x014047 (0x8beb bytes) 000856 1777911150.920445 mapread 0x00f42f thru 0x018b0d (0x96df bytes) 000860 1777911150.925302 write 0x02f87e thru 0x032dc4 (0x3547 bytes) 000862 1777911150.928322 trunc from 0x032dc5 to 0x00f6f4 (0x236d0 bytes) 000863 1777911150.941203 trunc from 0x00f6f5 to 0x021044 (0x11950 bytes) 000864 1777911150.955858 trunc from 0x021045 to 0x00b493 (0x15bb1 bytes) 000865 1777911150.977100 mapread 0x00a585 thru 0x00b493 (0xf0f bytes) 000867 1777911150.979698 read 0x000104 thru 0x0078cb (0x77c8 bytes) 000868 1777911150.989876 read 0x002774 thru 0x003ccd (0x155a bytes) 000869 1777911150.997831 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb494): Mirror 10: 0x8e5ae601 (stale) Mirror 13: 0x8e5ae601 (stale) Mirror 20: 0x8e5ae601 (stale) Mirror 21: 0x8e5ae601 (stale) Mirror 23: 0x8e5ae601 (stale) Mirror 24: 0x8e5ae601 (stale) Mirror 25: 0x8e5ae601 (stale) Mirror 26: 0x8e5ae601 (stale) Mirror 27: 0x8e5ae601 (stale) Mirror 28: 0x8e5ae601 (stale) Mirror 30: 0x8e5ae601 (stale) Mirror 31: 0x8e5ae601 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb494): Mirror 10: 0x8e5ae601 (stale) Mirror 13: 0x8e5ae601 (stale) Mirror 20: 0x8e5ae601 (stale) Mirror 21: 0x8e5ae601 (stale) Mirror 23: 0x8e5ae601 (stale) Mirror 24: 0x8e5ae601 (stale) Mirror 25: 0x8e5ae601 (stale) Mirror 26: 0x8e5ae601 (stale) Mirror 27: 0x8e5ae601 (stale) Mirror 28: 0x8e5ae601 (stale) Mirror 30: 0x8e5ae601 (stale) Mirror 31: 0x8e5ae601 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb494: skipped 000870 1777911151.514477 mapwrite 0x033df3 thru 0x038078 (0x4286 bytes) 000873 1777911151.624748 trunc from 0x038079 to 0x0226d0 (0x159a8 bytes) 000875 1777911151.642031 mapread 0x018abc thru 0x01e12c (0x5671 bytes) 000877 1777911151.654516 mapread 0x019abb thru 0x0226d0 (0x8c16 bytes) 000878 1777911151.663964 mapread 0x00ddb9 thru 0x0171c1 (0x9409 bytes) 000879 1777911151.674136 write 0x0371e5 thru 0x03ca56 (0x5872 bytes) 000882 1777911151.676929 write 0x003cc5 thru 0x005ab0 (0x1dec bytes) 000883 1777911151.693208 mapwrite 0x016b84 thru 0x01c65e (0x5adb bytes) 000886 1777911151.737877 write 0x03b3c4 thru 0x03ffff (0x4c3c bytes) 000887 1777911151.748087 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x9a6a6d2a (stale) Mirror 13: 0x9a6a6d2a (stale) Mirror 20: 0x9a6a6d2a (stale) Mirror 23: 0x9a6a6d2a (stale) Mirror 24: 0x9a6a6d2a (stale) Mirror 25: 0x9a6a6d2a (stale) Mirror 26: 0x9a6a6d2a (stale) Mirror 27: 0x9a6a6d2a (stale) Mirror 28: 0x9a6a6d2a (stale) Mirror 30: 0x9a6a6d2a (stale) Mirror 31: 0x9a6a6d2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x9a6a6d2a (stale) Mirror 13: 0x9a6a6d2a (stale) Mirror 20: 0x9a6a6d2a (stale) Mirror 23: 0x9a6a6d2a (stale) Mirror 24: 0x9a6a6d2a (stale) Mirror 25: 0x9a6a6d2a (stale) Mirror 26: 0x9a6a6d2a (stale) Mirror 27: 0x9a6a6d2a (stale) Mirror 28: 0x9a6a6d2a (stale) Mirror 30: 0x9a6a6d2a (stale) Mirror 31: 0x9a6a6d2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000888 1777911152.222236 write 0x0302d8 thru 0x03ab2e (0xa857 bytes) 000890 1777911152.232883 write 0x017cee thru 0x027604 (0xf917 bytes) 000891 1777911152.250518 read 0x01e3ab thru 0x02be89 (0xdadf bytes) 000892 1777911152.256001 mapread 0x028c9a thru 0x028e09 (0x170 bytes) 000895 1777911152.258736 trunc from 0x040000 to 0x038102 (0x7efd bytes) 000896 1777911152.281331 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38103): Mirror 10: 0xd8b1b6ff (stale) Mirror 13: 0xd8b1b6ff (stale) Mirror 20: 0xd8b1b6ff (stale) Mirror 23: 0xd8b1b6ff (stale) Mirror 24: 0xd8b1b6ff (stale) Mirror 25: 0xd8b1b6ff (stale) Mirror 26: 0xd8b1b6ff (stale) Mirror 27: 0xd8b1b6ff (stale) Mirror 28: 0xd8b1b6ff (stale) Mirror 30: 0xd8b1b6ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38103): Mirror 10: 0xd8b1b6ff (stale) Mirror 13: 0xd8b1b6ff (stale) Mirror 20: 0xd8b1b6ff (stale) Mirror 23: 0xd8b1b6ff (stale) Mirror 24: 0xd8b1b6ff (stale) Mirror 25: 0xd8b1b6ff (stale) Mirror 26: 0xd8b1b6ff (stale) Mirror 27: 0xd8b1b6ff (stale) Mirror 28: 0xd8b1b6ff (stale) Mirror 30: 0xd8b1b6ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38103: skipped 000897 1777911152.705028 read 0x03243f thru 0x038102 (0x5cc4 bytes) 000898 1777911152.723202 read 0x02de50 thru 0x038102 (0xa2b3 bytes) 000899 1777911152.732949 write 0x017750 thru 0x020c08 (0x94b9 bytes) 000900 1777911152.734617 trunc from 0x038103 to 0x011584 (0x26b7e bytes) 000901 1777911152.754543 read 0x007641 thru 0x01035b (0x8d1b bytes) 000902 1777911152.761583 MIRROR_RESYNC 11 mirrors 000904 1777911153.794712 mapread 0x00b3fb thru 0x00cf9e (0x1ba4 bytes) 000905 1777911153.807981 read 0x00831c thru 0x0094b5 (0x119a bytes) 000908 1777911153.815249 mapread 0x009f65 thru 0x00c3fd (0x2499 bytes) 000910 1777911153.829200 mapwrite 0x00efe4 thru 0x0147dc (0x57f9 bytes) 000911 1777911153.941508 trunc from 0x0147dd to 0x023b9b (0xf3bf bytes) 000912 1777911153.964101 mapwrite 0x0068a6 thru 0x00e4b0 (0x7c0b bytes) 000913 1777911154.016255 mapread 0x002223 thru 0x006993 (0x4771 bytes) 000914 1777911154.022247 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23b9c): Mirror 10: 0x9ae6ec2b (stale) Mirror 13: 0x9ae6ec2b (stale) Mirror 20: 0x9ae6ec2b (stale) Mirror 23: 0x9ae6ec2b (stale) Mirror 24: 0x9ae6ec2b (stale) Mirror 25: 0x9ae6ec2b (stale) Mirror 27: 0x9ae6ec2b (stale) Mirror 28: 0x9ae6ec2b (stale) Mirror 30: 0x9ae6ec2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23b9c): Mirror 10: 0x9ae6ec2b (stale) Mirror 13: 0x9ae6ec2b (stale) Mirror 20: 0x9ae6ec2b (stale) Mirror 23: 0x9ae6ec2b (stale) Mirror 24: 0x9ae6ec2b (stale) Mirror 25: 0x9ae6ec2b (stale) Mirror 27: 0x9ae6ec2b (stale) Mirror 28: 0x9ae6ec2b (stale) Mirror 30: 0x9ae6ec2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23b9c: skipped 000916 1777911154.452439 mapwrite 0x0274f4 thru 0x027673 (0x180 bytes) 000917 1777911154.526145 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x27674): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27674): Mirror 10: 0xb2581cde (stale) Mirror 13: 0xb2581cde (stale) Mirror 20: 0xb2581cde (stale) Mirror 23: 0xb2581cde (stale) Mirror 24: 0xb2581cde (stale) Mirror 25: 0xb2581cde (stale) Mirror 27: 0xb2581cde (stale) Mirror 28: 0xb2581cde (stale) Mirror 30: 0xb2581cde (stale) CRC-32 checksum value for chunk [0, 0x27674): Mirror 1: 0x7bae492 Mirror 31: 0x7bae492 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27674: skipped 000920 1777911155.243571 mapread 0x00d978 thru 0x0197a7 (0xbe30 bytes) 000922 1777911155.280958 read 0x015cc9 thru 0x01f2cf (0x9607 bytes) 000923 1777911155.284132 write 0x0274ff thru 0x0368fc (0xf3fe bytes) 000925 1777911155.347882 mapread 0x018fd1 thru 0x0212ac (0x82dc bytes) 000929 1777911155.356706 mapread 0x020168 thru 0x02b376 (0xb20f bytes) 000930 1777911155.358794 mapwrite 0x00a55f thru 0x00fb48 (0x55ea bytes) 000932 1777911155.405835 write 0x01eca9 thru 0x021693 (0x29eb bytes) 000933 1777911155.406835 mapread 0x014bd4 thru 0x0153f0 (0x81d bytes) 000935 1777911155.408981 read 0x012b09 thru 0x01a05f (0x7557 bytes) 000938 1777911155.410867 mapread 0x02448e thru 0x026872 (0x23e5 bytes) 000940 1777911155.412602 write 0x018960 thru 0x01b9e5 (0x3086 bytes) 000942 1777911155.414750 mapwrite 0x01a24a thru 0x020b71 (0x6928 bytes) 000943 1777911155.459539 mapwrite 0x02b6f9 thru 0x038fc2 (0xd8ca bytes) 000944 1777911155.543776 MIRROR_RESYNC 11 mirrors 000946 1777911156.750879 MIRROR_EXTEND to 12 mirrors 000948 1777911157.453969 mapread 0x02aaab thru 0x035f6e (0xb4c4 bytes) 000949 1777911157.489689 mapwrite 0x00c524 thru 0x01a893 (0xe370 bytes) 000950 1777911157.667057 MIRROR_RESYNC 12 mirrors 000951 1777911159.164127 mapread 0x020e3c thru 0x02cafc (0xbcc1 bytes) 000952 1777911159.193947 read 0x01cdcc thru 0x029e3c (0xd071 bytes) 000954 1777911159.195000 mapwrite 0x017f49 thru 0x023ebe (0xbf76 bytes) 000956 1777911159.376625 MIRROR_RESYNC 12 mirrors 000957 1777911160.711861 mapread 0x026bac thru 0x02a2f4 (0x3749 bytes) 000959 1777911160.730457 write 0x03e42a thru 0x03f952 (0x1529 bytes) 000962 1777911160.821852 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f953): Mirror 10: 0x2e1cb647 (stale) Mirror 13: 0x2e1cb647 (stale) Mirror 20: 0x2e1cb647 (stale) Mirror 23: 0x2e1cb647 (stale) Mirror 24: 0x2e1cb647 (stale) Mirror 27: 0x2e1cb647 (stale) Mirror 28: 0x2e1cb647 (stale) Mirror 30: 0x2e1cb647 (stale) Mirror 31: 0x2e1cb647 (stale) Mirror 32: 0x2e1cb647 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f953): Mirror 10: 0x2e1cb647 (stale) Mirror 13: 0x2e1cb647 (stale) Mirror 20: 0x2e1cb647 (stale) Mirror 23: 0x2e1cb647 (stale) Mirror 24: 0x2e1cb647 (stale) Mirror 27: 0x2e1cb647 (stale) Mirror 28: 0x2e1cb647 (stale) Mirror 30: 0x2e1cb647 (stale) Mirror 31: 0x2e1cb647 (stale) Mirror 32: 0x2e1cb647 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f953: skipped 000963 1777911161.336362 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3f953): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f953): Mirror 10: 0x2e1cb647 (stale) Mirror 13: 0x2e1cb647 (stale) Mirror 20: 0x2e1cb647 (stale) Mirror 23: 0x2e1cb647 (stale) Mirror 24: 0x2e1cb647 (stale) Mirror 27: 0x2e1cb647 (stale) Mirror 28: 0x2e1cb647 (stale) Mirror 30: 0x2e1cb647 (stale) Mirror 31: 0x2e1cb647 (stale) Mirror 32: 0x2e1cb647 (stale) CRC-32 checksum value for chunk [0, 0x3f953): Mirror 1: 0x959aa8ae Mirror 33: 0x959aa8ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f953: skipped 000964 1777911162.058133 read 0x01c48b thru 0x0275a4 (0xb11a bytes) 000967 1777911162.091233 write 0x02c46b thru 0x02cb55 (0x6eb bytes) 000968 1777911162.173816 trunc from 0x03f953 to 0x03b7d4 (0x417e bytes) 000969 1777911162.196120 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 10: 0x975a7733 (stale) Mirror 13: 0x975a7733 (stale) Mirror 20: 0x975a7733 (stale) Mirror 23: 0x975a7733 (stale) Mirror 24: 0x975a7733 (stale) Mirror 27: 0x975a7733 (stale) Mirror 28: 0x975a7733 (stale) Mirror 30: 0x975a7733 (stale) Mirror 31: 0x975a7733 (stale) Mirror 32: 0x975a7733 (stale) Mirror 33: 0x975a7733 (stale) CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 1: 0x5b691bef Mirror 34: 0x5b691bef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7d5: skipped 000970 1777911162.997505 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 13: 0x975a7733 (stale) Mirror 20: 0x975a7733 (stale) Mirror 23: 0x975a7733 (stale) Mirror 24: 0x975a7733 (stale) Mirror 27: 0x975a7733 (stale) Mirror 28: 0x975a7733 (stale) Mirror 30: 0x975a7733 (stale) Mirror 31: 0x975a7733 (stale) Mirror 32: 0x975a7733 (stale) Mirror 33: 0x975a7733 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 13: 0x975a7733 (stale) Mirror 20: 0x975a7733 (stale) Mirror 23: 0x975a7733 (stale) Mirror 24: 0x975a7733 (stale) Mirror 27: 0x975a7733 (stale) Mirror 28: 0x975a7733 (stale) Mirror 30: 0x975a7733 (stale) Mirror 31: 0x975a7733 (stale) Mirror 32: 0x975a7733 (stale) Mirror 33: 0x975a7733 (stale) CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 1: 0x5b691bef Mirror 34: 0x5b691bef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7d5: skipped 000971 1777911164.006864 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 13: 0x975a7733 (stale) Mirror 20: 0x975a7733 (stale) Mirror 23: 0x975a7733 (stale) Mirror 24: 0x975a7733 (stale) Mirror 27: 0x975a7733 (stale) Mirror 28: 0x975a7733 (stale) Mirror 30: 0x975a7733 (stale) Mirror 32: 0x975a7733 (stale) Mirror 33: 0x975a7733 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 13: 0x975a7733 (stale) Mirror 20: 0x975a7733 (stale) Mirror 23: 0x975a7733 (stale) Mirror 24: 0x975a7733 (stale) Mirror 27: 0x975a7733 (stale) Mirror 28: 0x975a7733 (stale) Mirror 30: 0x975a7733 (stale) Mirror 32: 0x975a7733 (stale) Mirror 33: 0x975a7733 (stale) CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 1: 0x5b691bef Mirror 34: 0x5b691bef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7d5: skipped 000972 1777911164.901382 read 0x00897b thru 0x01366f (0xacf5 bytes) 000973 1777911164.945787 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 13: 0x975a7733 (stale) Mirror 20: 0x975a7733 (stale) Mirror 23: 0x975a7733 (stale) Mirror 24: 0x975a7733 (stale) Mirror 27: 0x975a7733 (stale) Mirror 28: 0x975a7733 (stale) Mirror 30: 0x975a7733 (stale) Mirror 32: 0x975a7733 (stale) Mirror 33: 0x975a7733 (stale) CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 1: 0x5b691bef Mirror 34: 0x5b691bef Mirror 35: 0x5b691bef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7d5: skipped 000974 1777911165.752261 mapread 0x038f0b thru 0x03b7d4 (0x28ca bytes) 000975 1777911165.780544 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 13: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 13: 0x975a7733 (stale) Mirror 20: 0x975a7733 (stale) Mirror 23: 0x975a7733 (stale) Mirror 24: 0x975a7733 (stale) Mirror 27: 0x975a7733 (stale) Mirror 28: 0x975a7733 (stale) Mirror 30: 0x975a7733 (stale) Mirror 32: 0x975a7733 (stale) Mirror 33: 0x975a7733 (stale) CRC-32 checksum value for chunk [0, 0x3b7d5): Mirror 1: 0x5b691bef Mirror 34: 0x5b691bef Mirror 35: 0x5b691bef Mirror 36: 0x5b691bef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7d5: skipped 000977 1777911166.673325 read 0x00e255 thru 0x01878c (0xa538 bytes) 000978 1777911166.700971 mapread 0x0349b3 thru 0x0384e1 (0x3b2f bytes) 000981 1777911166.755578 trunc from 0x03b7d5 to 0x030259 (0xb57b bytes) 000984 1777911166.837056 read 0x02a482 thru 0x02daf5 (0x3674 bytes) 000985 1777911166.860362 write 0x0320c2 thru 0x034458 (0x2397 bytes) 000986 1777911166.862905 read 0x00937a thru 0x00c82a (0x34b1 bytes) 000988 1777911166.864460 write 0x006994 thru 0x010ea1 (0xa50e bytes) 000990 1777911166.866383 mapread 0x00a295 thru 0x016024 (0xbd90 bytes) 000992 1777911166.870644 mapwrite 0x03176b thru 0x03e257 (0xcaed bytes) 000993 1777911166.944982 read 0x02f649 thru 0x03607b (0x6a33 bytes) 000995 1777911166.946599 trunc from 0x03e258 to 0x00be37 (0x32420 bytes) 000997 1777911166.962135 trunc from 0x00be38 to 0x0179c3 (0xbb8c bytes) 001000 1777911166.981726 trunc from 0x0179c4 to 0x03cd49 (0x25386 bytes) All operations completed A-OK! PASS 70a (108s) == sanity-flr test 71: check mirror extend parent fid ==== 12:12:50 (1777911170) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 12:12:52 (1777911172) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x382:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175735 s, 11.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x382:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185995 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x383:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x382:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x383:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x382:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x368:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 12:12:57 (1777911177) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre2 Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8401248 .. Extending file size to 3838560 .. done Extending file size to 4293760 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 6485472 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8681664 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7852032 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8325728 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9021536 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8510720 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8657376 .. Extending file size to 8847488 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8962592 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56967 56968 56971 56973 56975 done 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) 192.168.204.160@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (73s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:14:10 (1777911250) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre2 Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3899648 .. Extending file size to 1052288 .. Extending file size to 5694144 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5780000 .. Extending file size to 7008992 .. Extending file size to 7663168 .. Extending file size to 6693088 .. Extending file size to 8732992 .. Extending file size to 6938688 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7532896 .. Extending file size to 8966240 .. Extending file size to 9199616 .. Extending file size to 8253600 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8573184 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8602752 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8909568 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58729 58730 58731 58732 58735 done 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) 192.168.204.160@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:15:19 (1777911319) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3747840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3748864 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2944 2205440 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 21504 3747840 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 20480 3748864 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0590197 s, 35.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184014 s, 11.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa50c096b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) PASS 200c (9s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:15:29 (1777911329) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0620453 s, 33.8 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 12:15:33 (1777911333) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x399:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:15:37 (1777911337) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125186 s, 16.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:15:41 (1777911341) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150847 s, 13.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x380:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x381:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x380:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x381:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39d:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:15:46 (1777911346) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219274 s, 9.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x382:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x382:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39f:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:16:13 (1777911373) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546486 s, 19.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x383:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x383:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x384:0x0] } PASS 204d (49s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:17:02 (1777911422) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123136 s, 17.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x385:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x385:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:17:52 (1777911472) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg460-server mds-ost sync done. Creating new pool test_204f oleg460-server: Pool lustre.test_204f created Adding targets to pool oleg460-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg460-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139606 s, 15.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x386:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x386:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg460-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg460-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg460-server: Pool lustre.test_204f destroyed PASS 204f (61s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:18:53 (1777911533) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:18:56 (1777911536) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:19:00 (1777911540) oleg460-server: Pool lustre.test_206 created Waiting 90s for '' Adding targets to pool oleg460-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg460-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg460-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg460-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg460-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:19:17 (1777911557) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377859 s, 27.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175438 s, 59.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1075c2f8 Mirror 3: 0x1075c2f8 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:19:21 (1777911561) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:19:23 (1777911563) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:19:25 (1777911565) Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0704359 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xca0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x13f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x13f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x13f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1090000 Read this (4 bytes): BBBB 192.168.204.160@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg460-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:19:49 (1777911589) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0147134 s, 557 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:19:53 (1777911593) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208176 s, 50.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x280 lmm_fid: [0x200000404:0x280:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x280 lmm_fid: [0x200000404:0x280:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x390:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834560 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-04 12:19:54.000000000 -0400 Modify: 2026-05-04 12:19:54.000000000 -0400 Change: 2026-05-04 12:19:55.000000000 -0400 Birth: 2026-05-04 12:19:54.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:19:57 (1777911597) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:19:59 (1777911599) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.359136 s, 114 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.048924 s, 83.7 kB/s size after second write -rw-r--r-- 1 root root 4096 May 4 12:20 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x282:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 4 12:20 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:20:03 (1777911603) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:20:07 (1777911607) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1457 sec =========== 12:20:12 (1777911612) === sanity-flr: start cleanup 12:20:13 (1777911613) === === sanity-flr: finish cleanup 12:20:17 (1777911617) ===