-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 04:44:27 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg331-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:44:46 (1787733886) === oleg331-client.virtnet: executing check_config_client /mnt/lustre oleg331-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg331-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f48d08b7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f48d08b7000.idle_timeout=debug disable quota as required oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:45:21 (1787733921) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7625] == sanity-flr test 0a: lfs mirror create with -N option == 04:45:23 (1787733923) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:45:40 (1787733940) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:45:43 (1787733943) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:45:47 (1787733947) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:46:14 (1787733974) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:46:17 (1787733977) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 04:46:20 (1787733980) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f48d08b7000.stats=clear osc.lustre-OST0001-osc-ffff8f48d08b7000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:46:27 (1787733987) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:46:37 (1787733997) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:46:46 (1787734006) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:46:51 (1787734011) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:47:00 (1787734020) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 04:47:10 (1787734030) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:47:18 (1787734038) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:47:23 (1787734043) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:47:30 (1787734050) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:47:40 (1787734060) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:47:46 (1787734066) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.20298 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180885 s, 5.8 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:47:57 (1787734077) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.31357 s, 40.1 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:48:04 (1787734084) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:48:15 (1787734095) Stopping /mnt/lustre-ost1 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f48d08b7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f48d08b7000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg331-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg331-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f48d08b7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f48d08b7000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg331-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (131s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:50:26 (1787734226) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.467179 s, 22.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f48d08b7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f48d08b7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg331-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:51:10 (1787734270) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f48d08b7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f48d08b7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg331-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f48d08b7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f48d08b7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg331-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (68s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:52:17 (1787734337) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.44923 s, 11.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 04:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.959s user 0m0.001s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f48d08b7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f48d08b7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.031s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg331-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg331-server oleg331-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f48d08b7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f48d08b7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.234s user 0m0.000s sys 0m0.181s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg331-server: oleg331-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg331-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg331-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f48d08b7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f48d08b7000.ost_server_uuid in FULL state after 0 sec PASS 33b (79s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:53:37 (1787734417) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:53:39 (1787734419) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:53:42 (1787734422) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 04:53:45 (1787734425) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 04:53:51 (1787734431) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.395171 s, 31.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:54:00 (1787734440) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189502 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286937 s, 36.5 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:54:06 (1787734446) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:54:16 (1787734456) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00296399 s, 345 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00484925 s, 211 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00739126 s, 139 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012853 s, 797 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 04:54:27 (1787734467) create test files with size 15733474 .. 220b708da7ce4d40565f3feb25c4d3be - b86d346a5ee25d277a99699be8ec3792 - b407e06efa4cdfc9ae2447603861287f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg331-client.virtnet /mnt/lustre (opts:) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 04:54:46 (1787734486) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:54:59 (1787734499) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:55:04 (1787734504) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103168 s, 10.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284735 s, 36.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 04:55:13 (1787734513) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109094 s, 19.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.27417 s, 15.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150886 s, 13.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 04:55:26 (1787734526) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 04:55:28 (1787734528) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:55:31 (1787734531) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504479 s, 20.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 04:55:37 (1787734537) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:55:40 (1787734540) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:55:49 (1787734549) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.323165 s, 32.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg331-client.virtnet /mnt/lustre (opts:) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:56:00 (1787734560) llite.lustre-ffff8f48c718d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.272107 s, 38.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg331-client.virtnet /mnt/lustre (opts:) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8f48c51dc000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:56:12 (1787734572) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (31s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:56:43 (1787734603) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:57:16 (1787734636) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:57:19 (1787734639) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.195935 s, 16.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.224257 s, 14.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8820d859 (nosync,stale) Mirror 2: 0x8820d859 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 602150a9d12d7c22cbb9ff46fd636b4a - mirror 1 checksum: a0753851ff9bfb7415c770d07bf36c07 - mirror 2 checksum: 602150a9d12d7c22cbb9ff46fd636b4a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T04:57:22-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T04:57:22-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 602150a9d12d7c22cbb9ff46fd636b4a - mirror 1 checksum: a0753851ff9bfb7415c770d07bf36c07 - mirror 2 checksum: a0753851ff9bfb7415c770d07bf36c07 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T04:57:22-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T04:57:22-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:57:30 (1787734650) Failing mds1 on oleg331-server Stopping /mnt/lustre-mds1 (opts:) on oleg331-server 04:57:35 (1787734655) shut down facet: mds1 facet_host: oleg331-server facet_failover_host: oleg331-server Failover mds1 to oleg331-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 04:57:59 (1787734679) targets are mounted 04:57:59 (1787734679) facet_failover done oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483426 s, 21.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:58:16 (1787734696) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:58:19 (1787734699) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:58:22 (1787734702) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00528033 s, 3.1 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:58:29 (1787734709) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:58:31 (1787734711) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00745319 s, 550 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0833467 s, 49.1 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:58:38 (1787734718) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:59:01 (1787734741) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:59:33 (1787734773) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:59:57 (1787734797) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:00:21 (1787734821) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 05:00:29 (1787734829) mirror create pid 54319 mirror split pid 54321 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54319 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54321 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 05:01:37 (1787734897) Seed set to 4899 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1787734899.546183 write 0x039ff2 thru 0x03ffff (0x600e bytes) truncating to largest ever: 0x21299 000005 1787734899.558787 trunc from 0x040000 to 0x021298 (0x1ed67 bytes) truncating to largest ever: 0x2f93a 000006 1787734899.579350 trunc from 0x021299 to 0x02f939 (0xe6a1 bytes) 000009 1787734899.598985 mapwrite 0x0205eb thru 0x024360 (0x3d76 bytes) 000012 1787734899.688690 mapwrite 0x02612a thru 0x035970 (0xf847 bytes) 000013 1787734899.802623 mapwrite 0x039997 thru 0x03bd82 (0x23ec bytes) 000015 1787734899.907350 write 0x00be67 thru 0x01a4d0 (0xe66a bytes) 000018 1787734899.931485 MIRROR_EXTEND to 2 mirrors 000019 1787734901.348419 write 0x0129be thru 0x01e29d (0xb8e0 bytes) 000020 1787734901.424346 mapwrite 0x037c75 thru 0x03ffff (0x838b bytes) 000021 1787734901.523507 mapread 0x02e38c thru 0x03b97b (0xd5f0 bytes) 000027 1787734901.536739 mapwrite 0x01b674 thru 0x020188 (0x4b15 bytes) 000028 1787734901.591604 write 0x035e7e thru 0x03f35e (0x94e1 bytes) 000029 1787734901.594404 mapwrite 0x02c676 thru 0x037cda (0xb665 bytes) 000031 1787734901.658150 mapread 0x01a57c thru 0x029d45 (0xf7ca bytes) 000032 1787734901.664090 mapwrite 0x0118aa thru 0x0142ca (0x2a21 bytes) 000033 1787734901.766498 MIRROR_RESYNC 2 mirrors 000036 1787734902.681308 mapwrite 0x01c27a thru 0x0295a0 (0xd327 bytes) 000037 1787734902.896084 mapread 0x03fc74 thru 0x03ffff (0x38c bytes) 000038 1787734902.918480 mapwrite 0x00aca6 thru 0x0115fd (0x6958 bytes) 000039 1787734903.000519 trunc from 0x040000 to 0x023365 (0x1cc9a bytes) 000040 1787734903.040648 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23366): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23366): Mirror 2: 0x75ebe969 (stale) CRC-32 checksum value for chunk [0, 0x23366): Mirror 1: 0x4d2d8e56 Mirror 3: 0x4d2d8e56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23366: skipped 000042 1787734904.244664 read 0x0153bc thru 0x01f995 (0xa5da bytes) 000043 1787734904.272525 mapwrite 0x0101ec thru 0x011aa2 (0x18b7 bytes) 000044 1787734904.583971 mapwrite 0x008099 thru 0x0132c2 (0xb22a bytes) 000045 1787734904.696592 mapread 0x0193ba thru 0x023365 (0x9fac bytes) 000046 1787734904.734892 trunc from 0x023366 to 0x005b80 (0x1d7e5 bytes) 000048 1787734904.762560 mapread 0x004fa0 thru 0x005b80 (0xbe1 bytes) 000049 1787734904.764579 write 0x02188c thru 0x028bec (0x7361 bytes) 000050 1787734904.771049 MIRROR_RESYNC 3 mirrors 000051 1787734905.645186 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000053 1787734906.110317 read 0x014b58 thru 0x016cdd (0x2186 bytes) 000054 1787734906.127150 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000055 1787734906.319057 write 0x010c77 thru 0x01d1d9 (0xc563 bytes) truncating to largest ever: 0x3cb79 000056 1787734906.357440 trunc from 0x028bed to 0x03cb78 (0x13f8c bytes) 000058 1787734906.391259 mapwrite 0x02f208 thru 0x0374c8 (0x82c1 bytes) 000059 1787734906.465888 read 0x010289 thru 0x01d888 (0xd600 bytes) 000062 1787734906.472343 read 0x032d08 thru 0x037f96 (0x528f bytes) 000063 1787734906.473245 trunc from 0x03cb79 to 0x001b36 (0x3b042 bytes) 000064 1787734906.497070 mapwrite 0x01d58c thru 0x025b9d (0x8612 bytes) 000066 1787734906.608606 mapread 0x020e28 thru 0x025b9d (0x4d76 bytes) 000069 1787734906.613072 mapread 0x02470e thru 0x025b9d (0x1490 bytes) 000075 1787734906.620301 mapread 0x01676f thru 0x01f550 (0x8de2 bytes) 000076 1787734906.621492 read 0x0061fd thru 0x00bd69 (0x5b6d bytes) 000078 1787734906.625426 mapread 0x02486c thru 0x025b9d (0x1332 bytes) 000079 1787734906.637607 MIRROR_EXTEND to 2 mirrors 000080 1787734907.618597 write 0x02bfeb thru 0x037675 (0xb68b bytes) 000082 1787734907.692717 write 0x00ce26 thru 0x0104d5 (0x36b0 bytes) 000083 1787734907.706563 MIRROR_RESYNC 2 mirrors 000084 1787734908.759356 mapwrite 0x02ce76 thru 0x030166 (0x32f1 bytes) 000087 1787734908.970242 mapread 0x00bcea thru 0x0128c5 (0x6bdc bytes) 000089 1787734908.999894 write 0x0377f3 thru 0x03ffff (0x880d bytes) 000094 1787734909.020620 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6eeb8f26 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc2661e4f Mirror 3: 0xc2661e4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000097 1787734910.533322 mapwrite 0x00059c thru 0x00571e (0x5183 bytes) 000098 1787734910.751951 trunc from 0x040000 to 0x013aa3 (0x2c55c bytes) 000099 1787734910.781498 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 2: 0x4a9c280f (stale) Mirror 3: 0x4a9c280f (stale) CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 1: 0xfc09996e Mirror 4: 0xfc09996e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13aa4: skipped 000100 1787734912.215156 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 2: 0x4a9c280f (stale) Mirror 3: 0x4a9c280f (stale) CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 1: 0xfc09996e Mirror 4: 0xfc09996e Mirror 5: 0xfc09996e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13aa4: skipped 000101 1787734913.428687 mapread 0x00c8e5 thru 0x013aa3 (0x71bf bytes) 000102 1787734913.449985 read 0x0133c5 thru 0x013aa3 (0x6df bytes) 000104 1787734913.455678 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 2: 0x4a9c280f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 2: 0x4a9c280f (stale) CRC-32 checksum value for chunk [0, 0x13aa4): Mirror 1: 0xfc09996e Mirror 4: 0xfc09996e Mirror 5: 0xfc09996e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13aa4: skipped 000105 1787734914.190858 read 0x00cc49 thru 0x01288e (0x5c46 bytes) 000109 1787734914.342947 mapwrite 0x032dea thru 0x0354c9 (0x26e0 bytes) 000110 1787734914.631431 read 0x010b52 thru 0x0147af (0x3c5e bytes) 000111 1787734914.659443 mapwrite 0x03021c thru 0x03b6c2 (0xb4a7 bytes) 000113 1787734914.772423 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b6c3): Mirror 4: 0x2fc7aeda (stale) Mirror 5: 0x2fc7aeda (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6c3): Mirror 4: 0x2fc7aeda (stale) Mirror 5: 0x2fc7aeda (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6c3: skipped 000115 1787734915.162269 read 0x023e7d thru 0x025038 (0x11bc bytes) 000116 1787734915.170784 mapread 0x0398f4 thru 0x03a1f7 (0x904 bytes) 000117 1787734915.186510 read 0x01439c thru 0x0216f6 (0xd35b bytes) 000119 1787734915.207106 write 0x036dbf thru 0x03de4a (0x708c bytes) 000120 1787734915.209267 write 0x024688 thru 0x032461 (0xddda bytes) 000121 1787734915.211591 trunc from 0x03de4b to 0x028e19 (0x15031 bytes) 000122 1787734915.254304 read 0x00aa89 thru 0x014d82 (0xa2fa bytes) 000123 1787734915.255517 trunc from 0x028e1a to 0x00a4e8 (0x1e931 bytes) 000125 1787734915.300553 MIRROR_RESYNC 3 mirrors 000127 1787734916.481562 mapwrite 0x037af9 thru 0x03f5d2 (0x7ada bytes) 000128 1787734916.803722 mapwrite 0x02646c thru 0x0289d8 (0x256d bytes) 000129 1787734916.930574 trunc from 0x03f5d3 to 0x004a2d (0x3aba5 bytes) 000130 1787734916.956337 mapwrite 0x0316b9 thru 0x03ae1d (0x9765 bytes) 000131 1787734917.068442 trunc from 0x03ae1e to 0x000baa (0x3a273 bytes) 000134 1787734917.098522 mapread 0x000926 thru 0x000baa (0x285 bytes) 000135 1787734917.108372 trunc from 0x000bab to 0x03b451 (0x3a8a7 bytes) 000137 1787734917.125554 mapwrite 0x0169d4 thru 0x021716 (0xad43 bytes) 000138 1787734917.230133 write 0x006744 thru 0x007bda (0x1497 bytes) 000139 1787734917.262673 MIRROR_RESYNC 3 mirrors 000140 1787734918.916795 trunc from 0x03b452 to 0x012f1f (0x28532 bytes) 000144 1787734919.049017 mapwrite 0x0039cd thru 0x009d5c (0x6390 bytes) 000145 1787734919.183703 write 0x031d14 thru 0x03f899 (0xdb86 bytes) 000146 1787734919.190978 read 0x010b83 thru 0x0133b1 (0x282f bytes) 000148 1787734919.224008 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f89a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f89a): Mirror 4: 0x7457a45c (stale) Mirror 5: 0x7457a45c (stale) CRC-32 checksum value for chunk [0, 0x3f89a): Mirror 1: 0x85dd33be Mirror 6: 0x85dd33be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f89a: skipped 000151 1787734920.466541 MIRROR_RESYNC 4 mirrors 000152 1787734922.019991 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000153 1787734922.493620 mapwrite 0x03e50a thru 0x03ffff (0x1af6 bytes) 000156 1787734922.734700 mapwrite 0x000ecc thru 0x010062 (0xf197 bytes) 000157 1787734922.819178 trunc from 0x040000 to 0x01de1b (0x221e4 bytes) 000159 1787734922.854788 write 0x03db2f thru 0x03ffff (0x24d1 bytes) 000160 1787734922.861771 MIRROR_RESYNC 3 mirrors 000161 1787734924.174779 MIRROR_RESYNC 3 mirrors 000162 1787734924.713778 read 0x00b7b1 thru 0x01717a (0xb9ca bytes) 000164 1787734924.723961 mapwrite 0x000078 thru 0x008a0b (0x8994 bytes) 000165 1787734924.919551 read 0x03c157 thru 0x03ef62 (0x2e0c bytes) 000166 1787734924.931300 mapwrite 0x03f151 thru 0x03ffff (0xeaf bytes) 000168 1787734924.972261 write 0x022940 thru 0x025445 (0x2b06 bytes) 000170 1787734924.975465 read 0x03fa56 thru 0x03ffff (0x5aa bytes) 000171 1787734924.976473 mapwrite 0x02aca4 thru 0x032397 (0x76f4 bytes) 000173 1787734925.022149 write 0x0036e9 thru 0x0041d7 (0xaef bytes) 000174 1787734925.023318 write 0x00414c thru 0x00e32b (0xa1e0 bytes) 000175 1787734925.025326 mapwrite 0x015c87 thru 0x018ebd (0x3237 bytes) 000176 1787734925.111623 mapwrite 0x02315d thru 0x02ed5b (0xbbff bytes) 000178 1787734925.236432 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xda869566 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xda869566 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000179 1787734925.582499 mapwrite 0x013b90 thru 0x01f838 (0xbca9 bytes) 000180 1787734925.653451 write 0x02aebc thru 0x03134e (0x6493 bytes) 000182 1787734925.675857 MIRROR_RESYNC 2 mirrors 000183 1787734927.170530 mapread 0x000849 thru 0x005c3c (0x53f4 bytes) 000184 1787734927.215066 MIRROR_EXTEND to 3 mirrors 000185 1787734928.197542 MIRROR_RESYNC 3 mirrors 000186 1787734928.654535 trunc from 0x040000 to 0x018764 (0x2789b bytes) 000187 1787734928.721405 mapread 0x00f861 thru 0x018764 (0x8f04 bytes) 000188 1787734928.740186 write 0x032efb thru 0x0395b9 (0x66bf bytes) 000189 1787734928.742251 mapread 0x02408a thru 0x029ab5 (0x5a2c bytes) 000190 1787734928.749294 trunc from 0x0395ba to 0x019be5 (0x1f9d4 bytes) 000192 1787734928.772689 mapread 0x0077d2 thru 0x014369 (0xcb98 bytes) 000193 1787734928.785000 mapwrite 0x036cfa thru 0x03b2ee (0x45f5 bytes) 000194 1787734928.920167 mapread 0x002102 thru 0x00b3a6 (0x92a5 bytes) 000195 1787734928.928290 write 0x02faa0 thru 0x03d506 (0xda67 bytes) truncating to largest ever: 0x3cfda 000197 1787734928.936342 trunc from 0x03d507 to 0x03cfd9 (0x52d bytes) 000198 1787734928.977432 mapwrite 0x039de1 thru 0x03ffff (0x621f bytes) 000199 1787734929.077605 MIRROR_RESYNC 3 mirrors 000202 1787734930.465719 read 0x03e985 thru 0x03ffff (0x167b bytes) 000203 1787734930.470700 trunc from 0x040000 to 0x0174a9 (0x28b56 bytes) 000204 1787734930.649931 mapwrite 0x032d6c thru 0x037d8d (0x5022 bytes) 000206 1787734930.795155 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37d8e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d8e): Mirror 6: 0xe8cb043 (stale) Mirror 7: 0xe8cb043 (stale) CRC-32 checksum value for chunk [0, 0x37d8e): Mirror 1: 0x7c401a4f Mirror 8: 0x7c401a4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d8e: skipped 000207 1787734932.031425 mapwrite 0x02869e thru 0x030d6a (0x86cd bytes) 000209 1787734932.374254 mapwrite 0x0373c9 thru 0x03ecde (0x7916 bytes) 000211 1787734932.486251 mapwrite 0x03e149 thru 0x03ffff (0x1eb7 bytes) 000212 1787734932.601073 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2019793d (stale) Mirror 7: 0x2019793d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2019793d (stale) Mirror 7: 0x2019793d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000213 1787734932.982164 MIRROR_RESYNC 3 mirrors 000215 1787734934.772248 mapwrite 0x023d43 thru 0x02de50 (0xa10e bytes) 000218 1787734935.032440 mapwrite 0x0082b7 thru 0x00c83f (0x4589 bytes) 000219 1787734935.131430 read 0x0113cc thru 0x014e34 (0x3a69 bytes) 000220 1787734935.142154 mapread 0x0145a8 thru 0x019dda (0x5833 bytes) 000223 1787734935.156564 mapread 0x0244dc thru 0x02862c (0x4151 bytes) 000225 1787734935.165083 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6a28bad9 (stale) Mirror 7: 0x6a28bad9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb1928489 Mirror 8: 0xb1928489 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000227 1787734936.295739 MIRROR_RESYNC 4 mirrors 000229 1787734937.963713 mapwrite 0x000d68 thru 0x00e826 (0xdabf bytes) 000234 1787734938.175652 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb1928489 (stale) Mirror 7: 0xb1928489 (stale) Mirror 8: 0xb1928489 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1b76ef87 Mirror 9: 0x1b76ef87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000235 1787734939.579647 mapread 0x03414a thru 0x03ffff (0xbeb6 bytes) 000238 1787734939.692293 mapwrite 0x023518 thru 0x02a22a (0x6d13 bytes) 000242 1787734939.944488 trunc from 0x040000 to 0x00744f (0x38bb0 bytes) 000244 1787734939.974671 write 0x03afb6 thru 0x03ffff (0x504a bytes) 000246 1787734939.984001 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb1928489 (stale) Mirror 7: 0xb1928489 (stale) Mirror 8: 0xb1928489 (stale) Mirror 9: 0x1b76ef87 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9d73d73 Mirror 10: 0xc9d73d73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000248 1787734941.459245 MIRROR_RESYNC 6 mirrors 000249 1787734943.076315 mapwrite 0x0235a7 thru 0x02ffe0 (0xca3a bytes) 000250 1787734943.280109 mapwrite 0x00f5a7 thru 0x010ecb (0x1925 bytes) 000251 1787734943.331169 MIRROR_RESYNC 6 mirrors 000253 1787734946.117223 read 0x01b7fe thru 0x020ba2 (0x53a5 bytes) 000254 1787734946.129387 mapwrite 0x02c3b0 thru 0x0324f7 (0x6148 bytes) 000255 1787734946.372440 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x236d8e54 (stale) Mirror 7: 0x236d8e54 (stale) Mirror 8: 0x236d8e54 (stale) Mirror 9: 0x236d8e54 (stale) Mirror 10: 0x236d8e54 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf53c3ffd Mirror 11: 0xf53c3ffd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000256 1787734948.209660 mapread 0x01956b thru 0x0228bb (0x9351 bytes) 000257 1787734948.231861 mapwrite 0x03cc8c thru 0x03ffff (0x3374 bytes) 000258 1787734948.373140 mapwrite 0x03ff74 thru 0x03ffff (0x8c bytes) 000260 1787734948.431488 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x236d8e54 (stale) Mirror 8: 0x236d8e54 (stale) Mirror 9: 0x236d8e54 (stale) Mirror 10: 0x236d8e54 (stale) Mirror 11: 0xf53c3ffd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x236d8e54 (stale) Mirror 8: 0x236d8e54 (stale) Mirror 9: 0x236d8e54 (stale) Mirror 10: 0x236d8e54 (stale) Mirror 11: 0xf53c3ffd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000262 1787734949.061507 read 0x03a2a5 thru 0x03ffff (0x5d5b bytes) 000263 1787734949.070287 write 0x013d6d thru 0x023323 (0xf5b7 bytes) 000264 1787734949.086419 mapwrite 0x03a708 thru 0x03ffff (0x58f8 bytes) 000268 1787734949.176785 write 0x01b604 thru 0x025471 (0x9e6e bytes) 000269 1787734949.188495 mapwrite 0x01dc85 thru 0x024351 (0x66cd bytes) 000271 1787734949.242425 trunc from 0x040000 to 0x0259b8 (0x1a647 bytes) 000273 1787734949.268428 mapwrite 0x0270ca thru 0x034498 (0xd3cf bytes) 000277 1787734949.444196 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34499): Mirror 6: 0x3eb53f48 (stale) Mirror 8: 0x3eb53f48 (stale) Mirror 9: 0x3eb53f48 (stale) Mirror 11: 0xdc75f54f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34499): Mirror 6: 0x3eb53f48 (stale) Mirror 8: 0x3eb53f48 (stale) Mirror 9: 0x3eb53f48 (stale) Mirror 11: 0xdc75f54f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34499: skipped 000279 1787734949.960260 mapwrite 0x03d5d2 thru 0x03ffff (0x2a2e bytes) 000281 1787734950.094393 write 0x01b48a thru 0x0200f0 (0x4c67 bytes) 000284 1787734950.125459 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x236d8e54 (stale) Mirror 8: 0x236d8e54 (stale) Mirror 9: 0x236d8e54 (stale) Mirror 11: 0xf53c3ffd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x334b564a Mirror 12: 0x334b564a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000286 1787734952.338618 mapwrite 0x001d76 thru 0x00a647 (0x88d2 bytes) 000287 1787734952.603154 read 0x01f2c8 thru 0x02420a (0x4f43 bytes) 000288 1787734952.622287 read 0x03223e thru 0x032786 (0x549 bytes) 000291 1787734952.647335 write 0x0227c2 thru 0x023bd9 (0x1418 bytes) 000292 1787734952.648329 read 0x02373e thru 0x024450 (0xd13 bytes) 000295 1787734952.651356 trunc from 0x040000 to 0x02f5d9 (0x10a26 bytes) 000297 1787734952.707190 read 0x02601e thru 0x02f5d9 (0x95bc bytes) 000298 1787734952.708017 mapwrite 0x01b437 thru 0x01cba2 (0x176c bytes) 000299 1787734952.831188 mapread 0x01a527 thru 0x0234d1 (0x8fab bytes) 000300 1787734952.846363 MIRROR_RESYNC 6 mirrors 000304 1787734956.165325 mapwrite 0x00ae1e thru 0x00b8ca (0xaad bytes) 000309 1787734956.369341 mapwrite 0x009f1a thru 0x00ef70 (0x5057 bytes) 000310 1787734956.449447 MIRROR_RESYNC 6 mirrors 000311 1787734958.955789 mapread 0x00706e thru 0x00fd1a (0x8cad bytes) 000313 1787734958.988986 mapwrite 0x003dda thru 0x009358 (0x557f bytes) 000316 1787734959.219125 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f5da): Mirror 6: 0xd1669fd3 (stale) Mirror 8: 0xd1669fd3 (stale) Mirror 9: 0xd1669fd3 (stale) Mirror 11: 0xd1669fd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5da): Mirror 6: 0xd1669fd3 (stale) Mirror 8: 0xd1669fd3 (stale) Mirror 9: 0xd1669fd3 (stale) Mirror 11: 0xd1669fd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5da: skipped 000317 1787734959.753643 write 0x0208b7 thru 0x028f16 (0x8660 bytes) 000318 1787734959.764366 trunc from 0x02f5da to 0x0333af (0x3dd6 bytes) 000319 1787734959.794860 mapwrite 0x03f323 thru 0x03ffff (0xcdd bytes) 000320 1787734959.946801 trunc from 0x040000 to 0x034733 (0xb8cc bytes) 000324 1787734959.980666 read 0x01201c thru 0x02165b (0xf640 bytes) 000326 1787734960.001542 mapwrite 0x0092b8 thru 0x009310 (0x59 bytes) 000327 1787734960.140689 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34734): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34734): Mirror 6: 0xf9166ea2 (stale) Mirror 8: 0xf9166ea2 (stale) Mirror 9: 0xf9166ea2 (stale) Mirror 11: 0xf9166ea2 (stale) CRC-32 checksum value for chunk [0, 0x34734): Mirror 1: 0xf88f6bd Mirror 12: 0xf88f6bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34734: skipped 000328 1787734961.407431 trunc from 0x034734 to 0x00be06 (0x2892d bytes) 000329 1787734961.587147 write 0x01bbc5 thru 0x0284d2 (0xc90e bytes) 000330 1787734961.591282 mapwrite 0x023cca thru 0x03171a (0xda51 bytes) 000331 1787734961.760299 mapwrite 0x0108e0 thru 0x01cb0b (0xc22c bytes) 000332 1787734961.867127 write 0x0101f5 thru 0x018eec (0x8cf8 bytes) 000335 1787734961.877936 mapwrite 0x012b37 thru 0x01de33 (0xb2fd bytes) 000337 1787734961.956211 write 0x032b64 thru 0x036cd6 (0x4173 bytes) 000338 1787734961.958552 write 0x039d9c thru 0x03ffff (0x6264 bytes) 000340 1787734961.960450 write 0x02dae9 thru 0x036d17 (0x922f bytes) 000341 1787734961.970088 mapwrite 0x0243e7 thru 0x032830 (0xe44a bytes) 000342 1787734962.048703 trunc from 0x040000 to 0x02e791 (0x1186e bytes) 000344 1787734962.073170 mapread 0x023f65 thru 0x02d580 (0x961c bytes) 000346 1787734962.077712 trunc from 0x02e792 to 0x010931 (0x1de60 bytes) 000347 1787734962.110281 trunc from 0x010932 to 0x01833d (0x7a0c bytes) 000348 1787734962.141137 MIRROR_RESYNC 6 mirrors 000349 1787734964.232039 mapwrite 0x0374a1 thru 0x03ade4 (0x3944 bytes) 000353 1787734964.438442 mapread 0x030023 thru 0x03ade4 (0xadc2 bytes) 000358 1787734964.460062 trunc from 0x03ade5 to 0x03cec7 (0x20e3 bytes) 000360 1787734964.492112 read 0x021832 thru 0x022d61 (0x1530 bytes) 000361 1787734964.515272 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cec8): Mirror 6: 0x641deff0 (stale) Mirror 9: 0x641deff0 (stale) Mirror 11: 0x641deff0 (stale) Mirror 12: 0x641deff0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cec8): Mirror 6: 0x641deff0 (stale) Mirror 9: 0x641deff0 (stale) Mirror 11: 0x641deff0 (stale) Mirror 12: 0x641deff0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cec8: skipped 000362 1787734965.103380 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cec8): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cec8): Mirror 6: 0x641deff0 (stale) Mirror 9: 0x641deff0 (stale) Mirror 11: 0x641deff0 (stale) Mirror 12: 0x641deff0 (stale) CRC-32 checksum value for chunk [0, 0x3cec8): Mirror 1: 0x215b3c9b Mirror 13: 0x215b3c9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cec8: skipped 000363 1787734966.452561 read 0x01f65b thru 0x02aad3 (0xb479 bytes) 000364 1787734966.472219 mapwrite 0x013a5e thru 0x018238 (0x47db bytes) 000366 1787734966.714265 mapwrite 0x02fb1a thru 0x038e96 (0x937d bytes) 000368 1787734966.796144 mapwrite 0x013f3c thru 0x01722e (0x32f3 bytes) 000369 1787734966.852680 read 0x01c327 thru 0x021a7a (0x5754 bytes) 000371 1787734966.867776 mapwrite 0x01f81d thru 0x02928b (0x9a6f bytes) 000374 1787734966.925094 mapwrite 0x035653 thru 0x03bd5b (0x6709 bytes) 000377 1787734967.003711 write 0x03a2cf thru 0x03ffff (0x5d31 bytes) 000378 1787734967.020337 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xa30702eb (stale) Mirror 9: 0xa30702eb (stale) Mirror 11: 0xa30702eb (stale) Mirror 12: 0xa30702eb (stale) Mirror 13: 0xea801399 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x94938fe1 Mirror 14: 0x94938fe1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000379 1787734968.638319 trunc from 0x040000 to 0x03760f (0x89f0 bytes) 000384 1787734968.727044 mapread 0x031501 thru 0x0369b6 (0x54b6 bytes) 000385 1787734968.761094 mapread 0x02d629 thru 0x0304e3 (0x2ebb bytes) 000392 1787734968.769269 read 0x018c86 thru 0x019e0a (0x1185 bytes) 000394 1787734968.778508 mapread 0x032132 thru 0x032ce0 (0xbaf bytes) 000395 1787734968.779711 mapread 0x017b68 thru 0x01e0bf (0x6558 bytes) 000397 1787734968.792639 trunc from 0x037610 to 0x03bc77 (0x4668 bytes) 000398 1787734968.819327 mapread 0x033acb thru 0x039f21 (0x6457 bytes) 000399 1787734968.839258 mapread 0x01c6c0 thru 0x024294 (0x7bd5 bytes) 000400 1787734968.869782 MIRROR_RESYNC 7 mirrors 000401 1787734973.044943 MIRROR_RESYNC 7 mirrors 000408 1787734973.693349 MIRROR_RESYNC 7 mirrors 000410 1787734974.252207 mapwrite 0x01c151 thru 0x02042d (0x42dd bytes) 000414 1787734974.481063 mapread 0x034189 thru 0x03bc77 (0x7aef bytes) 000417 1787734974.494181 mapwrite 0x03bbf2 thru 0x03ffff (0x440e bytes) 000420 1787734974.607611 trunc from 0x040000 to 0x0182a3 (0x27d5c bytes) 000421 1787734974.624206 mapwrite 0x02d4f5 thru 0x03c56f (0xf07b bytes) truncating to largest ever: 0x3efdb 000423 1787734974.713103 trunc from 0x03c570 to 0x03efda (0x2a6b bytes) 000424 1787734974.728416 mapread 0x03ae0e thru 0x03efda (0x41cd bytes) 000425 1787734974.734324 mapread 0x004ac2 thru 0x008f52 (0x4491 bytes) 000427 1787734974.753754 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3efdb): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efdb): Mirror 6: 0x82c696a9 (stale) Mirror 9: 0x82c696a9 (stale) Mirror 11: 0x82c696a9 (stale) Mirror 12: 0x82c696a9 (stale) Mirror 13: 0x82c696a9 (stale) Mirror 14: 0x82c696a9 (stale) CRC-32 checksum value for chunk [0, 0x3efdb): Mirror 1: 0xa9a3fdc2 Mirror 15: 0xa9a3fdc2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efdb: skipped 000429 1787734976.567541 mapwrite 0x0329df thru 0x03837d (0x599f bytes) 000430 1787734976.925092 mapwrite 0x017f5c thru 0x027337 (0xf3dc bytes) 000432 1787734977.091738 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3efdb): Mirror 6: 0x82c696a9 (stale) Mirror 9: 0x82c696a9 (stale) Mirror 11: 0x82c696a9 (stale) Mirror 12: 0x82c696a9 (stale) Mirror 13: 0x82c696a9 (stale) Mirror 14: 0x82c696a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efdb): Mirror 6: 0x82c696a9 (stale) Mirror 9: 0x82c696a9 (stale) Mirror 11: 0x82c696a9 (stale) Mirror 12: 0x82c696a9 (stale) Mirror 13: 0x82c696a9 (stale) Mirror 14: 0x82c696a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efdb: skipped 000433 1787734977.671567 trunc from 0x03efdb to 0x026f9b (0x1803f bytes) 000434 1787734977.689212 read 0x024767 thru 0x026f9b (0x2835 bytes) 000435 1787734977.696129 trunc from 0x026f9c to 0x002f51 (0x2404a bytes) 000436 1787734977.713234 read 0x000353 thru 0x002f51 (0x2bff bytes) 000437 1787734977.718714 trunc from 0x002f52 to 0x027444 (0x244f3 bytes) 000439 1787734977.737621 trunc from 0x027445 to 0x0362bc (0xee78 bytes) truncating to largest ever: 0x3f1e2 000440 1787734977.759360 trunc from 0x0362bd to 0x03f1e1 (0x8f25 bytes) 000441 1787734977.787664 read 0x01324f thru 0x018895 (0x5647 bytes) 000443 1787734977.800349 trunc from 0x03f1e2 to 0x00ae58 (0x34389 bytes) 000446 1787734977.818725 read 0x0007c0 thru 0x00ae58 (0xa699 bytes) 000447 1787734977.819770 trunc from 0x00ae59 to 0x018e0a (0xdfb2 bytes) 000448 1787734977.846728 trunc from 0x018e0b to 0x01c059 (0x324f bytes) 000451 1787734977.883108 MIRROR_RESYNC 7 mirrors 000452 1787734980.058406 MIRROR_RESYNC 7 mirrors 000454 1787734980.628610 write 0x01cd7b thru 0x01f734 (0x29ba bytes) 000459 1787734980.728988 mapwrite 0x00547f thru 0x0079ab (0x252d bytes) 000461 1787734980.846629 mapwrite 0x020a19 thru 0x023739 (0x2d21 bytes) 000464 1787734980.925908 trunc from 0x02373a to 0x01fed3 (0x3866 bytes) 000465 1787734980.958213 mapwrite 0x02c26d thru 0x02d40b (0x119f bytes) 000467 1787734981.043191 write 0x002374 thru 0x005add (0x376a bytes) 000469 1787734981.045730 mapread 0x02844d thru 0x02d40b (0x4fbf bytes) 000472 1787734981.048980 trunc from 0x02d40c to 0x013426 (0x19fe5 bytes) 000475 1787734981.071403 write 0x0277f1 thru 0x029453 (0x1c63 bytes) 000478 1787734981.075430 mapwrite 0x020881 thru 0x025e6b (0x55eb bytes) 000479 1787734981.129978 mapread 0x01b1e4 thru 0x01d377 (0x2194 bytes) 000480 1787734981.132211 mapwrite 0x003734 thru 0x012309 (0xebd6 bytes) 000481 1787734981.194791 read 0x0081c1 thru 0x00a515 (0x2355 bytes) 000482 1787734981.195800 write 0x02c45a thru 0x034fe7 (0x8b8e bytes) 000484 1787734981.213368 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 9: 0xbf4195a3 (stale) Mirror 11: 0xbf4195a3 (stale) Mirror 12: 0xbf4195a3 (stale) Mirror 13: 0xbf4195a3 (stale) Mirror 14: 0xbf4195a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 9: 0xbf4195a3 (stale) Mirror 11: 0xbf4195a3 (stale) Mirror 12: 0xbf4195a3 (stale) Mirror 13: 0xbf4195a3 (stale) Mirror 14: 0xbf4195a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fe8: skipped 000486 1787734981.767208 mapwrite 0x01a16c thru 0x01ccaa (0x2b3f bytes) 000489 1787734981.852945 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 9: 0xbf4195a3 (stale) Mirror 11: 0xbf4195a3 (stale) Mirror 12: 0xbf4195a3 (stale) Mirror 13: 0xbf4195a3 (stale) Mirror 14: 0xbf4195a3 (stale) CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 1: 0x201cdbe6 Mirror 15: 0x201cdbe6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fe8: skipped 000490 1787734983.450156 mapwrite 0x0194c0 thru 0x022c25 (0x9766 bytes) 000491 1787734983.738552 mapwrite 0x0190d8 thru 0x01a48d (0x13b6 bytes) 000492 1787734983.844202 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 9: 0xbf4195a3 (stale) Mirror 11: 0xbf4195a3 (stale) Mirror 12: 0xbf4195a3 (stale) Mirror 13: 0xbf4195a3 (stale) Mirror 14: 0xbf4195a3 (stale) Mirror 15: 0x201cdbe6 (stale) CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 1: 0x9ecdcdd0 Mirror 16: 0x9ecdcdd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fe8: skipped 000494 1787734985.532189 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 9: 0xbf4195a3 (stale) Mirror 11: 0xbf4195a3 (stale) Mirror 12: 0xbf4195a3 (stale) Mirror 13: 0xbf4195a3 (stale) Mirror 14: 0xbf4195a3 (stale) Mirror 15: 0x201cdbe6 (stale) CRC-32 checksum value for chunk [0, 0x34fe8): Mirror 1: 0x9ecdcdd0 Mirror 16: 0x9ecdcdd0 Mirror 17: 0x9ecdcdd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34fe8: skipped 000495 1787734986.682749 write 0x005630 thru 0x007a10 (0x23e1 bytes) 000496 1787734986.860084 MIRROR_RESYNC 9 mirrors 000497 1787734991.121935 mapread 0x025f87 thru 0x02fa8f (0x9b09 bytes) 000498 1787734991.162295 mapwrite 0x00cb99 thru 0x01a282 (0xd6ea bytes) 000501 1787734991.431455 write 0x0187b5 thru 0x0286ff (0xff4b bytes) 000502 1787734991.443955 mapwrite 0x03aaae thru 0x03aacc (0x1f bytes) 000503 1787734991.565736 read 0x037971 thru 0x03aacc (0x315c bytes) 000507 1787734991.570656 mapread 0x013c68 thru 0x0171be (0x3557 bytes) 000509 1787734991.572793 mapwrite 0x001913 thru 0x008a3c (0x712a bytes) 000510 1787734991.627724 mapwrite 0x01f765 thru 0x0259ee (0x628a bytes) 000511 1787734991.676441 mapwrite 0x0102c1 thru 0x0125ce (0x230e bytes) 000512 1787734991.715730 mapwrite 0x0035c4 thru 0x00eb46 (0xb583 bytes) 000513 1787734991.769542 trunc from 0x03aacd to 0x02100e (0x19abe bytes) 000514 1787734991.796790 read 0x01bc30 thru 0x02100e (0x53df bytes) 000517 1787734991.799797 write 0x00cc38 thru 0x01cb7d (0xff46 bytes) 000518 1787734991.810066 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2100f): Mirror 11: 0xfde204da (stale) Mirror 12: 0xfde204da (stale) Mirror 13: 0xfde204da (stale) Mirror 14: 0xfde204da (stale) Mirror 15: 0xfde204da (stale) Mirror 16: 0xfde204da (stale) Mirror 17: 0xfde204da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2100f): Mirror 11: 0xfde204da (stale) Mirror 12: 0xfde204da (stale) Mirror 13: 0xfde204da (stale) Mirror 14: 0xfde204da (stale) Mirror 15: 0xfde204da (stale) Mirror 16: 0xfde204da (stale) Mirror 17: 0xfde204da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2100f: skipped 000519 1787734992.400978 mapread 0x01b0fc thru 0x02100e (0x5f13 bytes) 000520 1787734992.426778 mapwrite 0x008599 thru 0x00daf3 (0x555b bytes) 000523 1787734992.504974 mapread 0x01991a thru 0x02100e (0x76f5 bytes) 000526 1787734992.512531 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2100f): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2100f): Mirror 11: 0xfde204da (stale) Mirror 12: 0xfde204da (stale) Mirror 13: 0xfde204da (stale) Mirror 14: 0xfde204da (stale) Mirror 15: 0xfde204da (stale) Mirror 16: 0xfde204da (stale) Mirror 17: 0xfde204da (stale) CRC-32 checksum value for chunk [0, 0x2100f): Mirror 1: 0xb034aa5f Mirror 18: 0xb034aa5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2100f: skipped 000530 1787734993.565123 MIRROR_RESYNC 9 mirrors 000531 1787734994.989705 trunc from 0x02100f to 0x037952 (0x16944 bytes) 000532 1787734995.102004 mapwrite 0x03a58f thru 0x03e117 (0x3b89 bytes) 000533 1787734995.225743 mapread 0x0163ca thru 0x020965 (0xa59c bytes) 000537 1787734995.279231 trunc from 0x03e118 to 0x02fded (0xe32a bytes) 000538 1787734995.302733 write 0x018b1a thru 0x0270e3 (0xe5ca bytes) 000539 1787734995.304257 write 0x0165f5 thru 0x01bc77 (0x5683 bytes) 000540 1787734995.314778 MIRROR_RESYNC 9 mirrors 000541 1787734997.507047 read 0x020562 thru 0x02bae4 (0xb583 bytes) 000542 1787734997.525082 MIRROR_RESYNC 9 mirrors 000545 1787734998.132455 mapwrite 0x038b66 thru 0x03bf3c (0x33d7 bytes) 000546 1787734998.318544 mapwrite 0x03136e thru 0x03eaaa (0xd73d bytes) 000547 1787734998.458143 write 0x011550 thru 0x01afb0 (0x9a61 bytes) 000548 1787734998.470619 mapwrite 0x012e22 thru 0x01c61b (0x97fa bytes) 000550 1787734998.566987 write 0x027222 thru 0x033a94 (0xc873 bytes) 000551 1787734998.585358 MIRROR_RESYNC 9 mirrors 000552 1787735000.388932 MIRROR_RESYNC 9 mirrors 000553 1787735001.120789 MIRROR_EXTEND to 10 mirrors 000554 1787735002.299383 trunc from 0x03eaab to 0x032d42 (0xbd68 bytes) 000556 1787735002.524381 read 0x0050ad thru 0x00dba0 (0x8af4 bytes) 000557 1787735002.547115 trunc from 0x032d43 to 0x03e35f (0xb61d bytes) 000559 1787735002.583982 trunc from 0x03e360 to 0x0233ea (0x1af75 bytes) 000560 1787735002.614925 mapwrite 0x027935 thru 0x033b2b (0xc1f7 bytes) 000566 1787735002.758030 trunc from 0x033b2c to 0x0076a1 (0x2c48a bytes) 000567 1787735002.792453 trunc from 0x0076a2 to 0x014b35 (0xd494 bytes) 000570 1787735002.812188 mapread 0x003b65 thru 0x0075d6 (0x3a72 bytes) 000573 1787735002.835767 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x14b36): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14b36): Mirror 11: 0x67a74b21 (stale) Mirror 12: 0x67a74b21 (stale) Mirror 13: 0x67a74b21 (stale) Mirror 14: 0x67a74b21 (stale) Mirror 15: 0x67a74b21 (stale) Mirror 16: 0x67a74b21 (stale) Mirror 17: 0x67a74b21 (stale) Mirror 18: 0x67a74b21 (stale) Mirror 19: 0x67a74b21 (stale) CRC-32 checksum value for chunk [0, 0x14b36): Mirror 1: 0x36b36da5 Mirror 20: 0x36b36da5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14b36: skipped 000574 1787735004.238780 trunc from 0x014b36 to 0x00a542 (0xa5f3 bytes) 000576 1787735004.385196 mapread 0x00a229 thru 0x00a542 (0x31a bytes) 000577 1787735004.394355 mapwrite 0x01cbd6 thru 0x020a29 (0x3e54 bytes) 000578 1787735004.559844 mapread 0x007a1a thru 0x00ba2e (0x4015 bytes) 000579 1787735004.570517 trunc from 0x020a2a to 0x0014a8 (0x1f581 bytes) 000581 1787735004.590748 read 0x000ce9 thru 0x0014a8 (0x7c0 bytes) 000582 1787735004.602101 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14a9): Mirror 11: 0xbb57ec4b (stale) Mirror 12: 0xbb57ec4b (stale) Mirror 13: 0xbb57ec4b (stale) Mirror 14: 0xbb57ec4b (stale) Mirror 15: 0xbb57ec4b (stale) Mirror 16: 0xbb57ec4b (stale) Mirror 17: 0xbb57ec4b (stale) Mirror 18: 0xbb57ec4b (stale) Mirror 20: 0xbb57ec4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14a9): Mirror 11: 0xbb57ec4b (stale) Mirror 12: 0xbb57ec4b (stale) Mirror 13: 0xbb57ec4b (stale) Mirror 14: 0xbb57ec4b (stale) Mirror 15: 0xbb57ec4b (stale) Mirror 16: 0xbb57ec4b (stale) Mirror 17: 0xbb57ec4b (stale) Mirror 18: 0xbb57ec4b (stale) Mirror 20: 0xbb57ec4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14a9: skipped 000583 1787735005.107457 mapwrite 0x00adf4 thru 0x00fd5c (0x4f69 bytes) 000584 1787735005.269237 trunc from 0x00fd5d to 0x0316c7 (0x2196b bytes) 000587 1787735005.309643 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x316c8): Mirror 11: 0x71ed57a8 (stale) Mirror 12: 0x71ed57a8 (stale) Mirror 13: 0x71ed57a8 (stale) Mirror 14: 0x71ed57a8 (stale) Mirror 15: 0x71ed57a8 (stale) Mirror 17: 0x71ed57a8 (stale) Mirror 18: 0x71ed57a8 (stale) Mirror 20: 0x2fa3c85b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x316c8): Mirror 11: 0x71ed57a8 (stale) Mirror 12: 0x71ed57a8 (stale) Mirror 13: 0x71ed57a8 (stale) Mirror 14: 0x71ed57a8 (stale) Mirror 15: 0x71ed57a8 (stale) Mirror 17: 0x71ed57a8 (stale) Mirror 18: 0x71ed57a8 (stale) Mirror 20: 0x2fa3c85b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x316c8: skipped 000588 1787735006.119404 mapwrite 0x0226d6 thru 0x026904 (0x422f bytes) 000589 1787735006.193152 write 0x00a840 thru 0x00c8cf (0x2090 bytes) 000592 1787735006.220385 mapread 0x00fe05 thru 0x01313a (0x3336 bytes) 000594 1787735006.244977 mapwrite 0x019d51 thru 0x01f0f6 (0x53a6 bytes) 000595 1787735006.386250 read 0x00a14e thru 0x00ae48 (0xcfb bytes) 000597 1787735006.397089 MIRROR_RESYNC 9 mirrors 000598 1787735011.239740 mapread 0x028216 thru 0x02ed0f (0x6afa bytes) 000600 1787735011.270431 mapread 0x00fa9f thru 0x0144bc (0x4a1e bytes) 000601 1787735011.287427 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000603 1787735011.839473 mapwrite 0x0358f4 thru 0x03b4d7 (0x5be4 bytes) 000604 1787735012.051654 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b4d8): Mirror 12: 0x38d75424 (stale) Mirror 13: 0x38d75424 (stale) Mirror 14: 0x38d75424 (stale) Mirror 15: 0x38d75424 (stale) Mirror 18: 0x38d75424 (stale) Mirror 20: 0x38d75424 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4d8): Mirror 12: 0x38d75424 (stale) Mirror 13: 0x38d75424 (stale) Mirror 14: 0x38d75424 (stale) Mirror 15: 0x38d75424 (stale) Mirror 18: 0x38d75424 (stale) Mirror 20: 0x38d75424 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4d8: skipped 000606 1787735012.738223 read 0x01083a thru 0x01a564 (0x9d2b bytes) 000610 1787735012.753905 mapread 0x015575 thru 0x019b22 (0x45ae bytes) 000615 1787735012.758589 mapwrite 0x02aa48 thru 0x031319 (0x68d2 bytes) 000619 1787735012.889217 MIRROR_RESYNC 7 mirrors 000620 1787735017.148706 trunc from 0x03b4d8 to 0x00d849 (0x2dc8e bytes) 000623 1787735017.258515 read 0x0017c4 thru 0x009a63 (0x82a0 bytes) 000624 1787735017.267553 mapread 0x008214 thru 0x00d849 (0x5636 bytes) 000625 1787735017.287396 read 0x0061c3 thru 0x00d849 (0x7687 bytes) 000628 1787735017.291043 trunc from 0x00d84a to 0x017139 (0x98f0 bytes) 000629 1787735017.317773 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1713a): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1713a): Mirror 12: 0xe8b90a1e (stale) Mirror 13: 0xe8b90a1e (stale) Mirror 14: 0xe8b90a1e (stale) Mirror 15: 0xe8b90a1e (stale) Mirror 18: 0xe8b90a1e (stale) Mirror 20: 0xe8b90a1e (stale) CRC-32 checksum value for chunk [0, 0x1713a): Mirror 1: 0x640a6bcc Mirror 21: 0x640a6bcc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1713a: skipped 000630 1787735018.475992 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1713a): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1713a): Mirror 12: 0xe8b90a1e (stale) Mirror 13: 0xe8b90a1e (stale) Mirror 14: 0xe8b90a1e (stale) Mirror 15: 0xe8b90a1e (stale) Mirror 18: 0xe8b90a1e (stale) Mirror 20: 0xe8b90a1e (stale) CRC-32 checksum value for chunk [0, 0x1713a): Mirror 1: 0x640a6bcc Mirror 21: 0x640a6bcc Mirror 22: 0x640a6bcc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1713a: skipped 000631 1787735019.956780 trunc from 0x01713a to 0x01348b (0x3cae bytes) 000633 1787735020.146682 read 0x00d20e thru 0x01348b (0x627e bytes) 000636 1787735020.157842 trunc from 0x01348c to 0x010995 (0x2af6 bytes) 000638 1787735020.186724 mapwrite 0x03ed26 thru 0x03ffff (0x12da bytes) 000639 1787735020.295238 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x37ca9bab (stale) Mirror 14: 0x37ca9bab (stale) Mirror 15: 0x37ca9bab (stale) Mirror 18: 0x37ca9bab (stale) Mirror 20: 0x37ca9bab (stale) Mirror 21: 0x305bb4a9 (stale) Mirror 22: 0x305bb4a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x37ca9bab (stale) Mirror 14: 0x37ca9bab (stale) Mirror 15: 0x37ca9bab (stale) Mirror 18: 0x37ca9bab (stale) Mirror 20: 0x37ca9bab (stale) Mirror 21: 0x305bb4a9 (stale) Mirror 22: 0x305bb4a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000640 1787735020.863541 mapwrite 0x010080 thru 0x019080 (0x9001 bytes) 000641 1787735020.928134 trunc from 0x040000 to 0x023741 (0x1c8be bytes) 000644 1787735020.953286 trunc from 0x023742 to 0x02454c (0xe0b bytes) 000646 1787735020.986063 trunc from 0x02454d to 0x02d489 (0x8f3d bytes) 000647 1787735021.022965 read 0x019b9c thru 0x01abbd (0x1022 bytes) 000649 1787735021.025636 mapread 0x027242 thru 0x02ab09 (0x38c8 bytes) 000650 1787735021.039550 trunc from 0x02d48a to 0x0243e8 (0x90a1 bytes) 000652 1787735021.093178 trunc from 0x0243e9 to 0x0327ea (0xe402 bytes) 000654 1787735021.118106 mapread 0x020f34 thru 0x028a73 (0x7b40 bytes) 000655 1787735021.138595 mapread 0x01544b thru 0x01a844 (0x53fa bytes) 000656 1787735021.148473 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x327eb): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x327eb): Mirror 13: 0x2681f2b (stale) Mirror 14: 0x2681f2b (stale) Mirror 15: 0x2681f2b (stale) Mirror 18: 0x2681f2b (stale) Mirror 20: 0x2681f2b (stale) Mirror 21: 0x6d56b722 (stale) Mirror 22: 0x6d56b722 (stale) CRC-32 checksum value for chunk [0, 0x327eb): Mirror 1: 0xb01a17c5 Mirror 23: 0xb01a17c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x327eb: skipped 000659 1787735022.345264 read 0x02211a thru 0x029d2f (0x7c16 bytes) 000660 1787735022.360753 read 0x004e4e thru 0x011583 (0xc736 bytes) 000661 1787735022.385840 mapread 0x01c4d5 thru 0x025e3f (0x996b bytes) 000662 1787735022.400280 mapread 0x00737f thru 0x011d73 (0xa9f5 bytes) 000665 1787735022.429945 mapwrite 0x00066b thru 0x003ce7 (0x367d bytes) 000666 1787735022.658995 trunc from 0x0327eb to 0x00e08b (0x2475f bytes) 000667 1787735022.717579 mapread 0x000a00 thru 0x0079fe (0x6fff bytes) 000670 1787735022.744602 read 0x0010a9 thru 0x00e08b (0xcfe3 bytes) 000672 1787735022.767565 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xe08c): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe08c): Mirror 13: 0xd921f611 (stale) Mirror 14: 0xd921f611 (stale) Mirror 15: 0xd921f611 (stale) Mirror 18: 0xd921f611 (stale) Mirror 20: 0xd921f611 (stale) Mirror 21: 0x719f954e (stale) Mirror 22: 0x719f954e (stale) Mirror 23: 0x719f954e (stale) CRC-32 checksum value for chunk [0, 0xe08c): Mirror 1: 0x701271e0 Mirror 24: 0x701271e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe08c: skipped 000673 1787735024.122802 mapwrite 0x026a25 thru 0x033200 (0xc7dc bytes) 000674 1787735024.304841 mapwrite 0x004836 thru 0x01341b (0xebe6 bytes) 000675 1787735024.386792 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33201): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33201): Mirror 13: 0x3ffb6bae (stale) Mirror 14: 0x3ffb6bae (stale) Mirror 15: 0x3ffb6bae (stale) Mirror 18: 0x3ffb6bae (stale) Mirror 20: 0x3ffb6bae (stale) Mirror 21: 0x6bd058bc (stale) Mirror 22: 0x6bd058bc (stale) Mirror 23: 0x22a0ab8d (stale) Mirror 24: 0x1530e3b0 (stale) CRC-32 checksum value for chunk [0, 0x33201): Mirror 1: 0x30a0e510 Mirror 25: 0x30a0e510 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33201: skipped 000678 1787735025.511709 mapwrite 0x003e9b thru 0x0106c1 (0xc827 bytes) 000682 1787735025.754496 mapread 0x00a309 thru 0x00fd36 (0x5a2e bytes) 000683 1787735025.766342 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33201): Mirror 13: 0x3ffb6bae (stale) Mirror 14: 0x3ffb6bae (stale) Mirror 15: 0x3ffb6bae (stale) Mirror 18: 0x3ffb6bae (stale) Mirror 20: 0x3ffb6bae (stale) Mirror 21: 0x6bd058bc (stale) Mirror 22: 0x6bd058bc (stale) Mirror 24: 0x1530e3b0 (stale) Mirror 25: 0x30a0e510 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33201): Mirror 13: 0x3ffb6bae (stale) Mirror 14: 0x3ffb6bae (stale) Mirror 15: 0x3ffb6bae (stale) Mirror 18: 0x3ffb6bae (stale) Mirror 20: 0x3ffb6bae (stale) Mirror 21: 0x6bd058bc (stale) Mirror 22: 0x6bd058bc (stale) Mirror 24: 0x1530e3b0 (stale) Mirror 25: 0x30a0e510 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33201: skipped 000684 1787735026.599397 write 0x038a5f thru 0x03ffff (0x75a1 bytes) 000686 1787735026.601420 write 0x0088b5 thru 0x009a5a (0x11a6 bytes) 000688 1787735026.646252 mapread 0x022275 thru 0x02c2df (0xa06b bytes) 000690 1787735026.677064 read 0x00ddf5 thru 0x01a1b4 (0xc3c0 bytes) 000691 1787735026.696084 mapwrite 0x03f4d0 thru 0x03ffff (0xb30 bytes) 000693 1787735026.801928 write 0x009c6b thru 0x012634 (0x89ca bytes) 000694 1787735026.807071 mapwrite 0x0168bf thru 0x01b46a (0x4bac bytes) 000695 1787735026.943720 mapwrite 0x01216d thru 0x01eec8 (0xcd5c bytes) 000697 1787735027.109601 read 0x033392 thru 0x03ffff (0xcc6e bytes) 000698 1787735027.151911 read 0x009789 thru 0x010890 (0x7108 bytes) 000699 1787735027.154502 read 0x03e8b8 thru 0x03ffff (0x1748 bytes) 000700 1787735027.172212 MIRROR_RESYNC 10 mirrors 000702 1787735031.109611 write 0x0293a4 thru 0x036a1c (0xd679 bytes) 000705 1787735031.201138 read 0x00cc3f thru 0x01bd4c (0xf10e bytes) 000706 1787735031.211778 mapread 0x02c686 thru 0x03be6b (0xf7e6 bytes) 000707 1787735031.234793 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xdedbcad2 (stale) Mirror 14: 0xdedbcad2 (stale) Mirror 15: 0xdedbcad2 (stale) Mirror 18: 0xdedbcad2 (stale) Mirror 20: 0xdedbcad2 (stale) Mirror 21: 0xdedbcad2 (stale) Mirror 22: 0xdedbcad2 (stale) Mirror 24: 0xdedbcad2 (stale) Mirror 25: 0xdedbcad2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73039acc Mirror 26: 0x73039acc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000710 1787735032.985391 mapwrite 0x03a7e1 thru 0x03fe02 (0x5622 bytes) 000711 1787735033.451147 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xdedbcad2 (stale) Mirror 14: 0xdedbcad2 (stale) Mirror 15: 0xdedbcad2 (stale) Mirror 18: 0xdedbcad2 (stale) Mirror 20: 0xdedbcad2 (stale) Mirror 21: 0xdedbcad2 (stale) Mirror 22: 0xdedbcad2 (stale) Mirror 24: 0xdedbcad2 (stale) Mirror 25: 0xdedbcad2 (stale) Mirror 26: 0x73039acc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3b81f03a Mirror 27: 0x3b81f03a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000712 1787735035.192519 mapwrite 0x007e9a thru 0x01747f (0xf5e6 bytes) 000714 1787735035.487186 read 0x0337c2 thru 0x036b6c (0x33ab bytes) 000716 1787735035.510228 MIRROR_RESYNC 12 mirrors 000718 1787735042.814160 MIRROR_EXTEND to 13 mirrors 000719 1787735044.966496 trunc from 0x040000 to 0x00c31d (0x33ce2 bytes) 000721 1787735045.215639 mapread 0x009b0e thru 0x00c31d (0x2810 bytes) 000723 1787735045.281202 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc31e): Mirror 13: 0x5fee0663 (stale) Mirror 14: 0x5fee0663 (stale) Mirror 15: 0x5fee0663 (stale) Mirror 18: 0x5fee0663 (stale) Mirror 20: 0x5fee0663 (stale) Mirror 21: 0x5fee0663 (stale) Mirror 22: 0x5fee0663 (stale) Mirror 24: 0x5fee0663 (stale) Mirror 26: 0x5fee0663 (stale) Mirror 27: 0x5fee0663 (stale) Mirror 28: 0x5fee0663 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc31e): Mirror 13: 0x5fee0663 (stale) Mirror 14: 0x5fee0663 (stale) Mirror 15: 0x5fee0663 (stale) Mirror 18: 0x5fee0663 (stale) Mirror 20: 0x5fee0663 (stale) Mirror 21: 0x5fee0663 (stale) Mirror 22: 0x5fee0663 (stale) Mirror 24: 0x5fee0663 (stale) Mirror 26: 0x5fee0663 (stale) Mirror 27: 0x5fee0663 (stale) Mirror 28: 0x5fee0663 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc31e: skipped 000724 1787735046.123119 mapwrite 0x033082 thru 0x03706c (0x3feb bytes) 000725 1787735046.239411 write 0x02a946 thru 0x0336fb (0x8db6 bytes) 000726 1787735046.249312 trunc from 0x03706d to 0x0135ed (0x23a7f bytes) 000727 1787735046.283002 write 0x03ad6e thru 0x03ffff (0x5292 bytes) 000730 1787735046.305634 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7863d0bf (stale) Mirror 15: 0x7863d0bf (stale) Mirror 18: 0x7863d0bf (stale) Mirror 20: 0x7863d0bf (stale) Mirror 21: 0x7863d0bf (stale) Mirror 22: 0x7863d0bf (stale) Mirror 24: 0x7863d0bf (stale) Mirror 26: 0x7863d0bf (stale) Mirror 27: 0x7863d0bf (stale) Mirror 28: 0x7863d0bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7863d0bf (stale) Mirror 15: 0x7863d0bf (stale) Mirror 18: 0x7863d0bf (stale) Mirror 20: 0x7863d0bf (stale) Mirror 21: 0x7863d0bf (stale) Mirror 22: 0x7863d0bf (stale) Mirror 24: 0x7863d0bf (stale) Mirror 26: 0x7863d0bf (stale) Mirror 27: 0x7863d0bf (stale) Mirror 28: 0x7863d0bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000732 1787735047.315615 trunc from 0x040000 to 0x00ac0e (0x353f1 bytes) 000735 1787735047.331218 mapwrite 0x00239f thru 0x004a63 (0x26c5 bytes) 000736 1787735047.438283 trunc from 0x00ac0f to 0x00cdec (0x21de bytes) 000737 1787735047.454119 mapread 0x000f79 thru 0x00cdec (0xbe74 bytes) 000738 1787735047.470296 trunc from 0x00cded to 0x01dad5 (0x10ce9 bytes) 000739 1787735047.493048 write 0x0153d6 thru 0x0232ff (0xdf2a bytes) 000740 1787735047.500626 read 0x01a651 thru 0x0232ff (0x8caf bytes) 000741 1787735047.501470 mapread 0x00d1cc thru 0x01336e (0x61a3 bytes) 000742 1787735047.517703 read 0x021a49 thru 0x0232ff (0x18b7 bytes) 000743 1787735047.520081 read 0x02211b thru 0x0232ff (0x11e5 bytes) 000744 1787735047.521729 trunc from 0x023300 to 0x019bee (0x9711 bytes) 000746 1787735047.563521 write 0x004e2b thru 0x013ca4 (0xee7a bytes) 000747 1787735047.572569 trunc from 0x019bef to 0x017aa1 (0x214d bytes) 000748 1787735047.614931 read 0x00412d thru 0x00b539 (0x740d bytes) 000749 1787735047.618023 write 0x0180fc thru 0x0199ab (0x18b0 bytes) 000750 1787735047.622199 trunc from 0x0199ac to 0x02c871 (0x12ec6 bytes) 000752 1787735047.663418 read 0x01d2cc thru 0x024efe (0x7c33 bytes) 000754 1787735047.688103 read 0x01132d thru 0x01e478 (0xd14c bytes) 000756 1787735047.697305 trunc from 0x02c872 to 0x03103d (0x47cc bytes) 000759 1787735047.754580 mapwrite 0x03b3cb thru 0x03df91 (0x2bc7 bytes) 000760 1787735047.933279 mapwrite 0x0196f0 thru 0x01da29 (0x433a bytes) 000761 1787735048.035343 trunc from 0x03df92 to 0x034d0b (0x9286 bytes) 000762 1787735048.089410 MIRROR_RESYNC 11 mirrors 000764 1787735052.449325 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000765 1787735053.534639 read 0x01c5ae thru 0x01d547 (0xf9a bytes) 000767 1787735053.557795 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) 000769 1787735054.306580 write 0x01524c thru 0x021dbf (0xcb74 bytes) 000776 1787735054.568207 read 0x02bb46 thru 0x030afb (0x4fb6 bytes) 000777 1787735054.586293 mapwrite 0x014771 thru 0x01638a (0x1c1a bytes) 000778 1787735054.795361 mapread 0x029ff5 thru 0x033204 (0x9210 bytes) 000780 1787735054.839758 mapwrite 0x02adc8 thru 0x033ed2 (0x910b bytes) 000781 1787735054.950501 trunc from 0x034d0c to 0x01c9fa (0x18311 bytes) 000782 1787735054.995188 MIRROR_RESYNC 9 mirrors 000784 1787735057.160272 mapread 0x00a2ec thru 0x014dff (0xab14 bytes) 000787 1787735057.202604 trunc from 0x01c9fb to 0x02a9e0 (0xdfe6 bytes) 000788 1787735057.442445 mapwrite 0x015edb thru 0x01a033 (0x4159 bytes) 000790 1787735057.587902 MIRROR_RESYNC 9 mirrors 000793 1787735060.656697 MIRROR_EXTEND to 10 mirrors 000794 1787735062.035124 mapread 0x01563c thru 0x023818 (0xe1dd bytes) 000795 1787735062.059230 MIRROR_EXTEND to 11 mirrors 000797 1787735062.975186 MIRROR_RESYNC 11 mirrors 000799 1787735063.669530 mapread 0x00a487 thru 0x00acb7 (0x831 bytes) 000801 1787735063.675370 mapwrite 0x02b386 thru 0x0345ea (0x9265 bytes) 000802 1787735063.941630 mapwrite 0x032b66 thru 0x0356d1 (0x2b6c bytes) 000803 1787735064.083043 read 0x026d4c thru 0x030073 (0x9328 bytes) 000805 1787735064.097443 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x356d2): Mirror 15: 0x371b0fd5 (stale) Mirror 18: 0x371b0fd5 (stale) Mirror 20: 0x371b0fd5 (stale) Mirror 22: 0x371b0fd5 (stale) Mirror 26: 0x371b0fd5 (stale) Mirror 27: 0x371b0fd5 (stale) Mirror 28: 0x371b0fd5 (stale) Mirror 29: 0x371b0fd5 (stale) Mirror 30: 0x371b0fd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x356d2): Mirror 15: 0x371b0fd5 (stale) Mirror 18: 0x371b0fd5 (stale) Mirror 20: 0x371b0fd5 (stale) Mirror 22: 0x371b0fd5 (stale) Mirror 26: 0x371b0fd5 (stale) Mirror 27: 0x371b0fd5 (stale) Mirror 28: 0x371b0fd5 (stale) Mirror 29: 0x371b0fd5 (stale) Mirror 30: 0x371b0fd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x356d2: skipped 000806 1787735064.896334 mapread 0x01eb65 thru 0x02b5b5 (0xca51 bytes) 000807 1787735064.943485 mapwrite 0x038d32 thru 0x03ffff (0x72ce bytes) 000808 1787735065.048471 read 0x015577 thru 0x01d572 (0x7ffc bytes) 000809 1787735065.059026 trunc from 0x040000 to 0x01536e (0x2ac91 bytes) 000812 1787735065.092361 mapread 0x0095ef thru 0x011715 (0x8127 bytes) 000813 1787735065.105307 mapwrite 0x01da9a thru 0x024334 (0x689b bytes) 000815 1787735065.211903 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24335): Mirror 15: 0x205dffe0 (stale) Mirror 18: 0x205dffe0 (stale) Mirror 22: 0x205dffe0 (stale) Mirror 26: 0x205dffe0 (stale) Mirror 27: 0x205dffe0 (stale) Mirror 28: 0x205dffe0 (stale) Mirror 29: 0x205dffe0 (stale) Mirror 30: 0x205dffe0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24335): Mirror 15: 0x205dffe0 (stale) Mirror 18: 0x205dffe0 (stale) Mirror 22: 0x205dffe0 (stale) Mirror 26: 0x205dffe0 (stale) Mirror 27: 0x205dffe0 (stale) Mirror 28: 0x205dffe0 (stale) Mirror 29: 0x205dffe0 (stale) Mirror 30: 0x205dffe0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24335: skipped 000816 1787735065.851721 trunc from 0x024335 to 0x035afc (0x117c8 bytes) 000821 1787735065.879462 trunc from 0x035afd to 0x006452 (0x2f6aa bytes) 000822 1787735065.902388 mapwrite 0x03f06f thru 0x03ffff (0xf91 bytes) 000823 1787735066.004567 write 0x0116fb thru 0x020693 (0xef99 bytes) 000825 1787735066.031119 write 0x0278c3 thru 0x031c61 (0xa39f bytes) 000826 1787735066.060884 trunc from 0x040000 to 0x037f6c (0x8093 bytes) 000830 1787735066.087576 mapread 0x02025b thru 0x02fb7a (0xf920 bytes) 000832 1787735066.107441 write 0x02cb66 thru 0x02f725 (0x2bc0 bytes) 000833 1787735066.111757 mapwrite 0x03d6be thru 0x03ffff (0x2942 bytes) 000835 1787735066.239320 mapread 0x0183fd thru 0x025e0b (0xda0f bytes) 000836 1787735066.245662 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x129873a (stale) Mirror 22: 0x129873a (stale) Mirror 26: 0x129873a (stale) Mirror 27: 0x129873a (stale) Mirror 28: 0x129873a (stale) Mirror 29: 0x129873a (stale) Mirror 30: 0x129873a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x129873a (stale) Mirror 22: 0x129873a (stale) Mirror 26: 0x129873a (stale) Mirror 27: 0x129873a (stale) Mirror 28: 0x129873a (stale) Mirror 29: 0x129873a (stale) Mirror 30: 0x129873a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000838 1787735066.894573 read 0x026cf5 thru 0x0360e8 (0xf3f4 bytes) 000840 1787735066.907600 mapwrite 0x031abe thru 0x034727 (0x2c6a bytes) 000841 1787735067.002557 read 0x01a169 thru 0x025f1b (0xbdb3 bytes) 000843 1787735067.027380 MIRROR_RESYNC 8 mirrors 000844 1787735071.356619 mapwrite 0x00ee75 thru 0x013ab8 (0x4c44 bytes) 000845 1787735071.610303 read 0x03c71b thru 0x03ffff (0x38e5 bytes) 000846 1787735071.622406 mapwrite 0x02f2ca thru 0x033d7a (0x4ab1 bytes) 000847 1787735071.755713 mapwrite 0x00ccfe thru 0x01c03e (0xf341 bytes) 000848 1787735071.918889 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x34587e45 (stale) Mirror 22: 0x34587e45 (stale) Mirror 26: 0x34587e45 (stale) Mirror 27: 0x34587e45 (stale) Mirror 28: 0x34587e45 (stale) Mirror 29: 0x34587e45 (stale) Mirror 30: 0x34587e45 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc1517972 Mirror 31: 0xc1517972 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000850 1787735073.647163 trunc from 0x040000 to 0x010ef3 (0x2f10c bytes) 000851 1787735073.841522 write 0x036026 thru 0x037634 (0x160f bytes) 000853 1787735073.848564 trunc from 0x037635 to 0x00a419 (0x2d21b bytes) 000855 1787735073.891721 write 0x001696 thru 0x00367c (0x1fe7 bytes) 000856 1787735073.902541 trunc from 0x00a41a to 0x00d09a (0x2c81 bytes) 000857 1787735073.922871 read 0x00ac33 thru 0x00d09a (0x2468 bytes) 000859 1787735073.933793 read 0x0029b5 thru 0x00a115 (0x7761 bytes) 000860 1787735073.964026 mapwrite 0x02a34b thru 0x02b206 (0xebc bytes) 000861 1787735074.112776 trunc from 0x02b207 to 0x00b149 (0x200bd bytes) 000862 1787735074.131243 trunc from 0x00b14a to 0x0084a7 (0x2ca2 bytes) 000863 1787735074.177754 write 0x01f63e thru 0x01f76d (0x130 bytes) 000864 1787735074.179507 mapread 0x001187 thru 0x00d5a8 (0xc422 bytes) 000866 1787735074.271200 mapread 0x01eb46 thru 0x01f76d (0xc28 bytes) 000867 1787735074.283598 read 0x00b362 thru 0x01a0d8 (0xed77 bytes) 000868 1787735074.323970 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1f76e): Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f76e): Mirror 18: 0x3639a35d (stale) Mirror 22: 0x3639a35d (stale) Mirror 26: 0x3639a35d (stale) Mirror 27: 0x3639a35d (stale) Mirror 28: 0x3639a35d (stale) Mirror 29: 0x3639a35d (stale) Mirror 30: 0x3639a35d (stale) Mirror 31: 0x816824d7 (stale) CRC-32 checksum value for chunk [0, 0x1f76e): Mirror 1: 0x5bc8d937 Mirror 32: 0x5bc8d937 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f76e: skipped 000869 1787735075.370406 mapread 0x0006b5 thru 0x0104e5 (0xfe31 bytes) 000870 1787735075.409726 write 0x023a62 thru 0x02d07d (0x961c bytes) 000871 1787735075.473671 read 0x02bebd thru 0x02d07d (0x11c1 bytes) 000872 1787735075.474506 mapread 0x021f09 thru 0x0297b6 (0x78ae bytes) 000874 1787735075.485116 mapwrite 0x01029d thru 0x01301e (0x2d82 bytes) 000875 1787735075.562158 mapwrite 0x00cc87 thru 0x00f533 (0x28ad bytes) 000876 1787735075.619116 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2d07e): Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d07e): Mirror 18: 0x833c1465 (stale) Mirror 22: 0x833c1465 (stale) Mirror 26: 0x833c1465 (stale) Mirror 27: 0x833c1465 (stale) Mirror 28: 0x833c1465 (stale) Mirror 29: 0x833c1465 (stale) Mirror 30: 0x833c1465 (stale) Mirror 31: 0xc819afba (stale) Mirror 32: 0x65a37e4b (stale) CRC-32 checksum value for chunk [0, 0x2d07e): Mirror 1: 0xb4a39e0e Mirror 33: 0xb4a39e0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d07e: skipped 000877 1787735077.363326 trunc from 0x02d07e to 0x002435 (0x2ac48 bytes) 000878 1787735077.479313 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2436): Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2436): Mirror 18: 0xc06f8135 (stale) Mirror 22: 0xc06f8135 (stale) Mirror 26: 0xc06f8135 (stale) Mirror 27: 0xc06f8135 (stale) Mirror 28: 0xc06f8135 (stale) Mirror 29: 0xc06f8135 (stale) Mirror 30: 0xc06f8135 (stale) Mirror 31: 0xc06f8135 (stale) Mirror 32: 0x509acc6a (stale) Mirror 33: 0x509acc6a (stale) CRC-32 checksum value for chunk [0, 0x2436): Mirror 1: 0x509acc6a Mirror 34: 0x509acc6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2436: skipped 000879 1787735078.969262 mapwrite 0x036a09 thru 0x03f746 (0x8d3e bytes) 000880 1787735079.368738 read 0x00c887 thru 0x0106f9 (0x3e73 bytes) 000881 1787735079.402926 MIRROR_RESYNC 12 mirrors 000885 1787735084.724221 mapread 0x00f210 thru 0x011d29 (0x2b1a bytes) 000886 1787735084.766449 read 0x02c777 thru 0x037efd (0xb787 bytes) 000887 1787735084.816916 trunc from 0x03f747 to 0x01f4d3 (0x20273 bytes) 000888 1787735084.934310 mapread 0x013a21 thru 0x015f15 (0x24f5 bytes) 000889 1787735084.948111 write 0x030a46 thru 0x03e029 (0xd5e4 bytes) 000891 1787735084.952070 write 0x0333e5 thru 0x039cbb (0x68d7 bytes) 000893 1787735084.957454 mapwrite 0x00df1c thru 0x0156eb (0x77d0 bytes) 000894 1787735085.108022 trunc from 0x03e02a to 0x0041a7 (0x39e82 bytes) 000895 1787735085.149261 mapread 0x000a72 thru 0x0041a7 (0x3736 bytes) 000897 1787735085.185170 read 0x001ec9 thru 0x0041a7 (0x22df bytes) 000901 1787735085.227082 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x41a8): Mirror 18: 0x180e36a6 (stale) Mirror 22: 0x180e36a6 (stale) Mirror 27: 0x180e36a6 (stale) Mirror 28: 0x180e36a6 (stale) Mirror 29: 0x180e36a6 (stale) Mirror 30: 0x180e36a6 (stale) Mirror 31: 0x180e36a6 (stale) Mirror 32: 0x180e36a6 (stale) Mirror 33: 0x180e36a6 (stale) Mirror 34: 0x180e36a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x41a8): Mirror 18: 0x180e36a6 (stale) Mirror 22: 0x180e36a6 (stale) Mirror 27: 0x180e36a6 (stale) Mirror 28: 0x180e36a6 (stale) Mirror 29: 0x180e36a6 (stale) Mirror 30: 0x180e36a6 (stale) Mirror 31: 0x180e36a6 (stale) Mirror 32: 0x180e36a6 (stale) Mirror 33: 0x180e36a6 (stale) Mirror 34: 0x180e36a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x41a8: skipped 000902 1787735085.973590 mapread 0x002630 thru 0x0041a7 (0x1b78 bytes) 000905 1787735086.008608 mapread 0x0023e3 thru 0x002f06 (0xb24 bytes) 000906 1787735086.010283 read 0x001f8b thru 0x0041a7 (0x221d bytes) 000908 1787735086.021260 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x41a8): Mirror 22: 0x180e36a6 (stale) Mirror 27: 0x180e36a6 (stale) Mirror 28: 0x180e36a6 (stale) Mirror 29: 0x180e36a6 (stale) Mirror 30: 0x180e36a6 (stale) Mirror 31: 0x180e36a6 (stale) Mirror 32: 0x180e36a6 (stale) Mirror 33: 0x180e36a6 (stale) Mirror 34: 0x180e36a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x41a8): Mirror 22: 0x180e36a6 (stale) Mirror 27: 0x180e36a6 (stale) Mirror 28: 0x180e36a6 (stale) Mirror 29: 0x180e36a6 (stale) Mirror 30: 0x180e36a6 (stale) Mirror 31: 0x180e36a6 (stale) Mirror 32: 0x180e36a6 (stale) Mirror 33: 0x180e36a6 (stale) Mirror 34: 0x180e36a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x41a8: skipped 000909 1787735086.802396 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x41a8): Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x41a8): Mirror 22: 0x180e36a6 (stale) Mirror 27: 0x180e36a6 (stale) Mirror 28: 0x180e36a6 (stale) Mirror 29: 0x180e36a6 (stale) Mirror 30: 0x180e36a6 (stale) Mirror 31: 0x180e36a6 (stale) Mirror 32: 0x180e36a6 (stale) Mirror 33: 0x180e36a6 (stale) Mirror 34: 0x180e36a6 (stale) CRC-32 checksum value for chunk [0, 0x41a8): Mirror 1: 0x180e36a6 Mirror 35: 0x180e36a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x41a8: skipped 000912 1787735088.164321 trunc from 0x0041a8 to 0x03b2d9 (0x37132 bytes) 000913 1787735088.269067 mapread 0x012927 thru 0x017941 (0x501b bytes) 000914 1787735088.311722 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3b2da): Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b2da): Mirror 22: 0x642eef0d (stale) Mirror 27: 0x642eef0d (stale) Mirror 28: 0x642eef0d (stale) Mirror 29: 0x642eef0d (stale) Mirror 30: 0x642eef0d (stale) Mirror 31: 0x642eef0d (stale) Mirror 32: 0x642eef0d (stale) Mirror 33: 0x642eef0d (stale) Mirror 34: 0x642eef0d (stale) Mirror 35: 0x4e8d1205 (stale) CRC-32 checksum value for chunk [0, 0x3b2da): Mirror 1: 0x4e8d1205 Mirror 36: 0x4e8d1205 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b2da: skipped 000915 1787735090.171132 write 0x0150c8 thru 0x021dcf (0xcd08 bytes) 000916 1787735090.425527 MIRROR_RESYNC 12 mirrors 000922 1787735095.668920 trunc from 0x03b2da to 0x03870d (0x2bcc bytes) 000923 1787735095.826200 read 0x006244 thru 0x00fcdc (0x9a99 bytes) 000925 1787735095.842316 write 0x020530 thru 0x02d8e1 (0xd3b2 bytes) 000926 1787735095.869312 MIRROR_RESYNC 12 mirrors 000928 1787735100.946862 mapread 0x011251 thru 0x01e194 (0xcf44 bytes) 000930 1787735101.024586 mapread 0x021dc0 thru 0x0271ba (0x53fb bytes) 000931 1787735101.055163 mapwrite 0x02f3ea thru 0x03579e (0x63b5 bytes) 000932 1787735101.317607 read 0x001c29 thru 0x00f66a (0xda42 bytes) 000933 1787735101.331585 trunc from 0x03870e to 0x038c3e (0x531 bytes) 000934 1787735101.370610 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c3f): Mirror 22: 0x45d6e33 (stale) Mirror 27: 0x45d6e33 (stale) Mirror 28: 0x45d6e33 (stale) Mirror 29: 0x45d6e33 (stale) Mirror 30: 0x45d6e33 (stale) Mirror 31: 0x45d6e33 (stale) Mirror 32: 0x45d6e33 (stale) Mirror 34: 0x45d6e33 (stale) Mirror 35: 0x45d6e33 (stale) Mirror 36: 0x45d6e33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c3f): Mirror 22: 0x45d6e33 (stale) Mirror 27: 0x45d6e33 (stale) Mirror 28: 0x45d6e33 (stale) Mirror 29: 0x45d6e33 (stale) Mirror 30: 0x45d6e33 (stale) Mirror 31: 0x45d6e33 (stale) Mirror 32: 0x45d6e33 (stale) Mirror 34: 0x45d6e33 (stale) Mirror 35: 0x45d6e33 (stale) Mirror 36: 0x45d6e33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c3f: skipped 000935 1787735102.190269 mapwrite 0x03394f thru 0x03ffff (0xc6b1 bytes) 000936 1787735102.503319 mapwrite 0x028f6f thru 0x03864b (0xf6dd bytes) 000937 1787735102.592267 read 0x00d296 thru 0x01518d (0x7ef8 bytes) 000940 1787735102.624802 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x609bbf4c (stale) Mirror 27: 0x609bbf4c (stale) Mirror 28: 0x609bbf4c (stale) Mirror 29: 0x609bbf4c (stale) Mirror 30: 0x609bbf4c (stale) Mirror 31: 0x609bbf4c (stale) Mirror 32: 0x609bbf4c (stale) Mirror 34: 0x609bbf4c (stale) Mirror 35: 0x609bbf4c (stale) Mirror 36: 0x609bbf4c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfbc8e41d Mirror 37: 0xfbc8e41d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000941 1787735104.307003 read 0x005f54 thru 0x014628 (0xe6d5 bytes) 000946 1787735104.379664 read 0x02b329 thru 0x02d1a3 (0x1e7b bytes) 000947 1787735104.419088 trunc from 0x040000 to 0x00ca30 (0x335cf bytes) 000948 1787735104.540376 mapwrite 0x011010 thru 0x01bfc8 (0xafb9 bytes) 000949 1787735104.753214 read 0x0028c4 thru 0x0070bc (0x47f9 bytes) 000951 1787735104.772148 write 0x0181c0 thru 0x021084 (0x8ec5 bytes) 000953 1787735104.776147 trunc from 0x021085 to 0x007036 (0x1a04e bytes) 000954 1787735104.798691 mapread 0x0015c1 thru 0x007036 (0x5a76 bytes) 000955 1787735104.800033 write 0x03e41c thru 0x03ffff (0x1be4 bytes) 000957 1787735104.803920 trunc from 0x040000 to 0x0042a1 (0x3bd5e bytes) 000958 1787735104.821307 write 0x024a49 thru 0x02e737 (0x9cef bytes) 000961 1787735104.825338 mapread 0x02572a thru 0x02b110 (0x59e7 bytes) 000962 1787735104.827151 trunc from 0x02e738 to 0x0013a6 (0x2d391 bytes) 000963 1787735104.847276 trunc from 0x0013a7 to 0x010141 (0xed9b bytes) 000964 1787735104.866342 read 0x00ddfc thru 0x010141 (0x2346 bytes) 000965 1787735104.875938 trunc from 0x010142 to 0x03a5be (0x2a47d bytes) 000966 1787735104.923513 write 0x03f9e2 thru 0x03ffff (0x61e bytes) 000969 1787735104.944082 MIRROR_RESYNC 12 mirrors 000970 1787735108.770529 read 0x02c4e2 thru 0x034425 (0x7f44 bytes) 000971 1787735108.794092 MIRROR_EXTEND to 13 mirrors 000972 1787735109.677423 write 0x02f449 thru 0x0341a9 (0x4d61 bytes) 000975 1787735109.759077 mapread 0x01b675 thru 0x01e460 (0x2dec bytes) 000976 1787735109.784530 mapwrite 0x01f8bd thru 0x0243fb (0x4b3f bytes) 000978 1787735109.853221 mapwrite 0x01c539 thru 0x01dc51 (0x1719 bytes) 000981 1787735109.934486 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x44519268 (stale) Mirror 27: 0x44519268 (stale) Mirror 28: 0x44519268 (stale) Mirror 29: 0x44519268 (stale) Mirror 30: 0x44519268 (stale) Mirror 31: 0x44519268 (stale) Mirror 32: 0x44519268 (stale) Mirror 34: 0x44519268 (stale) Mirror 35: 0x44519268 (stale) Mirror 36: 0x44519268 (stale) Mirror 37: 0x44519268 (stale) Mirror 38: 0x44519268 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xec113ede Mirror 39: 0xec113ede lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000983 1787735111.816110 read 0x01947c thru 0x02789c (0xe421 bytes) 000987 1787735111.852498 write 0x017528 thru 0x0271dc (0xfcb5 bytes) 000989 1787735111.914742 write 0x00324a thru 0x004dd6 (0x1b8d bytes) 000993 1787735111.924045 mapread 0x03f7ac thru 0x03ffff (0x854 bytes) 000995 1787735111.937657 MIRROR_RESYNC 14 mirrors 000998 1787735117.591349 trunc from 0x040000 to 0x02255a (0x1daa5 bytes) 001000 1787735117.675241 mapwrite 0x01e1dc thru 0x025d6d (0x7b92 bytes) All operations completed A-OK! PASS 70a (225s) == sanity-flr test 71: check mirror extend parent fid ==== 05:05:22 (1787735122) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 05:05:26 (1787735126) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245992 s, 8.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218102 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 05:05:35 (1787735135) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2375744 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2691008 .. done Extending file size to 6156384 .. Extending file size to 8639680 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8966784 .. Extending file size to 6865696 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7773952 .. Extending file size to 8839424 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8940288 .. Extending file size to 8981152 .. Extending file size to 9149376 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57508 57509 57512 57513 57514 done 192.168.203.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) 192.168.203.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:06:58 (1787735218) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8078272 .. Extending file size to 3124928 .. Extending file size to 8215424 .. Extending file size to 8494624 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8557344 .. Extending file size to 8854880 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9057792 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8968960 .. Extending file size to 9285376 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59127 59128 59129 59132 59133 done 192.168.203.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) 192.168.203.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:08:17 (1787735297) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134115 s, 15.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.462708 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8f71a734 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:08:38 (1787735318) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124218 s, 16.9 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 05:08:45 (1787735325) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:08:51 (1787735331) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199854 s, 10.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:08:59 (1787735339) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197399 s, 10.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:09:07 (1787735347) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191461 s, 11.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:09:39 (1787735379) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg331-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156347 s, 6.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:10:37 (1787735437) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg331-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.258309 s, 8.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:11:32 (1787735492) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg331-server mds-ost sync done. Creating new pool test_204f oleg331-server: Pool lustre.test_204f created Adding targets to pool oleg331-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg331-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138455 s, 15.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg331-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg331-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg331-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:12:48 (1787735568) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:12:55 (1787735575) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:13:01 (1787735581) oleg331-server: Pool lustre.test_206 created Adding targets to pool oleg331-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg331-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg331-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg331-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg331-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:13:22 (1787735602) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598829 s, 17.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603682 s, 17.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x25711c51 Mirror 3: 0x25711c51 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:13:32 (1787735612) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:13:35 (1787735615) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:13:38 (1787735618) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0973283 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2040000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f30000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f30000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f30000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1b00000 Read this (4 bytes): BBBB 192.168.203.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) PASS 209a (31s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:14:09 (1787735649) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00922125 s, 888 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:14:14 (1787735654) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412106 s, 25.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834445 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 05:14:15.000000000 -0400 Modify: 2026-08-26 05:14:15.000000000 -0400 Change: 2026-08-26 05:14:17.000000000 -0400 Birth: 2026-08-26 05:14:16.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:14:21 (1787735661) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:14:24 (1787735664) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.492924 s, 83.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0488641 s, 83.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 05:14 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 05:14 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:14:30 (1787735670) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:14:37 (1787735677) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1817 sec =========== 05:14:45 (1787735685) === sanity-flr: start cleanup 05:14:46 (1787735686) === === sanity-flr: finish cleanup 05:14:52 (1787735692) ===