-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 23:52:09 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg233-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:52:28 (1788839548) === oleg233-client.virtnet: executing check_config_client /mnt/lustre oleg233-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg233-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97f02015a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97f02015a800.idle_timeout=debug disable quota as required oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:53:01 (1788839581) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7638] == sanity-flr test 0a: lfs mirror create with -N option == 23:53:05 (1788839585) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:53:24 (1788839604) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:53:27 (1788839607) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:53:31 (1788839611) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:53:53 (1788839633) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:53:56 (1788839636) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 23:54:00 (1788839640) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff97f02015a800.stats=clear osc.lustre-OST0001-osc-ffff97f02015a800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:54:07 (1788839647) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:54:17 (1788839657) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:54:24 (1788839664) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:54:27 (1788839667) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:54:36 (1788839676) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 23:54:46 (1788839686) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:54:52 (1788839692) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:54:56 (1788839696) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:55:02 (1788839702) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:55:11 (1788839711) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:55:17 (1788839717) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13818 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152443 s, 6.9 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:55:26 (1788839726) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.314867 s, 40.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:55:34 (1788839734) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:55:43 (1788839743) Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97f02015a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97f02015a800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg233-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97f02015a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97f02015a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (127s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:57:50 (1788839870) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111497 s, 18.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97f02015a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97f02015a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (39s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:58:29 (1788839909) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97f02015a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97f02015a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97f02015a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97f02015a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (93s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:00:02 (1788840002) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.94446 s, 13.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 8 00:00 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.099s user 0m0.000s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97f02015a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97f02015a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.048s user 0m0.000s sys 0m0.168s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97f02015a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97f02015a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.118s user 0m0.002s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97f02015a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97f02015a800.ost_server_uuid in FULL state after 0 sec PASS 33b (85s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:01:27 (1788840087) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:01:30 (1788840090) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:01:34 (1788840094) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 00:01:37 (1788840097) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 00:01:44 (1788840104) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.48417 s, 26.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:01:53 (1788840113) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040527 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324984 s, 32.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:02:00 (1788840120) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:02:07 (1788840127) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115633 s, 886 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000732859 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105476 s, 971 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00120985 s, 846 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 00:02:16 (1788840136) create test files with size 15734831 .. 5ab8d058ae706849190b9104530ea50f - 78c9001bc86987487d3e61411be1ecf1 - 83b972342e2f485ab807cb8a4596bfa5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 00:02:37 (1788840157) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:02:53 (1788840173) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:03:01 (1788840181) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168608 s, 6.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730374 s, 14.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 00:03:11 (1788840191) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10274 s, 20.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.172293 s, 24.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254788 s, 8.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 00:03:25 (1788840205) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 00:03:29 (1788840209) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:03:31 (1788840211) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389751 s, 26.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 00:03:39 (1788840219) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:03:43 (1788840223) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:03:53 (1788840233) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.404552 s, 25.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:04:04 (1788840244) llite.lustre-ffff97f005ef9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.288085 s, 36.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff97f011619000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:04:19 (1788840259) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:04:56 (1788840296) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:05:28 (1788840328) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 00:05:31 (1788840331) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.279324 s, 11.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.179003 s, 17.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa76af976 (nosync,stale) Mirror 2: 0xa76af976 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 4dee349981ba184279cf9042175e20f0 - mirror 1 checksum: 10de17bb1ac3d4b5a4aca40b24d062a6 - mirror 2 checksum: 4dee349981ba184279cf9042175e20f0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T00:05:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T00:05:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4dee349981ba184279cf9042175e20f0 - mirror 1 checksum: 10de17bb1ac3d4b5a4aca40b24d062a6 - mirror 2 checksum: 10de17bb1ac3d4b5a4aca40b24d062a6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T00:05:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T00:05:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:05:43 (1788840343) Failing mds1 on oleg233-server Stopping /mnt/lustre-mds1 (opts:) on oleg233-server 00:05:49 (1788840349) shut down facet: mds1 facet_host: oleg233-server facet_failover_host: oleg233-server Failover mds1 to oleg233-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 00:06:12 (1788840372) targets are mounted 00:06:12 (1788840372) facet_failover done oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359286 s, 29.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:06:31 (1788840391) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:06:35 (1788840395) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:06:38 (1788840398) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0138183 s, 1.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:06:45 (1788840405) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:06:48 (1788840408) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00782519 s, 523 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0856723 s, 47.8 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:06:59 (1788840419) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:07:23 (1788840443) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:07:52 (1788840472) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:08:15 (1788840495) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:08:39 (1788840519) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 00:08:46 (1788840526) mirror create pid 54356 mirror split pid 54357 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54356 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54357 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 00:09:54 (1788840594) Seed set to 596 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1788840596.554288 mapwrite 0x0348df thru 0x03b3c0 (0x6ae2 bytes) 000003 1788840596.653038 mapread 0x02b7a8 thru 0x0382c8 (0xcb21 bytes) 000007 1788840596.665899 write 0x02eccf thru 0x03c424 (0xd756 bytes) 000008 1788840596.671181 MIRROR_EXTEND to 2 mirrors 000010 1788840597.431329 write 0x022b82 thru 0x02b471 (0x88f0 bytes) 000013 1788840597.504039 mapwrite 0x02686c thru 0x02697c (0x111 bytes) 000016 1788840597.550677 mapwrite 0x00b908 thru 0x00d212 (0x190b bytes) 000018 1788840597.605665 read 0x0069c8 thru 0x009288 (0x28c1 bytes) 000019 1788840597.607026 read 0x019c27 thru 0x01b0ee (0x14c8 bytes) truncating to largest ever: 0x336a0 000020 1788840597.621712 trunc from 0x03c425 to 0x03369f (0x8d85 bytes) 000022 1788840597.641466 read 0x010319 thru 0x016e8a (0x6b72 bytes) 000024 1788840597.642460 trunc from 0x0336a0 to 0x02f29d (0x4402 bytes) truncating to largest ever: 0x3dde9 000025 1788840597.657668 trunc from 0x02f29e to 0x03dde8 (0xeb4b bytes) 000026 1788840597.671327 mapread 0x00e40d thru 0x015b99 (0x778d bytes) 000028 1788840597.673282 mapwrite 0x0366fb thru 0x037cdb (0x15e1 bytes) 000029 1788840597.703521 write 0x037cde thru 0x03ffff (0x8322 bytes) 000030 1788840597.707434 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb334200c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe34966d8 Mirror 3: 0xe34966d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000032 1788840598.788352 trunc from 0x040000 to 0x02fdf0 (0x1020f bytes) truncating to largest ever: 0x3ed44 000033 1788840598.874098 trunc from 0x02fdf1 to 0x03ed43 (0xef53 bytes) 000034 1788840598.900189 mapwrite 0x010dab thru 0x01e7d9 (0xda2f bytes) 000035 1788840598.974155 write 0x002ed9 thru 0x004fc8 (0x20f0 bytes) 000036 1788840598.984004 write 0x00569a thru 0x011f2d (0xc894 bytes) 000039 1788840598.990551 read 0x03983b thru 0x03ed43 (0x5509 bytes) 000041 1788840599.000536 trunc from 0x03ed44 to 0x03015b (0xebe8 bytes) 000042 1788840599.018180 mapwrite 0x02c36e thru 0x037a25 (0xb6b8 bytes) 000043 1788840599.147053 write 0x0261ea thru 0x02db8b (0x79a2 bytes) 000045 1788840599.153760 write 0x0049fd thru 0x0144ec (0xfaf0 bytes) 000047 1788840599.170263 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37a26): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a26): Mirror 2: 0x2bccb8d3 (stale) Mirror 3: 0xa6f651b6 (stale) CRC-32 checksum value for chunk [0, 0x37a26): Mirror 1: 0x7e869391 Mirror 4: 0x7e869391 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a26: skipped 000048 1788840600.093285 mapread 0x007622 thru 0x012272 (0xac51 bytes) 000050 1788840600.134845 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37a26): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a26): Mirror 2: 0x2bccb8d3 (stale) Mirror 3: 0xa6f651b6 (stale) CRC-32 checksum value for chunk [0, 0x37a26): Mirror 1: 0x7e869391 Mirror 4: 0x7e869391 Mirror 5: 0x7e869391 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a26: skipped 000051 1788840601.102084 read 0x012c1e thru 0x015332 (0x2715 bytes) 000052 1788840601.107326 punch from 0x029b86 to 0x02bd25 (0x21a0 bytes) 000054 1788840601.182688 write 0x00b07e thru 0x012d69 (0x7cec bytes) 000055 1788840601.197392 mapwrite 0x006066 thru 0x009ec7 (0x3e62 bytes) 000057 1788840601.261797 trunc from 0x037a26 to 0x0005fe (0x37427 bytes) 000058 1788840601.306809 trunc from 0x0005ff to 0x0283f0 (0x27df2 bytes) 000060 1788840601.325042 read 0x0066b4 thru 0x015b39 (0xf486 bytes) 000062 1788840601.339166 mapwrite 0x018a74 thru 0x024eeb (0xc478 bytes) 000063 1788840601.389807 read 0x00439c thru 0x006654 (0x22b9 bytes) 000064 1788840601.391606 trunc from 0x0283f1 to 0x039a0a (0x1161a bytes) 000065 1788840601.411522 trunc from 0x039a0b to 0x00391f (0x360eb bytes) 000066 1788840601.424701 read 0x001a0e thru 0x00391f (0x1f12 bytes) 000067 1788840601.430594 MIRROR_RESYNC 5 mirrors 000068 1788840602.223286 write 0x000b7b thru 0x007c6b (0x70f1 bytes) 000069 1788840602.297789 punch from 0x00045f to 0x0033db (0x2f7d bytes) 000070 1788840602.314533 read 0x003692 thru 0x007c6b (0x45da bytes) 000072 1788840602.321778 read 0x0033f7 thru 0x007c6b (0x4875 bytes) 000073 1788840602.322298 write 0x021436 thru 0x029bdc (0x87a7 bytes) 000075 1788840602.323996 punch from 0x0114a9 to 0x01b620 (0xa178 bytes) 000076 1788840602.334615 mapread 0x00d376 thru 0x013302 (0x5f8d bytes) 000078 1788840602.344897 mapwrite 0x0136f5 thru 0x01a16f (0x6a7b bytes) 000080 1788840602.390429 mapwrite 0x02d5bb thru 0x0397cd (0xc213 bytes) 000081 1788840602.497699 write 0x008d75 thru 0x00ae6d (0x20f9 bytes) 000083 1788840602.499740 mapwrite 0x000498 thru 0x007d1d (0x7886 bytes) 000084 1788840602.564187 write 0x0077bc thru 0x01351e (0xbd63 bytes) 000085 1788840602.567238 trunc from 0x0397ce to 0x011782 (0x2804b bytes) 000086 1788840602.590815 punch from 0x00f6a2 to 0x011782 (0x20e1 bytes) 000087 1788840602.612657 MIRROR_RESYNC 5 mirrors 000088 1788840603.765120 read 0x00eab1 thru 0x011782 (0x2cd2 bytes) 000089 1788840603.770607 punch from 0x010777 to 0x011782 (0x100c bytes) 000090 1788840603.832080 write 0x02aa21 thru 0x02e393 (0x3973 bytes) 000091 1788840603.833300 mapread 0x023c33 thru 0x02e393 (0xa761 bytes) 000092 1788840603.849874 mapwrite 0x0362d3 thru 0x03926c (0x2f9a bytes) 000094 1788840603.929366 mapwrite 0x02325f thru 0x02cc03 (0x99a5 bytes) 000095 1788840603.977576 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3926d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3926d): Mirror 2: 0x7522502a (stale) Mirror 3: 0x7522502a (stale) Mirror 4: 0x7522502a (stale) Mirror 5: 0x7522502a (stale) CRC-32 checksum value for chunk [0, 0x3926d): Mirror 1: 0xa37b32ac Mirror 6: 0xa37b32ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3926d: skipped 000096 1788840604.985487 MIRROR_RESYNC 6 mirrors 000098 1788840606.139105 read 0x033029 thru 0x03926c (0x6244 bytes) 000099 1788840606.149265 read 0x0147f8 thru 0x020a7a (0xc283 bytes) 000101 1788840606.159550 trunc from 0x03926d to 0x023222 (0x1604a bytes) 000102 1788840606.376317 read 0x006e52 thru 0x00b1f9 (0x43a8 bytes) 000103 1788840606.394959 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23223): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23223): Mirror 2: 0x9b8e7289 (stale) Mirror 3: 0x9b8e7289 (stale) Mirror 4: 0x9b8e7289 (stale) Mirror 5: 0x9b8e7289 (stale) Mirror 6: 0x9b8e7289 (stale) CRC-32 checksum value for chunk [0, 0x23223): Mirror 1: 0x9b8e7289 Mirror 7: 0x9b8e7289 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23223: skipped 000105 1788840607.269518 punch from 0x014b3f to 0x023222 (0xe6e4 bytes) 000108 1788840607.335646 read 0x00c272 thru 0x00f6c3 (0x3452 bytes) 000109 1788840607.342630 mapwrite 0x0183c3 thru 0x0234bd (0xb0fb bytes) 000110 1788840607.463032 read 0x008f36 thru 0x013241 (0xa30c bytes) 000111 1788840607.472402 write 0x03abff thru 0x03b82f (0xc31 bytes) 000112 1788840607.473389 mapread 0x01aae7 thru 0x0275c5 (0xcadf bytes) 000114 1788840607.483761 mapread 0x00e4aa thru 0x01968a (0xb1e1 bytes) 000116 1788840607.485349 trunc from 0x03b830 to 0x025e1e (0x15a11 bytes) 000117 1788840607.505364 read 0x01f19a thru 0x025e1e (0x6c85 bytes) 000119 1788840607.509450 MIRROR_RESYNC 7 mirrors 000120 1788840608.913061 mapread 0x02253d thru 0x025e1e (0x38e2 bytes) 000122 1788840608.929834 trunc from 0x025e1f to 0x027941 (0x1b23 bytes) 000123 1788840609.026961 trunc from 0x027942 to 0x0130e2 (0x1485f bytes) 000124 1788840609.044248 trunc from 0x0130e3 to 0x017261 (0x417f bytes) 000125 1788840609.071300 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x17262): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17262): Mirror 2: 0x1333cf33 (stale) Mirror 3: 0x1333cf33 (stale) Mirror 4: 0x1333cf33 (stale) Mirror 5: 0x1333cf33 (stale) Mirror 6: 0x1333cf33 (stale) Mirror 7: 0x1333cf33 (stale) CRC-32 checksum value for chunk [0, 0x17262): Mirror 1: 0x1333cf33 Mirror 8: 0x1333cf33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17262: skipped 000131 1788840610.091628 write 0x00857e thru 0x01459e (0xc021 bytes) 000133 1788840610.179560 mapwrite 0x037470 thru 0x037b74 (0x705 bytes) 000134 1788840610.254738 write 0x023270 thru 0x02d645 (0xa3d6 bytes) 000135 1788840610.264468 write 0x0262b6 thru 0x027a3e (0x1789 bytes) 000136 1788840610.265582 read 0x02b13f thru 0x031a54 (0x6916 bytes) 000137 1788840610.272097 read 0x015063 thru 0x01ce33 (0x7dd1 bytes) 000139 1788840610.284474 read 0x020423 thru 0x024a2e (0x460c bytes) 000140 1788840610.285193 mapwrite 0x03c2ff thru 0x03ffff (0x3d01 bytes) 000142 1788840610.350427 mapread 0x023c0f thru 0x02d88f (0x9c81 bytes) 000143 1788840610.351790 mapwrite 0x00e6d2 thru 0x015c0b (0x753a bytes) 000146 1788840610.397860 trunc from 0x040000 to 0x00a35a (0x35ca5 bytes) 000147 1788840610.418734 write 0x024793 thru 0x025286 (0xaf4 bytes) 000148 1788840610.419730 read 0x0238e6 thru 0x025286 (0x19a1 bytes) 000149 1788840610.427395 read 0x0124c2 thru 0x017850 (0x538f bytes) 000150 1788840610.428081 mapwrite 0x00ffaa thru 0x010153 (0x1aa bytes) 000151 1788840610.466698 write 0x00e07c thru 0x010fd6 (0x2f5b bytes) 000152 1788840610.467737 read 0x012668 thru 0x01a487 (0x7e20 bytes) 000156 1788840610.470798 punch from 0x021f6a to 0x025286 (0x331d bytes) 000157 1788840610.483449 MIRROR_RESYNC 8 mirrors 000158 1788840612.068619 punch from 0x0027c2 to 0x0081c6 (0x5a05 bytes) 000159 1788840612.143856 trunc from 0x025287 to 0x01c495 (0x8df1 bytes) 000161 1788840612.186224 mapread 0x01b73a thru 0x01c495 (0xd5c bytes) 000163 1788840612.200378 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1c496): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c496): Mirror 2: 0xe7de18a3 (stale) Mirror 3: 0xe7de18a3 (stale) Mirror 4: 0xe7de18a3 (stale) Mirror 5: 0xe7de18a3 (stale) Mirror 6: 0xe7de18a3 (stale) Mirror 7: 0xe7de18a3 (stale) Mirror 8: 0xe7de18a3 (stale) CRC-32 checksum value for chunk [0, 0x1c496): Mirror 1: 0x800dcc13 Mirror 9: 0x800dcc13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c496: skipped 000164 1788840613.105390 write 0x01829f thru 0x01f6b1 (0x7413 bytes) 000167 1788840613.254432 read 0x0008a3 thru 0x0029d4 (0x2132 bytes) 000168 1788840613.261680 mapread 0x00c0fc thru 0x00ff5b (0x3e60 bytes) 000169 1788840613.298529 punch from 0x01e3f8 to 0x01f6b1 (0x12ba bytes) 000170 1788840613.324714 write 0x0053a9 thru 0x0078a3 (0x24fb bytes) 000171 1788840613.332731 trunc from 0x01f6b2 to 0x03be5a (0x1c7a9 bytes) 000173 1788840613.390612 mapwrite 0x017953 thru 0x0218d7 (0x9f85 bytes) 000174 1788840613.475388 MIRROR_RESYNC 9 mirrors 000175 1788840615.456084 trunc from 0x03be5b to 0x03073f (0xb71b bytes) 000176 1788840615.551806 punch from 0x017b91 to 0x02292f (0xad9f bytes) 000177 1788840615.556022 punch from 0x019112 to 0x019b08 (0x9f7 bytes) 000178 1788840615.559835 mapwrite 0x02ed15 thru 0x0309d8 (0x1cc4 bytes) 000179 1788840615.633304 punch from 0x014d50 to 0x01927c (0x452d bytes) 000180 1788840615.638800 mapread 0x02cc16 thru 0x0309d8 (0x3dc3 bytes) 000182 1788840615.640390 trunc from 0x0309d9 to 0x01c2c9 (0x1470f bytes) 000184 1788840615.654035 trunc from 0x01c2ca to 0x0024e6 (0x19de3 bytes) 000186 1788840615.667945 mapwrite 0x014e71 thru 0x017676 (0x2806 bytes) 000188 1788840615.723496 mapwrite 0x024efd thru 0x0291e1 (0x42e5 bytes) 000189 1788840615.776995 mapwrite 0x02fe66 thru 0x030315 (0x4b0 bytes) 000190 1788840615.837197 trunc from 0x030316 to 0x00dcaa (0x2266b bytes) 000195 1788840615.863385 trunc from 0x00dcab to 0x03e8bb (0x30c11 bytes) 000199 1788840615.885469 MIRROR_RESYNC 9 mirrors 000200 1788840617.419856 mapread 0x03767c thru 0x03a3b1 (0x2d36 bytes) 000201 1788840617.438427 read 0x009f0f thru 0x01738a (0xd47c bytes) 000202 1788840617.446553 punch from 0x002e1c to 0x0030c6 (0x2ab bytes) 000203 1788840617.511706 mapwrite 0x0016f0 thru 0x00f0b2 (0xd9c3 bytes) 000207 1788840617.609316 read 0x00bca6 thru 0x01436f (0x86ca bytes) 000208 1788840617.634119 mapwrite 0x03deb7 thru 0x03ffff (0x2149 bytes) 000210 1788840617.804612 write 0x0259a3 thru 0x02dbab (0x8209 bytes) 000211 1788840617.810118 punch from 0x02fdb8 to 0x03debb (0xe104 bytes) 000213 1788840617.859436 mapread 0x02557b thru 0x0287be (0x3244 bytes) 000214 1788840617.860969 mapwrite 0x032097 thru 0x034d5d (0x2cc7 bytes) 000215 1788840617.977689 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7d7cd6a3 (stale) Mirror 4: 0x7d7cd6a3 (stale) Mirror 5: 0x7d7cd6a3 (stale) Mirror 6: 0x7d7cd6a3 (stale) Mirror 7: 0x7d7cd6a3 (stale) Mirror 8: 0x7d7cd6a3 (stale) Mirror 9: 0x7d7cd6a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7d7cd6a3 (stale) Mirror 4: 0x7d7cd6a3 (stale) Mirror 5: 0x7d7cd6a3 (stale) Mirror 6: 0x7d7cd6a3 (stale) Mirror 7: 0x7d7cd6a3 (stale) Mirror 8: 0x7d7cd6a3 (stale) Mirror 9: 0x7d7cd6a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000216 1788840618.573609 punch from 0x011e81 to 0x012f65 (0x10e5 bytes) 000217 1788840618.587115 MIRROR_RESYNC 8 mirrors 000218 1788840620.157629 read 0x03861c thru 0x03ffff (0x79e4 bytes) 000221 1788840620.171523 write 0x0138fe thru 0x0238ee (0xfff1 bytes) 000223 1788840620.246551 read 0x00b476 thru 0x01127d (0x5e08 bytes) 000224 1788840620.256140 mapwrite 0x00a263 thru 0x0187c8 (0xe566 bytes) 000225 1788840620.297993 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xcfc9ca7c (stale) Mirror 5: 0xcfc9ca7c (stale) Mirror 6: 0xcfc9ca7c (stale) Mirror 7: 0xcfc9ca7c (stale) Mirror 8: 0xcfc9ca7c (stale) Mirror 9: 0xcfc9ca7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xcfc9ca7c (stale) Mirror 5: 0xcfc9ca7c (stale) Mirror 6: 0xcfc9ca7c (stale) Mirror 7: 0xcfc9ca7c (stale) Mirror 8: 0xcfc9ca7c (stale) Mirror 9: 0xcfc9ca7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000226 1788840620.744430 read 0x01f788 thru 0x0289a4 (0x921d bytes) 000227 1788840620.753642 punch from 0x022677 to 0x02da4f (0xb3d9 bytes) 000228 1788840620.763189 mapwrite 0x00e8d8 thru 0x01a42e (0xbb57 bytes) 000229 1788840620.896710 mapwrite 0x001aba thru 0x0079d9 (0x5f20 bytes) 000230 1788840620.986633 trunc from 0x040000 to 0x028a7e (0x17581 bytes) 000232 1788840621.003370 punch from 0x01f4a0 to 0x022bf2 (0x3753 bytes) 000235 1788840621.019235 punch from 0x011fab to 0x01ced5 (0xaf2b bytes) 000238 1788840621.028336 write 0x01a0ab thru 0x01c61d (0x2573 bytes) 000240 1788840621.040552 read 0x0187ee thru 0x0233ac (0xabbf bytes) 000241 1788840621.051259 mapwrite 0x011c97 thru 0x0219ea (0xfd54 bytes) 000242 1788840621.103798 punch from 0x00d69c to 0x01722c (0x9b91 bytes) 000244 1788840621.114535 write 0x03afd4 thru 0x03ffff (0x502c bytes) 000245 1788840621.120356 trunc from 0x040000 to 0x019ba3 (0x2645c bytes) 000246 1788840621.161124 punch from 0x018da4 to 0x019ba3 (0xe00 bytes) 000247 1788840621.169806 trunc from 0x019ba4 to 0x01811d (0x1a86 bytes) 000248 1788840621.190479 read 0x0067ec thru 0x009d55 (0x356a bytes) 000249 1788840621.191126 punch from 0x0159cd to 0x01811d (0x2751 bytes) 000251 1788840621.200696 punch from 0x003216 to 0x00abf5 (0x79e0 bytes) 000252 1788840621.211169 read 0x009695 thru 0x00e624 (0x4f90 bytes) 000253 1788840621.218796 mapwrite 0x0298a7 thru 0x02ad65 (0x14bf bytes) 000254 1788840621.350498 mapwrite 0x008556 thru 0x017812 (0xf2bd bytes) 000255 1788840621.452446 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ad66): Mirror 4: 0xc6510ac0 (stale) Mirror 5: 0xc6510ac0 (stale) Mirror 6: 0xc6510ac0 (stale) Mirror 8: 0xc6510ac0 (stale) Mirror 9: 0xc6510ac0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad66): Mirror 4: 0xc6510ac0 (stale) Mirror 5: 0xc6510ac0 (stale) Mirror 6: 0xc6510ac0 (stale) Mirror 8: 0xc6510ac0 (stale) Mirror 9: 0xc6510ac0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad66: skipped 000256 1788840621.806401 write 0x039b4c thru 0x03bc70 (0x2125 bytes) 000257 1788840621.809196 mapwrite 0x028950 thru 0x02fbca (0x727b bytes) 000259 1788840621.854127 read 0x009c4e thru 0x0116c4 (0x7a77 bytes) 000260 1788840621.859062 trunc from 0x03bc71 to 0x03d981 (0x1d11 bytes) 000261 1788840621.868849 mapwrite 0x033e85 thru 0x03f99f (0xbb1b bytes) 000262 1788840621.971056 MIRROR_RESYNC 6 mirrors 000264 1788840623.326257 write 0x038217 thru 0x03ffff (0x7de9 bytes) 000265 1788840623.414129 read 0x026374 thru 0x02b6c9 (0x5356 bytes) 000267 1788840623.432398 mapread 0x01b7d7 thru 0x023d59 (0x8583 bytes) 000269 1788840623.457176 write 0x02c9e2 thru 0x03c557 (0xfb76 bytes) 000270 1788840623.465389 punch from 0x007acd to 0x0168fd (0xee31 bytes) 000274 1788840623.494387 mapread 0x03ef96 thru 0x03ffff (0x106a bytes) 000275 1788840623.496518 read 0x025c60 thru 0x02eea5 (0x9246 bytes) 000276 1788840623.497467 read 0x023295 thru 0x029271 (0x5fdd bytes) 000278 1788840623.501937 trunc from 0x040000 to 0x00df39 (0x320c6 bytes) 000279 1788840623.547307 trunc from 0x00df3a to 0x01d464 (0xf52b bytes) 000280 1788840623.591929 write 0x036aa1 thru 0x03ffff (0x955f bytes) 000281 1788840623.597529 trunc from 0x040000 to 0x037266 (0x8d99 bytes) 000282 1788840623.615246 mapread 0x01ede5 thru 0x02e2c0 (0xf4dc bytes) 000285 1788840623.628171 mapread 0x02edc9 thru 0x02face (0xd06 bytes) 000286 1788840623.630505 write 0x017993 thru 0x02602c (0xe69a bytes) 000287 1788840623.633669 trunc from 0x037267 to 0x03e43a (0x71d4 bytes) 000288 1788840623.652507 mapwrite 0x00253d thru 0x006a1d (0x44e1 bytes) 000289 1788840623.699271 write 0x007d49 thru 0x01052f (0x87e7 bytes) 000290 1788840623.700427 read 0x03933d thru 0x03e43a (0x50fe bytes) 000292 1788840623.707963 write 0x00615e thru 0x009d15 (0x3bb8 bytes) 000293 1788840623.710245 trunc from 0x03e43b to 0x011f62 (0x2c4d8 bytes) 000294 1788840623.725844 mapwrite 0x03a89b thru 0x03ffff (0x5765 bytes) 000295 1788840623.809004 write 0x0314fe thru 0x03fa88 (0xe58b bytes) 000296 1788840623.814518 mapwrite 0x025215 thru 0x02d49b (0x8287 bytes) 000298 1788840623.910339 trunc from 0x040000 to 0x03e186 (0x1e79 bytes) 000299 1788840623.934303 punch from 0x00d3cc to 0x01af61 (0xdb96 bytes) 000300 1788840623.940889 mapread 0x026a5c thru 0x02c09f (0x5644 bytes) 000302 1788840623.942234 punch from 0x010f48 to 0x01991a (0x89d3 bytes) 000303 1788840623.946977 mapwrite 0x028e1e thru 0x02dc1e (0x4e01 bytes) 000304 1788840624.009380 trunc from 0x03e187 to 0x027501 (0x16c85 bytes) 000307 1788840624.048893 MIRROR_RESYNC 6 mirrors 000309 1788840625.575646 mapread 0x00dfbe thru 0x018271 (0xa2b4 bytes) 000310 1788840625.600609 punch from 0x009d6a to 0x0166e7 (0xc97e bytes) truncating to largest ever: 0x3f17a 000311 1788840625.665424 trunc from 0x027502 to 0x03f179 (0x17c78 bytes) 000312 1788840625.682725 MIRROR_RESYNC 6 mirrors 000314 1788840627.472855 punch from 0x02fa3a to 0x031d2a (0x22f1 bytes) 000315 1788840627.539472 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f17a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f17a): Mirror 4: 0xe70b033 (stale) Mirror 5: 0xe70b033 (stale) Mirror 6: 0xe70b033 (stale) Mirror 8: 0xe70b033 (stale) Mirror 9: 0xe70b033 (stale) CRC-32 checksum value for chunk [0, 0x3f17a): Mirror 1: 0xe70b033 Mirror 10: 0xe70b033 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f17a: skipped 000316 1788840628.547517 mapread 0x02f562 thru 0x03f179 (0xfc18 bytes) 000317 1788840628.589230 mapread 0x03b4fa thru 0x03f179 (0x3c80 bytes) 000318 1788840628.604511 punch from 0x007577 to 0x0128e7 (0xb371 bytes) 000319 1788840628.768125 mapread 0x02d036 thru 0x02ef8b (0x1f56 bytes) 000320 1788840628.780786 mapwrite 0x03f0f9 thru 0x03ffff (0xf07 bytes) 000321 1788840628.911386 read 0x004d0c thru 0x00a687 (0x597c bytes) 000322 1788840628.920265 read 0x0254e8 thru 0x0323b1 (0xceca bytes) 000324 1788840628.932806 mapread 0x011b0c thru 0x016b97 (0x508c bytes) 000325 1788840628.934683 punch from 0x0116f7 to 0x015346 (0x3c50 bytes) 000326 1788840628.939783 mapread 0x03e99a thru 0x03ffff (0x1666 bytes) 000327 1788840628.945837 punch from 0x000f1c to 0x009fd6 (0x90bb bytes) 000328 1788840628.953555 mapwrite 0x03b3ed thru 0x03ffff (0x4c13 bytes) 000330 1788840629.007607 mapwrite 0x00676d thru 0x00b502 (0x4d96 bytes) 000331 1788840629.095393 mapread 0x01d538 thru 0x01ede6 (0x18af bytes) 000332 1788840629.096566 trunc from 0x040000 to 0x0332c0 (0xcd3f bytes) 000335 1788840629.115307 read 0x02a774 thru 0x02e720 (0x3fad bytes) 000338 1788840629.117426 write 0x01a59a thru 0x01a8bd (0x324 bytes) 000339 1788840629.119271 trunc from 0x0332c1 to 0x0258d2 (0xd9ee bytes) 000340 1788840629.132310 punch from 0x020a75 to 0x022b6b (0x20f7 bytes) 000341 1788840629.140617 read 0x0168c2 thru 0x01f4f2 (0x8c31 bytes) 000342 1788840629.141380 mapwrite 0x03f546 thru 0x03ffff (0xaba bytes) 000344 1788840629.206543 punch from 0x00d922 to 0x0187ce (0xaead bytes) 000345 1788840629.212206 mapread 0x02a442 thru 0x02ff47 (0x5b06 bytes) 000347 1788840629.213732 punch from 0x011cc0 to 0x013ef2 (0x2233 bytes) 000348 1788840629.218735 mapwrite 0x006973 thru 0x016763 (0xfdf1 bytes) truncating to largest ever: 0x3faf2 000349 1788840629.301351 trunc from 0x040000 to 0x03faf1 (0x50e bytes) 000351 1788840629.315652 mapread 0x003714 thru 0x00f874 (0xc161 bytes) 000352 1788840629.316962 mapread 0x00923f thru 0x016291 (0xd053 bytes) 000353 1788840629.322679 MIRROR_RESYNC 7 mirrors 000356 1788840631.166654 trunc from 0x03faf2 to 0x00a8f8 (0x351f9 bytes) 000357 1788840631.270869 mapread 0x009309 thru 0x00a8f8 (0x15f0 bytes) 000358 1788840631.281233 trunc from 0x00a8f9 to 0x0383b5 (0x2dabd bytes) 000359 1788840631.309198 MIRROR_RESYNC 7 mirrors 000360 1788840632.658840 trunc from 0x0383b6 to 0x004347 (0x3406e bytes) 000361 1788840632.736286 MIRROR_RESYNC 7 mirrors 000362 1788840633.807836 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) 000365 1788840634.072789 mapwrite 0x01870d thru 0x01ca1c (0x4310 bytes) 000366 1788840634.248961 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ca1d): Mirror 4: 0xa5ea4616 (stale) Mirror 5: 0xa5ea4616 (stale) Mirror 6: 0xa5ea4616 (stale) Mirror 9: 0xa5ea4616 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca1d): Mirror 4: 0xa5ea4616 (stale) Mirror 5: 0xa5ea4616 (stale) Mirror 6: 0xa5ea4616 (stale) Mirror 9: 0xa5ea4616 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca1d: skipped 000368 1788840634.555397 punch from 0x006f4f to 0x013a63 (0xcb15 bytes) 000369 1788840634.560134 read 0x003dc3 thru 0x008eb9 (0x50f7 bytes) 000370 1788840634.568247 mapwrite 0x03e554 thru 0x03ffff (0x1aac bytes) 000371 1788840634.627305 mapread 0x02bfe1 thru 0x035d30 (0x9d50 bytes) 000372 1788840634.647161 write 0x0055e4 thru 0x0091ea (0x3c07 bytes) 000375 1788840634.658221 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde78ab7b (stale) Mirror 5: 0xde78ab7b (stale) Mirror 6: 0xde78ab7b (stale) Mirror 9: 0xde78ab7b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1621349a Mirror 10: 0x1621349a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000376 1788840635.558760 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde78ab7b (stale) Mirror 5: 0xde78ab7b (stale) Mirror 6: 0xde78ab7b (stale) Mirror 9: 0xde78ab7b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1621349a Mirror 10: 0x1621349a Mirror 11: 0x1621349a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000379 1788840636.373105 trunc from 0x040000 to 0x01f7d6 (0x20829 bytes) 000384 1788840636.469234 mapwrite 0x012156 thru 0x016162 (0x400d bytes) 000386 1788840636.531243 punch from 0x01b36c to 0x01f7d6 (0x446b bytes) 000391 1788840636.538793 punch from 0x00512d to 0x00e153 (0x9027 bytes) 000392 1788840636.542574 read 0x00331f thru 0x012d0a (0xf9ec bytes) 000394 1788840636.556811 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f7d7): Mirror 4: 0x1db9152c (stale) Mirror 5: 0x1db9152c (stale) Mirror 6: 0x1db9152c (stale) Mirror 10: 0x70bf0c6b (stale) Mirror 11: 0x70bf0c6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f7d7): Mirror 4: 0x1db9152c (stale) Mirror 5: 0x1db9152c (stale) Mirror 6: 0x1db9152c (stale) Mirror 10: 0x70bf0c6b (stale) Mirror 11: 0x70bf0c6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f7d7: skipped 000396 1788840636.872424 punch from 0x019c3c to 0x01f7d6 (0x5b9b bytes) 000397 1788840636.877319 mapread 0x01661b thru 0x01f7d6 (0x91bc bytes) 000398 1788840636.897088 mapwrite 0x007762 thru 0x0154cd (0xdd6c bytes) 000399 1788840636.944594 punch from 0x00c4c5 to 0x018b00 (0xc63c bytes) 000400 1788840636.949268 mapread 0x00247f thru 0x00505f (0x2be1 bytes) 000402 1788840636.952221 mapread 0x001522 thru 0x010ffe (0xfadd bytes) 000403 1788840636.959775 write 0x029bcf thru 0x02b1f8 (0x162a bytes) 000406 1788840636.966340 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b1f9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1f9): Mirror 4: 0xb44608d5 (stale) Mirror 5: 0xb44608d5 (stale) Mirror 6: 0xb44608d5 (stale) Mirror 10: 0x42141ac0 (stale) Mirror 11: 0x42141ac0 (stale) CRC-32 checksum value for chunk [0, 0x2b1f9): Mirror 1: 0xc6b798df Mirror 12: 0xc6b798df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1f9: skipped 000407 1788840638.263553 mapwrite 0x000d3a thru 0x004f6d (0x4234 bytes) 000412 1788840638.536876 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b1f9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1f9): Mirror 4: 0xb44608d5 (stale) Mirror 5: 0xb44608d5 (stale) Mirror 6: 0xb44608d5 (stale) Mirror 10: 0x42141ac0 (stale) Mirror 11: 0x42141ac0 (stale) Mirror 12: 0xc6b798df (stale) CRC-32 checksum value for chunk [0, 0x2b1f9): Mirror 1: 0x43c25cf0 Mirror 13: 0x43c25cf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1f9: skipped 000413 1788840639.892508 punch from 0x0245b4 to 0x02b1f8 (0x6c45 bytes) 000414 1788840639.968278 punch from 0x000e05 to 0x00a4e9 (0x96e5 bytes) 000415 1788840639.975267 mapread 0x01908d thru 0x0281f2 (0xf166 bytes) 000416 1788840640.012121 mapread 0x0092e9 thru 0x00bd51 (0x2a69 bytes) 000417 1788840640.021629 punch from 0x0065cc to 0x0131eb (0xcc20 bytes) 000418 1788840640.028051 mapwrite 0x0390ad thru 0x03ffff (0x6f53 bytes) 000419 1788840640.105303 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde78ab7b (stale) Mirror 5: 0xde78ab7b (stale) Mirror 6: 0xde78ab7b (stale) Mirror 10: 0x1621349a (stale) Mirror 11: 0x1621349a (stale) Mirror 12: 0x473e6dea (stale) Mirror 13: 0x9c69a48e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc42e0077 Mirror 14: 0xc42e0077 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000420 1788840641.297629 punch from 0x02da16 to 0x02db99 (0x184 bytes) 000424 1788840641.378610 mapwrite 0x03c301 thru 0x03dc8f (0x198f bytes) 000425 1788840641.434034 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xde78ab7b (stale) Mirror 5: 0xde78ab7b (stale) Mirror 6: 0xde78ab7b (stale) Mirror 10: 0x1621349a (stale) Mirror 11: 0x1621349a (stale) Mirror 12: 0x473e6dea (stale) Mirror 13: 0x9c69a48e (stale) Mirror 14: 0xc42e0077 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x16ac1488 Mirror 15: 0x16ac1488 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000427 1788840642.644568 mapwrite 0x00aa44 thru 0x019507 (0xeac4 bytes) 000428 1788840642.886874 mapread 0x03a949 thru 0x03ffff (0x56b7 bytes) 000432 1788840642.908682 punch from 0x037e74 to 0x03a07c (0x2209 bytes) 000433 1788840642.918558 read 0x02a35e thru 0x033665 (0x9308 bytes) 000435 1788840642.948178 write 0x00714b thru 0x014f39 (0xddef bytes) 000436 1788840642.953639 mapwrite 0x011ff1 thru 0x01f890 (0xd8a0 bytes) 000437 1788840643.044868 read 0x00f856 thru 0x016445 (0x6bf0 bytes) 000438 1788840643.045553 write 0x0395ee thru 0x03ffff (0x6a12 bytes) 000439 1788840643.055580 write 0x0068da thru 0x010589 (0x9cb0 bytes) 000441 1788840643.058201 punch from 0x01f844 to 0x02248c (0x2c49 bytes) 000443 1788840643.087006 trunc from 0x040000 to 0x021a92 (0x1e56d bytes) 000444 1788840643.162634 read 0x00aa9b thru 0x00e07c (0x35e2 bytes) 000445 1788840643.163420 mapread 0x017b18 thru 0x018368 (0x851 bytes) 000446 1788840643.170202 mapwrite 0x025e5b thru 0x035ad7 (0xfc7d bytes) 000447 1788840643.325806 write 0x01a0df thru 0x029c76 (0xfb98 bytes) 000448 1788840643.327174 write 0x00fe6a thru 0x015323 (0x54ba bytes) 000449 1788840643.328148 trunc from 0x035ad8 to 0x0293b1 (0xc726 bytes) 000450 1788840643.352497 mapwrite 0x02b766 thru 0x0339eb (0x8286 bytes) 000451 1788840643.448671 MIRROR_RESYNC 10 mirrors 000452 1788840645.399491 read 0x0273ce thru 0x02de80 (0x6ab3 bytes) 000454 1788840645.408605 read 0x01bd57 thru 0x02a634 (0xe8de bytes) 000455 1788840645.421076 punch from 0x03174b to 0x0339eb (0x22a1 bytes) 000456 1788840645.570628 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x339ec): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x339ec): Mirror 4: 0xde612ba9 (stale) Mirror 5: 0xde612ba9 (stale) Mirror 6: 0xde612ba9 (stale) Mirror 10: 0xde612ba9 (stale) Mirror 11: 0xde612ba9 (stale) Mirror 12: 0xde612ba9 (stale) Mirror 13: 0xde612ba9 (stale) Mirror 14: 0xde612ba9 (stale) Mirror 15: 0xde612ba9 (stale) CRC-32 checksum value for chunk [0, 0x339ec): Mirror 1: 0x7e19c611 Mirror 16: 0x7e19c611 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x339ec: skipped 000457 1788840646.968165 read 0x010c80 thru 0x019cab (0x902c bytes) 000458 1788840647.004043 mapwrite 0x007157 thru 0x00b7c1 (0x466b bytes) 000459 1788840647.320758 trunc from 0x0339ec to 0x00e753 (0x25298 bytes) 000460 1788840647.368727 trunc from 0x00e754 to 0x031085 (0x22932 bytes) 000461 1788840647.401464 write 0x00c21f thru 0x015e5f (0x9c41 bytes) 000463 1788840647.411433 mapwrite 0x01738b thru 0x025008 (0xdc7e bytes) 000464 1788840647.502492 write 0x000054 thru 0x004c05 (0x4bb2 bytes) 000465 1788840647.511261 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31086): Mirror 5: 0x7220dc39 (stale) Mirror 6: 0x7220dc39 (stale) Mirror 10: 0x7220dc39 (stale) Mirror 11: 0x7220dc39 (stale) Mirror 12: 0x7220dc39 (stale) Mirror 13: 0x7220dc39 (stale) Mirror 14: 0x7220dc39 (stale) Mirror 15: 0x7220dc39 (stale) Mirror 16: 0x7220dc39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31086): Mirror 5: 0x7220dc39 (stale) Mirror 6: 0x7220dc39 (stale) Mirror 10: 0x7220dc39 (stale) Mirror 11: 0x7220dc39 (stale) Mirror 12: 0x7220dc39 (stale) Mirror 13: 0x7220dc39 (stale) Mirror 14: 0x7220dc39 (stale) Mirror 15: 0x7220dc39 (stale) Mirror 16: 0x7220dc39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31086: skipped 000466 1788840648.101996 read 0x011c1b thru 0x01c6f4 (0xaada bytes) 000467 1788840648.121229 punch from 0x010df3 to 0x01a925 (0x9b33 bytes) 000468 1788840648.125845 mapread 0x0096a8 thru 0x0156da (0xc033 bytes) 000469 1788840648.158801 punch from 0x01b6d0 to 0x029d90 (0xe6c1 bytes) 000470 1788840648.163393 write 0x022e71 thru 0x0253bd (0x254d bytes) 000471 1788840648.183491 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31086): Mirror 5: 0x7220dc39 (stale) Mirror 6: 0x7220dc39 (stale) Mirror 10: 0x7220dc39 (stale) Mirror 11: 0x7220dc39 (stale) Mirror 12: 0x7220dc39 (stale) Mirror 13: 0x7220dc39 (stale) Mirror 15: 0x7220dc39 (stale) Mirror 16: 0x7220dc39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31086): Mirror 5: 0x7220dc39 (stale) Mirror 6: 0x7220dc39 (stale) Mirror 10: 0x7220dc39 (stale) Mirror 11: 0x7220dc39 (stale) Mirror 12: 0x7220dc39 (stale) Mirror 13: 0x7220dc39 (stale) Mirror 15: 0x7220dc39 (stale) Mirror 16: 0x7220dc39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31086: skipped 000472 1788840648.691421 trunc from 0x031086 to 0x03854f (0x74ca bytes) 000473 1788840648.701448 write 0x01cbdd thru 0x01f9c9 (0x2ded bytes) 000474 1788840648.708208 mapwrite 0x00df8d thru 0x01c546 (0xe5ba bytes) 000475 1788840648.785088 mapwrite 0x02e64d thru 0x036e35 (0x87e9 bytes) 000476 1788840648.844428 mapwrite 0x0393e8 thru 0x03ffff (0x6c18 bytes) 000478 1788840648.911645 mapread 0x02b76f thru 0x031e32 (0x66c4 bytes) 000480 1788840648.914306 trunc from 0x040000 to 0x02356f (0x1ca90 bytes) 000481 1788840648.934628 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x23570): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23570): Mirror 5: 0x32305b1b (stale) Mirror 6: 0x32305b1b (stale) Mirror 10: 0x32305b1b (stale) Mirror 11: 0x32305b1b (stale) Mirror 12: 0x32305b1b (stale) Mirror 13: 0x32305b1b (stale) Mirror 15: 0x32305b1b (stale) Mirror 16: 0x32305b1b (stale) CRC-32 checksum value for chunk [0, 0x23570): Mirror 1: 0x5a85d96c Mirror 17: 0x5a85d96c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23570: skipped 000484 1788840650.465343 mapread 0x00c99d thru 0x015d5e (0x93c2 bytes) 000485 1788840650.510382 write 0x034f57 thru 0x0358ad (0x957 bytes) 000486 1788840650.626665 read 0x01db93 thru 0x02d802 (0xfc70 bytes) 000487 1788840650.637709 trunc from 0x0358ae to 0x00f47c (0x26431 bytes) 000488 1788840650.659705 write 0x0184c3 thru 0x025a15 (0xd553 bytes) 000489 1788840650.661279 punch from 0x019b03 to 0x025a15 (0xbf13 bytes) 000490 1788840650.674722 trunc from 0x025a16 to 0x00b3b0 (0x1a665 bytes) 000492 1788840650.687034 mapwrite 0x032e10 thru 0x03bfd6 (0x91c7 bytes) 000493 1788840650.780347 write 0x0102f4 thru 0x017c6d (0x797a bytes) 000494 1788840650.795270 punch from 0x00e1e2 to 0x016526 (0x8345 bytes) 000495 1788840650.808227 write 0x026cca thru 0x0273b7 (0x6ee bytes) 000497 1788840650.817690 trunc from 0x03bfd7 to 0x00bcd8 (0x302fe bytes) 000498 1788840650.843032 punch from 0x00091a to 0x00bcd8 (0xb3bf bytes) 000499 1788840650.848092 punch from 0x004b37 to 0x00a86d (0x5d37 bytes) 000500 1788840650.854261 punch from 0x00689f to 0x00bcd8 (0x543a bytes) 000502 1788840650.860363 write 0x01af85 thru 0x01d978 (0x29f4 bytes) 000503 1788840650.862884 write 0x018b1b thru 0x026bf6 (0xe0dc bytes) 000504 1788840650.870062 mapwrite 0x01492a thru 0x01d280 (0x8957 bytes) 000505 1788840650.958724 write 0x01f96c thru 0x020645 (0xcda bytes) 000507 1788840650.960123 mapwrite 0x03a0d1 thru 0x03ffd4 (0x5f04 bytes) 000508 1788840651.077660 punch from 0x00d63c to 0x013e66 (0x682b bytes) 000511 1788840651.084000 mapread 0x02a69e thru 0x03725e (0xcbc1 bytes) 000512 1788840651.100246 punch from 0x030c28 to 0x03fc8e (0xf067 bytes) 000513 1788840651.104886 punch from 0x011a7e to 0x02110e (0xf691 bytes) 000514 1788840651.108813 read 0x03f465 thru 0x03fa83 (0x61f bytes) 000515 1788840651.116445 mapread 0x039871 thru 0x03c4e3 (0x2c73 bytes) 000516 1788840651.123610 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ffd5): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffd5): Mirror 5: 0x7a7b4c87 (stale) Mirror 6: 0x7a7b4c87 (stale) Mirror 10: 0x7a7b4c87 (stale) Mirror 11: 0x7a7b4c87 (stale) Mirror 12: 0x7a7b4c87 (stale) Mirror 13: 0x7a7b4c87 (stale) Mirror 15: 0x7a7b4c87 (stale) Mirror 16: 0xe1c6fe31 (stale) Mirror 17: 0x133ac433 (stale) CRC-32 checksum value for chunk [0, 0x3ffd5): Mirror 1: 0x26b82169 Mirror 18: 0x26b82169 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffd5: skipped 000517 1788840652.545450 read 0x00035f thru 0x0043aa (0x404c bytes) 000518 1788840652.563442 mapwrite 0x036ad0 thru 0x03ffff (0x9530 bytes) 000520 1788840652.756404 write 0x004164 thru 0x00fda1 (0xbc3e bytes) 000523 1788840652.775777 read 0x03f106 thru 0x03ffff (0xefa bytes) 000525 1788840652.777675 write 0x014403 thru 0x01d0ef (0x8ced bytes) 000526 1788840652.790583 punch from 0x010dbd to 0x01ec10 (0xde54 bytes) 000527 1788840652.808553 mapwrite 0x030778 thru 0x03ffff (0xf888 bytes) 000529 1788840652.866381 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc712003c (stale) Mirror 10: 0xc712003c (stale) Mirror 11: 0xc712003c (stale) Mirror 12: 0xc712003c (stale) Mirror 13: 0xc712003c (stale) Mirror 15: 0xc712003c (stale) Mirror 16: 0x262871a9 (stale) Mirror 17: 0x3f968fc5 (stale) Mirror 18: 0x38c1d85f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc712003c (stale) Mirror 10: 0xc712003c (stale) Mirror 11: 0xc712003c (stale) Mirror 12: 0xc712003c (stale) Mirror 13: 0xc712003c (stale) Mirror 15: 0xc712003c (stale) Mirror 16: 0x262871a9 (stale) Mirror 17: 0x3f968fc5 (stale) Mirror 18: 0x38c1d85f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000531 1788840653.443411 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xc712003c (stale) Mirror 11: 0xc712003c (stale) Mirror 12: 0xc712003c (stale) Mirror 13: 0xc712003c (stale) Mirror 15: 0xc712003c (stale) Mirror 16: 0x262871a9 (stale) Mirror 17: 0x3f968fc5 (stale) Mirror 18: 0x38c1d85f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xc712003c (stale) Mirror 11: 0xc712003c (stale) Mirror 12: 0xc712003c (stale) Mirror 13: 0xc712003c (stale) Mirror 15: 0xc712003c (stale) Mirror 16: 0x262871a9 (stale) Mirror 17: 0x3f968fc5 (stale) Mirror 18: 0x38c1d85f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000532 1788840653.987236 trunc from 0x040000 to 0x0272e3 (0x18d1c bytes) 000533 1788840654.004884 punch from 0x01f576 to 0x0272e3 (0x7d6e bytes) 000534 1788840654.011199 write 0x015677 thru 0x018567 (0x2ef1 bytes) 000536 1788840654.021095 read 0x0015c4 thru 0x0037ef (0x222c bytes) 000539 1788840654.027320 mapread 0x000243 thru 0x0023dd (0x219b bytes) 000540 1788840654.033605 mapread 0x00a9f2 thru 0x00f87c (0x4e8b bytes) 000541 1788840654.059132 mapread 0x000b16 thru 0x00b90e (0xadf9 bytes) 000543 1788840654.064421 read 0x00f2a7 thru 0x011c86 (0x29e0 bytes) 000544 1788840654.065421 mapread 0x013189 thru 0x01f74f (0xc5c7 bytes) 000546 1788840654.075582 write 0x01e0f7 thru 0x0285c1 (0xa4cb bytes) 000549 1788840654.078417 punch from 0x01e28e to 0x0285c1 (0xa334 bytes) 000550 1788840654.093005 write 0x028d93 thru 0x0353ac (0xc61a bytes) 000552 1788840654.100256 trunc from 0x0353ad to 0x01d16b (0x18241 bytes) 000556 1788840654.123243 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d16c): Mirror 10: 0x767951ff (stale) Mirror 12: 0x767951ff (stale) Mirror 13: 0x767951ff (stale) Mirror 15: 0x767951ff (stale) Mirror 16: 0x767951ff (stale) Mirror 17: 0xbc2cf869 (stale) Mirror 18: 0x5c189f1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d16c): Mirror 10: 0x767951ff (stale) Mirror 12: 0x767951ff (stale) Mirror 13: 0x767951ff (stale) Mirror 15: 0x767951ff (stale) Mirror 16: 0x767951ff (stale) Mirror 17: 0xbc2cf869 (stale) Mirror 18: 0x5c189f1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d16c: skipped 000557 1788840654.508174 mapread 0x013302 thru 0x015041 (0x1d40 bytes) 000558 1788840654.536308 trunc from 0x01d16c to 0x03c2f0 (0x1f185 bytes) 000559 1788840654.564689 mapread 0x00c8a1 thru 0x00cba4 (0x304 bytes) 000560 1788840654.593011 read 0x00b85c thru 0x014866 (0x900b bytes) 000561 1788840654.594437 read 0x024c61 thru 0x030946 (0xbce6 bytes) 000563 1788840654.618504 write 0x0283b0 thru 0x032053 (0x9ca4 bytes) 000564 1788840654.623452 mapwrite 0x035702 thru 0x03ffff (0xa8fe bytes) 000565 1788840654.726097 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xc712003c (stale) Mirror 12: 0xc712003c (stale) Mirror 13: 0xc712003c (stale) Mirror 15: 0xc712003c (stale) Mirror 16: 0x262871a9 (stale) Mirror 17: 0x3f968fc5 (stale) Mirror 18: 0x38c1d85f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x525ef8ee Mirror 19: 0x525ef8ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000566 1788840656.051873 read 0x039a83 thru 0x03ffff (0x657d bytes) 000567 1788840656.077994 read 0x011969 thru 0x01eb7d (0xd215 bytes) 000568 1788840656.105222 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xc712003c (stale) Mirror 12: 0xc712003c (stale) Mirror 13: 0xc712003c (stale) Mirror 15: 0xc712003c (stale) Mirror 16: 0x262871a9 (stale) Mirror 17: 0x3f968fc5 (stale) Mirror 18: 0x38c1d85f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x525ef8ee Mirror 19: 0x525ef8ee Mirror 20: 0x525ef8ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000569 1788840656.977432 mapread 0x022303 thru 0x031995 (0xf693 bytes) 000571 1788840657.009790 trunc from 0x040000 to 0x000b92 (0x3f46d bytes) 000573 1788840657.108045 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb93): Mirror 10: 0x5e933802 (stale) Mirror 12: 0x5e933802 (stale) Mirror 13: 0x5e933802 (stale) Mirror 15: 0x5e933802 (stale) Mirror 16: 0x5e933802 (stale) Mirror 17: 0xb3eab0ea (stale) Mirror 18: 0xd423b9ef (stale) Mirror 19: 0xd423b9ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb93): Mirror 10: 0x5e933802 (stale) Mirror 12: 0x5e933802 (stale) Mirror 13: 0x5e933802 (stale) Mirror 15: 0x5e933802 (stale) Mirror 16: 0x5e933802 (stale) Mirror 17: 0xb3eab0ea (stale) Mirror 18: 0xd423b9ef (stale) Mirror 19: 0xd423b9ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb93: skipped 000574 1788840657.392286 read 0x00048a thru 0x000b92 (0x709 bytes) 000575 1788840657.398477 punch from 0x000358 to 0x000b92 (0x83b bytes) 000576 1788840657.406335 trunc from 0x000b93 to 0x00584f (0x4cbd bytes) 000577 1788840657.427417 punch from 0x0034ef to 0x00584f (0x2361 bytes) 000578 1788840657.439265 read 0x002fbc thru 0x004dd0 (0x1e15 bytes) 000579 1788840657.456525 mapread 0x0000fd thru 0x00584f (0x5753 bytes) 000580 1788840657.462847 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5850): Mirror 10: 0xb947d027 (stale) Mirror 12: 0xb947d027 (stale) Mirror 13: 0xb947d027 (stale) Mirror 15: 0xb947d027 (stale) Mirror 16: 0xb947d027 (stale) Mirror 17: 0x8420e9b (stale) Mirror 19: 0x1600068b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5850): Mirror 10: 0xb947d027 (stale) Mirror 12: 0xb947d027 (stale) Mirror 13: 0xb947d027 (stale) Mirror 15: 0xb947d027 (stale) Mirror 16: 0xb947d027 (stale) Mirror 17: 0x8420e9b (stale) Mirror 19: 0x1600068b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5850: skipped 000581 1788840657.995998 mapread 0x002595 thru 0x00584f (0x32bb bytes) 000582 1788840658.012695 write 0x02047b thru 0x02f5af (0xf135 bytes) 000584 1788840658.015551 read 0x0098ab thru 0x00ad46 (0x149c bytes) 000587 1788840658.022421 write 0x003bdd thru 0x0040a3 (0x4c7 bytes) 000588 1788840658.023501 mapwrite 0x002d91 thru 0x0075c3 (0x4833 bytes) 000589 1788840658.080581 punch from 0x018f3b to 0x0237d6 (0xa89c bytes) 000593 1788840658.088109 write 0x006990 thru 0x00761f (0xc90 bytes) 000596 1788840658.090413 mapwrite 0x003de3 thru 0x005dce (0x1fec bytes) 000597 1788840658.128339 read 0x015b52 thru 0x02434b (0xe7fa bytes) 000598 1788840658.141336 punch from 0x02d1cb to 0x02f5af (0x23e5 bytes) 000600 1788840658.147178 read 0x02d53f thru 0x02f5af (0x2071 bytes) 000605 1788840658.156528 punch from 0x01b0cd to 0x01c539 (0x146d bytes) 000606 1788840658.166790 read 0x02cf27 thru 0x02f5af (0x2689 bytes) 000609 1788840658.170150 mapwrite 0x00da96 thru 0x00ea6e (0xfd9 bytes) 000610 1788840658.219067 mapread 0x025f0f thru 0x02a460 (0x4552 bytes) 000611 1788840658.220120 punch from 0x020525 to 0x02baab (0xb587 bytes) 000612 1788840658.224646 write 0x011cb8 thru 0x017709 (0x5a52 bytes) 000613 1788840658.238512 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f5b0): Mirror 10: 0x974d8f19 (stale) Mirror 13: 0x974d8f19 (stale) Mirror 15: 0x974d8f19 (stale) Mirror 16: 0x974d8f19 (stale) Mirror 17: 0xb8411c27 (stale) Mirror 19: 0x765d80a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5b0): Mirror 10: 0x974d8f19 (stale) Mirror 13: 0x974d8f19 (stale) Mirror 15: 0x974d8f19 (stale) Mirror 16: 0x974d8f19 (stale) Mirror 17: 0xb8411c27 (stale) Mirror 19: 0x765d80a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5b0: skipped 000615 1788840658.728640 write 0x0108bb thru 0x011d26 (0x146c bytes) 000617 1788840658.754370 punch from 0x002a92 to 0x008c6d (0x61dc bytes) 000620 1788840658.767735 punch from 0x011cce to 0x0196e9 (0x7a1c bytes) 000621 1788840658.773375 mapread 0x02b57e thru 0x02f5af (0x4032 bytes) 000622 1788840658.801611 mapread 0x006ce9 thru 0x013b93 (0xceab bytes) 000623 1788840658.826897 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f5b0): Mirror 13: 0x974d8f19 (stale) Mirror 15: 0x974d8f19 (stale) Mirror 16: 0x974d8f19 (stale) Mirror 17: 0xb8411c27 (stale) Mirror 19: 0x765d80a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5b0): Mirror 13: 0x974d8f19 (stale) Mirror 15: 0x974d8f19 (stale) Mirror 16: 0x974d8f19 (stale) Mirror 17: 0xb8411c27 (stale) Mirror 19: 0x765d80a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5b0: skipped 000625 1788840659.316531 read 0x029d80 thru 0x02ee0c (0x508d bytes) 000630 1788840659.332133 mapwrite 0x000693 thru 0x00840c (0x7d7a bytes) 000631 1788840659.427540 write 0x0396bb thru 0x03becf (0x2815 bytes) 000632 1788840659.433261 trunc from 0x03bed0 to 0x00c891 (0x2f63e bytes) 000633 1788840659.477653 mapwrite 0x00268b thru 0x0093fd (0x6d73 bytes) 000634 1788840659.553319 mapwrite 0x025631 thru 0x03049c (0xae6c bytes) 000635 1788840659.677325 trunc from 0x03049d to 0x00aa1d (0x25a7f bytes) 000636 1788840659.702896 read 0x002fd3 thru 0x00aa1d (0x7a4b bytes) 000637 1788840659.704941 punch from 0x003ab7 to 0x00aa1d (0x6f67 bytes) 000638 1788840659.722746 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xaa1e): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaa1e): Mirror 13: 0xaebc5890 (stale) Mirror 15: 0xaebc5890 (stale) Mirror 16: 0xaebc5890 (stale) Mirror 17: 0xdfdfd6f7 (stale) Mirror 19: 0xe64c7287 (stale) CRC-32 checksum value for chunk [0, 0xaa1e): Mirror 1: 0x40f9c28b Mirror 20: 0x40f9c28b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaa1e: skipped 000639 1788840660.882634 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xaa1e): Mirror 13: 0xaebc5890 (stale) Mirror 16: 0xaebc5890 (stale) Mirror 17: 0xdfdfd6f7 (stale) Mirror 19: 0xe64c7287 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaa1e): Mirror 13: 0xaebc5890 (stale) Mirror 16: 0xaebc5890 (stale) Mirror 17: 0xdfdfd6f7 (stale) Mirror 19: 0xe64c7287 (stale) CRC-32 checksum value for chunk [0, 0xaa1e): Mirror 1: 0x40f9c28b Mirror 20: 0x40f9c28b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaa1e: skipped 000640 1788840661.640156 trunc from 0x00aa1e to 0x03a825 (0x2fe08 bytes) 000641 1788840661.734790 MIRROR_RESYNC 6 mirrors 000643 1788840663.457113 mapread 0x0087cf thru 0x016575 (0xdda7 bytes) 000644 1788840663.507550 punch from 0x0107d7 to 0x01dc32 (0xd45c bytes) 000645 1788840663.566406 read 0x004a3c thru 0x00ce3b (0x8400 bytes) 000646 1788840663.575260 write 0x005624 thru 0x00ae62 (0x583f bytes) 000647 1788840663.582549 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a826): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a826): Mirror 13: 0xcc7d432a (stale) Mirror 16: 0xcc7d432a (stale) Mirror 17: 0xcc7d432a (stale) Mirror 19: 0xcc7d432a (stale) Mirror 20: 0xcc7d432a (stale) CRC-32 checksum value for chunk [0, 0x3a826): Mirror 1: 0xa0480a2a Mirror 21: 0xa0480a2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a826: skipped 000649 1788840665.232407 trunc from 0x03a826 to 0x006549 (0x342dc bytes) 000651 1788840665.372253 trunc from 0x00654a to 0x015637 (0xf0ee bytes) 000652 1788840665.415356 mapwrite 0x03ab71 thru 0x03ffff (0x548f bytes) 000653 1788840665.538997 mapread 0x034375 thru 0x03dbd6 (0x9862 bytes) 000654 1788840665.555562 read 0x00ff04 thru 0x0124a5 (0x25a2 bytes) 000655 1788840665.568378 mapread 0x00631e thru 0x00b6f2 (0x53d5 bytes) 000656 1788840665.578462 write 0x018137 thru 0x01f84f (0x7719 bytes) 000657 1788840665.592093 read 0x02b248 thru 0x034922 (0x96db bytes) 000658 1788840665.613557 write 0x03af9f thru 0x03ffff (0x5061 bytes) 000659 1788840665.615288 mapwrite 0x037e7b thru 0x03ffff (0x8185 bytes) 000661 1788840665.681815 mapwrite 0x038914 thru 0x03ffff (0x76ec bytes) 000662 1788840665.742470 mapread 0x00962c thru 0x00c7df (0x31b4 bytes) 000663 1788840665.744692 write 0x0282fc thru 0x033bcb (0xb8d0 bytes) 000664 1788840665.746127 write 0x03a1ae thru 0x03c4a3 (0x22f6 bytes) 000666 1788840665.747568 mapwrite 0x002979 thru 0x00abf9 (0x8281 bytes) 000670 1788840665.795094 trunc from 0x040000 to 0x00a2d5 (0x35d2a bytes) 000673 1788840665.822484 MIRROR_RESYNC 7 mirrors 000674 1788840667.442014 read 0x008b2e thru 0x00a2d5 (0x17a8 bytes) 000676 1788840667.452817 trunc from 0x00a2d6 to 0x0078ee (0x29e7 bytes) 000677 1788840667.546862 trunc from 0x0078ef to 0x025669 (0x1dd7b bytes) 000680 1788840667.595834 mapread 0x01c771 thru 0x02287d (0x610d bytes) 000681 1788840667.626705 trunc from 0x02566a to 0x02628c (0xc23 bytes) 000682 1788840667.654206 punch from 0x01f5c5 to 0x020906 (0x1342 bytes) 000685 1788840667.684317 mapread 0x009a87 thru 0x00fe55 (0x63cf bytes) 000686 1788840667.707761 trunc from 0x02628d to 0x0035cb (0x22cc1 bytes) 000687 1788840667.747281 trunc from 0x0035cc to 0x01ef85 (0x1b9ba bytes) 000688 1788840667.776838 trunc from 0x01ef86 to 0x02cb33 (0xdbae bytes) 000689 1788840667.789270 mapwrite 0x01c137 thru 0x027456 (0xb320 bytes) 000690 1788840667.889466 punch from 0x00e053 to 0x014854 (0x6802 bytes) 000691 1788840667.895424 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2cb34): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb34): Mirror 13: 0xabfbd690 (stale) Mirror 16: 0xabfbd690 (stale) Mirror 17: 0xabfbd690 (stale) Mirror 19: 0xabfbd690 (stale) Mirror 20: 0xabfbd690 (stale) Mirror 21: 0xabfbd690 (stale) CRC-32 checksum value for chunk [0, 0x2cb34): Mirror 1: 0xdf738622 Mirror 22: 0xdf738622 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb34: skipped 000692 1788840669.769717 mapread 0x016b50 thru 0x01d951 (0x6e02 bytes) 000697 1788840669.915428 read 0x02a5f3 thru 0x02cb33 (0x2541 bytes) 000698 1788840669.941843 write 0x0054ab thru 0x00d281 (0x7dd7 bytes) 000699 1788840670.093336 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2cb34): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb34): Mirror 13: 0xabfbd690 (stale) Mirror 16: 0xabfbd690 (stale) Mirror 17: 0xabfbd690 (stale) Mirror 19: 0xabfbd690 (stale) Mirror 20: 0xabfbd690 (stale) Mirror 21: 0xabfbd690 (stale) Mirror 22: 0xdf738622 (stale) CRC-32 checksum value for chunk [0, 0x2cb34): Mirror 1: 0x9f2777d6 Mirror 23: 0x9f2777d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb34: skipped 000700 1788840671.937256 punch from 0x005a7d to 0x008e37 (0x33bb bytes) 000701 1788840672.045346 mapread 0x0070c6 thru 0x00bef2 (0x4e2d bytes) 000702 1788840672.066423 punch from 0x01870a to 0x01dc09 (0x5500 bytes) 000703 1788840672.073006 mapwrite 0x01d547 thru 0x01e3c2 (0xe7c bytes) 000705 1788840672.161753 punch from 0x02956b to 0x02cb33 (0x35c9 bytes) 000706 1788840672.181040 read 0x029bcc thru 0x02cb33 (0x2f68 bytes) 000708 1788840672.208875 read 0x024cf2 thru 0x028a89 (0x3d98 bytes) 000709 1788840672.232779 write 0x032ab4 thru 0x03baaa (0x8ff7 bytes) 000711 1788840672.241518 punch from 0x0291d5 to 0x0325a0 (0x93cc bytes) 000712 1788840672.270275 mapread 0x019bb7 thru 0x025fa2 (0xc3ec bytes) 000713 1788840672.271477 write 0x00704e thru 0x0079c2 (0x975 bytes) 000714 1788840672.285302 MIRROR_RESYNC 9 mirrors 000717 1788840675.807639 write 0x02c8d8 thru 0x03b790 (0xeeb9 bytes) 000718 1788840675.970051 punch from 0x022465 to 0x0302ad (0xde49 bytes) 000719 1788840676.013975 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3baab): Mirror 13: 0x407895b7 (stale) Mirror 16: 0x407895b7 (stale) Mirror 17: 0x407895b7 (stale) Mirror 19: 0x407895b7 (stale) Mirror 20: 0x407895b7 (stale) Mirror 21: 0x407895b7 (stale) Mirror 22: 0x407895b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baab): Mirror 13: 0x407895b7 (stale) Mirror 16: 0x407895b7 (stale) Mirror 17: 0x407895b7 (stale) Mirror 19: 0x407895b7 (stale) Mirror 20: 0x407895b7 (stale) Mirror 21: 0x407895b7 (stale) Mirror 22: 0x407895b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baab: skipped 000720 1788840676.566399 read 0x03311b thru 0x03b23f (0x8125 bytes) 000721 1788840676.586246 punch from 0x00f83a to 0x0169ad (0x7174 bytes) 000722 1788840676.591580 punch from 0x02a411 to 0x037972 (0xd562 bytes) 000723 1788840676.596144 punch from 0x03a449 to 0x03b768 (0x1320 bytes) 000728 1788840676.608505 mapread 0x01fda1 thru 0x02aaea (0xad4a bytes) 000729 1788840676.658086 mapwrite 0x003794 thru 0x01206e (0xe8db bytes) 000730 1788840676.720450 mapread 0x00a92c thru 0x019ff4 (0xf6c9 bytes) 000731 1788840676.721382 write 0x03cd00 thru 0x03ffff (0x3300 bytes) 000732 1788840676.722341 write 0x004d0e thru 0x00702e (0x2321 bytes) 000734 1788840676.725369 punch from 0x01b787 to 0x028d46 (0xd5c0 bytes) 000735 1788840676.734992 mapread 0x0146a5 thru 0x017402 (0x2d5e bytes) 000737 1788840676.736714 punch from 0x030cd4 to 0x03ad12 (0xa03f bytes) 000739 1788840676.741375 trunc from 0x040000 to 0x03adf4 (0x520b bytes) 000740 1788840676.756548 mapread 0x0008a8 thru 0x00a275 (0x99ce bytes) 000744 1788840676.763630 write 0x006864 thru 0x012b1d (0xc2ba bytes) 000745 1788840676.773210 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3adf5): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3adf5): Mirror 13: 0xd9f3bc60 (stale) Mirror 16: 0xd9f3bc60 (stale) Mirror 17: 0xd9f3bc60 (stale) Mirror 19: 0xd9f3bc60 (stale) Mirror 20: 0xd9f3bc60 (stale) Mirror 21: 0xd9f3bc60 (stale) Mirror 22: 0xd9f3bc60 (stale) CRC-32 checksum value for chunk [0, 0x3adf5): Mirror 1: 0xeaf7bbb5 Mirror 23: 0xeaf7bbb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3adf5: skipped 000746 1788840677.780211 mapread 0x007d3a thru 0x01307d (0xb344 bytes) 000747 1788840677.822176 read 0x029c1b thru 0x030984 (0x6d6a bytes) 000749 1788840677.846278 punch from 0x02743a to 0x02d8ac (0x6473 bytes) 000751 1788840677.935889 trunc from 0x03adf5 to 0x00bde4 (0x2f010 bytes) 000752 1788840677.960820 punch from 0x0062a5 to 0x00bde4 (0x5b40 bytes) 000754 1788840677.967253 mapwrite 0x0099e1 thru 0x00efb4 (0x55d4 bytes) 000756 1788840678.089880 read 0x000de0 thru 0x00bf6b (0xb18c bytes) 000757 1788840678.106430 mapwrite 0x029577 thru 0x02de95 (0x491f bytes) 000758 1788840678.205120 read 0x0198de thru 0x022f88 (0x96ab bytes) 000760 1788840678.211077 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2de96): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de96): Mirror 13: 0x4b9e235a (stale) Mirror 16: 0x4b9e235a (stale) Mirror 17: 0x4b9e235a (stale) Mirror 19: 0x4b9e235a (stale) Mirror 20: 0x4b9e235a (stale) Mirror 21: 0x4b9e235a (stale) Mirror 22: 0x4b9e235a (stale) Mirror 23: 0x2445a69a (stale) CRC-32 checksum value for chunk [0, 0x2de96): Mirror 1: 0x3a69b7b1 Mirror 24: 0x3a69b7b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de96: skipped 000763 1788840679.351862 read 0x027c81 thru 0x02de95 (0x6215 bytes) 000764 1788840679.382225 punch from 0x020834 to 0x028de3 (0x85b0 bytes) 000768 1788840679.559120 mapread 0x029445 thru 0x02de95 (0x4a51 bytes) 000769 1788840679.603677 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2de96): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de96): Mirror 13: 0x4b9e235a (stale) Mirror 16: 0x4b9e235a (stale) Mirror 17: 0x4b9e235a (stale) Mirror 19: 0x4b9e235a (stale) Mirror 20: 0x4b9e235a (stale) Mirror 21: 0x4b9e235a (stale) Mirror 22: 0x4b9e235a (stale) Mirror 23: 0x2445a69a (stale) Mirror 24: 0x3a69b7b1 (stale) CRC-32 checksum value for chunk [0, 0x2de96): Mirror 1: 0x3a69b7b1 Mirror 25: 0x3a69b7b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de96: skipped 000770 1788840680.845222 MIRROR_RESYNC 11 mirrors 000771 1788840683.530650 write 0x025560 thru 0x031faa (0xca4b bytes) 000772 1788840683.601771 read 0x00c859 thru 0x015841 (0x8fe9 bytes) 000773 1788840683.611254 punch from 0x00d9fb to 0x01c4c8 (0xeace bytes) 000774 1788840683.627442 read 0x0194ad thru 0x027cac (0xe800 bytes) 000775 1788840683.637679 punch from 0x0088eb to 0x017487 (0xeb9d bytes) 000776 1788840683.642305 read 0x02a28f thru 0x02fc32 (0x59a4 bytes) 000779 1788840683.644270 mapwrite 0x027246 thru 0x031626 (0xa3e1 bytes) 000781 1788840683.706093 write 0x015069 thru 0x01baa4 (0x6a3c bytes) 000783 1788840683.724654 punch from 0x01f2be to 0x022d54 (0x3a97 bytes) 000786 1788840683.743836 punch from 0x01946f to 0x0216e9 (0x827b bytes) 000787 1788840683.748364 read 0x0302b4 thru 0x031faa (0x1cf7 bytes) 000789 1788840683.754342 punch from 0x002311 to 0x002d96 (0xa86 bytes) 000790 1788840683.762373 MIRROR_RESYNC 11 mirrors 000791 1788840685.344897 mapread 0x02ed25 thru 0x031faa (0x3286 bytes) 000792 1788840685.390459 punch from 0x02df1b to 0x031faa (0x4090 bytes) 000793 1788840685.497632 trunc from 0x031fab to 0x034292 (0x22e8 bytes) 000794 1788840685.539302 trunc from 0x034293 to 0x009d94 (0x2a4fe bytes) 000796 1788840685.586968 punch from 0x0012dc to 0x006b59 (0x587e bytes) 000799 1788840685.594195 write 0x027f81 thru 0x02d1c3 (0x5243 bytes) 000800 1788840685.595371 write 0x02bfd8 thru 0x02cccc (0xcf5 bytes) 000802 1788840685.602680 mapwrite 0x03d036 thru 0x03ffff (0x2fca bytes) 000803 1788840685.718662 punch from 0x01cb08 to 0x02a399 (0xd892 bytes) 000804 1788840685.735659 read 0x032a52 thru 0x03ffff (0xd5ae bytes) 000805 1788840685.749529 trunc from 0x040000 to 0x014766 (0x2b899 bytes) 000807 1788840685.768676 punch from 0x011b50 to 0x014766 (0x2c17 bytes) 000808 1788840685.793092 MIRROR_RESYNC 11 mirrors 000809 1788840688.707273 write 0x021492 thru 0x026f02 (0x5a71 bytes) 000810 1788840688.837853 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26f03): Mirror 13: 0x9c042043 (stale) Mirror 16: 0x9c042043 (stale) Mirror 17: 0x9c042043 (stale) Mirror 19: 0x9c042043 (stale) Mirror 20: 0x9c042043 (stale) Mirror 22: 0x9c042043 (stale) Mirror 23: 0x9c042043 (stale) Mirror 24: 0x9c042043 (stale) Mirror 25: 0x9c042043 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26f03): Mirror 13: 0x9c042043 (stale) Mirror 16: 0x9c042043 (stale) Mirror 17: 0x9c042043 (stale) Mirror 19: 0x9c042043 (stale) Mirror 20: 0x9c042043 (stale) Mirror 22: 0x9c042043 (stale) Mirror 23: 0x9c042043 (stale) Mirror 24: 0x9c042043 (stale) Mirror 25: 0x9c042043 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26f03: skipped 000812 1788840689.329106 mapwrite 0x034aff thru 0x03f711 (0xac13 bytes) 000813 1788840689.421714 punch from 0x032708 to 0x03f711 (0xd00a bytes) 000814 1788840689.433798 punch from 0x00300f to 0x008ae2 (0x5ad4 bytes) 000816 1788840689.439833 mapread 0x039902 thru 0x03f711 (0x5e10 bytes) 000818 1788840689.479802 mapread 0x033ab8 thru 0x038b32 (0x507b bytes) 000820 1788840689.491863 mapread 0x037830 thru 0x03f711 (0x7ee2 bytes) 000821 1788840689.500398 read 0x019df7 thru 0x01b740 (0x194a bytes) 000822 1788840689.521832 mapread 0x038bc8 thru 0x03b1a6 (0x25df bytes) 000823 1788840689.530176 trunc from 0x03f712 to 0x007b9a (0x37b77 bytes) 000824 1788840689.555792 write 0x005623 thru 0x00d313 (0x7cf1 bytes) 000825 1788840689.560846 trunc from 0x00d314 to 0x014da4 (0x7a91 bytes) 000826 1788840689.587141 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x14da5): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14da5): Mirror 13: 0x4f2ae4e1 (stale) Mirror 16: 0x4f2ae4e1 (stale) Mirror 17: 0x4f2ae4e1 (stale) Mirror 19: 0x4f2ae4e1 (stale) Mirror 20: 0x4f2ae4e1 (stale) Mirror 22: 0x4f2ae4e1 (stale) Mirror 23: 0x4f2ae4e1 (stale) Mirror 24: 0x4f2ae4e1 (stale) Mirror 25: 0x4f2ae4e1 (stale) CRC-32 checksum value for chunk [0, 0x14da5): Mirror 1: 0x9c63fecf Mirror 26: 0x9c63fecf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14da5: skipped 000827 1788840690.960448 mapread 0x00312a thru 0x00fd05 (0xcbdc bytes) 000829 1788840691.068442 MIRROR_RESYNC 11 mirrors 000830 1788840694.140350 punch from 0x002041 to 0x008a00 (0x69c0 bytes) 000833 1788840694.195744 mapread 0x0103d4 thru 0x01080f (0x43c bytes) 000837 1788840694.204060 MIRROR_RESYNC 11 mirrors 000839 1788840696.902699 write 0x020c32 thru 0x025185 (0x4554 bytes) 000840 1788840697.052377 punch from 0x008a25 to 0x0168e2 (0xdebe bytes) 000841 1788840697.076433 trunc from 0x025186 to 0x01ab39 (0xa64c bytes) 000842 1788840697.138245 trunc from 0x01ab3a to 0x023513 (0x89da bytes) 000844 1788840697.166663 punch from 0x00c7fb to 0x0147a8 (0x7fae bytes) 000846 1788840697.174162 write 0x014768 thru 0x014824 (0xbd bytes) 000847 1788840697.183455 punch from 0x0034c1 to 0x004988 (0x14c8 bytes) 000849 1788840697.214221 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23514): Mirror 13: 0x40b8e4eb (stale) Mirror 16: 0x40b8e4eb (stale) Mirror 17: 0x40b8e4eb (stale) Mirror 19: 0x40b8e4eb (stale) Mirror 20: 0x40b8e4eb (stale) Mirror 22: 0x40b8e4eb (stale) Mirror 23: 0x40b8e4eb (stale) Mirror 24: 0x40b8e4eb (stale) Mirror 25: 0x40b8e4eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23514): Mirror 13: 0x40b8e4eb (stale) Mirror 16: 0x40b8e4eb (stale) Mirror 17: 0x40b8e4eb (stale) Mirror 19: 0x40b8e4eb (stale) Mirror 20: 0x40b8e4eb (stale) Mirror 22: 0x40b8e4eb (stale) Mirror 23: 0x40b8e4eb (stale) Mirror 24: 0x40b8e4eb (stale) Mirror 25: 0x40b8e4eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23514: skipped 000850 1788840697.798789 mapwrite 0x020252 thru 0x02d72a (0xd4d9 bytes) 000851 1788840697.897741 read 0x0288a9 thru 0x02d72a (0x4e82 bytes) 000852 1788840697.898453 mapwrite 0x007556 thru 0x013729 (0xc1d4 bytes) 000853 1788840697.996382 mapread 0x01f9ec thru 0x02d20a (0xd81f bytes) 000855 1788840698.001114 write 0x0222ba thru 0x02e17d (0xbec4 bytes) 000856 1788840698.002687 trunc from 0x02e17e to 0x01c339 (0x11e44 bytes) 000857 1788840698.029073 write 0x01bb4a thru 0x01eabd (0x2f74 bytes) 000858 1788840698.037011 read 0x01a076 thru 0x01eabd (0x4a48 bytes) 000859 1788840698.045949 mapread 0x018471 thru 0x01eabd (0x664d bytes) 000860 1788840698.048819 punch from 0x00a06e to 0x0101bf (0x6152 bytes) 000861 1788840698.059789 mapread 0x018a82 thru 0x01eabd (0x603c bytes) 000863 1788840698.061307 mapread 0x0168fc thru 0x01eabd (0x81c2 bytes) 000864 1788840698.063817 read 0x016215 thru 0x01eabd (0x88a9 bytes) 000865 1788840698.065569 punch from 0x00e672 to 0x01d995 (0xf324 bytes) 000866 1788840698.080425 mapread 0x001228 thru 0x008a7d (0x7856 bytes) 000867 1788840698.090450 punch from 0x0006cd to 0x000eae (0x7e2 bytes) 000868 1788840698.101229 trunc from 0x01eabe to 0x0095a8 (0x15515 bytes) 000871 1788840698.143831 mapread 0x00509d thru 0x0095a8 (0x450c bytes) 000873 1788840698.152637 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x95a9): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x95a9): Mirror 13: 0x24541e46 (stale) Mirror 16: 0x24541e46 (stale) Mirror 17: 0x24541e46 (stale) Mirror 19: 0x24541e46 (stale) Mirror 20: 0x24541e46 (stale) Mirror 22: 0x24541e46 (stale) Mirror 23: 0x24541e46 (stale) Mirror 24: 0x24541e46 (stale) Mirror 25: 0x24541e46 (stale) CRC-32 checksum value for chunk [0, 0x95a9): Mirror 1: 0xe8146507 Mirror 26: 0xe8146507 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x95a9: skipped 000874 1788840698.909784 trunc from 0x0095a9 to 0x01dd71 (0x147c9 bytes) 000877 1788840699.011156 write 0x0240bd thru 0x02ddc7 (0x9d0b bytes) 000878 1788840699.012470 write 0x002871 thru 0x00f151 (0xc8e1 bytes) 000879 1788840699.024773 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ddc8): Mirror 13: 0xd2772f47 (stale) Mirror 16: 0xd2772f47 (stale) Mirror 17: 0xd2772f47 (stale) Mirror 20: 0xd2772f47 (stale) Mirror 22: 0xd2772f47 (stale) Mirror 23: 0xd2772f47 (stale) Mirror 24: 0xd2772f47 (stale) Mirror 25: 0xd2772f47 (stale) Mirror 26: 0xde0d5b2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddc8): Mirror 13: 0xd2772f47 (stale) Mirror 16: 0xd2772f47 (stale) Mirror 17: 0xd2772f47 (stale) Mirror 20: 0xd2772f47 (stale) Mirror 22: 0xd2772f47 (stale) Mirror 23: 0xd2772f47 (stale) Mirror 24: 0xd2772f47 (stale) Mirror 25: 0xd2772f47 (stale) Mirror 26: 0xde0d5b2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddc8: skipped 000880 1788840699.648676 read 0x019a66 thru 0x019b8d (0x128 bytes) 000882 1788840699.672931 trunc from 0x02ddc8 to 0x01b54e (0x12879 bytes) 000884 1788840699.702951 mapread 0x00f209 thru 0x016ac7 (0x78bf bytes) 000885 1788840699.741220 write 0x022acb thru 0x02c792 (0x9cc8 bytes) 000887 1788840699.743077 mapwrite 0x000596 thru 0x00ad21 (0xa78c bytes) 000889 1788840699.812547 punch from 0x02571c to 0x02c792 (0x7077 bytes) 000890 1788840699.821272 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c793): Mirror 13: 0x6dbe98c5 (stale) Mirror 16: 0x6dbe98c5 (stale) Mirror 17: 0x6dbe98c5 (stale) Mirror 20: 0x6dbe98c5 (stale) Mirror 23: 0x6dbe98c5 (stale) Mirror 24: 0x6dbe98c5 (stale) Mirror 25: 0x6dbe98c5 (stale) Mirror 26: 0xfd140700 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c793): Mirror 13: 0x6dbe98c5 (stale) Mirror 16: 0x6dbe98c5 (stale) Mirror 17: 0x6dbe98c5 (stale) Mirror 20: 0x6dbe98c5 (stale) Mirror 23: 0x6dbe98c5 (stale) Mirror 24: 0x6dbe98c5 (stale) Mirror 25: 0x6dbe98c5 (stale) Mirror 26: 0xfd140700 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c793: skipped 000891 1788840700.289100 punch from 0x00a18c to 0x01977d (0xf5f2 bytes) 000893 1788840700.302968 punch from 0x01a7ff to 0x027ea9 (0xd6ab bytes) 000894 1788840700.307770 trunc from 0x02c793 to 0x00409c (0x286f6 bytes) 000895 1788840700.355629 punch from 0x0024fa to 0x003bed (0x16f4 bytes) 000896 1788840700.372846 punch from 0x0004a5 to 0x00409c (0x3bf8 bytes) 000897 1788840700.421572 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x409d): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x409d): Mirror 13: 0x3d2370d3 (stale) Mirror 16: 0x3d2370d3 (stale) Mirror 17: 0x3d2370d3 (stale) Mirror 20: 0x3d2370d3 (stale) Mirror 23: 0x3d2370d3 (stale) Mirror 24: 0x3d2370d3 (stale) Mirror 25: 0x3d2370d3 (stale) Mirror 26: 0x8429fe36 (stale) CRC-32 checksum value for chunk [0, 0x409d): Mirror 1: 0x154cdd10 Mirror 27: 0x154cdd10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x409d: skipped 000899 1788840701.268676 mapread 0x0032d6 thru 0x00409c (0xdc7 bytes) 000900 1788840701.308171 read 0x003d3b thru 0x00409c (0x362 bytes) 000901 1788840701.314619 mapwrite 0x03eaed thru 0x03ffff (0x1513 bytes) 000902 1788840701.504286 read 0x00ac2d thru 0x0186ab (0xda7f bytes) 000903 1788840701.523521 trunc from 0x040000 to 0x03a809 (0x57f6 bytes) 000904 1788840701.549533 MIRROR_RESYNC 10 mirrors 000905 1788840703.553765 trunc from 0x03a80a to 0x024134 (0x166d5 bytes) 000906 1788840703.626757 read 0x01333e thru 0x0206fa (0xd3bd bytes) 000907 1788840703.638657 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x24135): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24135): Mirror 13: 0x54b42a06 (stale) Mirror 16: 0x54b42a06 (stale) Mirror 17: 0x54b42a06 (stale) Mirror 20: 0x54b42a06 (stale) Mirror 23: 0x54b42a06 (stale) Mirror 24: 0x54b42a06 (stale) Mirror 25: 0x54b42a06 (stale) Mirror 26: 0x54b42a06 (stale) Mirror 27: 0x54b42a06 (stale) CRC-32 checksum value for chunk [0, 0x24135): Mirror 1: 0x54b42a06 Mirror 28: 0x54b42a06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24135: skipped 000908 1788840704.861562 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x24135): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24135): Mirror 13: 0x54b42a06 (stale) Mirror 16: 0x54b42a06 (stale) Mirror 17: 0x54b42a06 (stale) Mirror 20: 0x54b42a06 (stale) Mirror 23: 0x54b42a06 (stale) Mirror 24: 0x54b42a06 (stale) Mirror 25: 0x54b42a06 (stale) Mirror 26: 0x54b42a06 (stale) Mirror 27: 0x54b42a06 (stale) CRC-32 checksum value for chunk [0, 0x24135): Mirror 1: 0x54b42a06 Mirror 28: 0x54b42a06 Mirror 29: 0x54b42a06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24135: skipped 000909 1788840705.861999 write 0x023131 thru 0x032284 (0xf154 bytes) 000910 1788840705.975396 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32285): Mirror 13: 0xd8473d54 (stale) Mirror 17: 0xd8473d54 (stale) Mirror 20: 0xd8473d54 (stale) Mirror 23: 0xd8473d54 (stale) Mirror 24: 0xd8473d54 (stale) Mirror 25: 0xd8473d54 (stale) Mirror 26: 0xd8473d54 (stale) Mirror 27: 0xd8473d54 (stale) Mirror 28: 0xd8473d54 (stale) Mirror 29: 0xd8473d54 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32285): Mirror 13: 0xd8473d54 (stale) Mirror 17: 0xd8473d54 (stale) Mirror 20: 0xd8473d54 (stale) Mirror 23: 0xd8473d54 (stale) Mirror 24: 0xd8473d54 (stale) Mirror 25: 0xd8473d54 (stale) Mirror 26: 0xd8473d54 (stale) Mirror 27: 0xd8473d54 (stale) Mirror 28: 0xd8473d54 (stale) Mirror 29: 0xd8473d54 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32285: skipped 000911 1788840706.510584 mapread 0x02ddfc thru 0x02faa4 (0x1ca9 bytes) 000912 1788840706.528552 mapread 0x018402 thru 0x01f756 (0x7355 bytes) 000913 1788840706.538589 trunc from 0x032285 to 0x033be6 (0x1962 bytes) 000915 1788840706.553531 mapread 0x0255e9 thru 0x026e6b (0x1883 bytes) 000916 1788840706.567140 MIRROR_RESYNC 11 mirrors 000917 1788840708.892667 trunc from 0x033be7 to 0x014398 (0x1f84e bytes) 000918 1788840708.986419 write 0x006b5b thru 0x00f4ed (0x8993 bytes) 000920 1788840708.998174 punch from 0x0039fe to 0x00a0de (0x66e1 bytes) 000921 1788840709.012850 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14399): Mirror 13: 0x47de0f8f (stale) Mirror 17: 0x47de0f8f (stale) Mirror 20: 0x47de0f8f (stale) Mirror 23: 0x47de0f8f (stale) Mirror 24: 0x47de0f8f (stale) Mirror 25: 0x47de0f8f (stale) Mirror 26: 0x47de0f8f (stale) Mirror 27: 0x47de0f8f (stale) Mirror 28: 0x47de0f8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14399): Mirror 13: 0x47de0f8f (stale) Mirror 17: 0x47de0f8f (stale) Mirror 20: 0x47de0f8f (stale) Mirror 23: 0x47de0f8f (stale) Mirror 24: 0x47de0f8f (stale) Mirror 25: 0x47de0f8f (stale) Mirror 26: 0x47de0f8f (stale) Mirror 27: 0x47de0f8f (stale) Mirror 28: 0x47de0f8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14399: skipped 000922 1788840709.482453 mapwrite 0x0167da thru 0x01a2ed (0x3b14 bytes) 000923 1788840709.569087 mapwrite 0x033e95 thru 0x034719 (0x885 bytes) 000924 1788840709.632576 trunc from 0x03471a to 0x03a598 (0x5e7f bytes) 000926 1788840709.650199 read 0x02c19a thru 0x036a21 (0xa888 bytes) 000928 1788840709.656564 write 0x00fa72 thru 0x01e5a6 (0xeb35 bytes) 000929 1788840709.666628 punch from 0x019121 to 0x024aa3 (0xb983 bytes) 000930 1788840709.680613 write 0x023e61 thru 0x029b28 (0x5cc8 bytes) 000932 1788840709.690465 read 0x00c966 thru 0x013daf (0x744a bytes) 000933 1788840709.700647 mapwrite 0x01e7b6 thru 0x02e516 (0xfd61 bytes) 000934 1788840709.755569 punch from 0x019a75 to 0x0291af (0xf73b bytes) 000936 1788840709.764469 MIRROR_RESYNC 10 mirrors 000937 1788840712.654625 mapread 0x0376e7 thru 0x03a598 (0x2eb2 bytes) 000938 1788840712.676532 mapread 0x00f3f6 thru 0x013696 (0x42a1 bytes) 000940 1788840712.690874 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) 000942 1788840713.064858 punch from 0x02f6bd to 0x03a598 (0xaedc bytes) 000944 1788840713.166546 read 0x001e98 thru 0x005451 (0x35ba bytes) 000945 1788840713.176542 write 0x013f41 thru 0x01a99a (0x6a5a bytes) 000946 1788840713.189985 mapread 0x017a22 thru 0x01c6aa (0x4c89 bytes) 000947 1788840713.213232 trunc from 0x03a599 to 0x00787c (0x32d1c bytes) 000948 1788840713.268120 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x787d): Mirror 20: 0xe56815ea (stale) Mirror 23: 0xe56815ea (stale) Mirror 24: 0xe56815ea (stale) Mirror 25: 0xe56815ea (stale) Mirror 26: 0xe56815ea (stale) Mirror 27: 0xe56815ea (stale) Mirror 28: 0xe56815ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x787d): Mirror 20: 0xe56815ea (stale) Mirror 23: 0xe56815ea (stale) Mirror 24: 0xe56815ea (stale) Mirror 25: 0xe56815ea (stale) Mirror 26: 0xe56815ea (stale) Mirror 27: 0xe56815ea (stale) Mirror 28: 0xe56815ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x787d: skipped 000950 1788840713.815947 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x787d): Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x787d): Mirror 20: 0xe56815ea (stale) Mirror 23: 0xe56815ea (stale) Mirror 24: 0xe56815ea (stale) Mirror 25: 0xe56815ea (stale) Mirror 26: 0xe56815ea (stale) Mirror 27: 0xe56815ea (stale) Mirror 28: 0xe56815ea (stale) CRC-32 checksum value for chunk [0, 0x787d): Mirror 1: 0xe56815ea Mirror 29: 0xe56815ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x787d: skipped 000951 1788840714.673107 mapread 0x00048d thru 0x00787c (0x73f0 bytes) 000952 1788840714.716867 MIRROR_RESYNC 9 mirrors 000954 1788840715.990726 trunc from 0x00787d to 0x0015da (0x62a2 bytes) 000955 1788840716.133824 punch from 0x000252 to 0x0015da (0x1389 bytes) 000956 1788840716.140827 write 0x005a86 thru 0x005d7b (0x2f6 bytes) 000960 1788840716.146107 mapwrite 0x03c005 thru 0x03f1e3 (0x31df bytes) 000962 1788840716.246617 MIRROR_RESYNC 9 mirrors 000963 1788840718.920247 write 0x03fdee thru 0x03ffff (0x212 bytes) 000965 1788840719.043474 read 0x03daef thru 0x03dfcc (0x4de bytes) 000967 1788840719.055836 punch from 0x034eb7 to 0x03ffff (0xb149 bytes) 000968 1788840719.076084 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x509df16d (stale) Mirror 23: 0x509df16d (stale) Mirror 24: 0x509df16d (stale) Mirror 25: 0x509df16d (stale) Mirror 26: 0x509df16d (stale) Mirror 27: 0x509df16d (stale) Mirror 28: 0x509df16d (stale) Mirror 29: 0x509df16d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x128f6d89 Mirror 30: 0x128f6d89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000969 1788840720.408902 write 0x0019d0 thru 0x00e7c4 (0xcdf5 bytes) 000970 1788840720.501864 read 0x01f025 thru 0x02153b (0x2517 bytes) 000971 1788840720.512258 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x509df16d (stale) Mirror 23: 0x509df16d (stale) Mirror 24: 0x509df16d (stale) Mirror 25: 0x509df16d (stale) Mirror 26: 0x509df16d (stale) Mirror 27: 0x509df16d (stale) Mirror 28: 0x509df16d (stale) Mirror 29: 0x509df16d (stale) Mirror 30: 0x128f6d89 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf395d64c Mirror 31: 0xf395d64c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000972 1788840721.571252 read 0x009aa1 thru 0x00ecf6 (0x5256 bytes) 000973 1788840721.583314 write 0x01d5a9 thru 0x024716 (0x716e bytes) 000977 1788840721.652368 mapwrite 0x0338e5 thru 0x036234 (0x2950 bytes) 000981 1788840721.703791 trunc from 0x040000 to 0x0119e7 (0x2e618 bytes) 000983 1788840721.748856 mapread 0x0017ec thru 0x00fc48 (0xe45d bytes) 000984 1788840721.763413 MIRROR_RESYNC 11 mirrors 000985 1788840724.078655 punch from 0x00e950 to 0x0119e7 (0x3098 bytes) 000989 1788840724.146870 write 0x013edd thru 0x0159e9 (0x1b0d bytes) 000990 1788840724.154393 MIRROR_RESYNC 11 mirrors 000992 1788840725.986619 trunc from 0x0159ea to 0x032a8d (0x1d0a4 bytes) 000994 1788840726.075494 mapread 0x02be49 thru 0x02be64 (0x1c bytes) 000995 1788840726.081832 write 0x00f1b6 thru 0x0193c8 (0xa213 bytes) 000998 1788840726.096548 mapread 0x00ace5 thru 0x01321c (0x8538 bytes) 000999 1788840726.112890 trunc from 0x032a8e to 0x02f5c6 (0x34c7 bytes) 001000 1788840726.135633 mapread 0x002393 thru 0x00e9f6 (0xc664 bytes) All operations completed A-OK! PASS 70a (137s) == sanity-flr test 71: check mirror extend parent fid ==== 00:12:12 (1788840732) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 00:12:15 (1788840735) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.326644 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12616 s, 8.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 00:12:24 (1788840744) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1978656 .. Extending file size to 735712 .. Extending file size to 7301088 .. Extending file size to 7313344 .. Extending file size to 8525120 .. Extending file size to 7405472 .. Extending file size to 8090528 .. Extending file size to 8630592 .. Extending file size to 8734240 .. done Extending file size to 9251200 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8916512 .. Extending file size to 9337632 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9385504 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57437 57438 57439 57443 57444 failed 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:13:42 (1788840822) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1378848 .. Extending file size to 6975872 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 3010528 .. Extending file size to 7735968 .. Extending file size to 4596576 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6601984 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8563360 .. Extending file size to 8760480 .. Extending file size to 8947744 .. Extending file size to 9220544 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59204 59205 59206 59209 59210 done 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:15:00 (1788840900) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148175 s, 14.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.344517 s, 6.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x49563e01 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:15:20 (1788840920) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122962 s, 17.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 00:15:27 (1788840927) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e9:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:15:36 (1788840936) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19668 s, 10.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:15:43 (1788840943) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271351 s, 7.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:15:52 (1788840952) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.427755 s, 4.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:16:25 (1788840985) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120689 s, 8.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:17:25 (1788841045) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.333093 s, 6.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:18:25 (1788841105) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. Creating new pool test_204f oleg233-server: Pool lustre.test_204f created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.457206 s, 4.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg233-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:19:41 (1788841181) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:19:48 (1788841188) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:19:55 (1788841195) oleg233-server: Pool lustre.test_206 created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg233-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:20:17 (1788841217) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604638 s, 17.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458206 s, 22.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x66cdeacf Mirror 3: 0x66cdeacf PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:20:27 (1788841227) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:20:31 (1788841231) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:20:36 (1788841236) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.12177 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2080000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xbd0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xbd0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xbd0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1680000 Read this (4 bytes): BBBB 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:21:11 (1788841271) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0328197 s, 250 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:21:18 (1788841278) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420393 s, 24.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20e lmm_fid: [0x200000404:0x20e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20e lmm_fid: [0x200000404:0x20e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834446 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 00:21:20.000000000 -0400 Modify: 2026-09-08 00:21:20.000000000 -0400 Change: 2026-09-08 00:21:21.000000000 -0400 Birth: 2026-09-08 00:21:20.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:21:27 (1788841287) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:21:30 (1788841290) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.61106 s, 67.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0503608 s, 81.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 8 00:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 8 00:21 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:21:40 (1788841300) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:21:48 (1788841308) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1787 sec =========== 00:21:57 (1788841317) === sanity-flr: start cleanup 00:21:59 (1788841319) === === sanity-flr: finish cleanup 00:22:05 (1788841325) ===