-----============= acceptance-small: sanity-flr ============----- Tue Sep 15 14:36:41 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg311-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:36:58 (1789497418) === oleg311-client.virtnet: executing check_config_client /mnt/lustre oleg311-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg311-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9189d0ebe800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9189d0ebe800.idle_timeout=debug disable quota as required oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:37:32 (1789497452) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7648] == sanity-flr test 0a: lfs mirror create with -N option == 14:37:35 (1789497455) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:37:53 (1789497473) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:37:57 (1789497477) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:38:00 (1789497480) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:38:28 (1789497508) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:38:31 (1789497511) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 14:38:35 (1789497515) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9189d0ebe800.stats=clear osc.lustre-OST0001-osc-ffff9189d0ebe800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:38:43 (1789497523) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:38:55 (1789497535) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:39:06 (1789497546) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:39:09 (1789497549) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:39:18 (1789497558) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 14:39:30 (1789497570) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:39:39 (1789497579) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:39:44 (1789497584) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:39:53 (1789497593) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:40:03 (1789497603) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:40:09 (1789497609) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 2.20976 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139698 s, 7.5 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:40:22 (1789497622) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.133275 s, 31.5 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:40:31 (1789497631) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:40:41 (1789497641) Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9189d0ebe800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9189d0ebe800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg311-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9189d0ebe800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9189d0ebe800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (139s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:43:00 (1789497780) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124023 s, 16.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9189d0ebe800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9189d0ebe800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:43:47 (1789497827) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9189d0ebe800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9189d0ebe800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9189d0ebe800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9189d0ebe800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:45:17 (1789497917) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.97845 s, 6.6 MB/s -rw-r--r-- 1 root root 52428800 Sep 15 14:45 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.326s user 0m0.001s sys 0m0.165s Stopping /mnt/lustre-ost1 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9189d0ebe800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9189d0ebe800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.909s user 0m0.001s sys 0m0.170s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg311-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg311-server oleg311-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9189d0ebe800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9189d0ebe800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.308s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg311-server: oleg311-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg311-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9189d0ebe800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9189d0ebe800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:46:58 (1789498018) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:47:01 (1789498021) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:47:04 (1789498024) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 14:47:08 (1789498028) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 14:47:16 (1789498036) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.647419 s, 19.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:47:31 (1789498051) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742063 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469759 s, 22.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:47:39 (1789498059) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:47:48 (1789498068) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000792639 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0043506 s, 235 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000790341 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109395 s, 936 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 14:48:00 (1789498080) create test files with size 15730047 .. d8f2f28b3a92f56763f12da019956f5b - 78236817f6684f04ef00be20d5c27f31 - 0cf9eb52e090e54fefe907f9c0b70f5c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 14:48:27 (1789498107) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:48:42 (1789498122) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:48:49 (1789498129) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127704 s, 8.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514083 s, 20.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 14:48:59 (1789498139) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143074 s, 14.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.441044 s, 9.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.337866 s, 6.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 14:49:18 (1789498158) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:49:22 (1789498162) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:49:26 (1789498166) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511308 s, 20.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 14:49:35 (1789498175) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:49:39 (1789498179) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:49:50 (1789498190) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.429963 s, 24.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:50:03 (1789498203) llite.lustre-ffff9189c7f57000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.311659 s, 33.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9189d1950000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:50:19 (1789498219) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:50:59 (1789498259) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:51:40 (1789498300) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:51:44 (1789498304) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.286418 s, 11.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.154057 s, 20.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe2358f14 (nosync,stale) Mirror 2: 0xe2358f14 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 9b9f056568967afa62477efc1df7480d - mirror 1 checksum: 9baa9fcec4bdac56282bc8bd63c739b3 - mirror 2 checksum: 9b9f056568967afa62477efc1df7480d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T14:51:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T14:51:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 9b9f056568967afa62477efc1df7480d - mirror 1 checksum: 9baa9fcec4bdac56282bc8bd63c739b3 - mirror 2 checksum: 9baa9fcec4bdac56282bc8bd63c739b3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T14:51:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T14:51:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:51:57 (1789498317) Failing mds1 on oleg311-server Stopping /mnt/lustre-mds1 (opts:) on oleg311-server 14:52:04 (1789498324) shut down facet: mds1 facet_host: oleg311-server facet_failover_host: oleg311-server Failover mds1 to oleg311-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 14:52:39 (1789498359) targets are mounted 14:52:39 (1789498359) facet_failover done oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392224 s, 26.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:53:00 (1789498380) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:53:04 (1789498384) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:53:08 (1789498388) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0153401 s, 1.1 MB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:53:19 (1789498399) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:53:23 (1789498403) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00750397 s, 546 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.154961 s, 26.4 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:53:33 (1789498413) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:53:57 (1789498437) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:54:26 (1789498466) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:54:49 (1789498489) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:55:14 (1789498514) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 14:55:22 (1789498522) mirror create pid 54354 mirror split pid 54357 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54354 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54357 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 14:56:29 (1789498589) Seed set to 8591 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1789498591.596450 mapwrite 0x037e3f thru 0x0392e0 (0x14a2 bytes) 000002 1789498591.795784 punch from 0x023aed to 0x025b2c (0x2040 bytes) 000007 1789498591.810162 mapread 0x03194e thru 0x0392e0 (0x7993 bytes) 000008 1789498591.829430 mapread 0x037600 thru 0x0382f7 (0xcf8 bytes) 000009 1789498591.832916 punch from 0x02cd13 to 0x02fd5b (0x3049 bytes) 000010 1789498591.839409 punch from 0x0375b9 to 0x0392e0 (0x1d28 bytes) 000011 1789498591.857351 read 0x0201cb thru 0x028f69 (0x8d9f bytes) 000012 1789498591.869580 write 0x00ee68 thru 0x011a26 (0x2bbf bytes) truncating to largest ever: 0x1faf8 000015 1789498591.887466 trunc from 0x0392e1 to 0x01faf7 (0x197e9 bytes) 000016 1789498591.913889 punch from 0x013cab to 0x01bbdf (0x7f35 bytes) 000017 1789498591.945115 MIRROR_EXTEND to 2 mirrors 000020 1789498593.405325 punch from 0x009df6 to 0x015892 (0xba9d bytes) truncating to largest ever: 0x338ed 000021 1789498593.502086 trunc from 0x01faf8 to 0x0338ec (0x13df5 bytes) 000024 1789498593.535225 read 0x0051af thru 0x009eb6 (0x4d08 bytes) 000026 1789498593.559485 mapwrite 0x0128b2 thru 0x0146bc (0x1e0b bytes) 000027 1789498593.631997 mapread 0x00ac8f thru 0x013aba (0x8e2c bytes) 000028 1789498593.644853 MIRROR_RESYNC 2 mirrors 000029 1789498594.451471 MIRROR_EXTEND to 3 mirrors 000030 1789498595.346380 write 0x00eb82 thru 0x0145de (0x5a5d bytes) 000031 1789498595.489448 trunc from 0x0338ed to 0x02e631 (0x52bb bytes) 000033 1789498595.529166 write 0x00b106 thru 0x017a9f (0xc99a bytes) 000034 1789498595.552965 mapwrite 0x0330f6 thru 0x03ffff (0xcf0a bytes) 000035 1789498595.744350 mapwrite 0x037b05 thru 0x03ffff (0x84fb bytes) 000036 1789498595.817737 mapwrite 0x00f5fc thru 0x0196cd (0xa0d2 bytes) 000037 1789498595.938695 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9e9a4ef9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9e9a4ef9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000038 1789498596.463938 trunc from 0x040000 to 0x02b210 (0x14def bytes) 000039 1789498596.513268 mapwrite 0x02cb3b thru 0x03b8f4 (0xedba bytes) 000040 1789498596.682626 read 0x016570 thru 0x01e451 (0x7ee2 bytes) 000043 1789498596.705229 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000044 1789498596.845292 punch from 0x002269 to 0x010d6f (0xeb07 bytes) 000045 1789498596.851616 mapread 0x0214ee thru 0x02920c (0x7d1f bytes) 000046 1789498596.885586 MIRROR_EXTEND to 2 mirrors 000047 1789498598.327880 mapwrite 0x00df58 thru 0x013292 (0x533b bytes) 000050 1789498598.639315 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b8f5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8f5): Mirror 2: 0xea050941 (stale) CRC-32 checksum value for chunk [0, 0x3b8f5): Mirror 1: 0x4fb92ee7 Mirror 3: 0x4fb92ee7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8f5: skipped 000052 1789498600.241724 trunc from 0x03b8f5 to 0x01641c (0x254d8 bytes) 000053 1789498600.433378 mapread 0x002620 thru 0x011d84 (0xf765 bytes) 000054 1789498600.523880 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1641d): Mirror 2: 0x16948343 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1641d): Mirror 2: 0x16948343 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1641d: skipped 000055 1789498600.944286 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000056 1789498601.258132 trunc from 0x01641d to 0x00769f (0xed7d bytes) 000058 1789498601.306738 read 0x004d0e thru 0x00769f (0x2992 bytes) 000059 1789498601.319414 write 0x02485f thru 0x02a637 (0x5dd9 bytes) 000063 1789498601.324952 mapwrite 0x035067 thru 0x03e3ed (0x9387 bytes) 000065 1789498601.512182 trunc from 0x03e3ee to 0x019383 (0x2506a bytes) 000066 1789498601.558766 read 0x0148ab thru 0x0183a9 (0x3aff bytes) 000068 1789498601.576516 mapwrite 0x000734 thru 0x00a310 (0x9bdd bytes) 000069 1789498601.701312 mapread 0x003836 thru 0x009d74 (0x653f bytes) 000070 1789498601.702934 write 0x00a7f3 thru 0x013845 (0x9053 bytes) 000072 1789498601.722411 MIRROR_EXTEND to 2 mirrors 000073 1789498602.932306 mapread 0x005094 thru 0x00fa43 (0xa9b0 bytes) 000074 1789498603.003454 write 0x005c08 thru 0x00fac9 (0x9ec2 bytes) 000075 1789498603.110931 write 0x036a4d thru 0x03ffff (0x95b3 bytes) 000076 1789498603.123915 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000077 1789498603.417656 read 0x031223 thru 0x03de91 (0xcc6f bytes) 000078 1789498603.433922 mapwrite 0x01aec8 thru 0x022adf (0x7c18 bytes) 000079 1789498603.514109 mapread 0x0299da thru 0x039464 (0xfa8b bytes) 000081 1789498603.527682 punch from 0x0171e2 to 0x01ea98 (0x78b7 bytes) 000083 1789498603.537037 mapread 0x030a11 thru 0x031bc9 (0x11b9 bytes) 000084 1789498603.538619 mapwrite 0x027178 thru 0x028fa4 (0x1e2d bytes) 000086 1789498603.588044 read 0x0063cd thru 0x01456d (0xe1a1 bytes) 000087 1789498603.609801 MIRROR_EXTEND to 2 mirrors 000088 1789498604.795611 trunc from 0x040000 to 0x00c738 (0x338c7 bytes) 000089 1789498604.875583 punch from 0x008e22 to 0x00c738 (0x3917 bytes) 000091 1789498604.881430 write 0x039c9e thru 0x039ca0 (0x3 bytes) 000092 1789498604.882551 mapread 0x023d17 thru 0x02ea76 (0xad60 bytes) 000094 1789498604.921088 write 0x03a310 thru 0x03ffff (0x5cf0 bytes) 000095 1789498604.922320 mapread 0x02dd8d thru 0x035026 (0x729a bytes) 000096 1789498604.926867 read 0x023315 thru 0x0326a7 (0xf393 bytes) 000097 1789498604.929544 trunc from 0x040000 to 0x013e8e (0x2c171 bytes) 000098 1789498604.969445 punch from 0x00a0a4 to 0x0106d9 (0x6636 bytes) 000099 1789498604.978217 MIRROR_RESYNC 2 mirrors 000100 1789498606.009266 trunc from 0x013e8f to 0x02cee4 (0x19056 bytes) 000102 1789498606.132496 mapread 0x0162b9 thru 0x02235a (0xc0a2 bytes) 000103 1789498606.172980 mapread 0x002d18 thru 0x003744 (0xa2d bytes) 000106 1789498606.186431 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 2: 0x8df4b0bf (stale) CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 1: 0x8df4b0bf Mirror 3: 0x8df4b0bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cee5: skipped 000107 1789498607.433386 write 0x02475d thru 0x0272f9 (0x2b9d bytes) 000110 1789498607.565675 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 2: 0x8df4b0bf (stale) Mirror 3: 0x8df4b0bf (stale) CRC-32 checksum value for chunk [0, 0x2cee5): Mirror 1: 0x3ae8365f Mirror 4: 0x3ae8365f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cee5: skipped 000111 1789498608.898516 mapread 0x007322 thru 0x00a52a (0x3209 bytes) 000112 1789498608.958907 mapread 0x026840 thru 0x028792 (0x1f53 bytes) 000114 1789498609.005312 read 0x00512e thru 0x010bef (0xbac2 bytes) 000115 1789498609.021186 punch from 0x021dd8 to 0x0256f6 (0x391f bytes) 000116 1789498609.188976 punch from 0x02c13e to 0x02cee4 (0xda7 bytes) 000118 1789498609.203234 mapread 0x017575 thru 0x022061 (0xaaed bytes) 000119 1789498609.243141 read 0x004736 thru 0x0066c9 (0x1f94 bytes) 000121 1789498609.255361 read 0x000e81 thru 0x00a1f9 (0x9379 bytes) 000122 1789498609.256577 trunc from 0x02cee5 to 0x00e58f (0x1e955 bytes) 000123 1789498609.321036 MIRROR_RESYNC 4 mirrors 000124 1789498610.732575 mapwrite 0x017461 thru 0x01a1e5 (0x2d85 bytes) 000125 1789498610.995210 trunc from 0x01a1e6 to 0x01bc3f (0x1a5a bytes) 000126 1789498611.031923 mapread 0x0177fc thru 0x019d13 (0x2518 bytes) 000127 1789498611.034033 mapwrite 0x00f1e1 thru 0x01effd (0xfe1d bytes) 000128 1789498611.261718 read 0x006714 thru 0x00a67a (0x3f67 bytes) 000129 1789498611.274356 read 0x01bfff thru 0x01db88 (0x1b8a bytes) 000130 1789498611.276394 mapread 0x00b5eb thru 0x010ff3 (0x5a09 bytes) 000132 1789498611.282416 mapwrite 0x0295f0 thru 0x031a70 (0x8481 bytes) 000133 1789498611.425950 read 0x0023b7 thru 0x010c37 (0xe881 bytes) 000135 1789498611.444753 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31a71): Mirror 3: 0xd9fd6718 (stale) Mirror 4: 0xd9fd6718 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31a71): Mirror 3: 0xd9fd6718 (stale) Mirror 4: 0xd9fd6718 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31a71: skipped 000136 1789498611.793179 write 0x012cf9 thru 0x0198fa (0x6c02 bytes) 000137 1789498611.814008 punch from 0x004884 to 0x013e4f (0xf5cc bytes) 000138 1789498611.851990 punch from 0x02adb7 to 0x031a70 (0x6cba bytes) 000139 1789498611.867504 punch from 0x01adcd to 0x01b941 (0xb75 bytes) 000140 1789498611.872339 write 0x022801 thru 0x030fc0 (0xe7c0 bytes) 000141 1789498611.894366 write 0x02b614 thru 0x036610 (0xaffd bytes) 000142 1789498611.902371 MIRROR_RESYNC 3 mirrors 000145 1789498613.315382 write 0x038daf thru 0x03ffff (0x7251 bytes) 000147 1789498613.414318 read 0x032947 thru 0x036626 (0x3ce0 bytes) 000148 1789498613.431680 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd8a07d75 (stale) Mirror 4: 0xd8a07d75 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1ce23a02 Mirror 5: 0x1ce23a02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000149 1789498614.791371 write 0x0309d1 thru 0x032e5b (0x248b bytes) 000150 1789498614.904771 read 0x03df1d thru 0x03ffff (0x20e3 bytes) 000151 1789498614.930974 mapread 0x00e6c8 thru 0x01d88f (0xf1c8 bytes) 000152 1789498614.967574 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd8a07d75 (stale) Mirror 5: 0x1ce23a02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd8a07d75 (stale) Mirror 5: 0x1ce23a02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000153 1789498615.354545 write 0x0372be thru 0x0386ac (0x13ef bytes) 000154 1789498615.381410 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd8a07d75 (stale) Mirror 5: 0x1ce23a02 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1c0261ed Mirror 6: 0x1c0261ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000155 1789498616.921697 punch from 0x037853 to 0x03f056 (0x7804 bytes) 000157 1789498617.049669 trunc from 0x040000 to 0x01f5cf (0x20a30 bytes) 000158 1789498617.074840 read 0x00f2bc thru 0x01728b (0x7fd0 bytes) 000159 1789498617.084649 read 0x002214 thru 0x0079ea (0x57d7 bytes) 000161 1789498617.098643 trunc from 0x01f5d0 to 0x030500 (0x10f31 bytes) 000162 1789498617.129792 MIRROR_RESYNC 4 mirrors 000163 1789498618.990038 punch from 0x015434 to 0x01ca5e (0x762b bytes) 000164 1789498619.110778 read 0x00adda thru 0x0171d0 (0xc3f7 bytes) 000165 1789498619.121335 read 0x002b52 thru 0x00d22c (0xa6db bytes) 000168 1789498619.139357 mapread 0x00d11a thru 0x015638 (0x851f bytes) 000170 1789498619.144784 punch from 0x005478 to 0x01000f (0xab98 bytes) 000171 1789498619.151504 trunc from 0x030501 to 0x00715c (0x293a4 bytes) 000172 1789498619.183563 write 0x032290 thru 0x0359bf (0x3730 bytes) 000174 1789498619.185353 read 0x010ae4 thru 0x01ed86 (0xe2a3 bytes) 000175 1789498619.203318 mapwrite 0x004076 thru 0x00876d (0x46f8 bytes) 000177 1789498619.270441 read 0x025c83 thru 0x031bbc (0xbf3a bytes) 000180 1789498619.286337 trunc from 0x0359c0 to 0x001f36 (0x33a89 bytes) 000181 1789498619.321183 mapwrite 0x0171e1 thru 0x027078 (0xfe98 bytes) 000182 1789498619.431740 mapread 0x02317e thru 0x027078 (0x3efb bytes) 000184 1789498619.449834 MIRROR_RESYNC 4 mirrors 000185 1789498620.821904 read 0x01e19f thru 0x02222c (0x408e bytes) 000189 1789498620.831308 read 0x023890 thru 0x027078 (0x37e9 bytes) 000190 1789498620.848263 read 0x019fdb thru 0x01e616 (0x463c bytes) 000191 1789498620.848727 punch from 0x0172da to 0x02669b (0xf3c2 bytes) 000192 1789498620.941095 mapread 0x027013 thru 0x027078 (0x66 bytes) 000193 1789498620.960225 trunc from 0x027079 to 0x025c14 (0x1464 bytes) 000194 1789498620.997942 mapwrite 0x037513 thru 0x03b430 (0x3f1e bytes) 000195 1789498621.145003 write 0x037389 thru 0x03ffff (0x8c77 bytes) 000197 1789498621.147235 read 0x0088a8 thru 0x01676d (0xdec6 bytes) 000200 1789498621.157759 write 0x02f022 thru 0x034317 (0x52f6 bytes) 000201 1789498621.165288 trunc from 0x040000 to 0x011bac (0x2e453 bytes) 000203 1789498621.187031 punch from 0x00b2b3 to 0x011bac (0x68fa bytes) 000204 1789498621.193328 mapwrite 0x02290e thru 0x03202b (0xf71e bytes) 000205 1789498621.347328 mapwrite 0x031128 thru 0x03fbf5 (0xeace bytes) 000207 1789498621.454824 mapwrite 0x032c26 thru 0x03ffff (0xd3da bytes) 000208 1789498621.550089 mapread 0x020c33 thru 0x020ec4 (0x292 bytes) 000209 1789498621.551439 mapread 0x01d481 thru 0x02c51a (0xf09a bytes) 000210 1789498621.554277 punch from 0x01ccf5 to 0x021727 (0x4a33 bytes) 000211 1789498621.564431 mapread 0x026468 thru 0x02e467 (0x8000 bytes) 000212 1789498621.566315 write 0x01a227 thru 0x023305 (0x90df bytes) 000214 1789498621.569474 mapwrite 0x037408 thru 0x03ffff (0x8bf8 bytes) 000215 1789498621.633950 write 0x0342b5 thru 0x0361c6 (0x1f12 bytes) 000217 1789498621.637026 read 0x02d06b thru 0x031d7a (0x4d10 bytes) 000219 1789498621.639624 mapread 0x0131f9 thru 0x016dfa (0x3c02 bytes) 000220 1789498621.644637 punch from 0x02a727 to 0x038429 (0xdd03 bytes) 000221 1789498621.674562 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf93eae75 (stale) Mirror 6: 0xf93eae75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf93eae75 (stale) Mirror 6: 0xf93eae75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000222 1789498622.124519 write 0x00f1f3 thru 0x01b02c (0xbe3a bytes) 000223 1789498622.148560 trunc from 0x040000 to 0x02cf6a (0x13095 bytes) 000224 1789498622.185493 mapread 0x02293b thru 0x02cf6a (0xa630 bytes) 000225 1789498622.241698 read 0x00fe8e thru 0x01d3dd (0xd550 bytes) 000226 1789498622.251696 write 0x03c929 thru 0x03ffff (0x36d7 bytes) 000228 1789498622.269989 write 0x0280e6 thru 0x032ea7 (0xadc2 bytes) 000229 1789498622.286786 punch from 0x03f349 to 0x03ffff (0xcb7 bytes) truncating to largest ever: 0x3a41a 000230 1789498622.310404 trunc from 0x040000 to 0x03a419 (0x5be6 bytes) 000231 1789498622.350638 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 4: 0xf732e9f7 (stale) Mirror 6: 0xf732e9f7 (stale) CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 1: 0xd935161d Mirror 7: 0xd935161d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a41a: skipped 000233 1789498623.666366 punch from 0x00fb57 to 0x018a36 (0x8ee0 bytes) 000235 1789498623.748215 mapread 0x03a019 thru 0x03a419 (0x401 bytes) 000236 1789498623.757691 mapread 0x01e885 thru 0x026e59 (0x85d5 bytes) 000237 1789498623.785010 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 6: 0xf732e9f7 (stale) Mirror 7: 0xd935161d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 6: 0xf732e9f7 (stale) Mirror 7: 0xd935161d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a41a: skipped 000238 1789498624.291898 mapread 0x0241e1 thru 0x0248cb (0x6eb bytes) 000239 1789498624.297372 write 0x02cc78 thru 0x035f2f (0x92b8 bytes) 000240 1789498624.310214 read 0x005593 thru 0x010a80 (0xb4ee bytes) 000241 1789498624.321295 punch from 0x010cb8 to 0x01a661 (0x99aa bytes) 000243 1789498624.354246 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 6: 0xf732e9f7 (stale) Mirror 7: 0xd935161d (stale) CRC-32 checksum value for chunk [0, 0x3a41a): Mirror 1: 0xf3256393 Mirror 8: 0xf3256393 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a41a: skipped 000244 1789498625.891050 trunc from 0x03a41a to 0x029081 (0x11398 bytes) 000245 1789498625.974735 read 0x024be5 thru 0x029081 (0x449d bytes) 000246 1789498625.982999 mapread 0x018475 thru 0x01d32d (0x4eb9 bytes) 000247 1789498626.019307 MIRROR_RESYNC 4 mirrors 000250 1789498627.801708 write 0x024bd2 thru 0x027215 (0x2644 bytes) 000251 1789498627.882888 write 0x03dfea thru 0x03ffff (0x2016 bytes) 000253 1789498627.886187 mapwrite 0x005ef8 thru 0x01084f (0xa958 bytes) 000256 1789498627.997505 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4dbc664e (stale) Mirror 7: 0x4dbc664e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4dbc664e (stale) Mirror 7: 0x4dbc664e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000257 1789498628.498577 mapread 0x027926 thru 0x035071 (0xd74c bytes) 000258 1789498628.537781 mapwrite 0x00ffb7 thru 0x017b8e (0x7bd8 bytes) 000259 1789498628.611755 write 0x00ec8a thru 0x0174f8 (0x886f bytes) 000261 1789498628.615282 write 0x0082de thru 0x01544f (0xd172 bytes) 000264 1789498628.620764 write 0x01e581 thru 0x02df19 (0xf999 bytes) 000265 1789498628.630793 punch from 0x00c37d to 0x0171ff (0xae83 bytes) 000266 1789498628.667759 trunc from 0x040000 to 0x021ecc (0x1e133 bytes) 000269 1789498628.714989 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x21ecd): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ecd): Mirror 6: 0xe9af93ef (stale) Mirror 7: 0xe9af93ef (stale) CRC-32 checksum value for chunk [0, 0x21ecd): Mirror 1: 0x44afe641 Mirror 8: 0x44afe641 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ecd: skipped 000270 1789498630.369651 mapwrite 0x007ebb thru 0x007ee7 (0x2d bytes) 000271 1789498630.699008 write 0x0307b0 thru 0x0373de (0x6c2f bytes) 000273 1789498630.700896 trunc from 0x0373df to 0x01c40e (0x1afd0 bytes) 000274 1789498630.722592 mapwrite 0x025e29 thru 0x033d48 (0xdf20 bytes) 000275 1789498630.821338 mapread 0x015ddd thru 0x018a34 (0x2c58 bytes) 000277 1789498630.832727 punch from 0x006072 to 0x00ed67 (0x8cf6 bytes) 000278 1789498630.838212 write 0x03d13e thru 0x03ffff (0x2ec2 bytes) 000281 1789498630.841124 punch from 0x037633 to 0x03ffff (0x89cd bytes) 000283 1789498630.851019 read 0x03a112 thru 0x03b9c0 (0x18af bytes) 000286 1789498630.859484 mapread 0x0093f4 thru 0x00e257 (0x4e64 bytes) 000288 1789498630.883152 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4dbc664e (stale) Mirror 7: 0x4dbc664e (stale) Mirror 8: 0x9ab5839d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x120c9555 Mirror 9: 0x120c9555 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000290 1789498632.554693 punch from 0x033ba7 to 0x037812 (0x3c6c bytes) 000292 1789498632.651806 trunc from 0x040000 to 0x036d06 (0x92f9 bytes) 000294 1789498632.692126 read 0x013229 thru 0x021c62 (0xea3a bytes) 000296 1789498632.701164 read 0x01ae15 thru 0x01f169 (0x4355 bytes) 000298 1789498632.702369 mapwrite 0x00d86a thru 0x01255d (0x4cf4 bytes) 000299 1789498632.841663 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36d07): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d07): Mirror 6: 0xc850c8c (stale) Mirror 7: 0xc850c8c (stale) Mirror 8: 0xcf92e64f (stale) Mirror 9: 0x3fe4f3fe (stale) CRC-32 checksum value for chunk [0, 0x36d07): Mirror 1: 0x823ed619 Mirror 10: 0x823ed619 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d07: skipped 000302 1789498634.661959 read 0x0183c1 thru 0x01df77 (0x5bb7 bytes) 000303 1789498634.691508 read 0x003535 thru 0x009cc5 (0x6791 bytes) 000304 1789498634.730880 trunc from 0x036d07 to 0x000ed7 (0x35e2f bytes) 000306 1789498634.853405 punch from 0x000e68 to 0x000ed7 (0x70 bytes) 000307 1789498634.862847 read 0x000399 thru 0x000ed7 (0xb3f bytes) 000308 1789498634.870794 mapread 0x0003c1 thru 0x000ed7 (0xb17 bytes) 000310 1789498634.874026 write 0x02263d thru 0x02f6d3 (0xd097 bytes) 000311 1789498634.876199 mapread 0x00774f thru 0x010d68 (0x961a bytes) 000312 1789498634.894727 write 0x01f245 thru 0x02d6cb (0xe487 bytes) 000313 1789498634.903012 write 0x009852 thru 0x01491d (0xb0cc bytes) 000315 1789498634.909040 write 0x018632 thru 0x01e7f6 (0x61c5 bytes) 000316 1789498634.918810 punch from 0x017a44 to 0x021271 (0x982e bytes) 000317 1789498634.953832 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 6: 0x7a99dd4 (stale) Mirror 7: 0x7a99dd4 (stale) Mirror 8: 0xce6c0e49 (stale) Mirror 9: 0x8547a178 (stale) Mirror 10: 0x83b33b48 (stale) CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 1: 0x487ba40c Mirror 11: 0x487ba40c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6d4: skipped 000318 1789498636.226566 punch from 0x00d450 to 0x011d99 (0x494a bytes) 000321 1789498636.407552 mapwrite 0x00fa4b thru 0x0158c3 (0x5e79 bytes) 000322 1789498636.520077 mapread 0x021aae thru 0x025950 (0x3ea3 bytes) 000323 1789498636.552633 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 7: 0x7a99dd4 (stale) Mirror 8: 0xce6c0e49 (stale) Mirror 9: 0x8547a178 (stale) Mirror 10: 0x83b33b48 (stale) Mirror 11: 0x487ba40c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6d4): Mirror 7: 0x7a99dd4 (stale) Mirror 8: 0xce6c0e49 (stale) Mirror 9: 0x8547a178 (stale) Mirror 10: 0x83b33b48 (stale) Mirror 11: 0x487ba40c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6d4: skipped 000324 1789498637.221572 trunc from 0x02f6d4 to 0x0356d6 (0x6003 bytes) 000326 1789498637.238544 trunc from 0x0356d7 to 0x004774 (0x30f62 bytes) 000328 1789498637.256012 mapread 0x0026f8 thru 0x00334c (0xc55 bytes) 000329 1789498637.270522 mapwrite 0x00603d thru 0x00ae98 (0x4e5c bytes) 000330 1789498637.369912 punch from 0x00a169 to 0x00ae98 (0xd30 bytes) 000331 1789498637.387688 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xae99): Mirror 7: 0xb4ea27b (stale) Mirror 8: 0xb639d4a8 (stale) Mirror 9: 0x36fe3a6b (stale) Mirror 11: 0x4156d49c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae99): Mirror 7: 0xb4ea27b (stale) Mirror 8: 0xb639d4a8 (stale) Mirror 9: 0x36fe3a6b (stale) Mirror 11: 0x4156d49c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae99: skipped 000332 1789498637.859984 MIRROR_RESYNC 5 mirrors 000334 1789498639.535699 MIRROR_RESYNC 5 mirrors 000336 1789498640.021049 read 0x003ab7 thru 0x00ae98 (0x73e2 bytes) 000337 1789498640.030700 trunc from 0x00ae99 to 0x004449 (0x6a4f bytes) 000338 1789498640.128825 punch from 0x00006a to 0x00026d (0x204 bytes) 000340 1789498640.135219 mapwrite 0x01560b thru 0x01d55f (0x7f55 bytes) 000341 1789498640.253689 punch from 0x0145e4 to 0x01d55f (0x8f7c bytes) 000343 1789498640.261442 mapwrite 0x03089d thru 0x03151a (0xc7e bytes) 000344 1789498640.377593 trunc from 0x03151b to 0x01848b (0x1908f bytes) 000345 1789498640.403571 trunc from 0x01848c to 0x025125 (0xcc9a bytes) 000346 1789498640.435227 read 0x01a74a thru 0x025125 (0xa9dc bytes) 000348 1789498640.452295 trunc from 0x025126 to 0x038c71 (0x13b4c bytes) 000349 1789498640.500570 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: 0x37c6355e (stale) Mirror 8: 0x37c6355e (stale) Mirror 9: 0x37c6355e (stale) Mirror 11: 0x37c6355e (stale) CRC-32 checksum value for chunk [0, 0x38c72): Mirror 1: 0x5f993e79 Mirror 12: 0x5f993e79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c72: skipped 000350 1789498641.854328 mapread 0x021ade thru 0x023e0c (0x232f bytes) 000351 1789498641.901466 read 0x00f462 thru 0x017b0c (0x86ab bytes) 000352 1789498641.923844 write 0x022c37 thru 0x02404a (0x1414 bytes) 000354 1789498642.037837 read 0x021f63 thru 0x026f12 (0x4fb0 bytes) 000356 1789498642.065443 read 0x0298de thru 0x031749 (0x7e6c bytes) 000359 1789498642.068110 mapwrite 0x0330d4 thru 0x037adf (0x4a0c bytes) 000360 1789498642.131246 punch from 0x00ca4d to 0x016e0d (0xa3c1 bytes) 000361 1789498642.148468 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: 0x37c6355e (stale) Mirror 8: 0x37c6355e (stale) Mirror 9: 0x37c6355e (stale) Mirror 11: 0x37c6355e (stale) Mirror 12: 0x5f993e79 (stale) CRC-32 checksum value for chunk [0, 0x38c72): Mirror 1: 0x2dc67401 Mirror 13: 0x2dc67401 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c72: skipped 000363 1789498643.668824 read 0x023603 thru 0x029b7d (0x657b bytes) 000365 1789498643.683429 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: 0x37c6355e (stale) Mirror 8: 0x37c6355e (stale) Mirror 9: 0x37c6355e (stale) Mirror 12: 0x5f993e79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c72): Mirror 7: 0x37c6355e (stale) Mirror 8: 0x37c6355e (stale) Mirror 9: 0x37c6355e (stale) Mirror 12: 0x5f993e79 (stale) CRC-32 checksum value for chunk [0, 0x38c72): Mirror 1: 0x2dc67401 Mirror 13: 0x2dc67401 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c72: skipped 000366 1789498644.620606 punch from 0x033cc1 to 0x038c71 (0x4fb1 bytes) 000367 1789498644.751223 trunc from 0x038c72 to 0x02ee63 (0x9e0e bytes) 000368 1789498644.772146 read 0x0024b9 thru 0x008d5e (0x68a6 bytes) 000369 1789498644.780473 mapread 0x00655b thru 0x007ff8 (0x1a9e bytes) 000370 1789498644.782698 mapwrite 0x018a35 thru 0x01bf29 (0x34f5 bytes) 000371 1789498644.851741 mapread 0x015d71 thru 0x01873d (0x29cd bytes) 000372 1789498644.854952 read 0x02db66 thru 0x02ee63 (0x12fe bytes) 000373 1789498644.872584 write 0x0349ba thru 0x039fba (0x5601 bytes) 000375 1789498644.875421 punch from 0x031ebb to 0x039fba (0x8100 bytes) 000376 1789498644.893268 mapread 0x009385 thru 0x011bc4 (0x8840 bytes) 000377 1789498644.894778 read 0x00a350 thru 0x010129 (0x5dda bytes) 000378 1789498644.896231 write 0x03a413 thru 0x03b90f (0x14fd bytes) 000379 1789498644.897471 mapwrite 0x0002e6 thru 0x00e466 (0xe181 bytes) 000380 1789498645.003635 mapread 0x020f9d thru 0x029979 (0x89dd bytes) 000381 1789498645.005621 punch from 0x02445e to 0x02ea76 (0xa619 bytes) 000387 1789498645.032998 punch from 0x015637 to 0x018cf4 (0x36be bytes) 000388 1789498645.043180 write 0x019fd3 thru 0x01f4ed (0x551b bytes) 000389 1789498645.056484 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b910): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b910): Mirror 7: 0x4eb460c3 (stale) Mirror 8: 0x4eb460c3 (stale) Mirror 9: 0x4eb460c3 (stale) Mirror 12: 0x7cda0fbc (stale) Mirror 13: 0xaafa21ad (stale) CRC-32 checksum value for chunk [0, 0x3b910): Mirror 1: 0xc0aa491e Mirror 14: 0xc0aa491e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b910: skipped 000390 1789498646.524207 mapwrite 0x03874c thru 0x03ca65 (0x431a bytes) 000392 1789498646.927817 punch from 0x00e404 to 0x012925 (0x4522 bytes) 000393 1789498646.944534 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ca66): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca66): Mirror 7: 0x69d49512 (stale) Mirror 8: 0x69d49512 (stale) Mirror 9: 0x69d49512 (stale) Mirror 12: 0x507dc8a8 (stale) Mirror 13: 0xf3a7f017 (stale) Mirror 14: 0x7f5bd3b8 (stale) CRC-32 checksum value for chunk [0, 0x3ca66): Mirror 1: 0x13007467 Mirror 15: 0x13007467 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca66: skipped 000394 1789498648.804177 punch from 0x022946 to 0x02fcd4 (0xd38f bytes) 000395 1789498648.921536 read 0x0046bc thru 0x01299e (0xe2e3 bytes) 000396 1789498648.952853 write 0x00dd87 thru 0x01a3e5 (0xc65f bytes) 000398 1789498648.973854 trunc from 0x03ca66 to 0x02a23a (0x1282b bytes) truncating to largest ever: 0x3ea68 000399 1789498649.006360 trunc from 0x02a23b to 0x03ea67 (0x1482d bytes) 000402 1789498649.039919 read 0x02eef4 thru 0x039b37 (0xac44 bytes) 000403 1789498649.055444 mapwrite 0x024b1e thru 0x032d96 (0xe279 bytes) 000404 1789498649.130419 punch from 0x026376 to 0x0361b1 (0xfe3c bytes) 000405 1789498649.135945 trunc from 0x03ea68 to 0x0085aa (0x364bd bytes) 000406 1789498649.178724 read 0x001b35 thru 0x0085aa (0x6a76 bytes) 000407 1789498649.179543 trunc from 0x0085ab to 0x02ce2f (0x24885 bytes) 000412 1789498649.220817 read 0x021c29 thru 0x025a4c (0x3e24 bytes) 000413 1789498649.242129 punch from 0x01326c to 0x0151f4 (0x1f89 bytes) 000414 1789498649.249814 mapread 0x002a7c thru 0x0104b6 (0xda3b bytes) 000415 1789498649.253731 mapread 0x0126b2 thru 0x01cc1d (0xa56c bytes) 000416 1789498649.270661 write 0x02f0e7 thru 0x037608 (0x8522 bytes) 000417 1789498649.273098 mapwrite 0x017377 thru 0x0221e4 (0xae6e bytes) 000418 1789498649.414180 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37609): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37609): Mirror 7: 0xd3dd7f0 (stale) Mirror 8: 0xd3dd7f0 (stale) Mirror 9: 0xd3dd7f0 (stale) Mirror 12: 0xdf422c6 (stale) Mirror 13: 0x1c928b5 (stale) Mirror 14: 0xfd026a81 (stale) Mirror 15: 0x9d6e4a (stale) CRC-32 checksum value for chunk [0, 0x37609): Mirror 1: 0xa8cadcde Mirror 16: 0xa8cadcde lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37609: skipped 000420 1789498651.359870 punch from 0x003515 to 0x00aa60 (0x754c bytes) 000422 1789498651.500985 write 0x025bf7 thru 0x026e6e (0x1278 bytes) 000423 1789498651.518099 mapread 0x02316b thru 0x02f68a (0xc520 bytes) 000424 1789498651.546190 mapwrite 0x021193 thru 0x0277ca (0x6638 bytes) 000426 1789498651.696719 trunc from 0x037609 to 0x026ba5 (0x10a63 bytes) 000427 1789498651.739036 punch from 0x0268e3 to 0x026ba5 (0x2c3 bytes) 000428 1789498651.754995 write 0x027f35 thru 0x028e2c (0xef8 bytes) 000429 1789498651.757786 mapread 0x00d608 thru 0x018bbc (0xb5b5 bytes) 000430 1789498651.793042 read 0x0066b7 thru 0x00aa18 (0x4362 bytes) 000436 1789498651.797533 mapwrite 0x00291e thru 0x0117e8 (0xeecb bytes) 000437 1789498651.858820 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28e2d): Mirror 8: 0x809a1d33 (stale) Mirror 9: 0x809a1d33 (stale) Mirror 12: 0x572d2168 (stale) Mirror 13: 0x2ef9b509 (stale) Mirror 14: 0xf838567b (stale) Mirror 15: 0xc89a731c (stale) Mirror 16: 0xc719c25c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28e2d): Mirror 8: 0x809a1d33 (stale) Mirror 9: 0x809a1d33 (stale) Mirror 12: 0x572d2168 (stale) Mirror 13: 0x2ef9b509 (stale) Mirror 14: 0xf838567b (stale) Mirror 15: 0xc89a731c (stale) Mirror 16: 0xc719c25c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28e2d: skipped 000438 1789498652.345856 mapwrite 0x030445 thru 0x033203 (0x2dbf bytes) 000440 1789498652.548532 punch from 0x03313a to 0x033203 (0xca bytes) 000441 1789498652.553127 trunc from 0x033204 to 0x013ad9 (0x1f72a bytes) 000443 1789498652.584975 read 0x000ef5 thru 0x005be0 (0x4cec bytes) 000445 1789498652.599543 mapwrite 0x030c1a thru 0x03ffff (0xf3e6 bytes) 000446 1789498652.790944 trunc from 0x040000 to 0x02bae9 (0x14516 bytes) 000448 1789498652.817396 punch from 0x01c669 to 0x02b2f1 (0xec89 bytes) 000449 1789498652.825042 read 0x00c452 thru 0x01501c (0x8bcb bytes) 000450 1789498652.838987 punch from 0x027dd7 to 0x02b3f8 (0x3622 bytes) 000451 1789498652.844357 read 0x0130e8 thru 0x01d047 (0x9f60 bytes) 000453 1789498652.845927 mapwrite 0x01d283 thru 0x02ad60 (0xdade bytes) 000454 1789498652.908522 punch from 0x01e25b to 0x027e57 (0x9bfd bytes) 000455 1789498652.920657 mapread 0x00a05e thru 0x0171fa (0xd19d bytes) 000457 1789498652.926375 punch from 0x00f201 to 0x01af3a (0xbd3a bytes) 000458 1789498652.938643 MIRROR_RESYNC 8 mirrors 000460 1789498656.901233 trunc from 0x02baea to 0x035906 (0x9e1d bytes) 000461 1789498656.986332 mapread 0x02ac86 thru 0x0357b3 (0xab2e bytes) 000462 1789498657.011030 trunc from 0x035907 to 0x018b26 (0x1cde0 bytes) 000463 1789498657.041619 trunc from 0x018b27 to 0x01938e (0x868 bytes) 000464 1789498657.061742 punch from 0x0123d2 to 0x017c9c (0x58cb bytes) 000466 1789498657.067158 trunc from 0x01938f to 0x020749 (0x73bb bytes) 000468 1789498657.089773 read 0x00f9e4 thru 0x0130ec (0x3709 bytes) 000469 1789498657.100627 mapwrite 0x0206eb thru 0x02e6a3 (0xdfb9 bytes) 000470 1789498657.203930 trunc from 0x02e6a4 to 0x0206a0 (0xe003 bytes) 000471 1789498657.220824 punch from 0x0106a2 to 0x01dd1f (0xd67e bytes) 000473 1789498657.226123 punch from 0x000b81 to 0x001813 (0xc93 bytes) 000474 1789498657.231547 write 0x02fa59 thru 0x03c8cb (0xce73 bytes) 000475 1789498657.233619 trunc from 0x03c8cc to 0x0307e2 (0xc0e9 bytes) 000477 1789498657.253926 read 0x01e7ff thru 0x029184 (0xa986 bytes) 000478 1789498657.269587 MIRROR_RESYNC 8 mirrors 000479 1789498660.631701 mapread 0x00222c thru 0x00ac9e (0x8a73 bytes) 000480 1789498660.678314 mapwrite 0x03b3a6 thru 0x03ffff (0x4c5a bytes) 000481 1789498660.899552 write 0x0322c7 thru 0x03ffff (0xdd39 bytes) 000483 1789498660.907901 read 0x0113cd thru 0x01e723 (0xd357 bytes) 000485 1789498660.928325 mapread 0x03574a thru 0x03665e (0xf15 bytes) 000487 1789498660.933412 mapread 0x0095c7 thru 0x00f067 (0x5aa1 bytes) 000490 1789498660.943286 write 0x002eb6 thru 0x0081ab (0x52f6 bytes) 000491 1789498660.949016 trunc from 0x040000 to 0x02314d (0x1ceb2 bytes) 000492 1789498660.967943 mapread 0x0153df thru 0x01d6de (0x8300 bytes) 000493 1789498660.972868 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2314e): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2314e): Mirror 8: 0xc31a0cbb (stale) Mirror 9: 0xc31a0cbb (stale) Mirror 12: 0xc31a0cbb (stale) Mirror 13: 0xc31a0cbb (stale) Mirror 14: 0xc31a0cbb (stale) Mirror 15: 0xc31a0cbb (stale) Mirror 16: 0xc31a0cbb (stale) CRC-32 checksum value for chunk [0, 0x2314e): Mirror 1: 0x2fa3d36f Mirror 17: 0x2fa3d36f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2314e: skipped 000496 1789498662.780539 MIRROR_RESYNC 9 mirrors 000498 1789498664.995847 punch from 0x00f752 to 0x01cc2f (0xd4de bytes) 000499 1789498665.058945 punch from 0x01799a to 0x01d4ff (0x5b66 bytes) 000502 1789498665.065337 read 0x01f1ff thru 0x02314d (0x3f4f bytes) 000503 1789498665.073267 punch from 0x01f92f to 0x02314d (0x381f bytes) 000504 1789498665.077561 punch from 0x006792 to 0x00a33a (0x3ba9 bytes) 000505 1789498665.082710 mapwrite 0x0205aa thru 0x02f78b (0xf1e2 bytes) 000507 1789498665.245922 trunc from 0x02f78c to 0x02c19e (0x35ed bytes) 000509 1789498665.276627 mapread 0x02a275 thru 0x02c19e (0x1f2a bytes) 000510 1789498665.278017 mapwrite 0x018782 thru 0x021c4b (0x94ca bytes) 000511 1789498665.341277 mapread 0x013fe4 thru 0x016cef (0x2d0c bytes) 000512 1789498665.350906 MIRROR_RESYNC 9 mirrors 000514 1789498668.883884 write 0x03ee3c thru 0x03ffff (0x11c4 bytes) 000515 1789498668.996895 MIRROR_RESYNC 9 mirrors 000516 1789498672.960874 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) 000517 1789498673.486856 MIRROR_RESYNC 8 mirrors 000518 1789498674.155273 trunc from 0x040000 to 0x01e729 (0x218d6 bytes) 000521 1789498674.303047 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1e72a): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e72a): Mirror 8: 0xebdfd5b2 (stale) Mirror 9: 0xebdfd5b2 (stale) Mirror 13: 0xebdfd5b2 (stale) Mirror 14: 0xebdfd5b2 (stale) Mirror 15: 0xebdfd5b2 (stale) Mirror 16: 0xebdfd5b2 (stale) Mirror 17: 0xebdfd5b2 (stale) CRC-32 checksum value for chunk [0, 0x1e72a): Mirror 1: 0xebdfd5b2 Mirror 18: 0xebdfd5b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e72a: skipped 000522 1789498676.208477 mapread 0x014aa3 thru 0x01867c (0x3bda bytes) 000523 1789498676.260756 punch from 0x00af64 to 0x016907 (0xb9a4 bytes) 000524 1789498676.420910 write 0x02abfd thru 0x03991e (0xed22 bytes) 000525 1789498676.434631 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3991f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3991f): Mirror 8: 0xe2fcd521 (stale) Mirror 9: 0xe2fcd521 (stale) Mirror 13: 0xe2fcd521 (stale) Mirror 14: 0xe2fcd521 (stale) Mirror 15: 0xe2fcd521 (stale) Mirror 16: 0xe2fcd521 (stale) Mirror 17: 0xe2fcd521 (stale) Mirror 18: 0xb72083d (stale) CRC-32 checksum value for chunk [0, 0x3991f): Mirror 1: 0xe8a10a57 Mirror 19: 0xe8a10a57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3991f: skipped 000527 1789498678.084978 write 0x03c8e1 thru 0x03ffff (0x371f bytes) 000528 1789498678.266721 mapread 0x038639 thru 0x03ffff (0x79c7 bytes) 000529 1789498678.299753 mapwrite 0x014ded thru 0x01c715 (0x7929 bytes) 000530 1789498678.374786 punch from 0x01e81e to 0x02c8a7 (0xe08a bytes) 000531 1789498678.379570 write 0x00e7d9 thru 0x019687 (0xaeaf bytes) 000532 1789498678.380790 mapwrite 0x0311ba thru 0x03cdfc (0xbc43 bytes) 000533 1789498678.439230 mapwrite 0x023553 thru 0x02a392 (0x6e40 bytes) 000535 1789498678.497369 mapread 0x01f691 thru 0x020ffe (0x196e bytes) 000537 1789498678.498700 mapwrite 0x0078c5 thru 0x014a34 (0xd170 bytes) 000539 1789498678.557208 mapwrite 0x00d8a8 thru 0x0199af (0xc108 bytes) 000540 1789498678.623728 trunc from 0x040000 to 0x002b64 (0x3d49b bytes) 000542 1789498678.666670 mapwrite 0x0123a6 thru 0x02165e (0xf2b9 bytes) 000543 1789498678.889418 mapread 0x00ed5b thru 0x01ea46 (0xfcec bytes) 000544 1789498678.890611 punch from 0x006dc8 to 0x00c7a5 (0x59de bytes) 000546 1789498678.913745 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2165f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2165f): Mirror 8: 0x56f074a3 (stale) Mirror 9: 0x56f074a3 (stale) Mirror 13: 0x56f074a3 (stale) Mirror 14: 0x56f074a3 (stale) Mirror 15: 0x56f074a3 (stale) Mirror 16: 0x56f074a3 (stale) Mirror 17: 0x56f074a3 (stale) Mirror 18: 0xa6de638d (stale) Mirror 19: 0x2e373916 (stale) CRC-32 checksum value for chunk [0, 0x2165f): Mirror 1: 0x16e594d9 Mirror 20: 0x16e594d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2165f: skipped 000547 1789498680.287414 trunc from 0x02165f to 0x03de2d (0x1c7cf bytes) 000548 1789498680.389950 mapread 0x02dcfb thru 0x03a867 (0xcb6d bytes) 000550 1789498680.419936 trunc from 0x03de2e to 0x006c0e (0x3721f bytes) 000551 1789498680.442524 read 0x00514a thru 0x006c0e (0x1ac5 bytes) 000552 1789498680.453512 mapread 0x001391 thru 0x006c0e (0x587e bytes) 000553 1789498680.461106 read 0x0027cc thru 0x002eb1 (0x6e6 bytes) 000556 1789498680.463213 read 0x00325d thru 0x006c0e (0x39b2 bytes) 000557 1789498680.463902 write 0x03538d thru 0x03ffff (0xac73 bytes) 000558 1789498680.466047 mapread 0x00300f thru 0x00ec4f (0xbc41 bytes) 000560 1789498680.477568 mapwrite 0x0283eb thru 0x02cdbc (0x49d2 bytes) 000561 1789498680.546484 mapwrite 0x0078ad thru 0x013fb7 (0xc70b bytes) 000562 1789498680.632755 punch from 0x022834 to 0x031c86 (0xf453 bytes) 000563 1789498680.651475 mapwrite 0x029d86 thru 0x036368 (0xc5e3 bytes) 000564 1789498680.805495 write 0x0141c4 thru 0x01fa24 (0xb861 bytes) 000565 1789498680.806883 trunc from 0x040000 to 0x017bd0 (0x2842f bytes) 000566 1789498680.841214 trunc from 0x017bd1 to 0x00e036 (0x9b9a bytes) 000567 1789498680.882569 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe037): Mirror 8: 0x6118ab47 (stale) Mirror 9: 0x6118ab47 (stale) Mirror 14: 0x6118ab47 (stale) Mirror 15: 0x6118ab47 (stale) Mirror 16: 0x6118ab47 (stale) Mirror 17: 0x6118ab47 (stale) Mirror 18: 0x6118ab47 (stale) Mirror 19: 0xdcf84285 (stale) Mirror 20: 0x68d1c0b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe037): Mirror 8: 0x6118ab47 (stale) Mirror 9: 0x6118ab47 (stale) Mirror 14: 0x6118ab47 (stale) Mirror 15: 0x6118ab47 (stale) Mirror 16: 0x6118ab47 (stale) Mirror 17: 0x6118ab47 (stale) Mirror 18: 0x6118ab47 (stale) Mirror 19: 0xdcf84285 (stale) Mirror 20: 0x68d1c0b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe037: skipped 000570 1789498681.418306 mapread 0x003751 thru 0x00e036 (0xa8e6 bytes) 000572 1789498681.438040 mapread 0x005f87 thru 0x00e036 (0x80b0 bytes) 000574 1789498681.439671 write 0x002c01 thru 0x007505 (0x4905 bytes) 000575 1789498681.441838 read 0x00031a thru 0x007e79 (0x7b60 bytes) 000576 1789498681.447796 punch from 0x00857c to 0x008d12 (0x797 bytes) 000577 1789498681.478183 MIRROR_RESYNC 10 mirrors 000579 1789498683.408454 mapread 0x00b283 thru 0x00e036 (0x2db4 bytes) 000581 1789498683.429498 trunc from 0x00e037 to 0x02d98b (0x1f955 bytes) 000582 1789498683.522153 trunc from 0x02d98c to 0x039e0e (0xc483 bytes) 000583 1789498683.551734 trunc from 0x039e0f to 0x018dda (0x21034 bytes) 000585 1789498683.574302 trunc from 0x018ddb to 0x033238 (0x1a45e bytes) 000586 1789498683.608145 mapwrite 0x0331b8 thru 0x033ee3 (0xd2c bytes) 000588 1789498683.718634 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33ee4): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ee4): Mirror 8: 0x8ec7d685 (stale) Mirror 9: 0x8ec7d685 (stale) Mirror 14: 0x8ec7d685 (stale) Mirror 15: 0x8ec7d685 (stale) Mirror 16: 0x8ec7d685 (stale) Mirror 17: 0x8ec7d685 (stale) Mirror 18: 0x8ec7d685 (stale) Mirror 19: 0x8ec7d685 (stale) Mirror 20: 0x8ec7d685 (stale) CRC-32 checksum value for chunk [0, 0x33ee4): Mirror 1: 0xdc90fb8b Mirror 21: 0xdc90fb8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ee4: skipped 000589 1789498685.114367 write 0x02dd29 thru 0x039392 (0xb66a bytes) 000590 1789498685.245834 read 0x0027bd thru 0x00d2ed (0xab31 bytes) 000591 1789498685.260287 read 0x01866b thru 0x01905c (0x9f2 bytes) 000592 1789498685.280191 trunc from 0x039393 to 0x028ff9 (0x10399 bytes) 000595 1789498685.310450 mapread 0x010fd1 thru 0x01cd1f (0xbd4f bytes) 000597 1789498685.312493 mapwrite 0x017078 thru 0x01c1bb (0x5144 bytes) 000600 1789498685.401594 read 0x009990 thru 0x014135 (0xa7a6 bytes) 000601 1789498685.402587 mapread 0x018692 thru 0x01f00f (0x697e bytes) 000602 1789498685.403574 punch from 0x0271be to 0x028ff9 (0x1e3c bytes) 000605 1789498685.421445 write 0x0173d0 thru 0x023c5f (0xc890 bytes) 000607 1789498685.425688 write 0x0359d6 thru 0x037eeb (0x2516 bytes) 000608 1789498685.427799 trunc from 0x037eec to 0x018518 (0x1f9d3 bytes) 000609 1789498685.446801 punch from 0x00a609 to 0x00ac02 (0x5fa bytes) 000611 1789498685.465670 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18519): Mirror 8: 0x18e610ab (stale) Mirror 9: 0x18e610ab (stale) Mirror 14: 0x18e610ab (stale) Mirror 16: 0x18e610ab (stale) Mirror 17: 0x18e610ab (stale) Mirror 18: 0x18e610ab (stale) Mirror 19: 0x18e610ab (stale) Mirror 20: 0x18e610ab (stale) Mirror 21: 0x18e610ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18519): Mirror 8: 0x18e610ab (stale) Mirror 9: 0x18e610ab (stale) Mirror 14: 0x18e610ab (stale) Mirror 16: 0x18e610ab (stale) Mirror 17: 0x18e610ab (stale) Mirror 18: 0x18e610ab (stale) Mirror 19: 0x18e610ab (stale) Mirror 20: 0x18e610ab (stale) Mirror 21: 0x18e610ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18519: skipped 000612 1789498685.966615 mapwrite 0x0243b1 thru 0x030037 (0xbc87 bytes) 000613 1789498686.196969 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x30038): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30038): Mirror 8: 0xc1a3c42c (stale) Mirror 9: 0xc1a3c42c (stale) Mirror 14: 0xc1a3c42c (stale) Mirror 16: 0xc1a3c42c (stale) Mirror 17: 0xc1a3c42c (stale) Mirror 18: 0xc1a3c42c (stale) Mirror 19: 0xc1a3c42c (stale) Mirror 20: 0xc1a3c42c (stale) Mirror 21: 0xc1a3c42c (stale) CRC-32 checksum value for chunk [0, 0x30038): Mirror 1: 0xa6b1d1d Mirror 22: 0xa6b1d1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30038: skipped 000614 1789498687.801575 punch from 0x023970 to 0x02c451 (0x8ae2 bytes) 000617 1789498687.905632 punch from 0x002b70 to 0x00c577 (0x9a08 bytes) 000620 1789498687.913270 read 0x017c33 thru 0x01ad2c (0x30fa bytes) 000621 1789498687.926444 write 0x03e147 thru 0x03ffff (0x1eb9 bytes) 000622 1789498687.927752 punch from 0x009c48 to 0x00a395 (0x74e bytes) 000623 1789498687.939738 read 0x02c89a thru 0x0304be (0x3c25 bytes) 000624 1789498687.959503 punch from 0x0144b3 to 0x01ff9a (0xbae8 bytes) 000626 1789498687.968193 mapread 0x0304a2 thru 0x03ffff (0xfb5e bytes) 000627 1789498687.970371 punch from 0x02cb41 to 0x034668 (0x7b28 bytes) 000628 1789498687.978724 mapwrite 0x03dbd5 thru 0x03ffff (0x242b bytes) 000630 1789498688.055039 read 0x009dbc thru 0x01186f (0x7ab4 bytes) 000631 1789498688.056719 punch from 0x014883 to 0x01f114 (0xa892 bytes) 000633 1789498688.067173 read 0x01bac8 thru 0x02aab2 (0xefeb bytes) 000634 1789498688.079045 punch from 0x020e6d to 0x023250 (0x23e4 bytes) 000635 1789498688.085479 mapwrite 0x037657 thru 0x03ffff (0x89a9 bytes) 000636 1789498688.184095 mapread 0x02854e thru 0x037d50 (0xf803 bytes) 000637 1789498688.186733 mapread 0x006269 thru 0x00f277 (0x900f bytes) 000639 1789498688.188657 punch from 0x02eb06 to 0x02f27b (0x776 bytes) 000640 1789498688.202259 trunc from 0x040000 to 0x0244cc (0x1bb33 bytes) 000641 1789498688.232142 punch from 0x01637f to 0x02402c (0xdcae bytes) 000642 1789498688.246924 mapwrite 0x03dfd5 thru 0x03ffff (0x202b bytes) 000643 1789498688.322812 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x341b57c0 (stale) Mirror 9: 0x341b57c0 (stale) Mirror 14: 0x341b57c0 (stale) Mirror 16: 0x341b57c0 (stale) Mirror 17: 0x341b57c0 (stale) Mirror 18: 0x341b57c0 (stale) Mirror 19: 0x341b57c0 (stale) Mirror 20: 0x341b57c0 (stale) Mirror 21: 0x67c1b114 (stale) Mirror 22: 0xf0a80cc8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x16762b4f Mirror 23: 0x16762b4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000644 1789498689.512279 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x341b57c0 (stale) Mirror 9: 0x341b57c0 (stale) Mirror 14: 0x341b57c0 (stale) Mirror 16: 0x341b57c0 (stale) Mirror 17: 0x341b57c0 (stale) Mirror 18: 0x341b57c0 (stale) Mirror 19: 0x341b57c0 (stale) Mirror 20: 0x341b57c0 (stale) Mirror 21: 0x67c1b114 (stale) Mirror 22: 0xf0a80cc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x341b57c0 (stale) Mirror 9: 0x341b57c0 (stale) Mirror 14: 0x341b57c0 (stale) Mirror 16: 0x341b57c0 (stale) Mirror 17: 0x341b57c0 (stale) Mirror 18: 0x341b57c0 (stale) Mirror 19: 0x341b57c0 (stale) Mirror 20: 0x341b57c0 (stale) Mirror 21: 0x67c1b114 (stale) Mirror 22: 0xf0a80cc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000645 1789498690.108988 write 0x0333e3 thru 0x0399be (0x65dc bytes) 000646 1789498690.145789 MIRROR_RESYNC 11 mirrors 000647 1789498696.784778 mapwrite 0x01140b thru 0x018d7a (0x7970 bytes) 000649 1789498697.025660 trunc from 0x040000 to 0x01a746 (0x258b9 bytes) 000650 1789498697.050659 write 0x01b9e0 thru 0x01e028 (0x2649 bytes) 000651 1789498697.053178 punch from 0x01a6f9 to 0x01e028 (0x3930 bytes) 000652 1789498697.069006 mapread 0x0026cc thru 0x00c299 (0x9bce bytes) 000653 1789498697.079290 read 0x011f60 thru 0x012520 (0x5c1 bytes) 000654 1789498697.080173 trunc from 0x01e029 to 0x008c58 (0x153d0 bytes) 000657 1789498697.104166 trunc from 0x008c59 to 0x00c98e (0x3d36 bytes) 000658 1789498697.126264 trunc from 0x00c98f to 0x02a8b5 (0x1df27 bytes) 000660 1789498697.143019 mapwrite 0x00a612 thru 0x0120d8 (0x7ac7 bytes) 000661 1789498697.189525 mapread 0x00167b thru 0x00df87 (0xc90d bytes) 000662 1789498697.191394 write 0x028900 thru 0x02ced5 (0x45d6 bytes) 000663 1789498697.196611 mapwrite 0x0131b6 thru 0x01a574 (0x73bf bytes) 000664 1789498697.254297 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ced6): Mirror 8: 0xfe3925ee (stale) Mirror 9: 0xfe3925ee (stale) Mirror 14: 0xfe3925ee (stale) Mirror 16: 0xfe3925ee (stale) Mirror 17: 0xfe3925ee (stale) Mirror 18: 0xfe3925ee (stale) Mirror 19: 0xfe3925ee (stale) Mirror 20: 0xfe3925ee (stale) Mirror 22: 0xfe3925ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ced6): Mirror 8: 0xfe3925ee (stale) Mirror 9: 0xfe3925ee (stale) Mirror 14: 0xfe3925ee (stale) Mirror 16: 0xfe3925ee (stale) Mirror 17: 0xfe3925ee (stale) Mirror 18: 0xfe3925ee (stale) Mirror 19: 0xfe3925ee (stale) Mirror 20: 0xfe3925ee (stale) Mirror 22: 0xfe3925ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ced6: skipped 000665 1789498698.263851 read 0x019ac9 thru 0x025d0f (0xc247 bytes) 000666 1789498698.284481 trunc from 0x02ced6 to 0x01a3a3 (0x12b32 bytes) 000667 1789498698.320854 MIRROR_RESYNC 10 mirrors 000670 1789498701.002757 mapread 0x004f09 thru 0x0086e5 (0x37dd bytes) 000672 1789498701.036356 trunc from 0x01a3a4 to 0x03284c (0x184a9 bytes) 000673 1789498701.111559 mapread 0x027656 thru 0x03284c (0xb1f7 bytes) 000674 1789498701.152715 trunc from 0x03284d to 0x017f6a (0x1a8e2 bytes) 000677 1789498701.190613 read 0x00c8c2 thru 0x017f6a (0xb6a9 bytes) 000678 1789498701.208769 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17f6b): Mirror 8: 0x3b9c271 (stale) Mirror 9: 0x3b9c271 (stale) Mirror 14: 0x3b9c271 (stale) Mirror 16: 0x3b9c271 (stale) Mirror 18: 0x3b9c271 (stale) Mirror 19: 0x3b9c271 (stale) Mirror 20: 0x3b9c271 (stale) Mirror 22: 0x3b9c271 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17f6b): Mirror 8: 0x3b9c271 (stale) Mirror 9: 0x3b9c271 (stale) Mirror 14: 0x3b9c271 (stale) Mirror 16: 0x3b9c271 (stale) Mirror 18: 0x3b9c271 (stale) Mirror 19: 0x3b9c271 (stale) Mirror 20: 0x3b9c271 (stale) Mirror 22: 0x3b9c271 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17f6b: skipped 000680 1789498701.840627 mapwrite 0x02665a thru 0x031787 (0xb12e bytes) 000681 1789498702.064722 punch from 0x01983b to 0x025d9e (0xc564 bytes) 000682 1789498702.087989 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x31788): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31788): Mirror 8: 0x55e37080 (stale) Mirror 9: 0x55e37080 (stale) Mirror 14: 0x55e37080 (stale) Mirror 16: 0x55e37080 (stale) Mirror 18: 0x55e37080 (stale) Mirror 19: 0x55e37080 (stale) Mirror 20: 0x55e37080 (stale) Mirror 22: 0x55e37080 (stale) CRC-32 checksum value for chunk [0, 0x31788): Mirror 1: 0x82e34b4a Mirror 23: 0x82e34b4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31788: skipped 000684 1789498703.616964 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31788): Mirror 8: 0x55e37080 (stale) Mirror 9: 0x55e37080 (stale) Mirror 16: 0x55e37080 (stale) Mirror 18: 0x55e37080 (stale) Mirror 19: 0x55e37080 (stale) Mirror 20: 0x55e37080 (stale) Mirror 22: 0x55e37080 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31788): Mirror 8: 0x55e37080 (stale) Mirror 9: 0x55e37080 (stale) Mirror 16: 0x55e37080 (stale) Mirror 18: 0x55e37080 (stale) Mirror 19: 0x55e37080 (stale) Mirror 20: 0x55e37080 (stale) Mirror 22: 0x55e37080 (stale) CRC-32 checksum value for chunk [0, 0x31788): Mirror 1: 0x82e34b4a Mirror 23: 0x82e34b4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31788: skipped 000686 1789498704.708746 mapwrite 0x0381e2 thru 0x038cbc (0xadb bytes) 000687 1789498705.016990 mapwrite 0x025269 thru 0x030021 (0xadb9 bytes) 000688 1789498705.170832 trunc from 0x038cbd to 0x012037 (0x26c85 bytes) 000690 1789498705.214805 write 0x009e8b thru 0x013022 (0x9198 bytes) 000691 1789498705.221842 mapread 0x002f5f thru 0x00f2a7 (0xc349 bytes) 000692 1789498705.236556 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x13023): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13023): Mirror 8: 0x96dee0f0 (stale) Mirror 9: 0x96dee0f0 (stale) Mirror 16: 0x96dee0f0 (stale) Mirror 18: 0x96dee0f0 (stale) Mirror 19: 0x96dee0f0 (stale) Mirror 20: 0x96dee0f0 (stale) Mirror 22: 0x96dee0f0 (stale) Mirror 23: 0x96dee0f0 (stale) CRC-32 checksum value for chunk [0, 0x13023): Mirror 1: 0x3e17d499 Mirror 24: 0x3e17d499 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13023: skipped 000693 1789498706.680735 read 0x006aca thru 0x00d63c (0x6b73 bytes) 000694 1789498706.715782 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13023): Mirror 8: 0x96dee0f0 (stale) Mirror 9: 0x96dee0f0 (stale) Mirror 16: 0x96dee0f0 (stale) Mirror 18: 0x96dee0f0 (stale) Mirror 19: 0x96dee0f0 (stale) Mirror 20: 0x96dee0f0 (stale) Mirror 22: 0x96dee0f0 (stale) Mirror 23: 0x96dee0f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13023): Mirror 8: 0x96dee0f0 (stale) Mirror 9: 0x96dee0f0 (stale) Mirror 16: 0x96dee0f0 (stale) Mirror 18: 0x96dee0f0 (stale) Mirror 19: 0x96dee0f0 (stale) Mirror 20: 0x96dee0f0 (stale) Mirror 22: 0x96dee0f0 (stale) Mirror 23: 0x96dee0f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13023: skipped 000695 1789498707.325604 write 0x027026 thru 0x02ee7e (0x7e59 bytes) 000696 1789498707.357688 trunc from 0x02ee7f to 0x00b34d (0x23b31 bytes) 000697 1789498707.394722 write 0x026b94 thru 0x03065a (0x9ac7 bytes) 000700 1789498707.405198 trunc from 0x03065b to 0x025bac (0xaaae bytes) 000701 1789498707.448664 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x25bad): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25bad): Mirror 8: 0xcb5c7828 (stale) Mirror 9: 0xcb5c7828 (stale) Mirror 16: 0xcb5c7828 (stale) Mirror 18: 0xcb5c7828 (stale) Mirror 19: 0xcb5c7828 (stale) Mirror 20: 0xcb5c7828 (stale) Mirror 22: 0xcb5c7828 (stale) Mirror 23: 0x752277ae (stale) CRC-32 checksum value for chunk [0, 0x25bad): Mirror 1: 0x7002ae93 Mirror 24: 0x7002ae93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25bad: skipped 000703 1789498708.946778 mapwrite 0x01bce9 thru 0x02aee9 (0xf201 bytes) 000704 1789498709.120190 mapwrite 0x021218 thru 0x02a87c (0x9665 bytes) 000705 1789498709.164609 trunc from 0x02aeea to 0x015f12 (0x14fd7 bytes) 000708 1789498709.187987 write 0x01f487 thru 0x028eee (0x9a68 bytes) 000709 1789498709.190478 write 0x024168 thru 0x02b2f7 (0x7190 bytes) 000710 1789498709.191907 mapread 0x00dcb4 thru 0x015f1f (0x826c bytes) 000711 1789498709.206050 MIRROR_RESYNC 10 mirrors 000713 1789498712.662979 mapwrite 0x0112c6 thru 0x015a22 (0x475d bytes) 000714 1789498712.897541 read 0x029be9 thru 0x02b2f7 (0x170f bytes) 000716 1789498712.907691 write 0x03fd49 thru 0x03ffff (0x2b7 bytes) 000717 1789498712.919048 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x26a06ab5 (stale) Mirror 9: 0x26a06ab5 (stale) Mirror 16: 0x26a06ab5 (stale) Mirror 19: 0x26a06ab5 (stale) Mirror 20: 0x26a06ab5 (stale) Mirror 22: 0x26a06ab5 (stale) Mirror 23: 0x26a06ab5 (stale) Mirror 24: 0x26a06ab5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x26a06ab5 (stale) Mirror 9: 0x26a06ab5 (stale) Mirror 16: 0x26a06ab5 (stale) Mirror 19: 0x26a06ab5 (stale) Mirror 20: 0x26a06ab5 (stale) Mirror 22: 0x26a06ab5 (stale) Mirror 23: 0x26a06ab5 (stale) Mirror 24: 0x26a06ab5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000718 1789498713.581531 trunc from 0x040000 to 0x02e0cc (0x11f33 bytes) 000720 1789498713.606238 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2e0cd): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0cd): Mirror 8: 0xec80f86d (stale) Mirror 9: 0xec80f86d (stale) Mirror 16: 0xec80f86d (stale) Mirror 19: 0xec80f86d (stale) Mirror 20: 0xec80f86d (stale) Mirror 22: 0xec80f86d (stale) Mirror 23: 0xec80f86d (stale) Mirror 24: 0xec80f86d (stale) CRC-32 checksum value for chunk [0, 0x2e0cd): Mirror 1: 0x3e59ac7a Mirror 25: 0x3e59ac7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0cd: skipped 000721 1789498715.444827 mapread 0x0150f5 thru 0x01d19f (0x80ab bytes) 000722 1789498715.509554 punch from 0x0237a6 to 0x02a5f4 (0x6e4f bytes) 000724 1789498715.656695 trunc from 0x02e0cd to 0x03aa74 (0xc9a8 bytes) 000728 1789498715.680243 punch from 0x0209ec to 0x02f6bc (0xecd1 bytes) 000733 1789498715.705831 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3aa75): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa75): Mirror 8: 0x7d008900 (stale) Mirror 9: 0x7d008900 (stale) Mirror 16: 0x7d008900 (stale) Mirror 19: 0x7d008900 (stale) Mirror 20: 0x7d008900 (stale) Mirror 22: 0x7d008900 (stale) Mirror 23: 0x7d008900 (stale) Mirror 24: 0x7d008900 (stale) Mirror 25: 0xe5092402 (stale) CRC-32 checksum value for chunk [0, 0x3aa75): Mirror 1: 0x33f63605 Mirror 26: 0x33f63605 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa75: skipped 000735 1789498717.408052 write 0x015c6a thru 0x01c2be (0x6655 bytes) 000736 1789498717.489290 trunc from 0x03aa75 to 0x0363bc (0x46b8 bytes) 000737 1789498717.515278 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x363bd): Mirror 8: 0xa714fb (stale) Mirror 16: 0xa714fb (stale) Mirror 19: 0xa714fb (stale) Mirror 20: 0xa714fb (stale) Mirror 22: 0xa714fb (stale) Mirror 23: 0xa714fb (stale) Mirror 24: 0xa714fb (stale) Mirror 25: 0xe764c5a3 (stale) Mirror 26: 0xe7bd67c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x363bd): Mirror 8: 0xa714fb (stale) Mirror 16: 0xa714fb (stale) Mirror 19: 0xa714fb (stale) Mirror 20: 0xa714fb (stale) Mirror 22: 0xa714fb (stale) Mirror 23: 0xa714fb (stale) Mirror 24: 0xa714fb (stale) Mirror 25: 0xe764c5a3 (stale) Mirror 26: 0xe7bd67c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x363bd: skipped 000738 1789498718.001941 read 0x01e181 thru 0x02073e (0x25be bytes) 000739 1789498718.008237 trunc from 0x0363bd to 0x001d28 (0x34694 bytes) 000742 1789498718.036004 punch from 0x000ed4 to 0x001218 (0x345 bytes) 000744 1789498718.043850 read 0x000bc6 thru 0x001d28 (0x1163 bytes) 000745 1789498718.050497 mapread 0x000e12 thru 0x001d28 (0xf17 bytes) 000746 1789498718.055677 MIRROR_RESYNC 10 mirrors 000748 1789498719.731994 mapwrite 0x01e73d thru 0x0215ff (0x2ec3 bytes) 000750 1789498719.901418 mapread 0x007504 thru 0x00bc38 (0x4735 bytes) 000751 1789498719.926343 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21600): Mirror 8: 0x8d51d19c (stale) Mirror 16: 0x8d51d19c (stale) Mirror 19: 0x8d51d19c (stale) Mirror 22: 0x8d51d19c (stale) Mirror 23: 0x8d51d19c (stale) Mirror 24: 0x8d51d19c (stale) Mirror 25: 0x8d51d19c (stale) Mirror 26: 0x8d51d19c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21600): Mirror 8: 0x8d51d19c (stale) Mirror 16: 0x8d51d19c (stale) Mirror 19: 0x8d51d19c (stale) Mirror 22: 0x8d51d19c (stale) Mirror 23: 0x8d51d19c (stale) Mirror 24: 0x8d51d19c (stale) Mirror 25: 0x8d51d19c (stale) Mirror 26: 0x8d51d19c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21600: skipped 000752 1789498720.482624 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21600): Mirror 8: 0x8d51d19c (stale) Mirror 16: 0x8d51d19c (stale) Mirror 19: 0x8d51d19c (stale) Mirror 22: 0x8d51d19c (stale) Mirror 23: 0x8d51d19c (stale) Mirror 24: 0x8d51d19c (stale) Mirror 25: 0x8d51d19c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21600): Mirror 8: 0x8d51d19c (stale) Mirror 16: 0x8d51d19c (stale) Mirror 19: 0x8d51d19c (stale) Mirror 22: 0x8d51d19c (stale) Mirror 23: 0x8d51d19c (stale) Mirror 24: 0x8d51d19c (stale) Mirror 25: 0x8d51d19c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21600: skipped 000756 1789498720.991705 MIRROR_RESYNC 8 mirrors 000759 1789498723.405944 mapwrite 0x0121f7 thru 0x022025 (0xfe2f bytes) 000760 1789498723.645450 read 0x008624 thru 0x012044 (0x9a21 bytes) 000761 1789498723.655207 write 0x00c1c2 thru 0x018b80 (0xc9bf bytes) 000762 1789498723.658165 mapread 0x00db5a thru 0x01d597 (0xfa3e bytes) 000764 1789498723.659900 write 0x02cd35 thru 0x032957 (0x5c23 bytes) 000765 1789498723.661041 mapread 0x013788 thru 0x013e84 (0x6fd bytes) 000766 1789498723.662440 read 0x02d296 thru 0x032957 (0x56c2 bytes) 000767 1789498723.663118 punch from 0x023d31 to 0x028fce (0x529e bytes) 000768 1789498723.684591 mapread 0x00928d thru 0x0141a9 (0xaf1d bytes) 000769 1789498723.685650 read 0x010127 thru 0x01a0b4 (0x9f8e bytes) 000770 1789498723.686247 mapread 0x019b2d thru 0x022949 (0x8e1d bytes) 000774 1789498723.696844 trunc from 0x032958 to 0x038663 (0x5d0c bytes) 000776 1789498723.735635 write 0x006206 thru 0x008f51 (0x2d4c bytes) 000778 1789498723.749383 trunc from 0x038664 to 0x0207f7 (0x17e6c bytes) 000779 1789498723.788798 read 0x000d02 thru 0x00cfaf (0xc2ae bytes) 000781 1789498723.792853 mapwrite 0x0278a8 thru 0x0285de (0xd37 bytes) 000782 1789498723.889125 punch from 0x002d55 to 0x00bf79 (0x9225 bytes) 000787 1789498723.899768 read 0x01eaea thru 0x0285de (0x9af5 bytes) 000788 1789498723.901396 write 0x027d09 thru 0x02cd3b (0x5033 bytes) 000789 1789498723.903124 read 0x01dfdb thru 0x028aa4 (0xaaca bytes) 000792 1789498723.906620 write 0x03aa1f thru 0x03ffff (0x55e1 bytes) 000793 1789498723.913203 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd189b47f (stale) Mirror 16: 0xd189b47f (stale) Mirror 19: 0xd189b47f (stale) Mirror 22: 0xd189b47f (stale) Mirror 23: 0xd189b47f (stale) Mirror 24: 0xd189b47f (stale) Mirror 25: 0xd189b47f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa84909e0 Mirror 26: 0xa84909e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1789498725.390711 mapread 0x016e49 thru 0x017b70 (0xd28 bytes) 000796 1789498725.433335 trunc from 0x040000 to 0x0378e4 (0x871b bytes) 000799 1789498725.539301 write 0x00790e thru 0x009c56 (0x2349 bytes) 000800 1789498725.553432 mapwrite 0x030805 thru 0x0354e2 (0x4cde bytes) 000801 1789498725.599523 write 0x0389d6 thru 0x03ffff (0x762a bytes) 000802 1789498725.601557 trunc from 0x040000 to 0x0367d8 (0x9827 bytes) 000803 1789498725.625567 write 0x02dcc7 thru 0x02e4dd (0x817 bytes) 000804 1789498725.626922 punch from 0x00b345 to 0x0123f2 (0x70ae bytes) 000805 1789498725.637859 write 0x039252 thru 0x03ffff (0x6dae bytes) 000806 1789498725.640650 write 0x03021a thru 0x038ac8 (0x88af bytes) 000807 1789498725.646540 mapread 0x03c980 thru 0x03ffff (0x3680 bytes) 000808 1789498725.648088 trunc from 0x040000 to 0x0104dc (0x2fb23 bytes) 000811 1789498725.668120 mapwrite 0x02a904 thru 0x02bd06 (0x1403 bytes) 000812 1789498725.788773 write 0x006de8 thru 0x01556e (0xe787 bytes) 000814 1789498725.805878 trunc from 0x02bd07 to 0x00f14b (0x1cbbb bytes) 000815 1789498725.836301 mapread 0x00522b thru 0x00f14b (0x9f21 bytes) 000816 1789498725.852545 read 0x00a985 thru 0x00e134 (0x37b0 bytes) 000817 1789498725.860205 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xf14c): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf14c): Mirror 8: 0x2e524804 (stale) Mirror 16: 0x2e524804 (stale) Mirror 19: 0x2e524804 (stale) Mirror 22: 0x2e524804 (stale) Mirror 23: 0x2e524804 (stale) Mirror 24: 0x2e524804 (stale) Mirror 25: 0x2e524804 (stale) Mirror 26: 0xecdf760b (stale) CRC-32 checksum value for chunk [0, 0xf14c): Mirror 1: 0xbe4057af Mirror 27: 0xbe4057af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf14c: skipped 000819 1789498727.123948 mapread 0x009210 thru 0x00ef91 (0x5d82 bytes) 000821 1789498727.202400 MIRROR_RESYNC 10 mirrors 000822 1789498729.466134 mapwrite 0x0142c2 thru 0x021743 (0xd482 bytes) 000823 1789498729.654955 MIRROR_RESYNC 10 mirrors 000824 1789498733.480350 MIRROR_RESYNC 10 mirrors 000825 1789498734.112685 MIRROR_EXTEND to 11 mirrors 000826 1789498734.989526 read 0x000b01 thru 0x005621 (0x4b21 bytes) 000828 1789498735.014293 read 0x01b116 thru 0x01b937 (0x822 bytes) 000831 1789498735.058824 write 0x01a4db thru 0x01fbb7 (0x56dd bytes) 000832 1789498735.180732 trunc from 0x021744 to 0x0289d6 (0x7293 bytes) 000833 1789498735.210577 mapread 0x01c1af thru 0x023a2f (0x7881 bytes) 000834 1789498735.231748 mapread 0x012969 thru 0x01e3dc (0xba74 bytes) 000836 1789498735.285910 write 0x01cf3a thru 0x01ff08 (0x2fcf bytes) 000837 1789498735.287903 write 0x027494 thru 0x034f0e (0xda7b bytes) 000838 1789498735.291507 punch from 0x0289fe to 0x034f0e (0xc511 bytes) 000840 1789498735.308680 write 0x0320d4 thru 0x03f61a (0xd547 bytes) 000841 1789498735.317296 punch from 0x026bab to 0x028f65 (0x23bb bytes) 000842 1789498735.328514 read 0x01801f thru 0x01c53d (0x451f bytes) 000843 1789498735.329801 write 0x03ef51 thru 0x03ff6c (0x101c bytes) 000846 1789498735.332325 trunc from 0x03ff6d to 0x0352d0 (0xac9c bytes) 000847 1789498735.355594 punch from 0x025a34 to 0x02aa51 (0x501e bytes) 000848 1789498735.362455 trunc from 0x0352d1 to 0x012f3e (0x22392 bytes) 000849 1789498735.400224 mapread 0x00a466 thru 0x012f3e (0x8ad9 bytes) 000850 1789498735.420776 write 0x0212ef thru 0x02ed48 (0xda5a bytes) 000851 1789498735.428284 read 0x016ba7 thru 0x01e65f (0x7ab9 bytes) 000853 1789498735.438002 punch from 0x01ca1a to 0x01f3fe (0x29e5 bytes) 000855 1789498735.477038 write 0x0042f4 thru 0x008b75 (0x4882 bytes) 000856 1789498735.483923 write 0x0201b6 thru 0x0266de (0x6529 bytes) 000857 1789498735.487490 punch from 0x00374c to 0x0136d1 (0xff86 bytes) 000859 1789498735.539154 MIRROR_RESYNC 11 mirrors 000860 1789498739.063489 mapwrite 0x037d69 thru 0x03ffff (0x8297 bytes) 000861 1789498739.276205 read 0x00decb thru 0x00f07b (0x11b1 bytes) 000862 1789498739.285474 trunc from 0x040000 to 0x001a64 (0x3e59b bytes) 000863 1789498739.335999 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a65): Mirror 8: 0x6c724eb0 (stale) Mirror 16: 0x6c724eb0 (stale) Mirror 19: 0x6c724eb0 (stale) Mirror 22: 0x6c724eb0 (stale) Mirror 24: 0x6c724eb0 (stale) Mirror 25: 0x6c724eb0 (stale) Mirror 26: 0x6c724eb0 (stale) Mirror 27: 0x6c724eb0 (stale) Mirror 28: 0x6c724eb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a65): Mirror 8: 0x6c724eb0 (stale) Mirror 16: 0x6c724eb0 (stale) Mirror 19: 0x6c724eb0 (stale) Mirror 22: 0x6c724eb0 (stale) Mirror 24: 0x6c724eb0 (stale) Mirror 25: 0x6c724eb0 (stale) Mirror 26: 0x6c724eb0 (stale) Mirror 27: 0x6c724eb0 (stale) Mirror 28: 0x6c724eb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a65: skipped 000864 1789498739.972341 mapwrite 0x016865 thru 0x026677 (0xfe13 bytes) 000865 1789498740.123488 trunc from 0x026678 to 0x0171b8 (0xf4bf bytes) 000866 1789498740.156109 write 0x0105d4 thru 0x01fd3d (0xf76a bytes) 000867 1789498740.177728 punch from 0x017cde to 0x01854f (0x872 bytes) 000869 1789498740.215379 trunc from 0x01fd3e to 0x03b04e (0x1b311 bytes) 000871 1789498740.263248 mapread 0x01da1a thru 0x02be22 (0xe409 bytes) 000873 1789498740.302271 trunc from 0x03b04f to 0x02ad2f (0x1031f bytes) 000874 1789498740.352448 mapread 0x008b4c thru 0x013460 (0xa915 bytes) 000875 1789498740.383543 MIRROR_RESYNC 10 mirrors 000876 1789498744.676644 mapwrite 0x021054 thru 0x02fa35 (0xe9e2 bytes) 000877 1789498744.979471 punch from 0x02f16f to 0x02fa35 (0x8c7 bytes) 000879 1789498745.002794 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2fa36): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa36): Mirror 8: 0xa19f195f (stale) Mirror 16: 0xa19f195f (stale) Mirror 19: 0xa19f195f (stale) Mirror 22: 0xa19f195f (stale) Mirror 24: 0xa19f195f (stale) Mirror 25: 0xa19f195f (stale) Mirror 26: 0xa19f195f (stale) Mirror 27: 0xa19f195f (stale) Mirror 28: 0xa19f195f (stale) CRC-32 checksum value for chunk [0, 0x2fa36): Mirror 1: 0x187573ec Mirror 29: 0x187573ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa36: skipped 000880 1789498746.645384 write 0x027b33 thru 0x02f117 (0x75e5 bytes) 000881 1789498746.784660 read 0x00393d thru 0x01083f (0xcf03 bytes) 000882 1789498746.798976 mapwrite 0x016645 thru 0x021f95 (0xb951 bytes) 000883 1789498746.906445 mapwrite 0x02f92a thru 0x03dc51 (0xe328 bytes) 000885 1789498747.061721 punch from 0x00c225 to 0x017bf6 (0xb9d2 bytes) 000887 1789498747.071443 write 0x02e253 thru 0x03d630 (0xf3de bytes) 000888 1789498747.077445 trunc from 0x03dc52 to 0x00214d (0x3bb04 bytes) 000889 1789498747.107834 read 0x00047b thru 0x00214d (0x1cd3 bytes) 000890 1789498747.115707 punch from 0x001843 to 0x00214d (0x90b bytes) 000893 1789498747.128960 punch from 0x001fa5 to 0x00214d (0x1a9 bytes) 000894 1789498747.138915 punch from 0x000ca0 to 0x00214d (0x14ae bytes) 000895 1789498747.144407 trunc from 0x00214e to 0x028c7a (0x26b2d bytes) 000896 1789498747.169311 read 0x01b862 thru 0x024bed (0x938c bytes) 000897 1789498747.196660 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28c7b): Mirror 8: 0x7a992f4 (stale) Mirror 16: 0x7a992f4 (stale) Mirror 19: 0x7a992f4 (stale) Mirror 22: 0x7a992f4 (stale) Mirror 24: 0x7a992f4 (stale) Mirror 25: 0x7a992f4 (stale) Mirror 26: 0x7a992f4 (stale) Mirror 27: 0x7a992f4 (stale) Mirror 28: 0x7a992f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c7b): Mirror 8: 0x7a992f4 (stale) Mirror 16: 0x7a992f4 (stale) Mirror 19: 0x7a992f4 (stale) Mirror 22: 0x7a992f4 (stale) Mirror 24: 0x7a992f4 (stale) Mirror 25: 0x7a992f4 (stale) Mirror 26: 0x7a992f4 (stale) Mirror 27: 0x7a992f4 (stale) Mirror 28: 0x7a992f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c7b: skipped 000898 1789498747.908455 write 0x0298a2 thru 0x036aab (0xd20a bytes) 000899 1789498747.910140 mapread 0x02fe65 thru 0x036aab (0x6c47 bytes) 000901 1789498747.917203 write 0x005e57 thru 0x00d95f (0x7b09 bytes) 000902 1789498747.936598 mapread 0x019c7b thru 0x023501 (0x9887 bytes) 000903 1789498747.967441 mapread 0x0198e6 thru 0x020b27 (0x7242 bytes) 000904 1789498747.974667 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36aac): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36aac): Mirror 8: 0xbdfcae08 (stale) Mirror 16: 0xbdfcae08 (stale) Mirror 19: 0xbdfcae08 (stale) Mirror 22: 0xbdfcae08 (stale) Mirror 24: 0xbdfcae08 (stale) Mirror 25: 0xbdfcae08 (stale) Mirror 26: 0xbdfcae08 (stale) Mirror 27: 0xbdfcae08 (stale) Mirror 28: 0xbdfcae08 (stale) CRC-32 checksum value for chunk [0, 0x36aac): Mirror 1: 0x2e24833d Mirror 29: 0x2e24833d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36aac: skipped 000905 1789498749.510515 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36aac): Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36aac): Mirror 8: 0xbdfcae08 (stale) Mirror 16: 0xbdfcae08 (stale) Mirror 19: 0xbdfcae08 (stale) Mirror 22: 0xbdfcae08 (stale) Mirror 24: 0xbdfcae08 (stale) Mirror 25: 0xbdfcae08 (stale) Mirror 26: 0xbdfcae08 (stale) Mirror 27: 0xbdfcae08 (stale) Mirror 28: 0xbdfcae08 (stale) CRC-32 checksum value for chunk [0, 0x36aac): Mirror 1: 0x2e24833d Mirror 29: 0x2e24833d Mirror 30: 0x2e24833d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36aac: skipped 000906 1789498750.643630 read 0x022a50 thru 0x02f488 (0xca39 bytes) 000907 1789498750.657586 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36aac): Mirror 16: 0xbdfcae08 (stale) Mirror 19: 0xbdfcae08 (stale) Mirror 22: 0xbdfcae08 (stale) Mirror 24: 0xbdfcae08 (stale) Mirror 25: 0xbdfcae08 (stale) Mirror 26: 0xbdfcae08 (stale) Mirror 27: 0xbdfcae08 (stale) Mirror 28: 0xbdfcae08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36aac): Mirror 16: 0xbdfcae08 (stale) Mirror 19: 0xbdfcae08 (stale) Mirror 22: 0xbdfcae08 (stale) Mirror 24: 0xbdfcae08 (stale) Mirror 25: 0xbdfcae08 (stale) Mirror 26: 0xbdfcae08 (stale) Mirror 27: 0xbdfcae08 (stale) Mirror 28: 0xbdfcae08 (stale) CRC-32 checksum value for chunk [0, 0x36aac): Mirror 1: 0x2e24833d Mirror 29: 0x2e24833d Mirror 30: 0x2e24833d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36aac: skipped 000908 1789498751.723724 punch from 0x036220 to 0x036aab (0x88c bytes) 000909 1789498751.859704 read 0x02526a thru 0x026e79 (0x1c10 bytes) 000912 1789498751.875198 write 0x019ea6 thru 0x0232ba (0x9415 bytes) 000913 1789498751.888896 trunc from 0x036aac to 0x02bf13 (0xab98 bytes) 000914 1789498751.921835 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 16: 0xfb806299 (stale) Mirror 19: 0xfb806299 (stale) Mirror 22: 0xfb806299 (stale) Mirror 24: 0xfb806299 (stale) Mirror 25: 0xfb806299 (stale) Mirror 26: 0xfb806299 (stale) Mirror 27: 0xfb806299 (stale) Mirror 28: 0xfb806299 (stale) Mirror 29: 0x2c4327cd (stale) Mirror 30: 0x2c4327cd (stale) CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 1: 0xd3b304bf Mirror 31: 0xd3b304bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf14: skipped 000916 1789498753.415921 write 0x01bae1 thru 0x02b1d9 (0xf6f9 bytes) 000917 1789498753.537595 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x193:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 16: 0xfb806299 (stale) Mirror 19: 0xfb806299 (stale) Mirror 24: 0xfb806299 (stale) Mirror 25: 0xfb806299 (stale) Mirror 26: 0xfb806299 (stale) Mirror 27: 0xfb806299 (stale) Mirror 28: 0xfb806299 (stale) Mirror 29: 0x2c4327cd (stale) Mirror 30: 0x2c4327cd (stale) Mirror 31: 0xd3b304bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf14): Mirror 16: 0xfb806299 (stale) Mirror 19: 0xfb806299 (stale) Mirror 24: 0xfb806299 (stale) Mirror 25: 0xfb806299 (stale) Mirror 26: 0xfb806299 (stale) Mirror 27: 0xfb806299 (stale) Mirror 28: 0xfb806299 (stale) Mirror 29: 0x2c4327cd (stale) Mirror 30: 0x2c4327cd (stale) Mirror 31: 0xd3b304bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf14: skipped 000918 1789498754.342540 write 0x006a73 thru 0x00b5b4 (0x4b42 bytes) 000919 1789498754.357915 trunc from 0x02bf14 to 0x002b46 (0x293cd bytes) 000920 1789498754.385971 read 0x00016b thru 0x002b46 (0x29dc bytes) 000921 1789498754.403103 read 0x0006d5 thru 0x002b46 (0x2472 bytes) 000922 1789498754.404803 write 0x03df82 thru 0x03ffff (0x207e bytes) 000923 1789498754.416451 write 0x03de1b thru 0x03ffff (0x21e5 bytes) 000924 1789498754.422450 mapwrite 0x020fec thru 0x0284cd (0x74e2 bytes) 000927 1789498754.522650 punch from 0x01d00d to 0x026f7f (0x9f73 bytes) 000928 1789498754.528873 read 0x0381f7 thru 0x03cef3 (0x4cfd bytes) 000929 1789498754.529703 trunc from 0x040000 to 0x002a0f (0x3d5f0 bytes) 000930 1789498754.556814 trunc from 0x002a10 to 0x02324c (0x2083d bytes) 000931 1789498754.578550 read 0x01bfa7 thru 0x01dbcf (0x1c29 bytes) 000932 1789498754.592046 read 0x013654 thru 0x021ed8 (0xe885 bytes) 000935 1789498754.599017 punch from 0x00798c to 0x014228 (0xc89d bytes) 000937 1789498754.604617 trunc from 0x02324d to 0x02b5f0 (0x83a4 bytes) 000938 1789498754.642957 trunc from 0x02b5f1 to 0x01fe57 (0xb799 bytes) 000940 1789498754.680599 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1fe58): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fe58): Mirror 16: 0x2ef9d1f3 (stale) Mirror 19: 0x2ef9d1f3 (stale) Mirror 24: 0x2ef9d1f3 (stale) Mirror 25: 0x2ef9d1f3 (stale) Mirror 26: 0x2ef9d1f3 (stale) Mirror 27: 0x2ef9d1f3 (stale) Mirror 28: 0x2ef9d1f3 (stale) Mirror 29: 0x4431285a (stale) Mirror 30: 0x4431285a (stale) Mirror 31: 0xab7ceef7 (stale) CRC-32 checksum value for chunk [0, 0x1fe58): Mirror 1: 0xc578b195 Mirror 32: 0xc578b195 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fe58: skipped 000941 1789498756.195807 read 0x010042 thru 0x01411d (0x40dc bytes) 000943 1789498756.216613 write 0x011a44 thru 0x02095e (0xef1b bytes) 000944 1789498756.306441 read 0x019e09 thru 0x02095e (0x6b56 bytes) 000945 1789498756.310726 punch from 0x014471 to 0x02095e (0xc4ee bytes) 000946 1789498756.326856 mapwrite 0x022be5 thru 0x02dee7 (0xb303 bytes) 000948 1789498756.447487 trunc from 0x02dee8 to 0x0287ee (0x56f9 bytes) 000952 1789498756.471574 mapwrite 0x03f055 thru 0x03ffff (0xfab bytes) 000953 1789498756.575211 MIRROR_RESYNC 12 mirrors 000954 1789498762.291382 write 0x013337 thru 0x014e79 (0x1b43 bytes) 000956 1789498762.400130 write 0x005109 thru 0x0116b3 (0xc5ab bytes) 000958 1789498762.416031 trunc from 0x040000 to 0x03a2e1 (0x5d1e bytes) 000959 1789498762.456514 mapread 0x0159e6 thru 0x022bad (0xd1c8 bytes) 000961 1789498762.518135 read 0x01fdf2 thru 0x02514e (0x535d bytes) 000962 1789498762.521990 write 0x02e4d7 thru 0x0347d3 (0x62fd bytes) 000963 1789498762.539621 punch from 0x0348f5 to 0x03a2e1 (0x59ed bytes) 000964 1789498762.565571 read 0x01f813 thru 0x027a1e (0x820c bytes) 000965 1789498762.566469 read 0x01eddb thru 0x01fa39 (0xc5f bytes) 000967 1789498762.569750 trunc from 0x03a2e2 to 0x03bd5b (0x1a7a bytes) 000968 1789498762.593697 mapwrite 0x0155fd thru 0x01b5a5 (0x5fa9 bytes) 000969 1789498762.670999 read 0x0349ae thru 0x03bd5b (0x73ae bytes) 000971 1789498762.682994 punch from 0x031365 to 0x033c7e (0x291a bytes) 000972 1789498762.693917 write 0x0004d0 thru 0x0031f6 (0x2d27 bytes) 000973 1789498762.702596 trunc from 0x03bd5c to 0x03be71 (0x116 bytes) 000974 1789498762.739315 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3be72): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be72): Mirror 16: 0x41159257 (stale) Mirror 19: 0x41159257 (stale) Mirror 24: 0x41159257 (stale) Mirror 25: 0x41159257 (stale) Mirror 26: 0x41159257 (stale) Mirror 27: 0x41159257 (stale) Mirror 28: 0x41159257 (stale) Mirror 29: 0x41159257 (stale) Mirror 30: 0x41159257 (stale) Mirror 31: 0x41159257 (stale) Mirror 32: 0x41159257 (stale) CRC-32 checksum value for chunk [0, 0x3be72): Mirror 1: 0xbae637a7 Mirror 33: 0xbae637a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be72: skipped 000975 1789498764.632621 punch from 0x03b54f to 0x03be71 (0x923 bytes) 000977 1789498764.751866 write 0x01f90f thru 0x02072c (0xe1e bytes) 000978 1789498764.774877 write 0x015ee4 thru 0x01fddd (0x9efa bytes) 000979 1789498764.785161 punch from 0x01811e to 0x01d068 (0x4f4b bytes) 000983 1789498764.826408 MIRROR_RESYNC 13 mirrors 000985 1789498773.421778 MIRROR_EXTEND to 14 mirrors 000987 1789498774.570410 mapread 0x005900 thru 0x00d25f (0x7960 bytes) 000988 1789498774.642211 mapwrite 0x01bdd9 thru 0x028eb7 (0xd0df bytes) 000989 1789498775.024721 mapread 0x02a12c thru 0x0386c9 (0xe59e bytes) 000990 1789498775.050701 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3be72): Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be72): Mirror 16: 0x8e481389 (stale) Mirror 19: 0x8e481389 (stale) Mirror 24: 0x8e481389 (stale) Mirror 25: 0x8e481389 (stale) Mirror 26: 0x8e481389 (stale) Mirror 27: 0x8e481389 (stale) Mirror 28: 0x8e481389 (stale) Mirror 29: 0x8e481389 (stale) Mirror 30: 0x8e481389 (stale) Mirror 31: 0x8e481389 (stale) Mirror 32: 0x8e481389 (stale) Mirror 33: 0x8e481389 (stale) Mirror 34: 0x8e481389 (stale) CRC-32 checksum value for chunk [0, 0x3be72): Mirror 1: 0xf4a249f6 Mirror 35: 0xf4a249f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be72: skipped 000992 1789498776.880603 punch from 0x032a1d to 0x036a67 (0x404b bytes) 000993 1789498776.983394 mapwrite 0x03673d thru 0x037969 (0x122d bytes) 000994 1789498777.043208 write 0x007a20 thru 0x00a9cc (0x2fad bytes) 000995 1789498777.055749 mapwrite 0x019fb2 thru 0x02734c (0xd39b bytes) 000996 1789498777.154782 trunc from 0x03be72 to 0x01e20e (0x1dc63 bytes) 000998 1789498777.190274 write 0x02cfea thru 0x0395cd (0xc5e4 bytes) 000999 1789498777.192970 write 0x00b1bd thru 0x00de22 (0x2c66 bytes) 001000 1789498777.227578 read 0x02ed17 thru 0x02f984 (0xc6e bytes) All operations completed A-OK! PASS 70a (192s) == sanity-flr test 71: check mirror extend parent fid ==== 14:59:42 (1789498782) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 14:59:45 (1789498785) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.282164 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143106 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 14:59:53 (1789498793) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8759008 .. Extending file size to 7467072 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8217248 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8248352 .. Extending file size to 8283264 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8332864 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9289280 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8939392 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57492 57493 57494 57495 57499 done 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) 192.168.203.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:01:12 (1789498872) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6136576 .. failed Extending file size to 6352320 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6794400 .. Extending file size to 6568608 .. Extending file size to 6968096 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8261344 .. Extending file size to 7873056 .. Extending file size to 8632608 .. Extending file size to 8686144 .. Extending file size to 8654816 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8706080 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8796448 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9282176 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58970 58971 58972 58974 58976 done 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) 192.168.203.111@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:02:37 (1789498957) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104142 s, 20.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.293991 s, 7.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xdf021274 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:02:56 (1789498976) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119482 s, 17.6 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 15:03:02 (1789498982) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:03:09 (1789498989) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261175 s, 8.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:03:17 (1789498997) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.304163 s, 6.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:03:25 (1789499005) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255993 s, 8.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:03:56 (1789499036) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140489 s, 7.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:04:51 (1789499091) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168083 s, 12.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:05:45 (1789499145) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. Creating new pool test_204f oleg311-server: Pool lustre.test_204f created Adding targets to pool oleg311-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg311-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.417955 s, 5.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg311-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg311-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg311-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:07:03 (1789499223) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:07:11 (1789499231) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:07:19 (1789499239) oleg311-server: Pool lustre.test_206 created Adding targets to pool oleg311-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg311-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg311-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg311-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg311-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:07:47 (1789499267) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921654 s, 11.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793171 s, 13.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5988e1cd Mirror 3: 0x5988e1cd PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:07:58 (1789499278) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:08:01 (1789499281) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:08:04 (1789499284) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.191998 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2460000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xed0000 Read this (4 bytes): BBBB 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:08:39 (1789499319) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0166331 s, 493 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:08:46 (1789499326) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394321 s, 26.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x213 lmm_fid: [0x200000404:0x213:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x213 lmm_fid: [0x200000404:0x213:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834451 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 15:08:47.000000000 -0400 Modify: 2026-09-15 15:08:47.000000000 -0400 Change: 2026-09-15 15:08:49.000000000 -0400 Birth: 2026-09-15 15:08:47.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:08:54 (1789499334) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:08:57 (1789499337) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.441639 s, 92.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0601316 s, 68.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 15 15:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x215:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 15 15:09 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:09:04 (1789499344) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:09:12 (1789499352) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1960 sec =========== 15:09:23 (1789499363) === sanity-flr: start cleanup 15:09:24 (1789499364) === === sanity-flr: finish cleanup 15:09:32 (1789499372) ===