-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 22:44:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg256-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:44:39 (1788835479) === oleg256-client.virtnet: executing check_config_client /mnt/lustre oleg256-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg256-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98ab20ea5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98ab20ea5000.idle_timeout=debug disable quota as required oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:45:31 (1788835531) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8226] == sanity-flr test 0a: lfs mirror create with -N option == 22:45:35 (1788835535) PASS 0a (23s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:45:57 (1788835557) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:46:01 (1788835561) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:46:06 (1788835566) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:46:33 (1788835593) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:46:38 (1788835598) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:46:42 (1788835602) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98ab20ea5000.stats=clear osc.lustre-OST0001-osc-ffff98ab20ea5000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:46:52 (1788835612) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:47:05 (1788835625) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:47:14 (1788835634) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:47:19 (1788835639) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (13s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:47:32 (1788835652) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (15s) == sanity-flr test 2: create components from existing files ========================================================== 22:47:47 (1788835667) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:47:57 (1788835677) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:48:07 (1788835687) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:48:18 (1788835698) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:48:31 (1788835711) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:48:39 (1788835719) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.9048 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153271 s, 6.8 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:48:51 (1788835731) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0794126 s, 52.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:49:00 (1788835740) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:49:12 (1788835752) Stopping /mnt/lustre-ost1 (opts:) on oleg256-server oleg256-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98ab20ea5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98ab20ea5000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg256-server: oleg256-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg256-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg256-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg256-server oleg256-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98ab20ea5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98ab20ea5000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg256-server: oleg256-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg256-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (149s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:51:41 (1788835901) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.210147 s, 15.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg256-server oleg256-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98ab20ea5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98ab20ea5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg256-server: oleg256-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg256-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:52:32 (1788835952) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg256-server oleg256-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98ab20ea5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98ab20ea5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg256-server: oleg256-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg256-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg256-server oleg256-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98ab20ea5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98ab20ea5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg256-server: oleg256-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg256-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (93s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:54:06 (1788836046) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.82773 s, 13.7 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 22:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.237s user 0m0.000s sys 0m0.174s Stopping /mnt/lustre-ost1 (opts:) on oleg256-server oleg256-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98ab20ea5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98ab20ea5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.561s user 0m0.001s sys 0m0.199s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg256-server: oleg256-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg256-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg256-server oleg256-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98ab20ea5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98ab20ea5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.315s user 0m0.001s sys 0m0.153s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg256-server: oleg256-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg256-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98ab20ea5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98ab20ea5000.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:55:50 (1788836150) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:55:53 (1788836153) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:55:57 (1788836157) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 22:56:00 (1788836160) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 22:56:09 (1788836169) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.625629 s, 20.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:56:22 (1788836182) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242208 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281246 s, 37.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:56:30 (1788836190) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:56:39 (1788836199) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147647 s, 694 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00157385 s, 651 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000906427 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00186213 s, 550 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 22:56:50 (1788836210) create test files with size 15757806 .. 3388d7773276157e05c86ff25ac33cce - 9ed605353f0dd0504f704be0b8a82e92 - 16754a6925b949d5e723df0d89505fde - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg256-client.virtnet /mnt/lustre (opts:) Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 22:57:19 (1788836239) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (22s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:57:40 (1788836260) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:57:49 (1788836269) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764318 s, 13.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xad:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411381 s, 25.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 22:57:58 (1788836278) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0546983 s, 38.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.379358 s, 11.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266174 s, 7.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 22:58:15 (1788836295) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 22:58:19 (1788836299) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:58:22 (1788836302) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567137 s, 18.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 22:58:29 (1788836309) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.235177 s, 13.4 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:58:40 (1788836320) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xba:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:58:52 (1788836332) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.439052 s, 23.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg256-client.virtnet /mnt/lustre (opts:) Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:59:04 (1788836344) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff98ab20076000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.324937 s, 32.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg256-client.virtnet /mnt/lustre (opts:) Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff98ab02cfe800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:59:22 (1788836362) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:00:09 (1788836409) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (47s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:00:56 (1788836456) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 23:01:02 (1788836462) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.314942 s, 10.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0814153 s, 38.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd1f06f6d (nosync,stale) Mirror 2: 0xd1f06f6d (nosync,stale) ** verify mirror 2 doesn't change original checksum: b84d3515e38f79e6bb0f33cf1b9e8dbc - mirror 1 checksum: fd947271bc111b80b53f028450e36a4d - mirror 2 checksum: b84d3515e38f79e6bb0f33cf1b9e8dbc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:01:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:01:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b84d3515e38f79e6bb0f33cf1b9e8dbc - mirror 1 checksum: fd947271bc111b80b53f028450e36a4d - mirror 2 checksum: fd947271bc111b80b53f028450e36a4d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:01:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:01:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:01:15 (1788836475) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399726 s, 26.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:01:24 (1788836484) Failing mds1 on oleg256-server Stopping /mnt/lustre-mds1 (opts:) on oleg256-server 23:01:38 (1788836498) shut down facet: mds1 facet_host: oleg256-server facet_failover_host: oleg256-server Failover mds1 to oleg256-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 23:02:05 (1788836525) targets are mounted 23:02:05 (1788836525) facet_failover done oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396124 s, 26.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:02:26 (1788836546) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0306127 s, 669 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0150708 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:02:39 (1788836559) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0753485 s, 272 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00935743 s, 2.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:02:52 (1788836572) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0062076 s, 2.6 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:03:02 (1788836582) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.104952 s, 195 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:03:12 (1788836592) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00679429 s, 603 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0269385 s, 152 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:03:20 (1788836600) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:03:49 (1788836629) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:04:19 (1788836659) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:04:43 (1788836683) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:05:09 (1788836709) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 23:05:16 (1788836716) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57640 mirror split pid 57642 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57640 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57642 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 23:06:26 (1788836786) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 6788 skipping zero size read 000003 1788836788.633024 mapwrite 0x039563 thru 0x03ffff (0x6a9d bytes) 000004 1788836788.732329 mapwrite 0x030e71 thru 0x03ffff (0xf18f bytes) 000005 1788836788.816889 punch from 0x037315 to 0x037eee (0xbda bytes) truncating to largest ever: 0x92b7 000006 1788836788.833533 trunc from 0x040000 to 0x0092b6 (0x36d49 bytes) 000007 1788836788.860399 write 0x02a48b thru 0x036621 (0xc197 bytes) 000008 1788836788.868184 mapread 0x013243 thru 0x017eae (0x4c6c bytes) 000009 1788836788.887083 write 0x03dac9 thru 0x03ffff (0x2537 bytes) fallocating to largest ever: 0xbdd8 000010 1788836788.888329 fallocate 0x003022 thru 0x00bdd7 (0x8db6 bytes) 000011 1788836788.914297 trunc from 0x03413e to 0x036f30 (0x2df3 bytes) 000012 1788836788.934523 read 0x026e15 thru 0x02ddd9 (0x6fc5 bytes) truncating to largest ever: 0x358aa 000013 1788836788.949887 trunc from 0x040000 to 0x0358a9 (0xa756 bytes) 000014 1788836788.984965 trunc from 0x02abed to 0x02ad2d (0x141 bytes) 000015 1788836788.994842 trunc from 0x02c16e to 0x0358a9 (0x973c bytes) 000016 1788836789.017387 MIRROR_EXTEND to 2 mirrors 000017 1788836790.081745 trunc from 0x0358aa to 0x02846e (0xd43b bytes) fallocating to largest ever: 0x3d478 000018 1788836790.145842 fallocate 0x02fd07 thru 0x03d477 (0xd771 bytes) 000019 1788836790.155352 punch from 0x01effb to 0x02846e (0x9474 bytes) 000020 1788836790.166865 mapwrite 0x02b167 thru 0x032fed (0x7e87 bytes) 000022 1788836790.263239 write 0x00643a thru 0x013c33 (0xd7fa bytes) 000023 1788836790.295695 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) 000024 1788836790.470331 punch from 0x021af1 to 0x024b82 (0x3092 bytes) 000025 1788836790.480767 trunc from 0x032fee to 0x019828 (0x197c5 bytes) 000026 1788836790.507042 trunc from 0x0169ee to 0x019828 (0x2e3b bytes) 000027 1788836790.511895 mapread 0x016e44 thru 0x019828 (0x29e5 bytes) 000028 1788836790.530938 punch from 0x005462 to 0x00f3f1 (0x9f90 bytes) 000029 1788836790.535693 read 0x006762 thru 0x00e067 (0x7906 bytes) 000030 1788836790.547101 trunc from 0x00de42 to 0x019828 (0xb9e7 bytes) 000031 1788836790.552819 read 0x019487 thru 0x019828 (0x3a2 bytes) 000032 1788836790.560067 trunc from 0x019829 to 0x013b43 (0x5ce5 bytes) 000034 1788836790.579233 punch from 0x00c5ef to 0x013b43 (0x7555 bytes) 000036 1788836790.585701 punch from 0x00943a to 0x013689 (0xa250 bytes) 000038 1788836790.590705 mapwrite 0x018ea3 thru 0x022e2c (0x9f8a bytes) 000040 1788836790.647845 mapwrite 0x006609 thru 0x01151f (0xaf17 bytes) 000041 1788836790.714326 write 0x016b2e thru 0x026274 (0xf747 bytes) 000042 1788836790.728043 MIRROR_EXTEND to 2 mirrors 000043 1788836791.391632 mapread 0x00152c thru 0x00470c (0x31e1 bytes) 000044 1788836791.420115 trunc from 0x00fa0e to 0x018d39 (0x932c bytes) 000045 1788836791.502221 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26275): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26275): Mirror 2: 0xe5cb7425 (stale) CRC-32 checksum value for chunk [0, 0x26275): Mirror 1: 0xf3ae8785 Mirror 3: 0xf3ae8785 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26275: skipped 000046 1788836792.477936 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26275): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26275): Mirror 2: 0xe5cb7425 (stale) CRC-32 checksum value for chunk [0, 0x26275): Mirror 1: 0xf3ae8785 Mirror 3: 0xf3ae8785 Mirror 4: 0xf3ae8785 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26275: skipped 000047 1788836793.700903 mapread 0x01b049 thru 0x024ef9 (0x9eb1 bytes) 000048 1788836793.755812 write 0x020b2d thru 0x0274d1 (0x69a5 bytes) 000049 1788836793.841176 trunc from 0x0274d2 to 0x005b4f (0x21982 bytes) 000051 1788836793.892746 read 0x000841 thru 0x005b4f (0x530f bytes) 000052 1788836793.907860 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x5b50): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5b50): Mirror 2: 0xcdb2416a (stale) Mirror 3: 0xcdb2416a (stale) Mirror 4: 0xcdb2416a (stale) CRC-32 checksum value for chunk [0, 0x5b50): Mirror 1: 0xcdb2416a Mirror 5: 0xcdb2416a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5b50: skipped 000053 1788836794.889554 fallocate 0x0279bb thru 0x02ddc2 (0x6408 bytes) 000054 1788836794.938820 trunc from 0x005b50 to 0x012a2d (0xcede bytes) fallocating to largest ever: 0x3dc1f 000055 1788836794.970505 fallocate 0x03a853 thru 0x03dc1e (0x33cc bytes) 000056 1788836794.979193 trunc from 0x01fa3b to 0x020de8 (0x13ae bytes) 000057 1788836794.984553 write 0x025b54 thru 0x02ad09 (0x51b6 bytes) 000058 1788836794.996561 punch from 0x0220a7 to 0x025572 (0x34cc bytes) 000060 1788836795.015397 mapread 0x034dea thru 0x03a242 (0x5459 bytes) 000061 1788836795.038088 write 0x002f81 thru 0x00a22e (0x72ae bytes) 000063 1788836795.049325 read 0x0255d1 thru 0x02c7ae (0x71de bytes) 000064 1788836795.063062 MIRROR_RESYNC 5 mirrors 000065 1788836796.915644 write 0x00b3c5 thru 0x00e9cd (0x3609 bytes) 000066 1788836796.953666 fallocate 0x039181 thru 0x03d18d (0x400d bytes) 000067 1788836796.962186 punch from 0x0281ca to 0x031968 (0x979f bytes) 000068 1788836796.971120 read 0x039f72 thru 0x03dc1e (0x3cad bytes) 000069 1788836796.984350 punch from 0x036313 to 0x03dc1e (0x790c bytes) 000071 1788836796.997200 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc1f): Mirror 2: 0x5c3e8e86 (stale) Mirror 3: 0x5c3e8e86 (stale) Mirror 4: 0x5c3e8e86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc1f): Mirror 2: 0x5c3e8e86 (stale) Mirror 3: 0x5c3e8e86 (stale) Mirror 4: 0x5c3e8e86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc1f: skipped 000072 1788836797.749249 fallocate 0x00f13c thru 0x010dc0 (0x1c85 bytes) 000073 1788836797.762788 punch from 0x012fb0 to 0x017302 (0x4353 bytes) 000074 1788836797.774508 write 0x01f8a8 thru 0x02672d (0x6e86 bytes) 000075 1788836797.790716 mapwrite 0x026fb7 thru 0x031bd5 (0xac1f bytes) 000077 1788836797.839228 mapread 0x011d7a thru 0x01fc58 (0xdedf bytes) fallocating to largest ever: 0x40000 000078 1788836797.867374 fallocate 0x03cf99 thru 0x03ffff (0x3067 bytes) 000079 1788836797.877803 trunc from 0x00d8d6 to 0x01ad4e (0xd479 bytes) 000080 1788836797.903553 read 0x034380 thru 0x03dc1e (0x989f bytes) 000081 1788836797.930685 trunc from 0x03dc1f to 0x021cb5 (0x1bf69 bytes) 000082 1788836797.950880 mapwrite 0x01fbf8 thru 0x0290e0 (0x94e9 bytes) 000083 1788836798.016909 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x290e1): Mirror 3: 0xa167c37 (stale) Mirror 4: 0xa167c37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x290e1): Mirror 3: 0xa167c37 (stale) Mirror 4: 0xa167c37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x290e1: skipped 000084 1788836798.378797 mapread 0x002083 thru 0x008923 (0x68a1 bytes) 000085 1788836798.409230 write 0x02e4bc thru 0x03050d (0x2052 bytes) 000086 1788836798.414630 fallocate 0x02129a thru 0x02deca (0xcc31 bytes) 000087 1788836798.425565 punch from 0x029886 to 0x03050d (0x6c88 bytes) 000088 1788836798.446200 mapread 0x0153b2 thru 0x01ad5e (0x59ad bytes) 000089 1788836798.460112 read 0x01bb0d thru 0x01f6a9 (0x3b9d bytes) 000090 1788836798.472578 trunc from 0x02524b to 0x03050d (0xb2c3 bytes) 000091 1788836798.484045 mapread 0x0039e6 thru 0x00fc89 (0xc2a4 bytes) 000092 1788836798.485088 write 0x029fdf thru 0x037e96 (0xdeb8 bytes) 000095 1788836798.493417 fallocate 0x01cc88 thru 0x0277f1 (0xab6a bytes) 000097 1788836798.501949 trunc from 0x037e97 to 0x012e73 (0x25023 bytes) 000098 1788836798.522498 punch from 0x00bd6d to 0x00fe0f (0x40a3 bytes) 000099 1788836798.539575 mapread 0x001580 thru 0x002eb0 (0x1931 bytes) 000101 1788836798.543175 punch from 0x00434f to 0x008f11 (0x4bc3 bytes) 000102 1788836798.553583 trunc from 0x0031e0 to 0x00ffa1 (0xcdc2 bytes) 000103 1788836798.558664 trunc from 0x012e74 to 0x0367fd (0x2398a bytes) 000104 1788836798.579809 write 0x032c4b thru 0x03d64d (0xaa03 bytes) 000105 1788836798.596691 mapwrite 0x034e79 thru 0x03ffff (0xb187 bytes) 000106 1788836798.661412 write 0x01b6f7 thru 0x026a06 (0xb310 bytes) 000107 1788836798.673852 mapwrite 0x03f71b thru 0x03ffff (0x8e5 bytes) 000108 1788836798.696021 read 0x011e11 thru 0x015b3b (0x3d2b bytes) 000109 1788836798.707136 punch from 0x00b11e to 0x01211d (0x7000 bytes) 000110 1788836798.713274 mapread 0x0318b1 thru 0x03b66e (0x9dbe bytes) 000111 1788836798.714369 write 0x016e22 thru 0x02699d (0xfb7c bytes) 000112 1788836798.716967 mapwrite 0x00f613 thru 0x01c17e (0xcb6c bytes) 000113 1788836798.756320 mapwrite 0x03e458 thru 0x03ffff (0x1ba8 bytes) 000114 1788836798.769837 fallocate 0x013809 thru 0x0140a2 (0x89a bytes) 000115 1788836798.774658 fallocate 0x03b97f thru 0x03ffff (0x4681 bytes) 000116 1788836798.779755 write 0x033755 thru 0x039346 (0x5bf2 bytes) 000117 1788836798.781873 punch from 0x0352a8 to 0x03bb98 (0x68f1 bytes) 000119 1788836798.801938 trunc from 0x032ded to 0x03ffff (0xd213 bytes) 000120 1788836798.808149 trunc from 0x040000 to 0x0169b0 (0x2964f bytes) 000122 1788836798.830594 punch from 0x004580 to 0x006a89 (0x250a bytes) 000123 1788836798.837773 fallocate 0x02dfd9 thru 0x02ebe1 (0xc09 bytes) 000124 1788836798.853258 mapread 0x0141bf thru 0x0194c3 (0x5305 bytes) 000125 1788836798.872353 mapwrite 0x01a258 thru 0x01ee88 (0x4c31 bytes) 000126 1788836798.927754 trunc from 0x01c126 to 0x02622b (0xa106 bytes) 000127 1788836798.934076 trunc from 0x004e32 to 0x012550 (0xd71f bytes) 000128 1788836798.940854 mapread 0x02bf34 thru 0x02ce85 (0xf52 bytes) 000129 1788836798.951670 mapwrite 0x00efbe thru 0x00f8cc (0x90f bytes) 000131 1788836798.975179 trunc from 0x02ebe2 to 0x014009 (0x1abd8 bytes) 000132 1788836799.001179 mapread 0x0124f5 thru 0x014009 (0x1b15 bytes) 000133 1788836799.007721 trunc from 0x01400a to 0x03d2c2 (0x292b9 bytes) 000134 1788836799.040342 mapwrite 0x024dd8 thru 0x029f9a (0x51c3 bytes) 000135 1788836799.102793 fallocate 0x036b8e thru 0x03c6fe (0x5b71 bytes) 000136 1788836799.111043 write 0x018eaa thru 0x024e53 (0xbfaa bytes) 000137 1788836799.118087 mapwrite 0x0146a2 thru 0x0177cc (0x312b bytes) 000138 1788836799.160850 write 0x02a900 thru 0x036e65 (0xc566 bytes) 000139 1788836799.172120 read 0x024b24 thru 0x0266fc (0x1bd9 bytes) 000140 1788836799.173010 trunc from 0x03d2c3 to 0x00ca20 (0x308a2 bytes) 000141 1788836799.197647 trunc from 0x00ca21 to 0x02479c (0x17d7c bytes) 000142 1788836799.224048 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2479d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2479d): Mirror 3: 0xe119e50c (stale) Mirror 4: 0xe119e50c (stale) CRC-32 checksum value for chunk [0, 0x2479d): Mirror 1: 0x8ad4c807 Mirror 5: 0x8ad4c807 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2479d: skipped 000143 1788836800.260012 punch from 0x01e78b to 0x02479c (0x6012 bytes) 000145 1788836800.344350 trunc from 0x0170b5 to 0x02479c (0xd6e8 bytes) 000146 1788836800.362232 mapwrite 0x0276d7 thru 0x03695f (0xf289 bytes) 000147 1788836800.460922 read 0x0234ec thru 0x02e5a8 (0xb0bd bytes) 000148 1788836800.461899 fallocate 0x01f69e thru 0x02852a (0x8e8d bytes) 000149 1788836800.474965 MIRROR_RESYNC 4 mirrors 000150 1788836802.143091 trunc from 0x036960 to 0x01ce0d (0x19b52 bytes) 000151 1788836802.189948 write 0x03cb3a thru 0x03ffff (0x34c6 bytes) 000154 1788836802.217234 MIRROR_RESYNC 4 mirrors 000155 1788836804.027521 mapread 0x00bdcf thru 0x014892 (0x8ac4 bytes) 000156 1788836804.094086 read 0x0209bc thru 0x027d9b (0x73e0 bytes) 000157 1788836804.113617 trunc from 0x006e99 to 0x00867e (0x17e6 bytes) 000158 1788836804.172181 fallocate 0x035b14 thru 0x03ffff (0xa4ec bytes) 000159 1788836804.188295 mapwrite 0x01093a thru 0x015f8f (0x5656 bytes) 000160 1788836804.239061 MIRROR_RESYNC 4 mirrors 000161 1788836806.218718 punch from 0x0230de to 0x028529 (0x544c bytes) 000163 1788836806.257923 fallocate 0x00a149 thru 0x017d69 (0xdc21 bytes) 000164 1788836806.262808 fallocate 0x025fa6 thru 0x02d3aa (0x7405 bytes) 000165 1788836806.267335 mapwrite 0x020361 thru 0x026c38 (0x68d8 bytes) 000166 1788836806.374998 trunc from 0x0293b8 to 0x02b1b5 (0x1dfe bytes) 000167 1788836806.387750 punch from 0x01d860 to 0x01ffb6 (0x2757 bytes) 000168 1788836806.399753 trunc from 0x01ee9d to 0x029077 (0xa1db bytes) 000169 1788836806.409942 mapread 0x03a935 thru 0x03e610 (0x3cdc bytes) 000170 1788836806.453721 mapread 0x031030 thru 0x03f61e (0xe5ef bytes) 000171 1788836806.482832 trunc from 0x040000 to 0x036d7e (0x9281 bytes) 000172 1788836806.533746 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36d7f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d7f): Mirror 3: 0xd43b7686 (stale) Mirror 4: 0xd43b7686 (stale) Mirror 5: 0xd43b7686 (stale) CRC-32 checksum value for chunk [0, 0x36d7f): Mirror 1: 0xd43b7686 Mirror 6: 0xd43b7686 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d7f: skipped 000173 1788836808.033398 read 0x00a3c1 thru 0x016493 (0xc0d3 bytes) 000175 1788836808.109025 read 0x00ff64 thru 0x01f75a (0xf7f7 bytes) 000176 1788836808.130865 trunc from 0x036d7f to 0x027e82 (0xeefc bytes) 000178 1788836808.253079 write 0x00779c thru 0x00e5f3 (0x6e58 bytes) 000179 1788836808.293128 trunc from 0x027e83 to 0x0310b4 (0x9232 bytes) 000180 1788836808.407530 MIRROR_RESYNC 5 mirrors 000181 1788836810.402907 mapread 0x0021cc thru 0x01031c (0xe151 bytes) 000182 1788836810.457142 punch from 0x01c909 to 0x02158e (0x4c86 bytes) 000184 1788836810.559913 fallocate 0x026d7f thru 0x0274d0 (0x752 bytes) 000185 1788836810.574541 trunc from 0x0061a7 to 0x00d377 (0x71d1 bytes) 000186 1788836810.594008 write 0x03de66 thru 0x03ffff (0x219a bytes) 000187 1788836810.598641 mapread 0x03d176 thru 0x03ffff (0x2e8a bytes) 000188 1788836810.609234 mapread 0x00e14f thru 0x01b6e8 (0xd59a bytes) 000189 1788836810.657438 fallocate 0x00abe0 thru 0x014197 (0x95b8 bytes) 000190 1788836810.665519 write 0x022736 thru 0x02b84b (0x9116 bytes) 000191 1788836810.692834 punch from 0x027a45 to 0x02c1d5 (0x4791 bytes) 000192 1788836810.726451 mapwrite 0x01c0de thru 0x026530 (0xa453 bytes) 000193 1788836810.823719 punch from 0x00d3a5 to 0x01931b (0xbf77 bytes) 000194 1788836810.835214 trunc from 0x040000 to 0x002a3f (0x3d5c0 bytes) 000195 1788836810.876451 trunc from 0x002a40 to 0x03b085 (0x38646 bytes) 000196 1788836810.908995 write 0x014352 thru 0x01f546 (0xb1f5 bytes) 000197 1788836810.942568 trunc from 0x0072f2 to 0x00d857 (0x6566 bytes) 000198 1788836810.971164 trunc from 0x0341d6 to 0x03422e (0x59 bytes) 000199 1788836810.995424 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b086): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b086): Mirror 3: 0x836c0a67 (stale) Mirror 4: 0x836c0a67 (stale) Mirror 5: 0x836c0a67 (stale) Mirror 6: 0x836c0a67 (stale) CRC-32 checksum value for chunk [0, 0x3b086): Mirror 1: 0xdd5cfe88 Mirror 7: 0xdd5cfe88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b086: skipped 000200 1788836812.978560 mapread 0x03588c thru 0x03b085 (0x57fa bytes) 000201 1788836813.026220 mapread 0x003743 thru 0x0070ce (0x398c bytes) 000202 1788836813.052787 punch from 0x011a01 to 0x01b8e1 (0x9ee1 bytes) 000204 1788836813.180632 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b086): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b086): Mirror 3: 0x836c0a67 (stale) Mirror 4: 0x836c0a67 (stale) Mirror 5: 0x836c0a67 (stale) Mirror 6: 0x836c0a67 (stale) Mirror 7: 0xdd5cfe88 (stale) CRC-32 checksum value for chunk [0, 0x3b086): Mirror 1: 0xbe6013b3 Mirror 8: 0xbe6013b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b086: skipped 000205 1788836814.852535 mapwrite 0x013a65 thru 0x0217ed (0xdd89 bytes) 000206 1788836814.957396 write 0x027cd5 thru 0x030d2e (0x905a bytes) 000207 1788836814.975741 mapwrite 0x010542 thru 0x010d21 (0x7e0 bytes) 000208 1788836814.991458 punch from 0x002025 to 0x007e0d (0x5de9 bytes) 000209 1788836814.996788 trunc from 0x01b999 to 0x01e267 (0x28cf bytes) 000210 1788836815.002765 read 0x011b79 thru 0x01a319 (0x87a1 bytes) 000211 1788836815.003631 punch from 0x032d1e to 0x03b085 (0x8368 bytes) 000212 1788836815.009015 mapread 0x0312c6 thru 0x033af1 (0x282c bytes) 000213 1788836815.017909 trunc from 0x007293 to 0x009c25 (0x2993 bytes) 000214 1788836815.022753 mapwrite 0x022edd thru 0x02bd97 (0x8ebb bytes) 000215 1788836815.084088 fallocate 0x031806 thru 0x03ede5 (0xd5e0 bytes) 000216 1788836815.096700 fallocate 0x025037 thru 0x030724 (0xb6ee bytes) 000217 1788836815.130939 mapwrite 0x020580 thru 0x02ecd6 (0xe757 bytes) 000218 1788836815.185664 mapwrite 0x0294f8 thru 0x037593 (0xe09c bytes) 000219 1788836815.242154 punch from 0x018cef to 0x02163d (0x894f bytes) 000220 1788836815.248783 fallocate 0x01d717 thru 0x02ca29 (0xf313 bytes) 000222 1788836815.262999 read 0x00d421 thru 0x01abf5 (0xd7d5 bytes) 000223 1788836815.279689 trunc from 0x010ba5 to 0x01d098 (0xc4f4 bytes) 000224 1788836815.298533 write 0x030002 thru 0x03e0a9 (0xe0a8 bytes) 000225 1788836815.300672 trunc from 0x03e0aa to 0x03d271 (0xe38 bytes) 000226 1788836815.350587 fallocate 0x012765 thru 0x013287 (0xb23 bytes) 000227 1788836815.361412 punch from 0x01517f to 0x019c57 (0x4ad9 bytes) 000228 1788836815.372603 mapread 0x01dbb9 thru 0x029fef (0xc437 bytes) 000229 1788836815.387203 fallocate 0x02169c thru 0x02a7c2 (0x9127 bytes) 000231 1788836815.402386 read 0x027059 thru 0x02a4df (0x3487 bytes) 000232 1788836815.403056 trunc from 0x03d272 to 0x032b21 (0xa750 bytes) 000233 1788836815.463072 punch from 0x0277b9 to 0x032b21 (0xb369 bytes) 000234 1788836815.491042 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32b22): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b22): Mirror 3: 0x868030ea (stale) Mirror 4: 0x868030ea (stale) Mirror 5: 0x868030ea (stale) Mirror 6: 0x868030ea (stale) Mirror 7: 0x525c400e (stale) Mirror 8: 0x2ee0d38d (stale) CRC-32 checksum value for chunk [0, 0x32b22): Mirror 1: 0xa30c34cd Mirror 9: 0xa30c34cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b22: skipped 000235 1788836817.070686 mapread 0x021435 thru 0x022b8d (0x1759 bytes) 000236 1788836817.131890 mapread 0x00c568 thru 0x01ae55 (0xe8ee bytes) 000237 1788836817.218430 read 0x01dfbc thru 0x026fa9 (0x8fee bytes) 000238 1788836817.267315 write 0x0002f5 thru 0x003b41 (0x384d bytes) 000239 1788836817.380571 trunc from 0x0028f2 to 0x00618b (0x389a bytes) 000240 1788836817.405937 fallocate 0x027b3b thru 0x0358f4 (0xddba bytes) 000241 1788836817.423970 mapwrite 0x022c3a thru 0x0278ff (0x4cc6 bytes) 000242 1788836817.520238 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32b22): Mirror 3: 0x868030ea (stale) Mirror 4: 0x868030ea (stale) Mirror 5: 0x868030ea (stale) Mirror 6: 0x868030ea (stale) Mirror 7: 0x525c400e (stale) Mirror 8: 0x2ee0d38d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b22): Mirror 3: 0x868030ea (stale) Mirror 4: 0x868030ea (stale) Mirror 5: 0x868030ea (stale) Mirror 6: 0x868030ea (stale) Mirror 7: 0x525c400e (stale) Mirror 8: 0x2ee0d38d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b22: skipped 000243 1788836818.419164 punch from 0x0232a8 to 0x02b327 (0x8080 bytes) 000244 1788836818.427391 mapread 0x02b0a9 thru 0x032b21 (0x7a79 bytes) 000245 1788836818.477585 trunc from 0x0014d0 to 0x00dc0f (0xc740 bytes) 000246 1788836818.490636 mapwrite 0x00779f thru 0x00e214 (0x6a76 bytes) 000247 1788836818.537571 read 0x026e0e thru 0x02e125 (0x7318 bytes) 000249 1788836818.539501 fallocate 0x00df3b thru 0x0131e5 (0x52ab bytes) 000250 1788836818.548135 punch from 0x01be92 to 0x01ee9f (0x300e bytes) 000252 1788836818.554954 write 0x0199c7 thru 0x023769 (0x9da3 bytes) 000254 1788836818.579733 read 0x00b44d thru 0x01668f (0xb243 bytes) 000255 1788836818.592593 read 0x0049bd thru 0x00cbe7 (0x822b bytes) 000257 1788836818.594875 trunc from 0x032b22 to 0x016634 (0x1c4ed bytes) 000258 1788836818.632840 write 0x028480 thru 0x02f367 (0x6ee8 bytes) 000259 1788836818.637180 fallocate 0x0120c0 thru 0x01538b (0x32cc bytes) 000260 1788836818.647881 mapwrite 0x0264e2 thru 0x02bca7 (0x57c6 bytes) 000261 1788836818.738128 write 0x004487 thru 0x00600c (0x1b86 bytes) 000262 1788836818.741159 mapread 0x025c2c thru 0x02bde3 (0x61b8 bytes) 000263 1788836818.747457 trunc from 0x02f368 to 0x03a6ff (0xb398 bytes) 000264 1788836818.792074 mapread 0x013c0e thru 0x01ad7b (0x716e bytes) 000265 1788836818.794666 trunc from 0x03a700 to 0x010d01 (0x299fe bytes) 000266 1788836818.852937 trunc from 0x002daf to 0x00c503 (0x9755 bytes) 000267 1788836818.869636 trunc from 0x010d02 to 0x01e91b (0xdc1a bytes) 000268 1788836818.894533 trunc from 0x01e91c to 0x02214c (0x3831 bytes) 000269 1788836818.934635 trunc from 0x019236 to 0x01fb78 (0x6943 bytes) 000270 1788836818.951580 MIRROR_RESYNC 7 mirrors 000271 1788836820.969638 MIRROR_RESYNC 7 mirrors 000272 1788836821.515091 fallocate 0x0214e1 thru 0x028405 (0x6f25 bytes) 000273 1788836821.580445 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2214d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2214d): Mirror 3: 0x578351f4 (stale) Mirror 4: 0x578351f4 (stale) Mirror 5: 0x578351f4 (stale) Mirror 6: 0x578351f4 (stale) Mirror 7: 0x578351f4 (stale) Mirror 8: 0x578351f4 (stale) CRC-32 checksum value for chunk [0, 0x2214d): Mirror 1: 0x578351f4 Mirror 9: 0x578351f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2214d: skipped 000274 1788836822.734685 trunc from 0x02214d to 0x03f9f3 (0x1d8a7 bytes) 000275 1788836822.798421 fallocate 0x0392c5 thru 0x03ffff (0x6d3b bytes) 000276 1788836822.811110 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6c4240cc (stale) Mirror 4: 0x6c4240cc (stale) Mirror 5: 0x6c4240cc (stale) Mirror 6: 0x6c4240cc (stale) Mirror 8: 0x6c4240cc (stale) Mirror 9: 0x6c4240cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6c4240cc (stale) Mirror 4: 0x6c4240cc (stale) Mirror 5: 0x6c4240cc (stale) Mirror 6: 0x6c4240cc (stale) Mirror 8: 0x6c4240cc (stale) Mirror 9: 0x6c4240cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000277 1788836823.283710 punch from 0x02f807 to 0x03b6d6 (0xbed0 bytes) 000278 1788836823.290365 write 0x016fd7 thru 0x024714 (0xd73e bytes) 000279 1788836823.305796 trunc from 0x005511 to 0x008ed6 (0x39c6 bytes) 000280 1788836823.315833 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6c4240cc (stale) Mirror 4: 0x6c4240cc (stale) Mirror 5: 0x6c4240cc (stale) Mirror 6: 0x6c4240cc (stale) Mirror 8: 0x6c4240cc (stale) Mirror 9: 0x6c4240cc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x34836ea7 Mirror 10: 0x34836ea7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000282 1788836824.628113 mapread 0x00542d thru 0x00b111 (0x5ce5 bytes) 000283 1788836824.679635 punch from 0x019a1a to 0x01a767 (0xd4e bytes) 000284 1788836824.742647 MIRROR_RESYNC 8 mirrors 000285 1788836827.367333 write 0x029772 thru 0x033dbd (0xa64c bytes) 000286 1788836827.409021 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc9b29d7b (stale) Mirror 4: 0xc9b29d7b (stale) Mirror 5: 0xc9b29d7b (stale) Mirror 6: 0xc9b29d7b (stale) Mirror 8: 0xc9b29d7b (stale) Mirror 9: 0xc9b29d7b (stale) Mirror 10: 0xc9b29d7b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x312fe43a Mirror 11: 0x312fe43a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000287 1788836828.638910 fallocate 0x035b14 thru 0x03a8bd (0x4daa bytes) 000289 1788836828.692947 write 0x03d266 thru 0x03ffff (0x2d9a bytes) 000290 1788836828.712690 punch from 0x0043fd to 0x0082a6 (0x3eaa bytes) 000291 1788836828.717568 read 0x023ac7 thru 0x02fb51 (0xc08b bytes) 000292 1788836828.728436 trunc from 0x040000 to 0x03ac9e (0x5361 bytes) 000293 1788836828.762169 trunc from 0x03ac9f to 0x03b8e4 (0xc46 bytes) 000294 1788836828.786002 trunc from 0x03b8e5 to 0x025149 (0x1679b bytes) 000295 1788836828.803974 write 0x02e808 thru 0x0378f9 (0x90f2 bytes) 000297 1788836828.806594 read 0x015b39 thru 0x0210e0 (0xb5a8 bytes) 000299 1788836828.830398 MIRROR_RESYNC 9 mirrors 000300 1788836832.178167 mapread 0x00b064 thru 0x012708 (0x76a5 bytes) 000301 1788836832.210408 read 0x0268a5 thru 0x034308 (0xda64 bytes) 000302 1788836832.225219 mapread 0x0372b0 thru 0x0378f9 (0x64a bytes) 000304 1788836832.234761 fallocate 0x013539 thru 0x01e488 (0xaf50 bytes) 000305 1788836832.372406 fallocate 0x019b5d thru 0x028509 (0xe9ad bytes) 000306 1788836832.390039 write 0x03083e thru 0x03244a (0x1c0d bytes) 000307 1788836832.414335 punch from 0x019e99 to 0x01acb6 (0xe1e bytes) 000308 1788836832.432695 write 0x03faf1 thru 0x03ffff (0x50f bytes) 000309 1788836832.442286 MIRROR_RESYNC 9 mirrors 000310 1788836836.605991 trunc from 0x040000 to 0x0178fc (0x28703 bytes) 000312 1788836836.634977 mapwrite 0x00915a thru 0x016e59 (0xdd00 bytes) 000313 1788836836.677715 mapread 0x00c6ed thru 0x0171b4 (0xaac8 bytes) 000314 1788836836.678722 read 0x000559 thru 0x006bd5 (0x667d bytes) 000315 1788836836.692373 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x178fd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178fd): Mirror 3: 0xcc0bd943 (stale) Mirror 4: 0xcc0bd943 (stale) Mirror 5: 0xcc0bd943 (stale) Mirror 6: 0xcc0bd943 (stale) Mirror 8: 0xcc0bd943 (stale) Mirror 9: 0xcc0bd943 (stale) Mirror 10: 0xcc0bd943 (stale) Mirror 11: 0xcc0bd943 (stale) CRC-32 checksum value for chunk [0, 0x178fd): Mirror 1: 0x80d39454 Mirror 12: 0x80d39454 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178fd: skipped 000317 1788836837.944772 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x178fd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178fd): Mirror 3: 0xcc0bd943 (stale) Mirror 4: 0xcc0bd943 (stale) Mirror 5: 0xcc0bd943 (stale) Mirror 6: 0xcc0bd943 (stale) Mirror 8: 0xcc0bd943 (stale) Mirror 9: 0xcc0bd943 (stale) Mirror 10: 0xcc0bd943 (stale) Mirror 11: 0xcc0bd943 (stale) CRC-32 checksum value for chunk [0, 0x178fd): Mirror 1: 0x80d39454 Mirror 12: 0x80d39454 Mirror 13: 0x80d39454 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178fd: skipped 000319 1788836839.513718 MIRROR_RESYNC 11 mirrors 000320 1788836841.622085 read 0x016708 thru 0x0178fc (0x11f5 bytes) 000322 1788836841.627444 fallocate 0x02f11f thru 0x035507 (0x63e9 bytes) 000323 1788836841.690314 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x178fd): Mirror 3: 0x80d39454 (stale) Mirror 5: 0x80d39454 (stale) Mirror 6: 0x80d39454 (stale) Mirror 8: 0x80d39454 (stale) Mirror 9: 0x80d39454 (stale) Mirror 10: 0x80d39454 (stale) Mirror 11: 0x80d39454 (stale) Mirror 12: 0x80d39454 (stale) Mirror 13: 0x80d39454 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178fd): Mirror 3: 0x80d39454 (stale) Mirror 5: 0x80d39454 (stale) Mirror 6: 0x80d39454 (stale) Mirror 8: 0x80d39454 (stale) Mirror 9: 0x80d39454 (stale) Mirror 10: 0x80d39454 (stale) Mirror 11: 0x80d39454 (stale) Mirror 12: 0x80d39454 (stale) Mirror 13: 0x80d39454 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178fd: skipped 000324 1788836842.584551 fallocate 0x005322 thru 0x005d4e (0xa2d bytes) 000326 1788836842.593796 trunc from 0x0178fd to 0x005a23 (0x11ed9 bytes) 000327 1788836842.643443 trunc from 0x004286 to 0x005a23 (0x179e bytes) 000328 1788836842.649494 trunc from 0x003960 to 0x005a23 (0x20c4 bytes) 000329 1788836842.654817 trunc from 0x005a24 to 0x00f8cf (0x9eac bytes) 000330 1788836842.698158 mapread 0x00c6f6 thru 0x00f8cf (0x31da bytes) 000331 1788836842.727127 trunc from 0x006185 to 0x00f8cf (0x974b bytes) 000332 1788836842.732283 mapwrite 0x01a2e8 thru 0x02969c (0xf3b5 bytes) 000333 1788836842.791817 write 0x02d713 thru 0x036754 (0x9042 bytes) 000334 1788836842.793419 trunc from 0x036755 to 0x00053d (0x36217 bytes) 000335 1788836842.815366 mapread 0x000528 thru 0x00053d (0x16 bytes) 000336 1788836842.822530 trunc from 0x000200 to 0x00053d (0x33e bytes) 000337 1788836842.828129 write 0x036e3e thru 0x0390de (0x22a1 bytes) 000338 1788836842.834069 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x390df): Mirror 3: 0x7b0c60 (stale) Mirror 5: 0x7b0c60 (stale) Mirror 6: 0x7b0c60 (stale) Mirror 8: 0x7b0c60 (stale) Mirror 9: 0x7b0c60 (stale) Mirror 10: 0x7b0c60 (stale) Mirror 11: 0x7b0c60 (stale) Mirror 13: 0x7b0c60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x390df): Mirror 3: 0x7b0c60 (stale) Mirror 5: 0x7b0c60 (stale) Mirror 6: 0x7b0c60 (stale) Mirror 8: 0x7b0c60 (stale) Mirror 9: 0x7b0c60 (stale) Mirror 10: 0x7b0c60 (stale) Mirror 11: 0x7b0c60 (stale) Mirror 13: 0x7b0c60 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x390df: skipped 000339 1788836843.580974 fallocate 0x02e98b thru 0x032d90 (0x4406 bytes) 000340 1788836843.590209 punch from 0x01eff4 to 0x020c3e (0x1c4b bytes) 000342 1788836843.599759 trunc from 0x0288ce to 0x02b187 (0x28ba bytes) 000343 1788836843.617607 fallocate 0x003e85 thru 0x013a55 (0xfbd1 bytes) 000344 1788836843.645741 read 0x007d40 thru 0x00fdec (0x80ad bytes) 000345 1788836843.665235 mapwrite 0x01ae21 thru 0x023012 (0x81f2 bytes) 000346 1788836843.721196 mapread 0x0174b0 thru 0x024326 (0xce77 bytes) 000347 1788836843.729217 MIRROR_RESYNC 9 mirrors 000348 1788836845.568663 mapwrite 0x00d899 thru 0x01ba4b (0xe1b3 bytes) 000349 1788836845.715389 trunc from 0x015078 to 0x022526 (0xd4af bytes) 000350 1788836845.726629 fallocate 0x039ad6 thru 0x03ffff (0x652a bytes) 000351 1788836845.751286 MIRROR_RESYNC 9 mirrors 000352 1788836848.868864 mapwrite 0x03f8f1 thru 0x03ffff (0x70f bytes) 000353 1788836848.934275 punch from 0x031117 to 0x03b98b (0xa875 bytes) 000354 1788836848.943062 write 0x01a265 thru 0x0240f9 (0x9e95 bytes) 000355 1788836848.955283 write 0x01169f thru 0x0211ce (0xfb30 bytes) 000356 1788836848.962241 read 0x004a6c thru 0x00e924 (0x9eb9 bytes) 000357 1788836848.974771 trunc from 0x008607 to 0x0106d6 (0x80d0 bytes) 000358 1788836848.987648 mapwrite 0x00d7a7 thru 0x018d01 (0xb55b bytes) 000359 1788836849.106600 mapwrite 0x021d2f thru 0x031116 (0xf3e8 bytes) 000360 1788836849.219885 trunc from 0x040000 to 0x035ba7 (0xa458 bytes) 000361 1788836849.247081 trunc from 0x01ceb4 to 0x02a16d (0xd2ba bytes) 000363 1788836849.255257 trunc from 0x035ba8 to 0x005e6c (0x2fd3b bytes) 000364 1788836849.272668 mapwrite 0x036a26 thru 0x039d47 (0x3322 bytes) 000365 1788836849.317136 write 0x0225ec thru 0x02b354 (0x8d69 bytes) 000367 1788836849.334967 punch from 0x023346 to 0x026c32 (0x38ed bytes) 000368 1788836849.353197 trunc from 0x011043 to 0x01adb7 (0x9d75 bytes) 000369 1788836849.361103 mapread 0x00a9cd thru 0x00d645 (0x2c79 bytes) 000370 1788836849.376454 write 0x036e24 thru 0x03c7f1 (0x59ce bytes) 000371 1788836849.379083 write 0x02b4db thru 0x031158 (0x5c7e bytes) 000372 1788836849.382403 read 0x01b34e thru 0x01fccb (0x497e bytes) 000373 1788836849.413144 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c7f2): Mirror 5: 0x96afab7b (stale) Mirror 6: 0x96afab7b (stale) Mirror 8: 0x96afab7b (stale) Mirror 9: 0x96afab7b (stale) Mirror 10: 0x96afab7b (stale) Mirror 11: 0x96afab7b (stale) Mirror 13: 0x96afab7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c7f2): Mirror 5: 0x96afab7b (stale) Mirror 6: 0x96afab7b (stale) Mirror 8: 0x96afab7b (stale) Mirror 9: 0x96afab7b (stale) Mirror 10: 0x96afab7b (stale) Mirror 11: 0x96afab7b (stale) Mirror 13: 0x96afab7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c7f2: skipped 000374 1788836850.130152 fallocate 0x01c406 thru 0x02a661 (0xe25c bytes) 000375 1788836850.136814 mapread 0x00e211 thru 0x00f2ad (0x109d bytes) 000376 1788836850.150196 mapread 0x004d3e thru 0x00527e (0x541 bytes) 000377 1788836850.162871 fallocate 0x01397b thru 0x0218e3 (0xdf69 bytes) 000378 1788836850.169489 mapread 0x01e99c thru 0x02614d (0x77b2 bytes) 000379 1788836850.183324 write 0x01fe5d thru 0x02caab (0xcc4f bytes) 000380 1788836850.208526 write 0x03c5a0 thru 0x03ffff (0x3a60 bytes) 000381 1788836850.216379 write 0x01b61f thru 0x01c82e (0x1210 bytes) 000382 1788836850.217721 punch from 0x021ed7 to 0x02a402 (0x852c bytes) 000383 1788836850.252623 fallocate 0x005697 thru 0x00aa9c (0x5406 bytes) 000384 1788836850.267832 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x50df4a7 (stale) Mirror 6: 0x50df4a7 (stale) Mirror 8: 0x50df4a7 (stale) Mirror 9: 0x50df4a7 (stale) Mirror 10: 0x50df4a7 (stale) Mirror 11: 0x50df4a7 (stale) Mirror 13: 0x50df4a7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa714872d Mirror 14: 0xa714872d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000386 1788836851.402913 mapwrite 0x018532 thru 0x0230ed (0xabbc bytes) 000387 1788836851.553299 write 0x03fc70 thru 0x03ffff (0x390 bytes) 000388 1788836851.585597 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x50df4a7 (stale) Mirror 6: 0x50df4a7 (stale) Mirror 8: 0x50df4a7 (stale) Mirror 9: 0x50df4a7 (stale) Mirror 10: 0x50df4a7 (stale) Mirror 11: 0x50df4a7 (stale) Mirror 13: 0x50df4a7 (stale) Mirror 14: 0xa714872d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x15f5c3a7 Mirror 15: 0x15f5c3a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000390 1788836852.905967 trunc from 0x016fb8 to 0x01a57a (0x35c3 bytes) 000391 1788836852.997626 MIRROR_RESYNC 10 mirrors 000392 1788836856.929456 MIRROR_EXTEND to 11 mirrors 000393 1788836858.385626 read 0x00a1cd thru 0x00c9c3 (0x27f7 bytes) 000394 1788836858.392445 mapwrite 0x0064f6 thru 0x00b1f3 (0x4cfe bytes) 000395 1788836858.574158 read 0x00ff0a thru 0x013c0a (0x3d01 bytes) 000396 1788836858.603369 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2bb7ecf6 (stale) Mirror 6: 0x2bb7ecf6 (stale) Mirror 8: 0x2bb7ecf6 (stale) Mirror 9: 0x2bb7ecf6 (stale) Mirror 10: 0x2bb7ecf6 (stale) Mirror 13: 0x2bb7ecf6 (stale) Mirror 14: 0x2bb7ecf6 (stale) Mirror 15: 0x2bb7ecf6 (stale) Mirror 16: 0x2bb7ecf6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2bb7ecf6 (stale) Mirror 6: 0x2bb7ecf6 (stale) Mirror 8: 0x2bb7ecf6 (stale) Mirror 9: 0x2bb7ecf6 (stale) Mirror 10: 0x2bb7ecf6 (stale) Mirror 13: 0x2bb7ecf6 (stale) Mirror 14: 0x2bb7ecf6 (stale) Mirror 15: 0x2bb7ecf6 (stale) Mirror 16: 0x2bb7ecf6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000397 1788836859.728393 write 0x00d158 thru 0x015d6f (0x8c18 bytes) 000398 1788836859.751335 mapwrite 0x035934 thru 0x037400 (0x1acd bytes) 000399 1788836859.814935 mapread 0x02df65 thru 0x0325d5 (0x4671 bytes) 000400 1788836859.826821 mapwrite 0x00e38b thru 0x019036 (0xacac bytes) 000401 1788836859.881991 trunc from 0x040000 to 0x01166b (0x2e994 bytes) 000402 1788836859.921309 fallocate 0x03b420 thru 0x03ffff (0x4be0 bytes) 000403 1788836859.947041 fallocate 0x0089f1 thru 0x014c6f (0xc27f bytes) 000404 1788836859.956722 fallocate 0x007aa7 thru 0x008c98 (0x11f2 bytes) 000406 1788836859.965662 mapread 0x03b9ce thru 0x03ffff (0x4632 bytes) 000407 1788836859.987934 punch from 0x009d27 to 0x010922 (0x6bfc bytes) 000408 1788836859.998786 trunc from 0x00dbdb to 0x01558f (0x79b5 bytes) 000409 1788836860.008326 punch from 0x01e589 to 0x02d095 (0xeb0d bytes) 000410 1788836860.021715 mapread 0x006aaa thru 0x0147b8 (0xdd0f bytes) 000412 1788836860.074429 trunc from 0x040000 to 0x020839 (0x1f7c6 bytes) 000413 1788836860.108919 read 0x01cf85 thru 0x01e610 (0x168c bytes) 000414 1788836860.124609 trunc from 0x013be8 to 0x015671 (0x1a8a bytes) 000415 1788836860.131348 write 0x03be33 thru 0x03ffff (0x41cd bytes) 000416 1788836860.135007 mapwrite 0x018de5 thru 0x023c36 (0xae52 bytes) 000417 1788836860.200940 read 0x01a308 thru 0x0224cb (0x81c4 bytes) 000418 1788836860.202738 punch from 0x024475 to 0x031a08 (0xd594 bytes) 000419 1788836860.217227 mapwrite 0x01316d thru 0x01dfdc (0xae70 bytes) 000420 1788836860.274020 read 0x0357bb thru 0x038e71 (0x36b7 bytes) 000422 1788836860.302007 write 0x03fdf2 thru 0x03ffff (0x20e bytes) 000423 1788836860.309302 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2bb7ecf6 (stale) Mirror 6: 0x2bb7ecf6 (stale) Mirror 8: 0x2bb7ecf6 (stale) Mirror 9: 0x2bb7ecf6 (stale) Mirror 10: 0x2bb7ecf6 (stale) Mirror 13: 0x2bb7ecf6 (stale) Mirror 14: 0x2bb7ecf6 (stale) Mirror 15: 0x2bb7ecf6 (stale) Mirror 16: 0x2bb7ecf6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x55dee79b Mirror 17: 0x55dee79b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000424 1788836862.126433 mapwrite 0x017811 thru 0x017e22 (0x612 bytes) 000425 1788836862.233554 mapwrite 0x016a69 thru 0x020647 (0x9bdf bytes) 000426 1788836862.303533 trunc from 0x022490 to 0x022e84 (0x9f5 bytes) 000427 1788836862.314277 read 0x022951 thru 0x03065f (0xdd0f bytes) 000428 1788836862.329428 mapread 0x03da04 thru 0x03ffff (0x25fc bytes) 000429 1788836862.341695 mapwrite 0x01772f thru 0x01b045 (0x3917 bytes) 000430 1788836862.361256 mapread 0x00ed77 thru 0x01617e (0x7408 bytes) 000432 1788836862.379020 MIRROR_RESYNC 11 mirrors 000433 1788836866.622324 trunc from 0x037be3 to 0x03ffff (0x841d bytes) 000434 1788836866.651208 read 0x029c7e thru 0x031c41 (0x7fc4 bytes) 000435 1788836866.668249 fallocate 0x039023 thru 0x03ffff (0x6fdd bytes) 000436 1788836866.673945 mapwrite 0x018b6b thru 0x028550 (0xf9e6 bytes) 000437 1788836866.745049 trunc from 0x040000 to 0x033ce2 (0xc31d bytes) 000438 1788836866.759631 read 0x005829 thru 0x00d80d (0x7fe5 bytes) 000439 1788836866.774156 trunc from 0x033ce3 to 0x014859 (0x1f489 bytes) 000440 1788836866.796226 write 0x028d0c thru 0x02d834 (0x4b29 bytes) 000442 1788836866.817782 mapread 0x023b54 thru 0x024b2a (0xfd7 bytes) 000443 1788836866.833174 trunc from 0x025a81 to 0x02985f (0x3ddf bytes) 000444 1788836866.849119 trunc from 0x02d835 to 0x0189ec (0x14e48 bytes) 000445 1788836866.883532 fallocate 0x0112b1 thru 0x01bcb8 (0xaa08 bytes) 000446 1788836866.894351 punch from 0x0041cd to 0x011c99 (0xdacd bytes) 000447 1788836866.908438 trunc from 0x0104af to 0x01256e (0x20c0 bytes) 000449 1788836866.923624 mapwrite 0x023651 thru 0x02b2b0 (0x7c60 bytes) 000450 1788836867.079127 trunc from 0x00a9c9 to 0x00dba0 (0x31d8 bytes) 000451 1788836867.099328 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2b2b1): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b2b1): Mirror 5: 0x9a4377f9 (stale) Mirror 6: 0x9a4377f9 (stale) Mirror 8: 0x9a4377f9 (stale) Mirror 9: 0x9a4377f9 (stale) Mirror 10: 0x9a4377f9 (stale) Mirror 13: 0x9a4377f9 (stale) Mirror 14: 0x9a4377f9 (stale) Mirror 15: 0x9a4377f9 (stale) Mirror 16: 0x9a4377f9 (stale) Mirror 17: 0x9a4377f9 (stale) CRC-32 checksum value for chunk [0, 0x2b2b1): Mirror 1: 0x8a22cd37 Mirror 18: 0x8a22cd37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b2b1: skipped 000452 1788836868.233960 mapread 0x004c99 thru 0x00fef9 (0xb261 bytes) 000454 1788836868.266997 punch from 0x0063f6 to 0x0112b8 (0xaec3 bytes) 000455 1788836868.320066 trunc from 0x0102c8 to 0x01d962 (0xd69b bytes) 000456 1788836868.328029 mapwrite 0x008651 thru 0x01378b (0xb13b bytes) 000457 1788836868.358175 mapwrite 0x006659 thru 0x00868f (0x2037 bytes) 000458 1788836868.372032 mapread 0x01be98 thru 0x023397 (0x7500 bytes) 000459 1788836868.384180 write 0x012857 thru 0x01d517 (0xacc1 bytes) 000460 1788836868.391943 fallocate 0x01a217 thru 0x01a5ad (0x397 bytes) 000461 1788836868.399245 mapread 0x020b28 thru 0x02b2b0 (0xa789 bytes) 000462 1788836868.413289 mapread 0x01ce74 thru 0x01de51 (0xfde bytes) 000463 1788836868.414037 write 0x0303ca thru 0x034a04 (0x463b bytes) 000464 1788836868.415131 trunc from 0x02f54d to 0x031770 (0x2224 bytes) 000465 1788836868.429015 fallocate 0x01e615 thru 0x02bd4d (0xd739 bytes) 000466 1788836868.432963 fallocate 0x017cd9 thru 0x01a662 (0x298a bytes) 000467 1788836868.437059 fallocate 0x02ce44 thru 0x037f01 (0xb0be bytes) 000468 1788836868.440797 write 0x031b06 thru 0x03e392 (0xc88d bytes) 000469 1788836868.444858 read 0x0312b2 thru 0x03baa9 (0xa7f8 bytes) 000470 1788836868.445380 fallocate 0x00ef8d thru 0x01e4c9 (0xf53d bytes) 000471 1788836868.450390 write 0x022e40 thru 0x02a3fe (0x75bf bytes) 000472 1788836868.451390 write 0x03ed19 thru 0x03ffff (0x12e7 bytes) 000473 1788836868.452513 fallocate 0x01073c thru 0x012235 (0x1afa bytes) 000474 1788836868.456327 mapread 0x015134 thru 0x0177f6 (0x26c3 bytes) 000475 1788836868.457746 punch from 0x001596 to 0x007e76 (0x68e1 bytes) 000476 1788836868.467226 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6642bdec (stale) Mirror 8: 0x6642bdec (stale) Mirror 9: 0x6642bdec (stale) Mirror 10: 0x6642bdec (stale) Mirror 13: 0x6642bdec (stale) Mirror 14: 0x6642bdec (stale) Mirror 15: 0x6642bdec (stale) Mirror 16: 0x6642bdec (stale) Mirror 17: 0x6642bdec (stale) Mirror 18: 0x39847023 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6642bdec (stale) Mirror 8: 0x6642bdec (stale) Mirror 9: 0x6642bdec (stale) Mirror 10: 0x6642bdec (stale) Mirror 13: 0x6642bdec (stale) Mirror 14: 0x6642bdec (stale) Mirror 15: 0x6642bdec (stale) Mirror 16: 0x6642bdec (stale) Mirror 17: 0x6642bdec (stale) Mirror 18: 0x39847023 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000477 1788836869.212955 punch from 0x010ab9 to 0x017bba (0x7102 bytes) 000478 1788836869.227755 mapread 0x028837 thru 0x0335e3 (0xadad bytes) 000479 1788836869.290496 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x6642bdec (stale) Mirror 9: 0x6642bdec (stale) Mirror 10: 0x6642bdec (stale) Mirror 13: 0x6642bdec (stale) Mirror 14: 0x6642bdec (stale) Mirror 15: 0x6642bdec (stale) Mirror 16: 0x6642bdec (stale) Mirror 17: 0x6642bdec (stale) Mirror 18: 0x39847023 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x6642bdec (stale) Mirror 9: 0x6642bdec (stale) Mirror 10: 0x6642bdec (stale) Mirror 13: 0x6642bdec (stale) Mirror 14: 0x6642bdec (stale) Mirror 15: 0x6642bdec (stale) Mirror 16: 0x6642bdec (stale) Mirror 17: 0x6642bdec (stale) Mirror 18: 0x39847023 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000480 1788836870.225095 write 0x03450b thru 0x03b010 (0x6b06 bytes) 000481 1788836870.244236 fallocate 0x0274fa thru 0x033ca5 (0xc7ac bytes) 000482 1788836870.250619 mapread 0x017bcb thru 0x024874 (0xccaa bytes) 000483 1788836870.284030 fallocate 0x03080f thru 0x03a82a (0xa01c bytes) 000484 1788836870.289556 trunc from 0x02f5f9 to 0x03e767 (0xf16f bytes) 000485 1788836870.305794 fallocate 0x012ea7 thru 0x013004 (0x15e bytes) 000486 1788836870.313285 fallocate 0x0229d0 thru 0x02933e (0x696f bytes) 000487 1788836870.319097 trunc from 0x040000 to 0x02a348 (0x15cb7 bytes) 000488 1788836870.342674 read 0x00ea3d thru 0x013905 (0x4ec9 bytes) 000489 1788836870.353507 fallocate 0x03e9b0 thru 0x03ffff (0x1650 bytes) 000491 1788836870.362598 fallocate 0x038370 thru 0x03d8df (0x5570 bytes) 000492 1788836870.368838 mapwrite 0x0247b9 thru 0x02eb15 (0xa35d bytes) 000493 1788836870.492298 mapread 0x00f22f thru 0x0134f1 (0x42c3 bytes) 000495 1788836870.501137 trunc from 0x02eb16 to 0x00868e (0x26487 bytes) 000496 1788836870.533194 punch from 0x002781 to 0x00868e (0x5f0e bytes) 000498 1788836870.545148 trunc from 0x00868f to 0x020271 (0x17be3 bytes) 000499 1788836870.620229 mapwrite 0x0279da thru 0x034989 (0xcfb0 bytes) 000501 1788836870.714712 fallocate 0x005fae thru 0x00ae9f (0x4ef2 bytes) 000502 1788836870.722691 read 0x00ffdf thru 0x01f661 (0xf683 bytes) 000503 1788836870.738667 trunc from 0x00622d to 0x012428 (0xc1fc bytes) 000506 1788836870.746419 punch from 0x0082a5 to 0x01276e (0xa4ca bytes) 000507 1788836870.751607 fallocate 0x02b17e thru 0x02e871 (0x36f4 bytes) 000508 1788836870.758156 mapwrite 0x02bf3b thru 0x038186 (0xc24c bytes) 000509 1788836870.824902 punch from 0x02c75c to 0x038186 (0xba2b bytes) 000510 1788836870.833966 read 0x005633 thru 0x015026 (0xf9f4 bytes) 000511 1788836870.859694 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x38187): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38187): Mirror 8: 0xfa95ea44 (stale) Mirror 9: 0xfa95ea44 (stale) Mirror 10: 0xfa95ea44 (stale) Mirror 13: 0xfa95ea44 (stale) Mirror 14: 0xfa95ea44 (stale) Mirror 15: 0xfa95ea44 (stale) Mirror 16: 0xfa95ea44 (stale) Mirror 17: 0xfa95ea44 (stale) Mirror 18: 0xf5d37376 (stale) CRC-32 checksum value for chunk [0, 0x38187): Mirror 1: 0x70379a2d Mirror 19: 0x70379a2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38187: skipped 000512 1788836872.266822 write 0x01a993 thru 0x02a867 (0xfed5 bytes) 000513 1788836872.412440 trunc from 0x034817 to 0x038186 (0x3970 bytes) 000514 1788836872.431260 mapread 0x022d3e thru 0x027503 (0x47c6 bytes) 000515 1788836872.432806 write 0x012196 thru 0x012579 (0x3e4 bytes) 000516 1788836872.443859 trunc from 0x038187 to 0x010061 (0x28125 bytes) 000517 1788836872.486858 mapwrite 0x025349 thru 0x0332b0 (0xdf68 bytes) 000518 1788836872.653776 trunc from 0x0332b1 to 0x009271 (0x2a03f bytes) 000519 1788836872.694042 mapwrite 0x00cb6b thru 0x00d10a (0x5a0 bytes) 000521 1788836872.741932 MIRROR_RESYNC 11 mirrors 000522 1788836875.286424 mapwrite 0x03aa01 thru 0x03ffff (0x55ff bytes) 000523 1788836875.397452 read 0x03e48f thru 0x03ffff (0x1b71 bytes) 000524 1788836875.398218 write 0x03151d thru 0x03dbc0 (0xc6a4 bytes) 000525 1788836875.406273 trunc from 0x01fe5c to 0x029bd3 (0x9d78 bytes) 000526 1788836875.411937 trunc from 0x040000 to 0x02e8d4 (0x1172b bytes) 000527 1788836875.442916 punch from 0x00ff83 to 0x019ad3 (0x9b51 bytes) 000528 1788836875.447876 punch from 0x017fd4 to 0x01b027 (0x3054 bytes) 000529 1788836875.455951 trunc from 0x02e8d5 to 0x02049e (0xe436 bytes) 000530 1788836875.484299 read 0x01a102 thru 0x02049e (0x639d bytes) 000531 1788836875.499766 mapread 0x010ff9 thru 0x01bf6a (0xaf72 bytes) 000532 1788836875.500874 mapread 0x01f16f thru 0x02049e (0x1330 bytes) 000533 1788836875.501875 trunc from 0x02049f to 0x03bf29 (0x1ba8b bytes) 000534 1788836875.534254 mapread 0x0196ac thru 0x0274b2 (0xde07 bytes) 000535 1788836875.548229 mapwrite 0x00fdbe thru 0x01a068 (0xa2ab bytes) 000536 1788836875.565946 read 0x008461 thru 0x014f20 (0xcac0 bytes) 000537 1788836875.566509 write 0x02f36a thru 0x032493 (0x312a bytes) 000538 1788836875.567445 fallocate 0x02268b thru 0x02b942 (0x92b8 bytes) 000539 1788836875.574063 trunc from 0x03bf2a to 0x00f88c (0x2c69d bytes) 000540 1788836875.601151 mapread 0x00aacd thru 0x00f88c (0x4dc0 bytes) 000541 1788836875.603015 trunc from 0x0020e8 to 0x0020fa (0x13 bytes) 000542 1788836875.609621 write 0x002ff0 thru 0x011d61 (0xed72 bytes) 000543 1788836875.617221 read 0x00b02b thru 0x00e958 (0x392e bytes) 000544 1788836875.617928 trunc from 0x011d62 to 0x01979b (0x7a3a bytes) 000546 1788836875.632122 read 0x00704b thru 0x00e612 (0x75c8 bytes) 000547 1788836875.632741 punch from 0x017415 to 0x01979b (0x2387 bytes) 000548 1788836875.635803 punch from 0x014ea3 to 0x01979b (0x48f9 bytes) 000549 1788836875.638786 write 0x03df27 thru 0x03ffff (0x20d9 bytes) 000550 1788836875.639824 mapwrite 0x032b31 thru 0x03af64 (0x8434 bytes) 000551 1788836875.662196 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xf682ead (stale) Mirror 10: 0xf682ead (stale) Mirror 13: 0xf682ead (stale) Mirror 14: 0xf682ead (stale) Mirror 15: 0xf682ead (stale) Mirror 16: 0xf682ead (stale) Mirror 17: 0xf682ead (stale) Mirror 18: 0xf682ead (stale) Mirror 19: 0xf682ead (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xf682ead (stale) Mirror 10: 0xf682ead (stale) Mirror 13: 0xf682ead (stale) Mirror 14: 0xf682ead (stale) Mirror 15: 0xf682ead (stale) Mirror 16: 0xf682ead (stale) Mirror 17: 0xf682ead (stale) Mirror 18: 0xf682ead (stale) Mirror 19: 0xf682ead (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000552 1788836876.238152 MIRROR_RESYNC 10 mirrors 000555 1788836878.758307 MIRROR_EXTEND to 11 mirrors 000557 1788836879.742643 mapwrite 0x01b865 thru 0x01f23c (0x39d8 bytes) 000558 1788836879.929401 trunc from 0x039512 to 0x03e088 (0x4b77 bytes) 000559 1788836879.938266 read 0x02d0f4 thru 0x030ff8 (0x3f05 bytes) 000560 1788836879.950035 trunc from 0x040000 to 0x038567 (0x7a98 bytes) 000561 1788836879.987105 trunc from 0x038568 to 0x015b4c (0x22a1b bytes) 000562 1788836880.013074 trunc from 0x0046c7 to 0x011c7f (0xd5b9 bytes) 000563 1788836880.020999 trunc from 0x014f80 to 0x015b4c (0xbcd bytes) 000564 1788836880.026973 write 0x0127f4 thru 0x02072a (0xdf37 bytes) 000565 1788836880.034541 mapwrite 0x0088ff thru 0x00ff1a (0x761c bytes) 000566 1788836880.062320 read 0x0188f8 thru 0x02072a (0x7e33 bytes) 000567 1788836880.063002 punch from 0x016308 to 0x01d155 (0x6e4e bytes) 000568 1788836880.069381 trunc from 0x00c918 to 0x00cdec (0x4d5 bytes) 000570 1788836880.083374 trunc from 0x00af12 to 0x00e40a (0x34f9 bytes) 000572 1788836880.094120 write 0x010a33 thru 0x01ecbf (0xe28d bytes) 000574 1788836880.099106 trunc from 0x02072b to 0x029d7f (0x9655 bytes) 000575 1788836880.138150 fallocate 0x00f254 thru 0x017396 (0x8143 bytes) 000576 1788836880.146256 trunc from 0x029d80 to 0x027d08 (0x2077 bytes) 000577 1788836880.185199 mapwrite 0x001951 thru 0x0073e2 (0x5a92 bytes) 000578 1788836880.225406 read 0x0171af thru 0x01d2c4 (0x6116 bytes) 000579 1788836880.226065 trunc from 0x027d09 to 0x026a5f (0x12a9 bytes) 000580 1788836880.252321 trunc from 0x021a62 to 0x026415 (0x49b4 bytes) 000582 1788836880.263871 mapread 0x00572c thru 0x00874f (0x3024 bytes) 000583 1788836880.264881 mapwrite 0x03b481 thru 0x03ffff (0x4b7f bytes) 000584 1788836880.326006 trunc from 0x01c8b6 to 0x01cbce (0x319 bytes) 000585 1788836880.349258 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa1a55e6b (stale) Mirror 10: 0xa1a55e6b (stale) Mirror 13: 0xa1a55e6b (stale) Mirror 14: 0xa1a55e6b (stale) Mirror 16: 0xa1a55e6b (stale) Mirror 17: 0xa1a55e6b (stale) Mirror 18: 0xa1a55e6b (stale) Mirror 19: 0xa1a55e6b (stale) Mirror 20: 0xa1a55e6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa1a55e6b (stale) Mirror 10: 0xa1a55e6b (stale) Mirror 13: 0xa1a55e6b (stale) Mirror 14: 0xa1a55e6b (stale) Mirror 16: 0xa1a55e6b (stale) Mirror 17: 0xa1a55e6b (stale) Mirror 18: 0xa1a55e6b (stale) Mirror 19: 0xa1a55e6b (stale) Mirror 20: 0xa1a55e6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000586 1788836880.977931 mapwrite 0x03973b thru 0x03dea8 (0x476e bytes) 000587 1788836881.005967 mapwrite 0x0323c7 thru 0x033b7a (0x17b4 bytes) 000588 1788836881.018551 mapread 0x011102 thru 0x0170b7 (0x5fb6 bytes) 000589 1788836881.028818 mapwrite 0x0124fb thru 0x0213a2 (0xeea8 bytes) 000590 1788836881.066565 punch from 0x00d9e9 to 0x0189c3 (0xafdb bytes) 000591 1788836881.073267 mapread 0x005294 thru 0x00bfc4 (0x6d31 bytes) 000592 1788836881.083018 trunc from 0x016bc9 to 0x0231b5 (0xc5ed bytes) 000593 1788836881.088051 fallocate 0x02dbd8 thru 0x034a2c (0x6e55 bytes) 000594 1788836881.092703 fallocate 0x033baf thru 0x036c58 (0x30aa bytes) 000595 1788836881.096455 trunc from 0x040000 to 0x01f5fd (0x20a02 bytes) 000596 1788836881.119967 mapread 0x01d255 thru 0x01f5fd (0x23a9 bytes) 000597 1788836881.145824 trunc from 0x01f5fe to 0x00c496 (0x13167 bytes) 000598 1788836881.162017 read 0x001af4 thru 0x001bf1 (0xfe bytes) 000599 1788836881.167356 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc497): Mirror 9: 0xe9610a (stale) Mirror 10: 0xe9610a (stale) Mirror 13: 0xe9610a (stale) Mirror 14: 0xe9610a (stale) Mirror 16: 0xe9610a (stale) Mirror 18: 0xe9610a (stale) Mirror 19: 0xe9610a (stale) Mirror 20: 0xe9610a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc497): Mirror 9: 0xe9610a (stale) Mirror 10: 0xe9610a (stale) Mirror 13: 0xe9610a (stale) Mirror 14: 0xe9610a (stale) Mirror 16: 0xe9610a (stale) Mirror 18: 0xe9610a (stale) Mirror 19: 0xe9610a (stale) Mirror 20: 0xe9610a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc497: skipped 000600 1788836881.671044 write 0x02a808 thru 0x02cd64 (0x255d bytes) 000601 1788836881.678233 mapwrite 0x01e4b5 thru 0x02a72d (0xc279 bytes) 000602 1788836881.739307 write 0x004d19 thru 0x00a623 (0x590b bytes) 000603 1788836881.750315 mapread 0x020846 thru 0x02cd64 (0xc51f bytes) 000604 1788836881.751293 write 0x000dbb thru 0x0103d1 (0xf617 bytes) 000605 1788836881.771818 mapread 0x001c79 thru 0x0119f2 (0xfd7a bytes) 000606 1788836881.791210 trunc from 0x02cd65 to 0x01bafb (0x11269 bytes) 000607 1788836881.807796 fallocate 0x00b290 thru 0x0134e1 (0x8252 bytes) 000608 1788836881.811345 trunc from 0x0045fd to 0x013770 (0xf174 bytes) 000609 1788836881.831134 write 0x0227c1 thru 0x02620e (0x3a4e bytes) 000610 1788836881.832549 mapwrite 0x02b5dc thru 0x02fe4a (0x486f bytes) 000611 1788836881.892034 mapread 0x022bda thru 0x02d754 (0xab7b bytes) 000612 1788836881.893526 trunc from 0x008811 to 0x013dab (0xb59b bytes) 000613 1788836881.905103 write 0x0320f7 thru 0x0369d7 (0x48e1 bytes) 000614 1788836881.906323 trunc from 0x0369d8 to 0x03faf9 (0x9122 bytes) 000615 1788836881.923131 mapread 0x0306b6 thru 0x03f988 (0xf2d3 bytes) 000616 1788836881.952118 punch from 0x034293 to 0x03e063 (0x9dd1 bytes) 000617 1788836881.972888 fallocate 0x004368 thru 0x00b9ea (0x7683 bytes) 000618 1788836881.979048 read 0x003f95 thru 0x010d53 (0xcdbf bytes) 000619 1788836882.009798 punch from 0x023f09 to 0x029993 (0x5a8b bytes) 000621 1788836882.017870 trunc from 0x01d933 to 0x01fc82 (0x2350 bytes) 000622 1788836882.023984 punch from 0x02f32c to 0x03ac14 (0xb8e9 bytes) 000624 1788836882.029229 trunc from 0x03fafa to 0x014e86 (0x2ac73 bytes) 000625 1788836882.043796 write 0x03b37d thru 0x03b5ed (0x271 bytes) 000626 1788836882.044917 fallocate 0x019680 thru 0x01c6f7 (0x3078 bytes) 000627 1788836882.048779 punch from 0x023221 to 0x025efa (0x2cda bytes) 000629 1788836882.051942 write 0x001ddb thru 0x00ae95 (0x90bb bytes) 000630 1788836882.057919 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3b5ee): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5ee): Mirror 9: 0x22a1b5d1 (stale) Mirror 10: 0x22a1b5d1 (stale) Mirror 13: 0x22a1b5d1 (stale) Mirror 14: 0x22a1b5d1 (stale) Mirror 16: 0x22a1b5d1 (stale) Mirror 18: 0x22a1b5d1 (stale) Mirror 19: 0x22a1b5d1 (stale) Mirror 20: 0x22a1b5d1 (stale) CRC-32 checksum value for chunk [0, 0x3b5ee): Mirror 1: 0xcdf82de4 Mirror 21: 0xcdf82de4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5ee: skipped 000631 1788836882.791590 fallocate 0x0301f5 thru 0x035b0b (0x5917 bytes) 000632 1788836882.827797 write 0x014a6f thru 0x023cec (0xf27e bytes) 000633 1788836882.858385 read 0x00ff5f thru 0x017e5e (0x7f00 bytes) 000634 1788836882.876954 trunc from 0x0172f3 to 0x01796a (0x678 bytes) 000635 1788836882.898460 trunc from 0x03145e to 0x035c34 (0x47d7 bytes) 000637 1788836882.932995 MIRROR_RESYNC 10 mirrors 000638 1788836885.365588 MIRROR_EXTEND to 11 mirrors 000639 1788836886.730143 write 0x03181c thru 0x036fcc (0x57b1 bytes) 000640 1788836886.815301 mapwrite 0x010680 thru 0x018ebc (0x883d bytes) 000641 1788836886.930242 trunc from 0x02235f to 0x02a458 (0x80fa bytes) 000642 1788836886.938672 mapread 0x01127f thru 0x01131d (0x9f bytes) 000643 1788836886.948213 write 0x0164f2 thru 0x025e0c (0xf91b bytes) 000644 1788836886.971011 fallocate 0x03e3c8 thru 0x03ffff (0x1c38 bytes) 000645 1788836886.979080 trunc from 0x0364f7 to 0x03ffff (0x9b09 bytes) 000646 1788836886.993843 trunc from 0x040000 to 0x01aa59 (0x255a6 bytes) 000647 1788836887.024876 punch from 0x002e8b to 0x005778 (0x28ee bytes) 000648 1788836887.030647 punch from 0x00f4a5 to 0x012257 (0x2db3 bytes) 000649 1788836887.035073 punch from 0x018791 to 0x01aa59 (0x22c9 bytes) 000650 1788836887.050142 fallocate 0x03d733 thru 0x03ffff (0x28cd bytes) 000651 1788836887.055005 fallocate 0x0101d4 thru 0x018ebe (0x8ceb bytes) 000652 1788836887.060349 fallocate 0x010f9c thru 0x01f09a (0xe0ff bytes) 000653 1788836887.066261 write 0x01d045 thru 0x025eaf (0x8e6b bytes) 000655 1788836887.077564 trunc from 0x040000 to 0x03af52 (0x50ad bytes) 000656 1788836887.098277 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af53): Mirror 9: 0x576cf899 (stale) Mirror 10: 0x576cf899 (stale) Mirror 14: 0x576cf899 (stale) Mirror 16: 0x576cf899 (stale) Mirror 18: 0x576cf899 (stale) Mirror 19: 0x576cf899 (stale) Mirror 20: 0x576cf899 (stale) Mirror 21: 0x576cf899 (stale) Mirror 22: 0x576cf899 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af53): Mirror 9: 0x576cf899 (stale) Mirror 10: 0x576cf899 (stale) Mirror 14: 0x576cf899 (stale) Mirror 16: 0x576cf899 (stale) Mirror 18: 0x576cf899 (stale) Mirror 19: 0x576cf899 (stale) Mirror 20: 0x576cf899 (stale) Mirror 21: 0x576cf899 (stale) Mirror 22: 0x576cf899 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af53: skipped 000657 1788836887.705591 mapread 0x018b85 thru 0x0284a1 (0xf91d bytes) 000659 1788836887.739353 mapread 0x01a0b3 thru 0x021cf6 (0x7c44 bytes) 000660 1788836887.741063 trunc from 0x03af53 to 0x0249f5 (0x1655d bytes) 000661 1788836887.770939 punch from 0x00a14d to 0x0188a3 (0xe757 bytes) 000662 1788836887.779186 fallocate 0x024a3c thru 0x030b32 (0xc0f7 bytes) 000664 1788836887.787763 trunc from 0x0087d4 to 0x013a32 (0xb25f bytes) 000665 1788836887.792970 trunc from 0x01b03d to 0x01c148 (0x110c bytes) 000667 1788836887.799227 mapwrite 0x003882 thru 0x0039f0 (0x16f bytes) 000668 1788836887.825947 read 0x016a86 thru 0x018df3 (0x236e bytes) 000669 1788836887.831938 mapread 0x021302 thru 0x02172a (0x429 bytes) 000670 1788836887.832784 write 0x0266bd thru 0x026c06 (0x54a bytes) 000671 1788836887.840245 mapwrite 0x0308d4 thru 0x03cdab (0xc4d8 bytes) 000672 1788836887.918815 mapwrite 0x00bd0d thru 0x01a3a5 (0xe699 bytes) 000673 1788836887.997884 trunc from 0x03cdac to 0x004ae2 (0x382c9 bytes) 000674 1788836888.053005 punch from 0x0003d5 to 0x004ae2 (0x470e bytes) 000675 1788836888.058729 trunc from 0x004937 to 0x004ae2 (0x1ac bytes) 000676 1788836888.072686 punch from 0x003152 to 0x004ae2 (0x1991 bytes) 000677 1788836888.079851 punch from 0x00414f to 0x004ae2 (0x994 bytes) 000678 1788836888.114060 punch from 0x002f88 to 0x004ae2 (0x1b5b bytes) 000679 1788836888.132399 punch from 0x002ff5 to 0x004ae2 (0x1aee bytes) 000680 1788836888.155007 mapread 0x001039 thru 0x002391 (0x1359 bytes) 000681 1788836888.176686 mapwrite 0x02ecc8 thru 0x031fcb (0x3304 bytes) 000683 1788836888.301593 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31fcc): Mirror 9: 0xeea697c9 (stale) Mirror 10: 0xeea697c9 (stale) Mirror 14: 0xeea697c9 (stale) Mirror 18: 0xeea697c9 (stale) Mirror 19: 0xeea697c9 (stale) Mirror 20: 0xeea697c9 (stale) Mirror 21: 0xeea697c9 (stale) Mirror 22: 0xeea697c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31fcc): Mirror 9: 0xeea697c9 (stale) Mirror 10: 0xeea697c9 (stale) Mirror 14: 0xeea697c9 (stale) Mirror 18: 0xeea697c9 (stale) Mirror 19: 0xeea697c9 (stale) Mirror 20: 0xeea697c9 (stale) Mirror 21: 0xeea697c9 (stale) Mirror 22: 0xeea697c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31fcc: skipped 000684 1788836889.422952 punch from 0x003313 to 0x008d04 (0x59f2 bytes) 000685 1788836889.430431 trunc from 0x0159ea to 0x0250eb (0xf702 bytes) 000686 1788836889.436435 read 0x009cfd thru 0x010255 (0x6559 bytes) 000687 1788836889.448275 mapread 0x008dd1 thru 0x0176d3 (0xe903 bytes) 000688 1788836889.469872 mapwrite 0x01c4ae thru 0x02b1b1 (0xed04 bytes) 000689 1788836889.509689 read 0x00b9f1 thru 0x011a72 (0x6082 bytes) 000690 1788836889.510454 read 0x01998f thru 0x0201d4 (0x6846 bytes) 000691 1788836889.511688 mapread 0x016b4d thru 0x019178 (0x262c bytes) 000692 1788836889.512959 fallocate 0x019f57 thru 0x01fb95 (0x5c3f bytes) 000693 1788836889.518189 read 0x01b62d thru 0x01dea9 (0x287d bytes) 000694 1788836889.519123 trunc from 0x031fcc to 0x01f93b (0x12690 bytes) 000695 1788836889.544097 mapread 0x012564 thru 0x014317 (0x1db4 bytes) 000696 1788836889.547004 trunc from 0x01e646 to 0x01f93b (0x12f6 bytes) 000697 1788836889.552714 read 0x004f11 thru 0x00abca (0x5cba bytes) 000698 1788836889.562175 read 0x00339e thru 0x0062ae (0x2f11 bytes) 000699 1788836889.563041 trunc from 0x01f93c to 0x020a1c (0x10e1 bytes) 000700 1788836889.579836 punch from 0x01e374 to 0x020a1c (0x26a9 bytes) 000701 1788836889.586703 mapwrite 0x03ac36 thru 0x03ffff (0x53ca bytes) 000702 1788836889.645105 read 0x02a0d4 thru 0x033f84 (0x9eb1 bytes) 000703 1788836889.646873 trunc from 0x040000 to 0x0224d8 (0x1db27 bytes) 000705 1788836889.663748 punch from 0x02244b to 0x0224d8 (0x8e bytes) 000706 1788836889.669320 punch from 0x02056d to 0x0224d8 (0x1f6c bytes) 000707 1788836889.678125 trunc from 0x0224d9 to 0x03558f (0x130b7 bytes) 000708 1788836889.698002 trunc from 0x035590 to 0x037fad (0x2a1e bytes) 000709 1788836889.716217 fallocate 0x017201 thru 0x023109 (0xbf09 bytes) 000711 1788836889.728997 MIRROR_RESYNC 9 mirrors 000712 1788836892.141228 trunc from 0x01b6d1 to 0x02a5a0 (0xeed0 bytes) 000713 1788836892.198716 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fae): Mirror 9: 0xfe9caa91 (stale) Mirror 10: 0xfe9caa91 (stale) Mirror 14: 0xfe9caa91 (stale) Mirror 18: 0xfe9caa91 (stale) Mirror 19: 0xfe9caa91 (stale) Mirror 20: 0xfe9caa91 (stale) Mirror 21: 0xfe9caa91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fae): Mirror 9: 0xfe9caa91 (stale) Mirror 10: 0xfe9caa91 (stale) Mirror 14: 0xfe9caa91 (stale) Mirror 18: 0xfe9caa91 (stale) Mirror 19: 0xfe9caa91 (stale) Mirror 20: 0xfe9caa91 (stale) Mirror 21: 0xfe9caa91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fae: skipped 000714 1788836892.930743 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37fae): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fae): Mirror 9: 0xfe9caa91 (stale) Mirror 10: 0xfe9caa91 (stale) Mirror 14: 0xfe9caa91 (stale) Mirror 18: 0xfe9caa91 (stale) Mirror 19: 0xfe9caa91 (stale) Mirror 20: 0xfe9caa91 (stale) Mirror 21: 0xfe9caa91 (stale) CRC-32 checksum value for chunk [0, 0x37fae): Mirror 1: 0x81c8f94b Mirror 22: 0x81c8f94b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fae: skipped 000716 1788836894.152268 mapread 0x02bd0a thru 0x036379 (0xa670 bytes) 000717 1788836894.214449 punch from 0x02aacf to 0x031511 (0x6a43 bytes) 000718 1788836894.287329 trunc from 0x037fae to 0x034bab (0x3402 bytes) 000720 1788836894.353167 write 0x029761 thru 0x02fa39 (0x62d9 bytes) 000721 1788836894.364269 trunc from 0x00ccdc to 0x011718 (0x4a3d bytes) 000722 1788836894.369725 mapwrite 0x002493 thru 0x003707 (0x1275 bytes) 000723 1788836894.445234 mapread 0x013d88 thru 0x01e84c (0xaac5 bytes) 000724 1788836894.489786 trunc from 0x017b13 to 0x01c277 (0x4765 bytes) 000725 1788836894.498004 punch from 0x00bf42 to 0x00fbea (0x3ca9 bytes) 000726 1788836894.517987 punch from 0x004402 to 0x006d48 (0x2947 bytes) 000727 1788836894.524692 trunc from 0x031ba7 to 0x034bab (0x3005 bytes) 000728 1788836894.533586 fallocate 0x00b276 thru 0x015400 (0xa18b bytes) 000729 1788836894.543440 write 0x025d96 thru 0x0324d8 (0xc743 bytes) 000730 1788836894.556937 trunc from 0x005d5c to 0x007f0c (0x21b1 bytes) 000731 1788836894.562033 trunc from 0x018665 to 0x023e7c (0xb818 bytes) 000732 1788836894.569918 trunc from 0x034bac to 0x00319d (0x31a0e bytes) 000733 1788836894.611145 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x319e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x319e): Mirror 9: 0x93a834ad (stale) Mirror 10: 0x93a834ad (stale) Mirror 14: 0x93a834ad (stale) Mirror 18: 0x93a834ad (stale) Mirror 19: 0x93a834ad (stale) Mirror 20: 0x93a834ad (stale) Mirror 21: 0x93a834ad (stale) Mirror 22: 0x93a834ad (stale) CRC-32 checksum value for chunk [0, 0x319e): Mirror 1: 0x615f098f Mirror 23: 0x615f098f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x319e: skipped 000734 1788836895.545095 mapread 0x000e62 thru 0x00319d (0x233c bytes) 000735 1788836895.592741 trunc from 0x002aa0 to 0x00319d (0x6fe bytes) 000736 1788836895.648131 punch from 0x0005d7 to 0x00319d (0x2bc7 bytes) 000737 1788836895.654088 trunc from 0x001856 to 0x00319d (0x1948 bytes) 000738 1788836895.660191 read 0x000091 thru 0x00319d (0x310d bytes) 000739 1788836895.671830 MIRROR_RESYNC 10 mirrors 000740 1788836897.322216 punch from 0x0005f8 to 0x00319d (0x2ba6 bytes) 000741 1788836897.387891 fallocate 0x010165 thru 0x011f76 (0x1e12 bytes) 000742 1788836897.404624 trunc from 0x011f77 to 0x039b4b (0x27bd5 bytes) 000743 1788836897.438909 trunc from 0x039b4c to 0x0037cd (0x3637e bytes) 000744 1788836897.459701 trunc from 0x0037ce to 0x01aa2c (0x1725f bytes) 000745 1788836897.478360 punch from 0x00c5c4 to 0x00d4fe (0xf3b bytes) 000746 1788836897.484980 punch from 0x0030a5 to 0x00df83 (0xaedf bytes) 000747 1788836897.491592 write 0x01047a thru 0x01e7b5 (0xe33c bytes) 000748 1788836897.518018 MIRROR_RESYNC 10 mirrors 000749 1788836900.173161 punch from 0x004167 to 0x00856c (0x4406 bytes) 000750 1788836900.225829 trunc from 0x01e7b6 to 0x033cc7 (0x15512 bytes) 000751 1788836900.262751 trunc from 0x033cc8 to 0x001bd5 (0x320f2 bytes) 000752 1788836900.286969 mapread 0x00013b thru 0x001bd5 (0x1a9b bytes) 000754 1788836900.302138 read 0x001473 thru 0x001bd5 (0x763 bytes) 000755 1788836900.302706 write 0x02721d thru 0x028fc6 (0x1daa bytes) 000756 1788836900.303650 mapwrite 0x010483 thru 0x019cf9 (0x9877 bytes) 000757 1788836900.342434 trunc from 0x01441f to 0x022e7f (0xea61 bytes) 000758 1788836900.350083 trunc from 0x028fc7 to 0x028e5d (0x169 bytes) 000759 1788836900.370425 trunc from 0x021a6e to 0x0280bf (0x6652 bytes) 000760 1788836900.378659 mapread 0x027c97 thru 0x028e5d (0x11c7 bytes) 000762 1788836900.391034 read 0x017bc6 thru 0x0267f3 (0xec2e bytes) 000763 1788836900.411429 fallocate 0x001e39 thru 0x00d9cd (0xbb95 bytes) 000764 1788836900.417330 mapwrite 0x030c9d thru 0x031e96 (0x11fa bytes) 000765 1788836900.477269 mapwrite 0x021e91 thru 0x0307a0 (0xe910 bytes) 000766 1788836900.577332 trunc from 0x0295e2 to 0x029772 (0x191 bytes) 000767 1788836900.595112 mapread 0x0234a1 thru 0x02ed28 (0xb888 bytes) 000768 1788836900.618958 punch from 0x01636e to 0x01c6d1 (0x6364 bytes) 000769 1788836900.626023 mapwrite 0x00a8b6 thru 0x013410 (0x8b5b bytes) 000770 1788836900.669354 punch from 0x022406 to 0x02b16c (0x8d67 bytes) 000771 1788836900.683069 read 0x01530b thru 0x01fc83 (0xa979 bytes) 000772 1788836900.712149 read 0x014b7b thru 0x016016 (0x149c bytes) 000773 1788836900.713308 punch from 0x01b6b6 to 0x01b780 (0xcb bytes) 000774 1788836900.731925 trunc from 0x030ad0 to 0x031e96 (0x13c7 bytes) 000775 1788836900.741813 punch from 0x015201 to 0x01bb09 (0x6909 bytes) 000776 1788836900.757019 punch from 0x02c6b5 to 0x02e22f (0x1b7b bytes) 000777 1788836900.767757 trunc from 0x02bd7a to 0x02d700 (0x1987 bytes) 000778 1788836900.779961 mapread 0x02c5a5 thru 0x031e96 (0x58f2 bytes) 000779 1788836900.795721 mapwrite 0x0182fb thru 0x01f7aa (0x74b0 bytes) 000780 1788836900.845127 read 0x011757 thru 0x0132b6 (0x1b60 bytes) 000781 1788836900.851383 punch from 0x02a706 to 0x031e96 (0x7791 bytes) 000782 1788836900.861121 mapread 0x01c37d thru 0x024aa3 (0x8727 bytes) 000783 1788836900.862203 write 0x038bf7 thru 0x03ffff (0x7409 bytes) 000784 1788836900.882654 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xffda116b (stale) Mirror 10: 0xffda116b (stale) Mirror 14: 0xffda116b (stale) Mirror 18: 0xffda116b (stale) Mirror 19: 0xffda116b (stale) Mirror 20: 0xffda116b (stale) Mirror 21: 0xffda116b (stale) Mirror 22: 0xffda116b (stale) Mirror 23: 0xffda116b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x62fc16cd Mirror 24: 0x62fc16cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000785 1788836902.563949 mapwrite 0x00977a thru 0x01242c (0x8cb3 bytes) 000786 1788836902.751084 punch from 0x012d8d to 0x0157f8 (0x2a6c bytes) 000787 1788836902.755339 trunc from 0x040000 to 0x021145 (0x1eeba bytes) 000788 1788836902.774680 write 0x010b76 thru 0x01bc09 (0xb094 bytes) 000790 1788836902.779354 mapwrite 0x0040db thru 0x011bcf (0xdaf5 bytes) 000791 1788836902.799135 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21146): Mirror 10: 0x8f896f6f (stale) Mirror 14: 0x8f896f6f (stale) Mirror 18: 0x8f896f6f (stale) Mirror 19: 0x8f896f6f (stale) Mirror 20: 0x8f896f6f (stale) Mirror 21: 0x8f896f6f (stale) Mirror 22: 0x8f896f6f (stale) Mirror 23: 0x8f896f6f (stale) Mirror 24: 0x1281ae22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21146): Mirror 10: 0x8f896f6f (stale) Mirror 14: 0x8f896f6f (stale) Mirror 18: 0x8f896f6f (stale) Mirror 19: 0x8f896f6f (stale) Mirror 20: 0x8f896f6f (stale) Mirror 21: 0x8f896f6f (stale) Mirror 22: 0x8f896f6f (stale) Mirror 23: 0x8f896f6f (stale) Mirror 24: 0x1281ae22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21146: skipped 000792 1788836903.678214 trunc from 0x004bcd to 0x00a7e7 (0x5c1b bytes) 000793 1788836903.689674 write 0x03eb55 thru 0x03ffff (0x14ab bytes) 000794 1788836903.702157 punch from 0x0278ff to 0x036245 (0xe947 bytes) 000795 1788836903.713431 trunc from 0x040000 to 0x02860e (0x179f1 bytes) 000796 1788836903.744842 trunc from 0x02860f to 0x0087c2 (0x1fe4c bytes) 000797 1788836903.770372 punch from 0x001e4b to 0x0087c2 (0x6978 bytes) 000798 1788836903.776214 fallocate 0x03f25d thru 0x03ffff (0xda3 bytes) 000799 1788836903.783029 mapread 0x00dc6e thru 0x019240 (0xb5d3 bytes) 000800 1788836903.864969 mapread 0x026831 thru 0x03195c (0xb12c bytes) 000801 1788836903.880408 punch from 0x0258b8 to 0x02d36d (0x7ab6 bytes) 000802 1788836903.884001 mapwrite 0x0169f9 thru 0x026942 (0xff4a bytes) 000803 1788836903.913578 fallocate 0x029589 thru 0x038992 (0xf40a bytes) 000804 1788836903.930579 mapread 0x013ec0 thru 0x014702 (0x843 bytes) 000805 1788836903.933299 trunc from 0x036040 to 0x03c44d (0x640e bytes) 000806 1788836903.941607 read 0x03496c thru 0x03ffff (0xb694 bytes) 000807 1788836903.954719 mapread 0x032c70 thru 0x03c0df (0x9470 bytes) 000809 1788836903.957786 trunc from 0x0351d2 to 0x03dfeb (0x8e1a bytes) 000810 1788836903.963696 mapwrite 0x0113fe thru 0x01c306 (0xaf09 bytes) 000811 1788836903.984458 trunc from 0x02b6fe to 0x036e4b (0xb74e bytes) 000813 1788836903.991304 trunc from 0x010229 to 0x01c715 (0xc4ed bytes) 000814 1788836903.998680 punch from 0x013f22 to 0x01cfba (0x9099 bytes) 000815 1788836904.005286 punch from 0x02234f to 0x0264fa (0x41ac bytes) 000816 1788836904.009800 trunc from 0x040000 to 0x00b35f (0x34ca0 bytes) 000817 1788836904.030899 fallocate 0x00c3a1 thru 0x013f81 (0x7be1 bytes) 000818 1788836904.038614 read 0x0079a0 thru 0x00b35f (0x39c0 bytes) 000820 1788836904.050565 read 0x007c6a thru 0x009f3a (0x22d1 bytes) 000822 1788836904.051847 mapwrite 0x02e260 thru 0x035a2f (0x77d0 bytes) 000823 1788836904.128691 write 0x022365 thru 0x0294d7 (0x7173 bytes) 000825 1788836904.131773 read 0x00374c thru 0x004714 (0xfc9 bytes) 000826 1788836904.137849 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35a30): Mirror 14: 0x5dd6c5ea (stale) Mirror 18: 0x5dd6c5ea (stale) Mirror 19: 0x5dd6c5ea (stale) Mirror 20: 0x5dd6c5ea (stale) Mirror 21: 0x5dd6c5ea (stale) Mirror 22: 0x5dd6c5ea (stale) Mirror 23: 0x5dd6c5ea (stale) Mirror 24: 0x619a6048 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a30): Mirror 14: 0x5dd6c5ea (stale) Mirror 18: 0x5dd6c5ea (stale) Mirror 19: 0x5dd6c5ea (stale) Mirror 20: 0x5dd6c5ea (stale) Mirror 21: 0x5dd6c5ea (stale) Mirror 22: 0x5dd6c5ea (stale) Mirror 23: 0x5dd6c5ea (stale) Mirror 24: 0x619a6048 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a30: skipped 000827 1788836905.127967 trunc from 0x035a30 to 0x01b42b (0x1a604 bytes) 000828 1788836905.175102 write 0x033a55 thru 0x034ca6 (0x1252 bytes) 000829 1788836905.207151 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x34ca7): Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ca7): Mirror 14: 0x8ffb0515 (stale) Mirror 18: 0x8ffb0515 (stale) Mirror 19: 0x8ffb0515 (stale) Mirror 20: 0x8ffb0515 (stale) Mirror 21: 0x8ffb0515 (stale) Mirror 22: 0x8ffb0515 (stale) Mirror 23: 0x8ffb0515 (stale) Mirror 24: 0x425dee36 (stale) CRC-32 checksum value for chunk [0, 0x34ca7): Mirror 1: 0x1dd310b6 Mirror 25: 0x1dd310b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ca7: skipped 000830 1788836906.808874 punch from 0x03349c to 0x033ae5 (0x64a bytes) 000831 1788836906.902667 fallocate 0x00b493 thru 0x0136bf (0x822d bytes) 000832 1788836906.911168 trunc from 0x02b666 to 0x034ca6 (0x9641 bytes) 000833 1788836906.927667 mapread 0x0266ab thru 0x0308e9 (0xa23f bytes) 000835 1788836906.968829 trunc from 0x034ca7 to 0x009ee6 (0x2adc0 bytes) 000836 1788836907.003771 write 0x0196de thru 0x01cae5 (0x3408 bytes) 000837 1788836907.005093 write 0x00e4fc thru 0x01b0c7 (0xcbcc bytes) 000839 1788836907.013814 fallocate 0x015370 thru 0x0187ef (0x3480 bytes) 000840 1788836907.019754 trunc from 0x00d282 to 0x01c6ba (0xf439 bytes) 000842 1788836907.043587 fallocate 0x01cd6d thru 0x02079b (0x3a2f bytes) 000843 1788836907.050711 punch from 0x00f112 to 0x01c6d3 (0xd5c2 bytes) 000844 1788836907.060622 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cae6): Mirror 14: 0x319d4f8a (stale) Mirror 18: 0x319d4f8a (stale) Mirror 20: 0x319d4f8a (stale) Mirror 21: 0x319d4f8a (stale) Mirror 22: 0x319d4f8a (stale) Mirror 23: 0x319d4f8a (stale) Mirror 24: 0x747a5510 (stale) Mirror 25: 0xc8b43add (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cae6): Mirror 14: 0x319d4f8a (stale) Mirror 18: 0x319d4f8a (stale) Mirror 20: 0x319d4f8a (stale) Mirror 21: 0x319d4f8a (stale) Mirror 22: 0x319d4f8a (stale) Mirror 23: 0x319d4f8a (stale) Mirror 24: 0x747a5510 (stale) Mirror 25: 0xc8b43add (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cae6: skipped 000845 1788836907.742302 mapread 0x001207 thru 0x00fdbb (0xebb5 bytes) 000846 1788836907.772114 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cae6): Mirror 18: 0x319d4f8a (stale) Mirror 20: 0x319d4f8a (stale) Mirror 21: 0x319d4f8a (stale) Mirror 22: 0x319d4f8a (stale) Mirror 23: 0x319d4f8a (stale) Mirror 24: 0x747a5510 (stale) Mirror 25: 0xc8b43add (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cae6): Mirror 18: 0x319d4f8a (stale) Mirror 20: 0x319d4f8a (stale) Mirror 21: 0x319d4f8a (stale) Mirror 22: 0x319d4f8a (stale) Mirror 23: 0x319d4f8a (stale) Mirror 24: 0x747a5510 (stale) Mirror 25: 0xc8b43add (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cae6: skipped 000847 1788836908.442361 trunc from 0x012e3e to 0x01cae5 (0x9ca8 bytes) 000849 1788836908.448298 read 0x01411e thru 0x01cae5 (0x89c8 bytes) 000850 1788836908.459062 fallocate 0x014ec9 thru 0x017265 (0x239d bytes) 000851 1788836908.464930 mapwrite 0x036071 thru 0x03ffff (0x9f8f bytes) 000852 1788836908.546847 trunc from 0x006d3f to 0x0121d3 (0xb495 bytes) 000853 1788836908.567835 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xffda116b (stale) Mirror 21: 0xffda116b (stale) Mirror 22: 0xffda116b (stale) Mirror 23: 0xffda116b (stale) Mirror 24: 0x62fc16cd (stale) Mirror 25: 0xaaee26eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xffda116b (stale) Mirror 21: 0xffda116b (stale) Mirror 22: 0xffda116b (stale) Mirror 23: 0xffda116b (stale) Mirror 24: 0x62fc16cd (stale) Mirror 25: 0xaaee26eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000854 1788836909.210163 mapwrite 0x027c03 thru 0x0283a2 (0x7a0 bytes) 000855 1788836909.237582 write 0x02555c thru 0x0293ef (0x3e94 bytes) 000856 1788836909.248580 mapread 0x0382bc thru 0x03fa4c (0x7791 bytes) 000857 1788836909.256560 mapwrite 0x01a246 thru 0x025979 (0xb734 bytes) 000858 1788836909.293665 read 0x031b05 thru 0x038a52 (0x6f4e bytes) 000860 1788836909.294898 trunc from 0x031355 to 0x036057 (0x4d03 bytes) 000861 1788836909.301607 fallocate 0x0171d4 thru 0x0226dc (0xb509 bytes) 000862 1788836909.306628 trunc from 0x040000 to 0x017191 (0x28e6e bytes) 000863 1788836909.328080 write 0x027d90 thru 0x02964d (0x18be bytes) 000865 1788836909.330892 mapread 0x00317d thru 0x00b258 (0x80dc bytes) 000866 1788836909.340137 trunc from 0x0184a0 to 0x01c502 (0x4063 bytes) 000867 1788836909.345629 mapwrite 0x006cce thru 0x00e765 (0x7a98 bytes) 000869 1788836909.365112 mapwrite 0x01a20d thru 0x01a25f (0x53 bytes) 000870 1788836909.399228 fallocate 0x026e72 thru 0x030258 (0x93e7 bytes) 000871 1788836909.404689 punch from 0x0009a4 to 0x00f815 (0xee72 bytes) 000873 1788836909.411104 mapread 0x002a64 thru 0x00f728 (0xccc5 bytes) 000874 1788836909.428707 punch from 0x00e926 to 0x01326a (0x4945 bytes) 000875 1788836909.433759 write 0x014d3f thru 0x020791 (0xba53 bytes) 000876 1788836909.440102 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30259): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30259): Mirror 20: 0xf992dc5e (stale) Mirror 21: 0xf992dc5e (stale) Mirror 22: 0xf992dc5e (stale) Mirror 23: 0xf992dc5e (stale) Mirror 24: 0x897ecea4 (stale) Mirror 25: 0xe45cde9e (stale) CRC-32 checksum value for chunk [0, 0x30259): Mirror 1: 0xda58325d Mirror 26: 0xda58325d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30259: skipped 000878 1788836910.508691 mapread 0x00d3c3 thru 0x019b4d (0xc78b bytes) 000879 1788836910.568928 trunc from 0x005379 to 0x00c1da (0x6e62 bytes) 000880 1788836910.634757 read 0x020fbe thru 0x029c05 (0x8c48 bytes) 000882 1788836910.645105 trunc from 0x030259 to 0x000167 (0x300f1 bytes) 000883 1788836910.706858 trunc from 0x00004b to 0x000167 (0x11d bytes) 000884 1788836910.713671 fallocate 0x01b891 thru 0x021bff (0x636f bytes) 000885 1788836910.720208 read 0x003e1b thru 0x009299 (0x547f bytes) 000887 1788836910.735739 trunc from 0x01f09f to 0x021bff (0x2b61 bytes) 000888 1788836910.742271 punch from 0x003fee to 0x009f0d (0x5f20 bytes) 000889 1788836910.753329 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x21c00): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21c00): Mirror 20: 0x5ff180c7 (stale) Mirror 21: 0x5ff180c7 (stale) Mirror 22: 0x5ff180c7 (stale) Mirror 23: 0x5ff180c7 (stale) Mirror 24: 0x45802a88 (stale) Mirror 25: 0x46cd28ff (stale) Mirror 26: 0x868524d0 (stale) CRC-32 checksum value for chunk [0, 0x21c00): Mirror 1: 0x46cd28ff Mirror 27: 0x46cd28ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21c00: skipped 000890 1788836912.550027 read 0x01e7b7 thru 0x021202 (0x2a4c bytes) 000891 1788836912.581873 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21c00): Mirror 20: 0x5ff180c7 (stale) Mirror 22: 0x5ff180c7 (stale) Mirror 23: 0x5ff180c7 (stale) Mirror 24: 0x45802a88 (stale) Mirror 25: 0x46cd28ff (stale) Mirror 26: 0x868524d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21c00): Mirror 20: 0x5ff180c7 (stale) Mirror 22: 0x5ff180c7 (stale) Mirror 23: 0x5ff180c7 (stale) Mirror 24: 0x45802a88 (stale) Mirror 25: 0x46cd28ff (stale) Mirror 26: 0x868524d0 (stale) CRC-32 checksum value for chunk [0, 0x21c00): Mirror 1: 0x46cd28ff Mirror 27: 0x46cd28ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21c00: skipped 000892 1788836913.811793 mapwrite 0x0137f5 thru 0x01d701 (0x9f0d bytes) 000894 1788836914.024084 mapread 0x018caa thru 0x01e6a9 (0x5a00 bytes) 000896 1788836914.027996 fallocate 0x0247ff thru 0x029ecb (0x56cd bytes) 000897 1788836914.035034 mapwrite 0x025395 thru 0x03190a (0xc576 bytes) 000898 1788836914.236938 write 0x00fdfb thru 0x01ab77 (0xad7d bytes) 000899 1788836914.240576 mapwrite 0x020134 thru 0x022e8f (0x2d5c bytes) 000900 1788836914.283234 fallocate 0x01fe6e thru 0x02e79a (0xe92d bytes) 000901 1788836914.297045 punch from 0x02779a to 0x03190a (0xa171 bytes) 000902 1788836914.335573 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3190b): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3190b): Mirror 20: 0xfe8b543 (stale) Mirror 22: 0xfe8b543 (stale) Mirror 23: 0xfe8b543 (stale) Mirror 24: 0xcbe6c59a (stale) Mirror 25: 0xddef38d2 (stale) Mirror 26: 0x9cf435b4 (stale) Mirror 27: 0xddef38d2 (stale) CRC-32 checksum value for chunk [0, 0x3190b): Mirror 1: 0x4fcd4113 Mirror 28: 0x4fcd4113 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3190b: skipped 000903 1788836915.913231 trunc from 0x03190b to 0x01ff9e (0x1196c bytes) 000904 1788836916.084318 read 0x00b4f3 thru 0x01ab80 (0xf68e bytes) 000906 1788836916.112126 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1ff9f): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff9f): Mirror 20: 0x62a71f96 (stale) Mirror 22: 0x62a71f96 (stale) Mirror 23: 0x62a71f96 (stale) Mirror 24: 0x37c66db6 (stale) Mirror 25: 0x5e9490d7 (stale) Mirror 26: 0x6f0bd1d5 (stale) Mirror 27: 0x5e9490d7 (stale) Mirror 28: 0xcf1260b6 (stale) CRC-32 checksum value for chunk [0, 0x1ff9f): Mirror 1: 0xcf1260b6 Mirror 29: 0xcf1260b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff9f: skipped 000907 1788836917.199378 fallocate 0x03af8a thru 0x03e242 (0x32b9 bytes) 000908 1788836917.247279 read 0x003c7f thru 0x00d825 (0x9ba7 bytes) 000909 1788836917.261972 fallocate 0x01fb8c thru 0x02f170 (0xf5e5 bytes) 000910 1788836917.272903 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1ff9f): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff9f): Mirror 20: 0x62a71f96 (stale) Mirror 22: 0x62a71f96 (stale) Mirror 23: 0x62a71f96 (stale) Mirror 24: 0x37c66db6 (stale) Mirror 25: 0x5e9490d7 (stale) Mirror 26: 0x6f0bd1d5 (stale) Mirror 27: 0x5e9490d7 (stale) Mirror 28: 0xcf1260b6 (stale) Mirror 29: 0xcf1260b6 (stale) CRC-32 checksum value for chunk [0, 0x1ff9f): Mirror 1: 0xcf1260b6 Mirror 30: 0xcf1260b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff9f: skipped 000911 1788836918.408672 fallocate 0x019011 thru 0x020627 (0x7617 bytes) 000912 1788836918.494603 punch from 0x0096f2 to 0x01924e (0xfb5d bytes) 000913 1788836918.504148 trunc from 0x020628 to 0x01a24f (0x63d8 bytes) 000914 1788836918.540892 mapwrite 0x02637a thru 0x031655 (0xb2dc bytes) 000915 1788836918.636798 read 0x030f1e thru 0x031655 (0x738 bytes) 000916 1788836918.638769 mapwrite 0x002778 thru 0x010070 (0xd8f9 bytes) 000917 1788836918.687832 mapwrite 0x032c01 thru 0x036972 (0x3d72 bytes) 000918 1788836918.748825 mapwrite 0x0070ef thru 0x013e3d (0xcd4f bytes) 000919 1788836918.780418 fallocate 0x02a865 thru 0x02d047 (0x27e3 bytes) 000920 1788836918.791196 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36973): Mirror 20: 0xeb58a39 (stale) Mirror 22: 0xeb58a39 (stale) Mirror 23: 0xeb58a39 (stale) Mirror 25: 0x3cc6d21b (stale) Mirror 26: 0x50bd2670 (stale) Mirror 27: 0xf7f0c680 (stale) Mirror 28: 0x9fef90cd (stale) Mirror 29: 0xf80ea6c1 (stale) Mirror 30: 0xf80ea6c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36973): Mirror 20: 0xeb58a39 (stale) Mirror 22: 0xeb58a39 (stale) Mirror 23: 0xeb58a39 (stale) Mirror 25: 0x3cc6d21b (stale) Mirror 26: 0x50bd2670 (stale) Mirror 27: 0xf7f0c680 (stale) Mirror 28: 0x9fef90cd (stale) Mirror 29: 0xf80ea6c1 (stale) Mirror 30: 0xf80ea6c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36973: skipped 000921 1788836919.788372 punch from 0x02bbfb to 0x0342a0 (0x86a6 bytes) 000922 1788836919.794428 trunc from 0x036973 to 0x00cac8 (0x29eaa bytes) 000923 1788836919.835367 mapread 0x007a31 thru 0x00a997 (0x2f67 bytes) 000924 1788836919.861088 trunc from 0x00cac9 to 0x036a23 (0x29f5b bytes) 000925 1788836919.880415 punch from 0x0008b0 to 0x00ab79 (0xa2ca bytes) 000926 1788836919.884391 write 0x0366f2 thru 0x03ffff (0x990e bytes) 000927 1788836919.889103 write 0x026a34 thru 0x030c95 (0xa262 bytes) 000928 1788836919.896847 mapread 0x0102a0 thru 0x013ea3 (0x3c04 bytes) 000929 1788836919.906377 mapread 0x00743a thru 0x012f7f (0xbb46 bytes) 000930 1788836919.912450 mapread 0x0271ad thru 0x02c986 (0x57da bytes) 000931 1788836919.913014 mapwrite 0x01b3db thru 0x020158 (0x4d7e bytes) 000932 1788836919.937513 fallocate 0x02fdd3 thru 0x037602 (0x7830 bytes) 000933 1788836919.942614 trunc from 0x040000 to 0x03d3bb (0x2c44 bytes) 000934 1788836919.960840 mapread 0x00a048 thru 0x019518 (0xf4d1 bytes) 000935 1788836919.962138 fallocate 0x00c110 thru 0x014d0f (0x8c00 bytes) 000936 1788836919.971179 mapread 0x01676d thru 0x01711d (0x9b1 bytes) 000937 1788836919.979719 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d3bc): Mirror 20: 0xa22d86c8 (stale) Mirror 22: 0xa22d86c8 (stale) Mirror 23: 0xa22d86c8 (stale) Mirror 25: 0x1ef8aa3e (stale) Mirror 26: 0x49f19e87 (stale) Mirror 27: 0x233b026f (stale) Mirror 28: 0x7c3dbf51 (stale) Mirror 29: 0xa89869e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d3bc): Mirror 20: 0xa22d86c8 (stale) Mirror 22: 0xa22d86c8 (stale) Mirror 23: 0xa22d86c8 (stale) Mirror 25: 0x1ef8aa3e (stale) Mirror 26: 0x49f19e87 (stale) Mirror 27: 0x233b026f (stale) Mirror 28: 0x7c3dbf51 (stale) Mirror 29: 0xa89869e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d3bc: skipped 000938 1788836920.875234 write 0x002ff3 thru 0x00aa79 (0x7a87 bytes) 000939 1788836920.898650 MIRROR_RESYNC 9 mirrors 000940 1788836924.842939 fallocate 0x01f596 thru 0x022d6f (0x37da bytes) 000941 1788836924.884083 read 0x0016e4 thru 0x0038be (0x21db bytes) 000942 1788836924.889180 trunc from 0x009cf4 to 0x019956 (0xfc63 bytes) 000943 1788836924.895414 mapread 0x0096e7 thru 0x0104f7 (0x6e11 bytes) 000944 1788836924.919189 trunc from 0x03d3bc to 0x01bd1c (0x2169f bytes) 000945 1788836924.962818 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bd1d): Mirror 20: 0xec3ecc31 (stale) Mirror 22: 0xec3ecc31 (stale) Mirror 25: 0xec3ecc31 (stale) Mirror 26: 0xec3ecc31 (stale) Mirror 27: 0xec3ecc31 (stale) Mirror 28: 0xec3ecc31 (stale) Mirror 29: 0xec3ecc31 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd1d): Mirror 20: 0xec3ecc31 (stale) Mirror 22: 0xec3ecc31 (stale) Mirror 25: 0xec3ecc31 (stale) Mirror 26: 0xec3ecc31 (stale) Mirror 27: 0xec3ecc31 (stale) Mirror 28: 0xec3ecc31 (stale) Mirror 29: 0xec3ecc31 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd1d: skipped 000946 1788836925.877902 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1bd1d): Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd1d): Mirror 20: 0xec3ecc31 (stale) Mirror 22: 0xec3ecc31 (stale) Mirror 25: 0xec3ecc31 (stale) Mirror 26: 0xec3ecc31 (stale) Mirror 27: 0xec3ecc31 (stale) Mirror 28: 0xec3ecc31 (stale) Mirror 29: 0xec3ecc31 (stale) CRC-32 checksum value for chunk [0, 0x1bd1d): Mirror 1: 0xfac5d198 Mirror 30: 0xfac5d198 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd1d: skipped 000947 1788836927.229850 trunc from 0x018b4f to 0x01bd1c (0x31ce bytes) 000948 1788836927.320406 write 0x00805c thru 0x00b1e9 (0x318e bytes) 000949 1788836927.346588 trunc from 0x004d07 to 0x010497 (0xb791 bytes) 000950 1788836927.372632 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bd1d): Mirror 22: 0xec3ecc31 (stale) Mirror 25: 0xec3ecc31 (stale) Mirror 26: 0xec3ecc31 (stale) Mirror 27: 0xec3ecc31 (stale) Mirror 28: 0xec3ecc31 (stale) Mirror 29: 0xec3ecc31 (stale) Mirror 30: 0xfac5d198 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bd1d): Mirror 22: 0xec3ecc31 (stale) Mirror 25: 0xec3ecc31 (stale) Mirror 26: 0xec3ecc31 (stale) Mirror 27: 0xec3ecc31 (stale) Mirror 28: 0xec3ecc31 (stale) Mirror 29: 0xec3ecc31 (stale) Mirror 30: 0xfac5d198 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bd1d: skipped 000951 1788836928.045130 mapwrite 0x01d7ca thru 0x020f3e (0x3775 bytes) 000952 1788836928.137501 trunc from 0x020f3f to 0x01108f (0xfeaf bytes) 000953 1788836928.191950 trunc from 0x011090 to 0x034788 (0x236f9 bytes) 000954 1788836928.226868 trunc from 0x034789 to 0x01977d (0x1b00b bytes) 000955 1788836928.264782 trunc from 0x00b28a to 0x018316 (0xd08d bytes) 000956 1788836928.278370 read 0x0001c2 thru 0x009764 (0x95a3 bytes) 000957 1788836928.296065 punch from 0x018c8b to 0x01977d (0xaf3 bytes) 000958 1788836928.300544 trunc from 0x01977e to 0x02eea8 (0x1572b bytes) 000959 1788836928.356718 trunc from 0x02eea9 to 0x027f31 (0x6f77 bytes) 000960 1788836928.431664 mapwrite 0x002b19 thru 0x0129c3 (0xfeab bytes) 000962 1788836928.479698 trunc from 0x027f32 to 0x00609e (0x21e93 bytes) 000963 1788836928.504750 trunc from 0x00609f to 0x022ca0 (0x1cc02 bytes) 000964 1788836928.524819 mapread 0x007e14 thru 0x00e8a6 (0x6a93 bytes) 000965 1788836928.542413 read 0x01b2f9 thru 0x02024a (0x4f52 bytes) 000966 1788836928.569270 punch from 0x002794 to 0x00c4fe (0x9d6b bytes) 000967 1788836928.583493 punch from 0x00da98 to 0x01b3f5 (0xd95e bytes) 000968 1788836928.588460 mapread 0x00fedc thru 0x0164f8 (0x661d bytes) 000969 1788836928.684991 mapwrite 0x039ac7 thru 0x03ffff (0x6539 bytes) 000970 1788836928.779277 trunc from 0x040000 to 0x004a0e (0x3b5f1 bytes) 000971 1788836928.842878 trunc from 0x002df6 to 0x0032fd (0x508 bytes) 000972 1788836928.857380 punch from 0x0026af to 0x004a0e (0x2360 bytes) 000973 1788836928.883309 MIRROR_RESYNC 8 mirrors 000974 1788836930.912405 mapread 0x002032 thru 0x004a0e (0x29dd bytes) 000976 1788836930.936492 punch from 0x00464a to 0x004a0e (0x3c5 bytes) 000977 1788836930.986596 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x4a0f): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4a0f): Mirror 22: 0x51dbc83d (stale) Mirror 25: 0x51dbc83d (stale) Mirror 26: 0x51dbc83d (stale) Mirror 27: 0x51dbc83d (stale) Mirror 28: 0x51dbc83d (stale) Mirror 29: 0x51dbc83d (stale) Mirror 30: 0x51dbc83d (stale) CRC-32 checksum value for chunk [0, 0x4a0f): Mirror 1: 0x51dbc83d Mirror 31: 0x51dbc83d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4a0f: skipped 000978 1788836931.982411 fallocate 0x00d8ce thru 0x012b37 (0x526a bytes) 000980 1788836932.031229 write 0x0226ba thru 0x02cf1d (0xa864 bytes) 000981 1788836932.043147 read 0x013735 thru 0x01ba1b (0x82e7 bytes) 000982 1788836932.051603 mapread 0x017c83 thru 0x017cda (0x58 bytes) 000983 1788836932.052350 trunc from 0x02cf1e to 0x0390b9 (0xc19c bytes) 000984 1788836932.070540 fallocate 0x015858 thru 0x023c0c (0xe3b5 bytes) 000986 1788836932.083587 trunc from 0x0390ba to 0x01f901 (0x197b8 bytes) 000987 1788836932.100912 write 0x01e31c thru 0x020a81 (0x2766 bytes) 000988 1788836932.106584 mapwrite 0x02fbe1 thru 0x03177b (0x1b9b bytes) 000989 1788836932.144104 trunc from 0x03177c to 0x012cca (0x1eab1 bytes) 000990 1788836932.153712 trunc from 0x012ccb to 0x03bae4 (0x28e1a bytes) 000991 1788836932.168385 punch from 0x019b5c to 0x02536b (0xb810 bytes) 000993 1788836932.172695 trunc from 0x03bae5 to 0x0120d8 (0x29a0c bytes) 000994 1788836932.198725 trunc from 0x0120d9 to 0x017d92 (0x5cba bytes) 000996 1788836932.212277 punch from 0x00aa51 to 0x013d2f (0x92df bytes) 000997 1788836932.217710 mapwrite 0x03dc31 thru 0x03ffff (0x23cf bytes) 000999 1788836932.249298 read 0x02724a thru 0x035b27 (0xe8de bytes) 001000 1788836932.262655 read 0x03910e thru 0x03ffff (0x6ef2 bytes) All operations completed A-OK! PASS 70a (152s) == sanity-flr test 71: check mirror extend parent fid ==== 23:08:58 (1788836938) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x2d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 23:09:09 (1788836949) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213973 s, 9.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162993 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xee:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x10b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xef:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 23:09:20 (1788836960) Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre2 Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8684512 .. Extending file size to 5711712 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6459296 .. Extending file size to 6756768 .. Extending file size to 7065824 .. Extending file size to 9202112 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61368 61369 61371 61374 61375 done 192.168.202.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg256-client.virtnet /mnt/lustre2 (opts:) 192.168.202.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg256-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:10:45 (1788837045) Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre2 Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7513600 .. done Extending file size to 1050560 .. Extending file size to 5203904 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8413152 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9141568 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8765088 .. Extending file size to 9072352 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62976 62977 62980 62981 62982 192.168.202.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg256-client.virtnet /mnt/lustre2 (opts:) 192.168.202.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg256-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:12:07 (1788837127) Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2256 1269048 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18772 3588248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24632 3582388 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43404 7170636 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2256 1269048 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18772 3588248 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24632 3582388 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43404 7170636 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152456 s, 13.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.214843 s, 9.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5ee599f1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg256-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:12:31 (1788837151) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0767841 s, 27.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 23:12:39 (1788837159) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x122:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x123:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x104:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:12:48 (1788837168) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0792964 s, 26.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:12:55 (1788837175) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106322 s, 19.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x105:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x124:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x105:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x106:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:13:03 (1788837183) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140365 s, 14.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:13:37 (1788837217) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg256-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080294 s, 13.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f9:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:14:31 (1788837271) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg256-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139396 s, 15.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:15:25 (1788837325) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg256-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg256-server: Pool lustre.test_204f created Adding targets to pool oleg256-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg256-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115442 s, 18.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg256-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg256-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg256-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:16:39 (1788837399) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:16:47 (1788837407) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:16:56 (1788837416) oleg256-server: Pool lustre.test_206 created Adding targets to pool oleg256-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg256-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg256-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg256-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg256-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:17:26 (1788837446) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110963 s, 9.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025744 s, 40.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbe828a72 Mirror 3: 0xbe828a72 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:17:36 (1788837456) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:17:39 (1788837459) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:17:43 (1788837463) Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.12008 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x970000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x17c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x17c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x17c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1f70000 Read this (4 bytes): BBBB 192.168.202.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg256-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:18:20 (1788837500) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0135022 s, 607 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:18:26 (1788837506) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383597 s, 27.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611229 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:18:27.000000000 -0400 Modify: 2026-09-07 23:18:27.000000000 -0400 Change: 2026-09-07 23:18:28.000000000 -0400 Birth: 2026-09-07 23:18:26.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:18:34 (1788837514) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513461 s, 20.4 MB/s oleg256-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg256-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg256-server: O/280000401/d20/134: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:18:57 (1788837537) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.205426 s, 199 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0252103 s, 162 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 23:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 23:19 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:19:06 (1788837546) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:19:14 (1788837554) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2101 sec =========== 23:19:23 (1788837563) === sanity-flr: start cleanup 23:19:25 (1788837565) === === sanity-flr: finish cleanup 23:19:33 (1788837573) ===