-----============= acceptance-small: sanity-flr ============----- Wed Sep 16 04:01:26 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg154-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:01:43 (1789545703) === oleg154-client.virtnet: executing check_config_client /mnt/lustre oleg154-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg154-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8dc011ccc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8dc011ccc000.idle_timeout=debug disable quota as required oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:02:20 (1789545740) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8230] == sanity-flr test 0a: lfs mirror create with -N option == 04:02:22 (1789545742) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:02:37 (1789545757) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:02:39 (1789545759) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:02:42 (1789545762) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:03:05 (1789545785) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:03:08 (1789545788) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 04:03:11 (1789545791) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8dc011ccc000.stats=clear osc.lustre-OST0001-osc-ffff8dc011ccc000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:03:19 (1789545799) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:03:30 (1789545810) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:03:40 (1789545820) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:03:43 (1789545823) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:03:50 (1789545830) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 04:03:58 (1789545838) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:04:06 (1789545846) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:04:14 (1789545854) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:04:22 (1789545862) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:04:32 (1789545872) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:04:38 (1789545878) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.763192 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645581 s, 16.2 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:04:47 (1789545887) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.438012 s, 31.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:04:55 (1789545895) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:05:06 (1789545906) Stopping /mnt/lustre-ost1 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dc011ccc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dc011ccc000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg154-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg154-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dc011ccc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dc011ccc000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg154-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (125s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:07:11 (1789546031) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.263951 s, 27.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dc011ccc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dc011ccc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg154-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:07:45 (1789546065) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dc011ccc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dc011ccc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg154-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dc011ccc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dc011ccc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg154-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (57s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:08:42 (1789546122) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.85099 s, 28.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 16 04:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.674s user 0m0.000s sys 0m0.144s Stopping /mnt/lustre-ost1 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dc011ccc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dc011ccc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.827s user 0m0.000s sys 0m0.174s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg154-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg154-server oleg154-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dc011ccc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dc011ccc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.840s user 0m0.000s sys 0m0.160s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg154-server: oleg154-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg154-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dc011ccc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dc011ccc000.ost_server_uuid in FULL state after 0 sec PASS 33b (76s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:09:58 (1789546198) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:10:01 (1789546201) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:10:03 (1789546203) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:10:06 (1789546206) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 04:10:12 (1789546212) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.408886 s, 30.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:10:21 (1789546221) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034661 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028419 s, 36.9 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:10:27 (1789546227) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:10:33 (1789546233) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000794435 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000563855 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00201352 s, 509 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119786 s, 855 kB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 04:10:40 (1789546240) create test files with size 15752820 .. b19084015886dd2533edc3dac9f20f92 - df8c5c70f59c1bdf57af6d5139ac28b5 - 1a79793d11cb28c62cca1def97e22f14 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre PASS 37 (15s) == sanity-flr test 38: resync ============================ 04:10:56 (1789546256) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (10s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:11:06 (1789546266) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:11:10 (1789546270) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616172 s, 17.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198237 s, 52.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (6s) == sanity-flr test 41: lfs mirror resync check =========== 04:11:16 (1789546276) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0994974 s, 21.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.241095 s, 17.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0999945 s, 21.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (10s) == sanity-flr test 42: lfs mirror verify ================= 04:11:26 (1789546286) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 04:11:29 (1789546289) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:11:32 (1789546292) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196668 s, 53.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 04:11:37 (1789546297) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.195238 s, 16.1 MB/s PASS 44a (7s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:11:44 (1789546304) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:11:52 (1789546312) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.264801 s, 39.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:12:00 (1789546320) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8dc0102c4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.148889 s, 70.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg154-client.virtnet /mnt/lustre (opts:) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8dc0032b6000.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:12:10 (1789546330) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (31s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:12:41 (1789546361) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:13:12 (1789546392) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 04:13:14 (1789546394) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.197269 s, 15.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.117641 s, 26.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x23c273cc (nosync,stale) Mirror 2: 0x23c273cc (nosync,stale) ** verify mirror 2 doesn't change original checksum: a56d535c1b6fdc2865b1b39690b370c8 - mirror 1 checksum: c3a552e724ebc41a7de30012f1aad175 - mirror 2 checksum: a56d535c1b6fdc2865b1b39690b370c8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T04:13:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T04:13:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a56d535c1b6fdc2865b1b39690b370c8 - mirror 1 checksum: c3a552e724ebc41a7de30012f1aad175 - mirror 2 checksum: c3a552e724ebc41a7de30012f1aad175 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T04:13:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T04:13:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:13:23 (1789546403) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183296 s, 57.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (5s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:13:28 (1789546408) Failing mds1 on oleg154-server Stopping /mnt/lustre-mds1 (opts:) on oleg154-server 04:13:39 (1789546419) shut down facet: mds1 facet_host: oleg154-server facet_failover_host: oleg154-server Failover mds1 to oleg154-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 04:13:58 (1789546438) targets are mounted 04:13:58 (1789546438) facet_failover done oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214634 s, 48.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (43s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:14:11 (1789546451) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0366299 s, 559 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0308695 s, 663 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (7s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:14:18 (1789546458) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.028196 s, 726 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00618549 s, 3.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (6s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:14:24 (1789546464) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00463395 s, 3.5 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:14:29 (1789546469) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0574285 s, 357 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:14:34 (1789546474) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0074316 s, 551 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0163383 s, 251 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:14:38 (1789546478) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (13s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:14:51 (1789546491) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (23s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:15:14 (1789546514) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:15:34 (1789546534) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:15:55 (1789546555) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 04:15:59 (1789546559) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57541 mirror split pid 57542 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7851: 57541 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57542 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 04:17:04 (1789546624) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 6626 truncating to largest ever: 0xa5e2 000001 1789546626.226794 trunc from 00000000 to 0x00a5e1 (0xa5e2 bytes) 000003 1789546626.248365 trunc from 0x00882d to 0x00a5e1 (0x1db5 bytes) 000004 1789546626.259267 trunc from 0x004540 to 0x00a5e1 (0x60a2 bytes) 000005 1789546626.266043 mapread 0x0076ba thru 0x00a5e1 (0x2f28 bytes) 000006 1789546626.290076 read 0x003ca7 thru 0x0090c9 (0x5423 bytes) 000007 1789546626.290768 read 0x0068cc thru 0x00a5e1 (0x3d16 bytes) 000008 1789546626.291642 read 0x003be6 thru 0x007108 (0x3523 bytes) 000009 1789546626.292317 write 0x02c1a9 thru 0x02c78d (0x5e5 bytes) 000011 1789546626.294870 punch from 0x0091ca to 0x015c4b (0xca82 bytes) 000014 1789546626.300970 mapread 0x01b533 thru 0x02177b (0x6249 bytes) 000015 1789546626.309245 write 0x02b7d9 thru 0x032fec (0x7814 bytes) 000018 1789546626.312835 mapread 0x00a852 thru 0x018013 (0xd7c2 bytes) 000019 1789546626.314036 read 0x017557 thru 0x02330d (0xbdb7 bytes) 000020 1789546626.315372 read 0x0047b1 thru 0x008ee5 (0x4735 bytes) 000021 1789546626.317278 mapread 0x02d366 thru 0x032fec (0x5c87 bytes) 000022 1789546626.320017 read 0x028954 thru 0x029fd3 (0x1680 bytes) 000023 1789546626.320895 trunc from 0x022166 to 0x022e78 (0xd13 bytes) fallocating to largest ever: 0xfc52 000024 1789546626.324741 fallocate 0x006e82 thru 0x00fc51 (0x8dd0 bytes) 000025 1789546626.328975 mapread 0x00c98e thru 0x00d420 (0xa93 bytes) truncating to largest ever: 0x134de 000026 1789546626.330456 trunc from 0x032fed to 0x0134dd (0x1fb0f bytes) 000027 1789546626.344825 read 0x00d829 thru 0x0133c3 (0x5b9b bytes) 000028 1789546626.345993 read 0x01115b thru 0x0134dd (0x2383 bytes) fallocating to largest ever: 0x32616 000029 1789546626.348064 fallocate 0x024e7f thru 0x032615 (0xd797 bytes) 000030 1789546626.352110 read 0x011e46 thru 0x0134dd (0x1698 bytes) 000031 1789546626.356040 MIRROR_EXTEND to 2 mirrors 000032 1789546626.758507 trunc from 0x00631f to 0x009dd1 (0x3ab3 bytes) 000033 1789546626.795653 trunc from 0x0052d7 to 0x00c5d0 (0x72fa bytes) 000034 1789546626.801771 punch from 0x01333f to 0x0134dd (0x19f bytes) 000035 1789546626.805457 write 0x037185 thru 0x03bac0 (0x493c bytes) 000036 1789546626.812707 trunc from 0x03bac1 to 0x028b46 (0x12f7a bytes) 000037 1789546626.853066 mapwrite 0x01ece8 thru 0x029295 (0xa5ae bytes) 000038 1789546626.921045 write 0x03b180 thru 0x03ffff (0x4e80 bytes) 000039 1789546626.923306 punch from 0x032513 to 0x0390f2 (0x6be0 bytes) 000040 1789546626.930463 mapread 0x01f136 thru 0x029d49 (0xac14 bytes) 000041 1789546626.932528 read 0x00d81f thru 0x00ffc8 (0x27aa bytes) 000042 1789546626.947030 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) 000043 1789546627.118970 fallocate 0x01e163 thru 0x029807 (0xb6a5 bytes) 000044 1789546627.124086 punch from 0x022fdd to 0x032756 (0xf77a bytes) 000045 1789546627.128373 trunc from 0x040000 to 0x0180f4 (0x27f0b bytes) 000046 1789546627.144318 punch from 0x012dbb to 0x0180f4 (0x533a bytes) 000047 1789546627.148445 punch from 0x012f17 to 0x0180f4 (0x51de bytes) 000048 1789546627.152766 write 0x0057a1 thru 0x014668 (0xeec8 bytes) 000049 1789546627.166212 mapread 0x00bdd9 thru 0x0180f4 (0xc31c bytes) 000050 1789546627.177329 trunc from 0x0180f5 to 0x02827e (0x1018a bytes) 000051 1789546627.188157 write 0x036af8 thru 0x03e288 (0x7791 bytes) 000053 1789546627.189600 trunc from 0x03898b to 0x03a1b0 (0x1826 bytes) 000054 1789546627.203116 punch from 0x00625d to 0x00d272 (0x7016 bytes) 000055 1789546627.213951 MIRROR_EXTEND to 2 mirrors 000056 1789546627.706640 write 0x01ccd2 thru 0x0231ea (0x6519 bytes) 000057 1789546627.740154 punch from 0x01abb9 to 0x02111c (0x6564 bytes) 000059 1789546627.750212 mapwrite 0x01fe2d thru 0x026466 (0x663a bytes) 000060 1789546627.781779 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e289): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e289): Mirror 2: 0x2f124082 (stale) CRC-32 checksum value for chunk [0, 0x3e289): Mirror 1: 0x62101e58 Mirror 3: 0x62101e58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e289: skipped 000061 1789546628.422144 trunc from 0x03e289 to 0x022c07 (0x1b681 bytes) 000062 1789546628.457276 trunc from 0x00d2e9 to 0x018642 (0xb35a bytes) fallocating to largest ever: 0x3f965 000063 1789546628.462126 fallocate 0x0379c9 thru 0x03f964 (0x7f9c bytes) 000064 1789546628.465456 read 0x025a51 thru 0x02e09e (0x864e bytes) 000065 1789546628.471690 write 0x0142a6 thru 0x021923 (0xd67e bytes) 000066 1789546628.483339 trunc from 0x01aaef to 0x024f1a (0xa42c bytes) 000067 1789546628.503585 trunc from 0x03f965 to 0x01dafb (0x21e69 bytes) 000068 1789546628.536636 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dafc): Mirror 3: 0x3fed00e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dafc): Mirror 3: 0x3fed00e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dafc: skipped 000069 1789546628.785893 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1dafc): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dafc): Mirror 3: 0x3fed00e1 (stale) CRC-32 checksum value for chunk [0, 0x1dafc): Mirror 1: 0x4e35c6f8 Mirror 4: 0x4e35c6f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dafc: skipped 000070 1789546629.424076 write 0x035c92 thru 0x03add0 (0x513f bytes) 000071 1789546629.469317 trunc from 0x02c874 to 0x03add0 (0xe55d bytes) 000072 1789546629.482113 write 0x003676 thru 0x00a8a2 (0x722d bytes) 000073 1789546629.490144 mapwrite 0x01dde6 thru 0x0217b5 (0x39d0 bytes) 000074 1789546629.521393 mapread 0x032725 thru 0x035522 (0x2dfe bytes) 000075 1789546629.532184 mapwrite 0x032b18 thru 0x03beb5 (0x939e bytes) 000076 1789546629.573861 trunc from 0x00439b to 0x012f97 (0xebfd bytes) 000077 1789546629.577979 write 0x0226dc thru 0x02b664 (0x8f89 bytes) 000078 1789546629.579669 trunc from 0x02d383 to 0x031a8d (0x470b bytes) 000079 1789546629.583584 read 0x01af36 thru 0x01e8ba (0x3985 bytes) 000080 1789546629.584764 write 0x028604 thru 0x02b84b (0x3248 bytes) 000081 1789546629.585636 write 0x004936 thru 0x010dec (0xc4b7 bytes) 000083 1789546629.595008 mapwrite 0x01da94 thru 0x02cdf4 (0xf361 bytes) fallocating to largest ever: 0x40000 000084 1789546629.613307 fallocate 0x032267 thru 0x03ffff (0xdd99 bytes) 000085 1789546629.617442 mapwrite 0x03d373 thru 0x03f48e (0x211c bytes) 000086 1789546629.633351 punch from 0x035a84 to 0x03dd88 (0x8305 bytes) 000087 1789546629.637586 trunc from 0x017f79 to 0x01c2f2 (0x437a bytes) 000088 1789546629.641255 mapread 0x03ac0d thru 0x03ffff (0x53f3 bytes) 000089 1789546629.665510 trunc from 0x040000 to 0x01a1d2 (0x25e2d bytes) 000090 1789546629.681837 mapread 0x016a6e thru 0x01a1d2 (0x3765 bytes) 000091 1789546629.691181 punch from 0x00b83c to 0x014464 (0x8c29 bytes) 000092 1789546629.695251 mapwrite 0x00c49c thru 0x011058 (0x4bbd bytes) 000093 1789546629.719324 read 0x002d96 thru 0x008f43 (0x61ae bytes) 000094 1789546629.728223 read 0x013c33 thru 0x01a1d2 (0x65a0 bytes) 000095 1789546629.729237 write 0x000c00 thru 0x00fa90 (0xee91 bytes) 000096 1789546629.731811 mapread 0x01593a thru 0x01a1d2 (0x4899 bytes) 000097 1789546629.732540 read 0x0105f7 thru 0x01a1d2 (0x9bdc bytes) 000098 1789546629.734659 mapwrite 0x00452b thru 0x0054ee (0xfc4 bytes) 000099 1789546629.749447 punch from 0x007c20 to 0x00f756 (0x7b37 bytes) 000100 1789546629.754155 trunc from 0x012ae4 to 0x01a1d2 (0x76ef bytes) 000101 1789546629.761097 trunc from 0x01a1d3 to 0x034144 (0x19f72 bytes) 000102 1789546629.779057 read 0x015797 thru 0x020ec7 (0xb731 bytes) 000103 1789546629.791460 trunc from 0x03382e to 0x034144 (0x917 bytes) 000104 1789546629.800361 trunc from 0x017ac7 to 0x01c616 (0x4b50 bytes) 000105 1789546629.808631 mapread 0x031ac4 thru 0x034144 (0x2681 bytes) 000106 1789546629.821829 read 0x02c764 thru 0x034118 (0x79b5 bytes) 000107 1789546629.822601 mapread 0x02c511 thru 0x034144 (0x7c34 bytes) 000108 1789546629.824246 fallocate 0x006703 thru 0x0078e4 (0x11e2 bytes) 000109 1789546629.829991 trunc from 0x024052 to 0x033717 (0xf6c6 bytes) 000110 1789546629.836311 read 0x021478 thru 0x0256d6 (0x425f bytes) 000111 1789546629.846837 fallocate 0x0100bf thru 0x017b21 (0x7a63 bytes) 000112 1789546629.852936 trunc from 0x004774 to 0x00f9cd (0xb25a bytes) 000113 1789546629.859324 trunc from 0x034145 to 0x00459f (0x2fba5 bytes) 000115 1789546629.874809 write 0x0329ad thru 0x033a88 (0x10dc bytes) 000116 1789546629.875947 trunc from 0x025a3d to 0x02dd77 (0x833b bytes) 000117 1789546629.883954 fallocate 0x01d4bb thru 0x01ff66 (0x2aac bytes) 000118 1789546629.900166 read 0x02b004 thru 0x033a88 (0x8a85 bytes) 000119 1789546629.928256 write 0x025cdc thru 0x02b92f (0x5c54 bytes) 000120 1789546629.929859 mapread 0x02c069 thru 0x02dc74 (0x1c0c bytes) 000123 1789546629.937791 write 0x00d1b6 thru 0x011f28 (0x4d73 bytes) 000124 1789546629.939721 fallocate 0x0267ec thru 0x02cc74 (0x6489 bytes) 000127 1789546629.944847 write 0x024e7d thru 0x0335da (0xe75e bytes) 000128 1789546629.946180 punch from 0x003dc0 to 0x01029d (0xc4de bytes) 000129 1789546629.967280 write 0x010878 thru 0x017da5 (0x752e bytes) 000130 1789546629.971338 mapwrite 0x03d226 thru 0x03ffff (0x2dda bytes) 000131 1789546630.019256 trunc from 0x040000 to 0x01d315 (0x22cea bytes) 000132 1789546630.033335 punch from 0x001d08 to 0x002cb3 (0xfac bytes) 000133 1789546630.038309 trunc from 0x01d316 to 0x024836 (0x7521 bytes) 000134 1789546630.053820 fallocate 0x034858 thru 0x0352ab (0xa54 bytes) 000136 1789546630.058877 read 0x004ed6 thru 0x0052a6 (0x3d1 bytes) 000137 1789546630.086141 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24837): Mirror 3: 0x3eb58c91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24837): Mirror 3: 0x3eb58c91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24837: skipped 000138 1789546630.308892 mapread 0x00c3da thru 0x017ad4 (0xb6fb bytes) 000139 1789546630.341162 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24837): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24837): Mirror 3: 0x3eb58c91 (stale) CRC-32 checksum value for chunk [0, 0x24837): Mirror 1: 0x87b89e Mirror 4: 0x87b89e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24837: skipped 000140 1789546630.875137 mapread 0x011d11 thru 0x0125a3 (0x893 bytes) 000141 1789546630.892748 read 0x017c81 thru 0x01ff9d (0x831d bytes) 000142 1789546630.906126 mapwrite 0x0343cc thru 0x03c342 (0x7f77 bytes) 000143 1789546630.965561 write 0x00a141 thru 0x0160f9 (0xbfb9 bytes) 000144 1789546630.975234 write 0x01274d thru 0x0217fe (0xf0b2 bytes) 000145 1789546630.983253 trunc from 0x038179 to 0x03c342 (0x41ca bytes) 000146 1789546630.987404 trunc from 0x01201c to 0x0177df (0x57c4 bytes) 000147 1789546630.998803 fallocate 0x02a6ce thru 0x02efc0 (0x48f3 bytes) 000148 1789546631.002992 write 0x00e818 thru 0x011f88 (0x3771 bytes) 000149 1789546631.004305 trunc from 0x0012ed to 0x010b96 (0xf8aa bytes) 000150 1789546631.013924 read 0x01b65b thru 0x02ab4a (0xf4f0 bytes) 000151 1789546631.022986 fallocate 0x02e8f5 thru 0x02ffbb (0x16c7 bytes) 000152 1789546631.026968 mapwrite 0x00be43 thru 0x0197bd (0xd97b bytes) 000153 1789546631.049825 write 0x016a03 thru 0x0245cc (0xdbca bytes) 000154 1789546631.051114 write 0x036455 thru 0x03e811 (0x83bd bytes) 000155 1789546631.052256 write 0x006a2b thru 0x00ff3e (0x9514 bytes) 000156 1789546631.054250 read 0x037087 thru 0x03bc7e (0x4bf8 bytes) 000157 1789546631.055142 fallocate 0x01928d thru 0x01d6b3 (0x4427 bytes) 000158 1789546631.060693 mapwrite 0x01c5fe thru 0x023d44 (0x7747 bytes) 000159 1789546631.082651 fallocate 0x01ac19 thru 0x027414 (0xc7fc bytes) 000160 1789546631.088252 punch from 0x02e167 to 0x02f3fb (0x1295 bytes) 000161 1789546631.092200 fallocate 0x024ece thru 0x025b0a (0xc3d bytes) 000163 1789546631.095868 mapread 0x007233 thru 0x0082b6 (0x1084 bytes) 000164 1789546631.097019 trunc from 0x03e812 to 0x001157 (0x3d6ba bytes) 000167 1789546631.118347 write 0x02e35f thru 0x02f2fb (0xf9d bytes) 000168 1789546631.119424 write 0x015725 thru 0x01abd5 (0x54b1 bytes) 000169 1789546631.131035 MIRROR_RESYNC 3 mirrors 000170 1789546631.731873 mapwrite 0x0071fd thru 0x00c7e4 (0x55e8 bytes) 000171 1789546631.810968 fallocate 0x02d624 thru 0x0351fb (0x7bd8 bytes) 000172 1789546631.816046 punch from 0x0210a9 to 0x02e4ec (0xd444 bytes) 000174 1789546631.821158 fallocate 0x00a294 thru 0x00dd47 (0x3ab4 bytes) 000175 1789546631.825552 mapread 0x02e0fb thru 0x02f2fb (0x1201 bytes) 000176 1789546631.831679 mapread 0x00ff51 thru 0x0111a2 (0x1252 bytes) 000177 1789546631.838139 trunc from 0x0216c3 to 0x025a94 (0x43d2 bytes) 000178 1789546631.846810 MIRROR_RESYNC 3 mirrors 000179 1789546632.452203 punch from 0x027a19 to 0x02f2fb (0x78e3 bytes) 000180 1789546632.480768 write 0x01e409 thru 0x0258f1 (0x74e9 bytes) 000181 1789546632.491698 write 0x026e0a thru 0x0335e4 (0xc7db bytes) 000182 1789546632.505386 MIRROR_RESYNC 3 mirrors 000183 1789546633.189064 write 0x00bb5e thru 0x010dac (0x524f bytes) 000184 1789546633.223246 mapwrite 0x0273d3 thru 0x0309a5 (0x95d3 bytes) 000185 1789546633.273368 read 0x01d94e thru 0x02ccdb (0xf38e bytes) 000186 1789546633.282792 fallocate 0x0214d4 thru 0x02efcf (0xdafc bytes) 000187 1789546633.291338 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x335e5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335e5): Mirror 3: 0x379d5993 (stale) Mirror 4: 0x379d5993 (stale) CRC-32 checksum value for chunk [0, 0x335e5): Mirror 1: 0xbe7dabac Mirror 5: 0xbe7dabac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335e5: skipped 000188 1789546633.891108 fallocate 0x01a0cf thru 0x02695f (0xc891 bytes) 000189 1789546633.932454 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x335e5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335e5): Mirror 3: 0x379d5993 (stale) Mirror 4: 0x379d5993 (stale) Mirror 5: 0xbe7dabac (stale) CRC-32 checksum value for chunk [0, 0x335e5): Mirror 1: 0xbe7dabac Mirror 6: 0xbe7dabac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335e5: skipped 000190 1789546634.522276 mapwrite 0x037e93 thru 0x03b6e0 (0x384e bytes) 000191 1789546634.585234 read 0x02f6ac thru 0x03b6e0 (0xc035 bytes) 000193 1789546634.589678 mapwrite 0x018edc thru 0x024ca4 (0xbdc9 bytes) 000194 1789546634.630291 write 0x001dec thru 0x008883 (0x6a98 bytes) 000195 1789546634.639976 punch from 0x016f71 to 0x01d74e (0x67de bytes) 000196 1789546634.652852 MIRROR_RESYNC 5 mirrors 000197 1789546635.683014 fallocate 0x013ed6 thru 0x0204fc (0xc627 bytes) 000198 1789546635.712420 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b6e1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6e1): Mirror 3: 0x3920bbf (stale) Mirror 4: 0x3920bbf (stale) Mirror 5: 0x3920bbf (stale) Mirror 6: 0x3920bbf (stale) CRC-32 checksum value for chunk [0, 0x3b6e1): Mirror 1: 0x3920bbf Mirror 7: 0x3920bbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6e1: skipped 000199 1789546636.473276 read 0x00dec4 thru 0x01bd7b (0xdeb8 bytes) 000200 1789546636.486578 mapwrite 0x00d06a thru 0x019c89 (0xcc20 bytes) 000201 1789546636.609297 trunc from 0x0130de to 0x02171d (0xe640 bytes) 000202 1789546636.613998 punch from 0x0117a0 to 0x0173f2 (0x5c53 bytes) 000204 1789546636.620287 fallocate 0x02af09 thru 0x035599 (0xa691 bytes) 000205 1789546636.626956 write 0x03f61e thru 0x03ffff (0x9e2 bytes) 000206 1789546636.631850 fallocate 0x03824e thru 0x03ffff (0x7db2 bytes) 000208 1789546636.639773 trunc from 0x040000 to 0x0145f8 (0x2ba07 bytes) 000210 1789546636.671050 write 0x010977 thru 0x01ea96 (0xe120 bytes) 000211 1789546636.680017 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1ea97): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ea97): Mirror 3: 0x783a6969 (stale) Mirror 4: 0x783a6969 (stale) Mirror 5: 0x783a6969 (stale) Mirror 6: 0x783a6969 (stale) Mirror 7: 0x783a6969 (stale) CRC-32 checksum value for chunk [0, 0x1ea97): Mirror 1: 0xb2d56c46 Mirror 8: 0xb2d56c46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ea97: skipped 000212 1789546637.349547 punch from 0x01b471 to 0x01ddf9 (0x2989 bytes) 000213 1789546637.386876 mapwrite 0x014919 thru 0x021e23 (0xd50b bytes) 000214 1789546637.434695 read 0x01885d thru 0x01f1e8 (0x698c bytes) 000215 1789546637.437784 write 0x0101d7 thru 0x0114ef (0x1319 bytes) 000216 1789546637.439515 trunc from 0x01c5c9 to 0x01d198 (0xbd0 bytes) 000217 1789546637.450425 trunc from 0x021e24 to 0x03d9f9 (0x1bbd6 bytes) 000218 1789546637.465192 mapwrite 0x00284b thru 0x00799e (0x5154 bytes) 000219 1789546637.486192 mapread 0x00184d thru 0x005e82 (0x4636 bytes) 000220 1789546637.487509 write 0x02a7a0 thru 0x02f766 (0x4fc7 bytes) 000221 1789546637.496195 fallocate 0x020d06 thru 0x025078 (0x4373 bytes) 000222 1789546637.500940 trunc from 0x03d9fa to 0x024605 (0x193f4 bytes) 000223 1789546637.514915 fallocate 0x012375 thru 0x0147c6 (0x2452 bytes) 000224 1789546637.518928 punch from 0x00e0af to 0x019d14 (0xbc66 bytes) 000225 1789546637.522594 write 0x009ee2 thru 0x00fbe6 (0x5d05 bytes) 000227 1789546637.527755 punch from 0x0061c7 to 0x010e60 (0xac9a bytes) 000228 1789546637.541771 fallocate 0x030f9e thru 0x03a618 (0x967b bytes) 000229 1789546637.545854 punch from 0x038794 to 0x03a618 (0x1e85 bytes) 000230 1789546637.550412 trunc from 0x02bd4d to 0x02fb0c (0x3dc0 bytes) 000231 1789546637.558104 mapwrite 0x034ed7 thru 0x03729a (0x23c4 bytes) 000232 1789546637.584343 write 0x03bb01 thru 0x03dc21 (0x2121 bytes) 000233 1789546637.585178 trunc from 0x003d80 to 0x00ba60 (0x7ce1 bytes) 000234 1789546637.592065 read 0x00b4c9 thru 0x00e733 (0x326b bytes) 000235 1789546637.605220 trunc from 0x03047a to 0x031c98 (0x181f bytes) 000236 1789546637.615345 fallocate 0x0035f1 thru 0x00ede3 (0xb7f3 bytes) 000237 1789546637.621974 mapwrite 0x038d85 thru 0x03a866 (0x1ae2 bytes) 000238 1789546637.656235 MIRROR_RESYNC 7 mirrors 000239 1789546638.726769 trunc from 0x03dc22 to 0x001ee2 (0x3bd3f bytes) 000240 1789546638.762103 punch from 0x000107 to 0x001ee2 (0x1ddc bytes) 000241 1789546638.767154 read 0x001e66 thru 0x001ee2 (0x7d bytes) 000243 1789546638.771134 read 0x00038d thru 0x001ee2 (0x1b56 bytes) 000244 1789546638.774707 punch from 0x000e05 to 0x001ee2 (0x10de bytes) 000245 1789546638.778590 mapwrite 0x01d7ff thru 0x020d7b (0x357d bytes) 000246 1789546638.816885 trunc from 0x020d7c to 0x000917 (0x20464 bytes) 000247 1789546638.826085 trunc from 0x0008cd to 0x000917 (0x4b bytes) 000248 1789546638.831137 trunc from 0x000401 to 0x000917 (0x517 bytes) 000249 1789546638.835765 trunc from 0x0004a2 to 0x000917 (0x476 bytes) 000250 1789546638.838843 mapwrite 0x022238 thru 0x025503 (0x32cc bytes) 000251 1789546638.869981 mapwrite 0x02daa9 thru 0x0398c4 (0xbe1c bytes) 000252 1789546638.910969 read 0x01743e thru 0x02262e (0xb1f1 bytes) 000253 1789546638.918589 trunc from 0x0398c5 to 0x016643 (0x23281 bytes) 000254 1789546638.935850 read 0x013839 thru 0x016643 (0x2e0b bytes) 000255 1789546638.936737 trunc from 0x016644 to 0x02537c (0xed39 bytes) 000256 1789546638.947941 write 0x03ae14 thru 0x03ffff (0x51ec bytes) 000257 1789546638.949247 write 0x032c37 thru 0x03b2de (0x86a8 bytes) 000259 1789546638.959695 trunc from 0x040000 to 0x00623c (0x39dc3 bytes) 000260 1789546638.982259 MIRROR_RESYNC 7 mirrors 000261 1789546639.707861 trunc from 0x00085a to 0x00623c (0x59e3 bytes) 000262 1789546639.733405 mapwrite 0x03dfc3 thru 0x03ffff (0x203d bytes) 000263 1789546639.779775 trunc from 0x031d02 to 0x0337aa (0x1aa9 bytes) 000264 1789546639.782726 trunc from 0x039d63 to 0x03ffff (0x629d bytes) 000265 1789546639.786657 mapread 0x019e6b thru 0x0264e4 (0xc67a bytes) 000266 1789546639.803914 mapread 0x02c161 thru 0x03932f (0xd1cf bytes) 000267 1789546639.832119 write 0x022f03 thru 0x02f68d (0xc78b bytes) 000268 1789546639.833798 write 0x01b21e thru 0x01bfd6 (0xdb9 bytes) 000269 1789546639.834512 punch from 0x00af50 to 0x00c6bc (0x176d bytes) 000270 1789546639.837799 read 0x011b07 thru 0x01e383 (0xc87d bytes) 000271 1789546639.842432 trunc from 0x03bb37 to 0x03d271 (0x173b bytes) 000272 1789546639.847385 mapread 0x03e3b2 thru 0x03ffff (0x1c4e bytes) 000273 1789546639.854940 mapwrite 0x031c50 thru 0x03b7bb (0x9b6c bytes) 000274 1789546639.869717 write 0x026066 thru 0x02f4bc (0x9457 bytes) 000275 1789546639.874178 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9adf613 Mirror 9: 0x9adf613 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000276 1789546640.667879 trunc from 0x01dc62 to 0x022979 (0x4d18 bytes) 000277 1789546640.708044 punch from 0x02c59a to 0x02fc14 (0x367b bytes) 000278 1789546640.714785 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0x9adf613 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe61b25be Mirror 10: 0xe61b25be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000279 1789546641.535782 read 0x023b49 thru 0x02e8d1 (0xad89 bytes) 000281 1789546641.565001 trunc from 0x040000 to 0x034d3c (0xb2c3 bytes) 000282 1789546641.616393 trunc from 0x034d3d to 0x01f5c5 (0x15777 bytes) 000283 1789546641.643817 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1f5c6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f5c6): Mirror 3: 0xf3827a41 (stale) Mirror 4: 0xf3827a41 (stale) Mirror 5: 0xf3827a41 (stale) Mirror 6: 0xf3827a41 (stale) Mirror 7: 0xf3827a41 (stale) Mirror 8: 0xf3827a41 (stale) Mirror 9: 0x5a8d0202 (stale) Mirror 10: 0x5a8d0202 (stale) CRC-32 checksum value for chunk [0, 0x1f5c6): Mirror 1: 0x5a8d0202 Mirror 11: 0x5a8d0202 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f5c6: skipped 000284 1789546642.405412 write 0x017131 thru 0x022924 (0xb7f4 bytes) 000285 1789546642.442606 read 0x004603 thru 0x013121 (0xeb1f bytes) 000286 1789546642.451224 read 0x00b484 thru 0x0105e5 (0x5162 bytes) 000287 1789546642.456910 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22925): Mirror 3: 0x193ab6e (stale) Mirror 4: 0x193ab6e (stale) Mirror 5: 0x193ab6e (stale) Mirror 6: 0x193ab6e (stale) Mirror 7: 0x193ab6e (stale) Mirror 8: 0x193ab6e (stale) Mirror 9: 0x321cf4e2 (stale) Mirror 11: 0x321cf4e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22925): Mirror 3: 0x193ab6e (stale) Mirror 4: 0x193ab6e (stale) Mirror 5: 0x193ab6e (stale) Mirror 6: 0x193ab6e (stale) Mirror 7: 0x193ab6e (stale) Mirror 8: 0x193ab6e (stale) Mirror 9: 0x321cf4e2 (stale) Mirror 11: 0x321cf4e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22925: skipped 000288 1789546642.892948 read 0x001d95 thru 0x001ddd (0x49 bytes) 000289 1789546642.900139 fallocate 0x005af4 thru 0x012646 (0xcb53 bytes) 000290 1789546642.906353 fallocate 0x03a0b3 thru 0x03ddf0 (0x3d3e bytes) 000291 1789546642.915148 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22925): Mirror 3: 0x193ab6e (stale) Mirror 4: 0x193ab6e (stale) Mirror 6: 0x193ab6e (stale) Mirror 7: 0x193ab6e (stale) Mirror 8: 0x193ab6e (stale) Mirror 9: 0x321cf4e2 (stale) Mirror 11: 0x321cf4e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22925): Mirror 3: 0x193ab6e (stale) Mirror 4: 0x193ab6e (stale) Mirror 6: 0x193ab6e (stale) Mirror 7: 0x193ab6e (stale) Mirror 8: 0x193ab6e (stale) Mirror 9: 0x321cf4e2 (stale) Mirror 11: 0x321cf4e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22925: skipped 000292 1789546643.266602 punch from 0x000fc0 to 0x001a25 (0xa66 bytes) 000294 1789546643.274500 mapread 0x022195 thru 0x022924 (0x790 bytes) 000295 1789546643.278667 read 0x013e77 thru 0x01671a (0x28a4 bytes) 000296 1789546643.288272 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x22925): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22925): Mirror 3: 0x193ab6e (stale) Mirror 4: 0x193ab6e (stale) Mirror 6: 0x193ab6e (stale) Mirror 7: 0x193ab6e (stale) Mirror 8: 0x193ab6e (stale) Mirror 9: 0x321cf4e2 (stale) Mirror 11: 0x321cf4e2 (stale) CRC-32 checksum value for chunk [0, 0x22925): Mirror 1: 0xf32af323 Mirror 12: 0xf32af323 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22925: skipped 000297 1789546643.864758 read 0x01e10a thru 0x020567 (0x245e bytes) 000298 1789546643.874890 write 0x012590 thru 0x01a9a6 (0x8417 bytes) 000299 1789546643.914120 read 0x017613 thru 0x017eca (0x8b8 bytes) 000300 1789546643.919909 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x22925): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22925): Mirror 3: 0x193ab6e (stale) Mirror 4: 0x193ab6e (stale) Mirror 6: 0x193ab6e (stale) Mirror 7: 0x193ab6e (stale) Mirror 8: 0x193ab6e (stale) Mirror 9: 0x321cf4e2 (stale) Mirror 11: 0x321cf4e2 (stale) Mirror 12: 0xf32af323 (stale) CRC-32 checksum value for chunk [0, 0x22925): Mirror 1: 0xc87d7765 Mirror 13: 0xc87d7765 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22925: skipped 000301 1789546644.589575 trunc from 0x002a73 to 0x0056e9 (0x2c77 bytes) 000302 1789546644.632165 read 0x000f9c thru 0x008ae8 (0x7b4d bytes) 000303 1789546644.638724 read 0x000585 thru 0x004079 (0x3af5 bytes) 000304 1789546644.639198 mapread 0x014367 thru 0x0226e6 (0xe380 bytes) 000305 1789546644.644582 read 0x01dec9 thru 0x02121c (0x3354 bytes) 000306 1789546644.645108 trunc from 0x003108 to 0x007ec0 (0x4db9 bytes) 000307 1789546644.652441 trunc from 0x015ad3 to 0x0215f1 (0xbb1f bytes) 000308 1789546644.658265 mapwrite 0x03b684 thru 0x03ffff (0x497c bytes) 000309 1789546644.711137 mapwrite 0x030354 thru 0x03fd0a (0xf9b7 bytes) 000310 1789546644.728193 mapread 0x03ad81 thru 0x03ffff (0x527f bytes) 000311 1789546644.729209 mapwrite 0x004869 thru 0x006868 (0x2000 bytes) 000312 1789546644.738398 write 0x035c23 thru 0x03ffff (0xa3dd bytes) 000313 1789546644.739672 fallocate 0x01d145 thru 0x025d2f (0x8beb bytes) 000315 1789546644.744972 trunc from 0x040000 to 0x000cd6 (0x3f329 bytes) 000316 1789546644.761229 write 0x0143e6 thru 0x02001c (0xbc37 bytes) 000317 1789546644.763555 trunc from 0x004a5c to 0x007807 (0x2dac bytes) 000319 1789546644.769402 mapread 0x003658 thru 0x00da40 (0xa3e9 bytes) 000320 1789546644.812790 trunc from 0x02001d to 0x03ff2f (0x1ff13 bytes) 000321 1789546644.824157 trunc from 0x03ff30 to 0x01c053 (0x23edc bytes) 000322 1789546644.835250 read 0x01a2bf thru 0x01c053 (0x1d95 bytes) 000323 1789546644.835889 trunc from 0x00665c to 0x0124e3 (0xbe88 bytes) 000324 1789546644.840469 trunc from 0x00ff06 to 0x015666 (0x5761 bytes) 000327 1789546644.853355 fallocate 0x023af5 thru 0x02538a (0x1896 bytes) 000328 1789546644.856752 fallocate 0x028205 thru 0x035305 (0xd101 bytes) 000329 1789546644.859928 punch from 0x010a52 to 0x01a5c7 (0x9b76 bytes) 000330 1789546644.862861 read 0x0163ab thru 0x01c053 (0x5ca9 bytes) 000331 1789546644.868128 trunc from 0x01c054 to 0x01ba88 (0x5cb bytes) 000332 1789546644.875822 trunc from 0x01ba89 to 0x00739e (0x146ea bytes) 000333 1789546644.886233 punch from 0x0041d6 to 0x00739e (0x31c9 bytes) 000334 1789546644.891149 mapread 0x001f4c thru 0x00739e (0x5453 bytes) 000335 1789546644.903162 write 0x035626 thru 0x03ac40 (0x561b bytes) 000337 1789546644.905661 trunc from 0x01d999 to 0x0216bd (0x3d25 bytes) 000339 1789546644.908402 mapwrite 0x02548e thru 0x03387c (0xe3ef bytes) 000340 1789546644.934185 read 0x035e7d thru 0x03ac40 (0x4dc4 bytes) 000341 1789546644.935012 mapread 0x003f98 thru 0x00d3e8 (0x9451 bytes) 000342 1789546644.954319 trunc from 0x02342e to 0x024376 (0xf49 bytes) 000345 1789546644.965808 punch from 0x00e8b3 to 0x016836 (0x7f84 bytes) 000346 1789546644.972224 MIRROR_RESYNC 10 mirrors 000347 1789546646.292013 punch from 0x00b79e to 0x018951 (0xd1b4 bytes) 000348 1789546646.316449 mapwrite 0x030bd4 thru 0x03ffff (0xf42c bytes) 000349 1789546646.376738 MIRROR_RESYNC 10 mirrors 000350 1789546647.342456 trunc from 0x019433 to 0x0219db (0x85a9 bytes) 000352 1789546647.364340 read 0x01977c thru 0x01d5fa (0x3e7f bytes) 000353 1789546647.370207 punch from 0x00b4a4 to 0x00b545 (0xa2 bytes) 000354 1789546647.376824 MIRROR_RESYNC 10 mirrors 000355 1789546648.436281 trunc from 0x0356f0 to 0x035d91 (0x6a2 bytes) 000357 1789546648.469055 write 0x00e6c0 thru 0x01bd8e (0xd6cf bytes) 000358 1789546648.477858 mapwrite 0x016d79 thru 0x01e8bb (0x7b43 bytes) 000360 1789546648.513868 trunc from 0x021a6f to 0x02bdf0 (0xa382 bytes) 000361 1789546648.520915 mapread 0x0219af thru 0x025c0c (0x425e bytes) 000362 1789546648.532874 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa9712786 (stale) Mirror 4: 0xa9712786 (stale) Mirror 7: 0xa9712786 (stale) Mirror 8: 0xa9712786 (stale) Mirror 9: 0xa9712786 (stale) Mirror 11: 0xa9712786 (stale) Mirror 12: 0xa9712786 (stale) Mirror 13: 0xa9712786 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa9712786 (stale) Mirror 4: 0xa9712786 (stale) Mirror 7: 0xa9712786 (stale) Mirror 8: 0xa9712786 (stale) Mirror 9: 0xa9712786 (stale) Mirror 11: 0xa9712786 (stale) Mirror 12: 0xa9712786 (stale) Mirror 13: 0xa9712786 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000363 1789546648.983692 mapwrite 0x0302c5 thru 0x0325b5 (0x22f1 bytes) 000364 1789546649.029052 mapwrite 0x03b136 thru 0x03b332 (0x1fd bytes) 000365 1789546649.051663 read 0x009707 thru 0x015b82 (0xc47c bytes) 000366 1789546649.058257 trunc from 0x040000 to 0x017b1f (0x284e0 bytes) 000367 1789546649.075082 punch from 0x00e183 to 0x011a52 (0x38d0 bytes) 000368 1789546649.078373 trunc from 0x017b20 to 0x019329 (0x180a bytes) 000369 1789546649.087062 punch from 0x00ac4c to 0x00cd10 (0x20c5 bytes) 000370 1789546649.090916 write 0x029aae thru 0x0381ef (0xe742 bytes) 000372 1789546649.092978 mapwrite 0x01de8c thru 0x02cf59 (0xf0ce bytes) 000373 1789546649.119330 fallocate 0x0012d9 thru 0x010cad (0xf9d5 bytes) 000374 1789546649.124566 mapread 0x031728 thru 0x032deb (0x16c4 bytes) 000375 1789546649.125528 trunc from 0x02a852 to 0x0381ef (0xd99e bytes) 000376 1789546649.134076 MIRROR_RESYNC 9 mirrors 000377 1789546650.721574 punch from 0x02e20c to 0x0381ef (0x9fe4 bytes) 000378 1789546650.747300 mapwrite 0x00ccf0 thru 0x017b4c (0xae5d bytes) 000379 1789546650.799368 trunc from 0x00fa13 to 0x010a83 (0x1071 bytes) 000380 1789546650.805441 trunc from 0x007054 to 0x008d11 (0x1cbe bytes) 000381 1789546650.812353 punch from 0x011ee1 to 0x014eab (0x2fcb bytes) 000382 1789546650.818854 fallocate 0x01ecc7 thru 0x0289f9 (0x9d33 bytes) 000383 1789546650.824038 punch from 0x0193bd to 0x01b464 (0x20a8 bytes) 000385 1789546650.838324 fallocate 0x02e305 thru 0x036fb0 (0x8cac bytes) 000386 1789546650.846074 trunc from 0x0381f0 to 0x0326a8 (0x5b47 bytes) 000387 1789546650.873693 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x326a9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326a9): Mirror 3: 0xec65a232 (stale) Mirror 4: 0xec65a232 (stale) Mirror 7: 0xec65a232 (stale) Mirror 8: 0xec65a232 (stale) Mirror 9: 0xec65a232 (stale) Mirror 11: 0xec65a232 (stale) Mirror 12: 0xec65a232 (stale) Mirror 13: 0xec65a232 (stale) CRC-32 checksum value for chunk [0, 0x326a9): Mirror 1: 0x2e971496 Mirror 14: 0x2e971496 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326a9: skipped 000389 1789546651.689400 punch from 0x01e37b to 0x028413 (0xa099 bytes) 000392 1789546651.723950 punch from 0x01afc0 to 0x023c52 (0x8c93 bytes) 000393 1789546651.732799 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326a9): Mirror 3: 0xec65a232 (stale) Mirror 4: 0xec65a232 (stale) Mirror 7: 0xec65a232 (stale) Mirror 9: 0xec65a232 (stale) Mirror 11: 0xec65a232 (stale) Mirror 12: 0xec65a232 (stale) Mirror 13: 0xec65a232 (stale) Mirror 14: 0x2e971496 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326a9): Mirror 3: 0xec65a232 (stale) Mirror 4: 0xec65a232 (stale) Mirror 7: 0xec65a232 (stale) Mirror 9: 0xec65a232 (stale) Mirror 11: 0xec65a232 (stale) Mirror 12: 0xec65a232 (stale) Mirror 13: 0xec65a232 (stale) Mirror 14: 0x2e971496 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326a9: skipped 000394 1789546652.150414 mapwrite 0x0056ae thru 0x00ed9c (0x96ef bytes) 000395 1789546652.193841 mapwrite 0x0096cd thru 0x0114f7 (0x7e2b bytes) 000396 1789546652.215430 fallocate 0x010f62 thru 0x01c37e (0xb41d bytes) 000397 1789546652.218627 fallocate 0x032b32 thru 0x03a305 (0x77d4 bytes) 000399 1789546652.222993 read 0x006476 thru 0x0068a0 (0x42b bytes) 000400 1789546652.223457 mapwrite 0x021ef1 thru 0x022570 (0x680 bytes) 000401 1789546652.240533 mapwrite 0x01e17d thru 0x02e15d (0xffe1 bytes) 000402 1789546652.265911 trunc from 0x0326a9 to 0x03f7bd (0xd115 bytes) 000403 1789546652.287971 fallocate 0x00583c thru 0x0154ae (0xfc73 bytes) 000404 1789546652.293578 punch from 0x0274ca to 0x0308fe (0x9435 bytes) 000405 1789546652.299626 trunc from 0x02fc6a to 0x0367fb (0x6b92 bytes) 000406 1789546652.304281 punch from 0x00fcdb to 0x01ed52 (0xf078 bytes) 000408 1789546652.309286 punch from 0x005e64 to 0x010c27 (0xadc4 bytes) 000409 1789546652.312708 fallocate 0x03e94a thru 0x03ffff (0x16b6 bytes) 000410 1789546652.317872 mapwrite 0x037a60 thru 0x03ffff (0x85a0 bytes) 000411 1789546652.361286 mapwrite 0x036e60 thru 0x03ffff (0x91a0 bytes) 000413 1789546652.384277 mapread 0x01cc33 thru 0x02498a (0x7d58 bytes) 000414 1789546652.394678 mapwrite 0x023738 thru 0x0303d8 (0xcca1 bytes) 000415 1789546652.417628 read 0x01a104 thru 0x01ed37 (0x4c34 bytes) 000416 1789546652.421769 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x17d2d921 (stale) Mirror 4: 0x17d2d921 (stale) Mirror 9: 0x17d2d921 (stale) Mirror 11: 0x17d2d921 (stale) Mirror 12: 0x17d2d921 (stale) Mirror 13: 0x17d2d921 (stale) Mirror 14: 0x8b50e9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x17d2d921 (stale) Mirror 4: 0x17d2d921 (stale) Mirror 9: 0x17d2d921 (stale) Mirror 11: 0x17d2d921 (stale) Mirror 12: 0x17d2d921 (stale) Mirror 13: 0x17d2d921 (stale) Mirror 14: 0x8b50e9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000417 1789546652.839304 mapread 0x00f9d1 thru 0x016f77 (0x75a7 bytes) 000418 1789546652.869206 mapwrite 0x0129a6 thru 0x022572 (0xfbcd bytes) 000419 1789546652.904420 trunc from 0x013b7a to 0x017d7e (0x4205 bytes) 000420 1789546652.914664 MIRROR_RESYNC 8 mirrors 000421 1789546654.622256 write 0x0390cd thru 0x03f922 (0x6856 bytes) 000422 1789546654.652446 mapread 0x0214a1 thru 0x02b489 (0x9fe9 bytes) 000423 1789546654.664122 fallocate 0x0092a3 thru 0x00c795 (0x34f3 bytes) 000424 1789546654.666666 trunc from 0x006ffb to 0x00a6fe (0x3704 bytes) 000425 1789546654.671186 trunc from 0x03c85b to 0x03d782 (0xf28 bytes) 000426 1789546654.681356 mapread 0x03c470 thru 0x03ffff (0x3b90 bytes) 000427 1789546654.689808 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7ed48329 (stale) Mirror 9: 0x7ed48329 (stale) Mirror 11: 0x7ed48329 (stale) Mirror 12: 0x7ed48329 (stale) Mirror 13: 0x7ed48329 (stale) Mirror 14: 0x7ed48329 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7ed48329 (stale) Mirror 9: 0x7ed48329 (stale) Mirror 11: 0x7ed48329 (stale) Mirror 12: 0x7ed48329 (stale) Mirror 13: 0x7ed48329 (stale) Mirror 14: 0x7ed48329 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000428 1789546655.023925 trunc from 0x040000 to 0x02f52e (0x10ad1 bytes) 000429 1789546655.046457 punch from 0x02f3ca to 0x02f52e (0x165 bytes) 000430 1789546655.052648 mapread 0x01bd8d thru 0x028916 (0xcb8a bytes) 000431 1789546655.084408 write 0x0184ef thru 0x02807e (0xfb90 bytes) 000432 1789546655.085799 mapread 0x00de06 thru 0x0177d1 (0x99cc bytes) 000433 1789546655.100125 punch from 0x02da3b to 0x02f52e (0x1af4 bytes) 000434 1789546655.104917 mapwrite 0x0152fe thru 0x01ef65 (0x9c68 bytes) 000435 1789546655.132460 punch from 0x0289ec to 0x02f52e (0x6b43 bytes) 000436 1789546655.140940 write 0x005461 thru 0x00a03e (0x4bde bytes) 000437 1789546655.146474 mapread 0x008151 thru 0x00b440 (0x32f0 bytes) 000439 1789546655.149094 fallocate 0x02c35a thru 0x034b34 (0x87db bytes) 000440 1789546655.154230 write 0x039ef2 thru 0x03b495 (0x15a4 bytes) 000441 1789546655.155985 mapread 0x01e0bf thru 0x02bd61 (0xdca3 bytes) 000442 1789546655.167861 trunc from 0x03b496 to 0x012c6c (0x28829 bytes) 000443 1789546655.185041 punch from 0x00b434 to 0x012c6c (0x7839 bytes) 000445 1789546655.191283 trunc from 0x007425 to 0x00a957 (0x3533 bytes) 000446 1789546655.201516 trunc from 0x00e918 to 0x00fb48 (0x1231 bytes) 000447 1789546655.209911 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x12c6d): Mirror 4: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12c6d): Mirror 4: 0x62c2f097 (stale) Mirror 9: 0x62c2f097 (stale) Mirror 11: 0x62c2f097 (stale) Mirror 12: 0x62c2f097 (stale) Mirror 13: 0x62c2f097 (stale) Mirror 14: 0x62c2f097 (stale) CRC-32 checksum value for chunk [0, 0x12c6d): Mirror 1: 0x9ed2ce46 Mirror 15: 0x9ed2ce46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12c6d: skipped 000448 1789546655.784790 write 0x039ad9 thru 0x03ffff (0x6527 bytes) 000449 1789546655.811561 write 0x03a30b thru 0x03ffff (0x5cf5 bytes) 000450 1789546655.812456 trunc from 0x0301e8 to 0x031186 (0xf9f bytes) 000451 1789546655.821678 MIRROR_RESYNC 8 mirrors 000453 1789546657.426323 mapread 0x002852 thru 0x00649c (0x3c4b bytes) 000454 1789546657.440958 trunc from 0x040000 to 0x01a6da (0x25925 bytes) 000455 1789546657.474881 trunc from 0x01a6db to 0x035a0e (0x1b334 bytes) 000456 1789546657.503882 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35a0f): Mirror 4: 0xf1d8df0a (stale) Mirror 11: 0xf1d8df0a (stale) Mirror 12: 0xf1d8df0a (stale) Mirror 13: 0xf1d8df0a (stale) Mirror 14: 0xf1d8df0a (stale) Mirror 15: 0xf1d8df0a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a0f): Mirror 4: 0xf1d8df0a (stale) Mirror 11: 0xf1d8df0a (stale) Mirror 12: 0xf1d8df0a (stale) Mirror 13: 0xf1d8df0a (stale) Mirror 14: 0xf1d8df0a (stale) Mirror 15: 0xf1d8df0a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a0f: skipped 000457 1789546657.845255 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35a0f): Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a0f): Mirror 4: 0xf1d8df0a (stale) Mirror 11: 0xf1d8df0a (stale) Mirror 12: 0xf1d8df0a (stale) Mirror 13: 0xf1d8df0a (stale) Mirror 14: 0xf1d8df0a (stale) Mirror 15: 0xf1d8df0a (stale) CRC-32 checksum value for chunk [0, 0x35a0f): Mirror 1: 0xf1d8df0a Mirror 16: 0xf1d8df0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a0f: skipped 000458 1789546658.511098 fallocate 0x00a939 thru 0x00cd58 (0x2420 bytes) 000459 1789546658.546680 punch from 0x021b35 to 0x02e5ca (0xca96 bytes) 000461 1789546658.549451 mapwrite 0x00a8df thru 0x00ae0c (0x52e bytes) 000462 1789546658.561085 mapread 0x01ec93 thru 0x0257b5 (0x6b23 bytes) 000463 1789546658.568753 punch from 0x01ac6f to 0x01ae71 (0x203 bytes) 000464 1789546658.572860 punch from 0x0027b6 to 0x00902b (0x6876 bytes) 000465 1789546658.576296 trunc from 0x035a0f to 0x00c982 (0x2908c bytes) 000466 1789546658.593745 mapread 0x00bdf0 thru 0x00c982 (0xb93 bytes) 000467 1789546658.603769 trunc from 0x00c983 to 0x02420c (0x1788a bytes) 000468 1789546658.614190 trunc from 0x007254 to 0x00d21b (0x5fc8 bytes) 000469 1789546658.618168 mapwrite 0x03e683 thru 0x03ffff (0x197d bytes) 000470 1789546658.647889 fallocate 0x03ad16 thru 0x03ffff (0x52ea bytes) 000471 1789546658.651166 punch from 0x038d6b to 0x03ffff (0x7295 bytes) 000472 1789546658.653469 mapread 0x03145c thru 0x038246 (0x6deb bytes) 000473 1789546658.660875 write 0x01efdb thru 0x02e0bc (0xf0e2 bytes) 000474 1789546658.667321 write 0x01453c thru 0x014cfd (0x7c2 bytes) 000475 1789546658.670990 mapread 0x00c4f6 thru 0x018a31 (0xc53c bytes) 000476 1789546658.684742 punch from 0x026c9f to 0x0283a5 (0x1707 bytes) 000477 1789546658.697144 read 0x0231d7 thru 0x02e1b4 (0xafde bytes) 000478 1789546658.702233 read 0x028d26 thru 0x02e008 (0x52e3 bytes) 000479 1789546658.702866 read 0x004246 thru 0x006879 (0x2634 bytes) 000480 1789546658.704169 trunc from 0x015a7d to 0x021f77 (0xc4fb bytes) 000481 1789546658.708644 write 0x009457 thru 0x00dcf2 (0x489c bytes) 000482 1789546658.710568 mapwrite 0x02cc3f thru 0x03b908 (0xecca bytes) 000484 1789546658.748675 punch from 0x00fc28 to 0x014153 (0x452c bytes) 000485 1789546658.755335 mapwrite 0x019c82 thru 0x01bcc4 (0x2043 bytes) 000486 1789546658.771810 read 0x01c5d9 thru 0x02935b (0xcd83 bytes) 000487 1789546658.773396 write 0x01ef83 thru 0x02ab5c (0xbbda bytes) 000488 1789546658.775055 trunc from 0x040000 to 0x033710 (0xc8ef bytes) 000489 1789546658.790162 trunc from 0x033711 to 0x005778 (0x2df98 bytes) 000490 1789546658.811244 MIRROR_RESYNC 8 mirrors 000491 1789546659.658293 trunc from 0x001b14 to 0x005778 (0x3c65 bytes) 000492 1789546659.680980 mapwrite 0x006b1c thru 0x0090d9 (0x25be bytes) 000493 1789546659.718396 mapwrite 0x01c316 thru 0x0282b6 (0xbfa1 bytes) 000494 1789546659.785083 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x282b7): Mirror 4: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x282b7): Mirror 4: 0xffa155e (stale) Mirror 11: 0xffa155e (stale) Mirror 12: 0xffa155e (stale) Mirror 13: 0xffa155e (stale) Mirror 14: 0xffa155e (stale) Mirror 15: 0xffa155e (stale) Mirror 16: 0xffa155e (stale) CRC-32 checksum value for chunk [0, 0x282b7): Mirror 1: 0x76fe5a18 Mirror 17: 0x76fe5a18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x282b7: skipped 000495 1789546660.494043 trunc from 0x016305 to 0x018441 (0x213d bytes) 000496 1789546660.522540 read 0x0230da thru 0x0282b6 (0x51dd bytes) 000497 1789546660.528463 mapwrite 0x0328ed thru 0x037dcf (0x54e3 bytes) 000498 1789546660.586853 fallocate 0x011c2e thru 0x01507f (0x3452 bytes) 000499 1789546660.591332 mapread 0x00e0b1 thru 0x013c06 (0x5b56 bytes) 000500 1789546660.603939 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37dd0): Mirror 11: 0x55f4e459 (stale) Mirror 12: 0x55f4e459 (stale) Mirror 13: 0x55f4e459 (stale) Mirror 14: 0x55f4e459 (stale) Mirror 15: 0x55f4e459 (stale) Mirror 16: 0x55f4e459 (stale) Mirror 17: 0x52cfc80d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37dd0): Mirror 11: 0x55f4e459 (stale) Mirror 12: 0x55f4e459 (stale) Mirror 13: 0x55f4e459 (stale) Mirror 14: 0x55f4e459 (stale) Mirror 15: 0x55f4e459 (stale) Mirror 16: 0x55f4e459 (stale) Mirror 17: 0x52cfc80d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37dd0: skipped 000501 1789546661.039526 write 0x00cf24 thru 0x0181ae (0xb28b bytes) 000502 1789546661.047096 punch from 0x037359 to 0x037dcf (0xa77 bytes) 000503 1789546661.051320 trunc from 0x037dd0 to 0x007692 (0x3073d bytes) 000504 1789546661.064330 trunc from 0x007693 to 0x010dff (0x976d bytes) 000505 1789546661.075470 write 0x02b518 thru 0x035601 (0xa0ea bytes) 000506 1789546661.084822 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x35602): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35602): Mirror 11: 0x1475051c (stale) Mirror 12: 0x1475051c (stale) Mirror 13: 0x1475051c (stale) Mirror 14: 0x1475051c (stale) Mirror 15: 0x1475051c (stale) Mirror 16: 0x1475051c (stale) Mirror 17: 0x759a2c63 (stale) CRC-32 checksum value for chunk [0, 0x35602): Mirror 1: 0xca4a8a6a Mirror 18: 0xca4a8a6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35602: skipped 000507 1789546661.834424 read 0x02d04f thru 0x031879 (0x482b bytes) 000508 1789546661.846530 mapwrite 0x036e14 thru 0x03bf10 (0x50fd bytes) 000509 1789546661.923760 mapread 0x0146ac thru 0x01f690 (0xafe5 bytes) 000510 1789546661.940351 mapread 0x00166b thru 0x00ac77 (0x960d bytes) 000511 1789546661.948592 punch from 0x01e388 to 0x020389 (0x2002 bytes) 000512 1789546661.951432 fallocate 0x009ee4 thru 0x017bf5 (0xdd12 bytes) 000513 1789546661.955768 trunc from 0x03bf11 to 0x0291fe (0x12d12 bytes) 000514 1789546661.968363 punch from 0x015bed to 0x0227a6 (0xcbba bytes) 000515 1789546661.972322 read 0x0132ba thru 0x01f88d (0xc5d4 bytes) 000516 1789546661.987329 mapwrite 0x0234b9 thru 0x023b44 (0x68c bytes) 000518 1789546662.010429 fallocate 0x005e9b thru 0x009bb9 (0x3d1f bytes) 000519 1789546662.019848 fallocate 0x01450a thru 0x01d0c7 (0x8bbe bytes) 000520 1789546662.030895 write 0x016991 thru 0x026067 (0xf6d7 bytes) 000521 1789546662.032955 punch from 0x01e4cf to 0x02458c (0x60be bytes) 000522 1789546662.051513 write 0x004a66 thru 0x00ac6a (0x6205 bytes) 000523 1789546662.052576 fallocate 0x023b71 thru 0x0334e6 (0xf976 bytes) 000524 1789546662.058232 trunc from 0x0291ff to 0x02febe (0x6cc0 bytes) 000525 1789546662.078419 fallocate 0x03c720 thru 0x03e4fc (0x1ddd bytes) 000526 1789546662.084998 write 0x01c583 thru 0x02302a (0x6aa8 bytes) 000527 1789546662.102025 trunc from 0x03e4fd to 0x02e914 (0xfbe8 bytes) 000528 1789546662.112538 read 0x0225ff thru 0x02a066 (0x7a68 bytes) 000529 1789546662.119011 mapread 0x017694 thru 0x019539 (0x1ea6 bytes) 000530 1789546662.120176 fallocate 0x018e7a thru 0x0253c6 (0xc54d bytes) 000531 1789546662.124215 write 0x00166c thru 0x008825 (0x71ba bytes) 000532 1789546662.125971 fallocate 0x027958 thru 0x031d26 (0xa3cf bytes) 000533 1789546662.132746 trunc from 0x011452 to 0x016394 (0x4f43 bytes) 000534 1789546662.143684 MIRROR_RESYNC 9 mirrors 000535 1789546663.667475 mapwrite 0x0327a3 thru 0x03cd84 (0xa5e2 bytes) 000536 1789546663.739672 mapwrite 0x00d7cb thru 0x017da4 (0xa5da bytes) 000537 1789546663.770052 write 0x015d36 thru 0x021e68 (0xc133 bytes) 000538 1789546663.775842 mapread 0x022ca0 thru 0x02f2c2 (0xc623 bytes) 000539 1789546663.788641 punch from 0x00e757 to 0x00ff55 (0x17ff bytes) 000540 1789546663.793409 trunc from 0x029f9b to 0x036034 (0xc09a bytes) 000541 1789546663.798013 trunc from 0x0256cd to 0x025fe8 (0x91c bytes) 000542 1789546663.802724 trunc from 0x02a0f1 to 0x0362d8 (0xc1e8 bytes) 000543 1789546663.807514 mapwrite 0x01a233 thru 0x028e29 (0xebf7 bytes) 000544 1789546663.840747 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 12: 0x11c772d6 (stale) Mirror 13: 0x11c772d6 (stale) Mirror 14: 0x11c772d6 (stale) Mirror 15: 0x11c772d6 (stale) Mirror 16: 0x11c772d6 (stale) Mirror 17: 0x11c772d6 (stale) Mirror 18: 0x11c772d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 12: 0x11c772d6 (stale) Mirror 13: 0x11c772d6 (stale) Mirror 14: 0x11c772d6 (stale) Mirror 15: 0x11c772d6 (stale) Mirror 16: 0x11c772d6 (stale) Mirror 17: 0x11c772d6 (stale) Mirror 18: 0x11c772d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd85: skipped 000545 1789546664.193317 mapwrite 0x0091bb thru 0x009c13 (0xa59 bytes) 000547 1789546664.206165 fallocate 0x00ce97 thru 0x00df6a (0x10d4 bytes) 000548 1789546664.210334 mapread 0x014125 thru 0x021694 (0xd570 bytes) 000549 1789546664.225243 read 0x029efa thru 0x02d42d (0x3534 bytes) 000550 1789546664.233226 fallocate 0x02a1ed thru 0x03a142 (0xff56 bytes) 000551 1789546664.238216 read 0x009a86 thru 0x00c30b (0x2886 bytes) 000552 1789546664.246954 read 0x03ab37 thru 0x03cd84 (0x224e bytes) 000553 1789546664.247617 read 0x02ea6e thru 0x0359e8 (0x6f7b bytes) 000554 1789546664.251782 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 12: 0x11c772d6 (stale) Mirror 13: 0x11c772d6 (stale) Mirror 14: 0x11c772d6 (stale) Mirror 15: 0x11c772d6 (stale) Mirror 16: 0x11c772d6 (stale) Mirror 17: 0x11c772d6 (stale) Mirror 18: 0x11c772d6 (stale) CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 1: 0xec072410 Mirror 19: 0xec072410 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd85: skipped 000556 1789546665.039253 fallocate 0x02530d thru 0x0310cb (0xbdbf bytes) 000557 1789546665.069475 MIRROR_RESYNC 9 mirrors 000558 1789546666.272673 punch from 0x013459 to 0x0225dc (0xf184 bytes) 000559 1789546666.305677 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 12: 0xec072410 (stale) Mirror 13: 0xec072410 (stale) Mirror 14: 0xec072410 (stale) Mirror 15: 0xec072410 (stale) Mirror 16: 0xec072410 (stale) Mirror 17: 0xec072410 (stale) Mirror 18: 0xec072410 (stale) Mirror 19: 0xec072410 (stale) CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 1: 0xba54c3f7 Mirror 20: 0xba54c3f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd85: skipped 000560 1789546667.092452 fallocate 0x03c9b0 thru 0x03ffff (0x3650 bytes) 000561 1789546667.147592 MIRROR_RESYNC 10 mirrors 000562 1789546668.865364 punch from 0x01b8eb to 0x029b5c (0xe272 bytes) 000564 1789546668.894733 mapread 0x02f0cb thru 0x03c0d5 (0xd00b bytes) 000565 1789546668.925318 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 12: 0xba54c3f7 (stale) Mirror 13: 0xba54c3f7 (stale) Mirror 14: 0xba54c3f7 (stale) Mirror 15: 0xba54c3f7 (stale) Mirror 16: 0xba54c3f7 (stale) Mirror 17: 0xba54c3f7 (stale) Mirror 18: 0xba54c3f7 (stale) Mirror 19: 0xba54c3f7 (stale) Mirror 20: 0xba54c3f7 (stale) CRC-32 checksum value for chunk [0, 0x3cd85): Mirror 1: 0x19a8054a Mirror 21: 0x19a8054a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd85: skipped 000566 1789546669.788997 mapwrite 0x02d9be thru 0x03d88a (0xfecd bytes) 000568 1789546669.839047 trunc from 0x023f92 to 0x031790 (0xd7ff bytes) 000569 1789546669.842279 trunc from 0x00a7b8 to 0x00ef8b (0x47d4 bytes) 000570 1789546669.844955 fallocate 0x035928 thru 0x03cf3f (0x7618 bytes) 000571 1789546669.847438 mapread 0x006aa7 thru 0x010eb8 (0xa412 bytes) 000572 1789546669.852114 punch from 0x03c47f to 0x03d88a (0x140c bytes) 000573 1789546669.854441 trunc from 0x03d88b to 0x03a21d (0x366d bytes) 000574 1789546669.865665 trunc from 0x0220ec to 0x023ca3 (0x1bb8 bytes) 000575 1789546669.873169 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3a21e): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a21e): Mirror 12: 0x1557721 (stale) Mirror 13: 0x1557721 (stale) Mirror 14: 0x1557721 (stale) Mirror 15: 0x1557721 (stale) Mirror 16: 0x1557721 (stale) Mirror 17: 0x1557721 (stale) Mirror 18: 0x1557721 (stale) Mirror 19: 0x1557721 (stale) Mirror 20: 0x1557721 (stale) Mirror 21: 0x75cb33cf (stale) CRC-32 checksum value for chunk [0, 0x3a21e): Mirror 1: 0xffd8053f Mirror 22: 0xffd8053f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a21e: skipped 000576 1789546670.617027 write 0x01bace thru 0x025441 (0x9974 bytes) 000577 1789546670.661410 trunc from 0x01abe7 to 0x0255d2 (0xa9ec bytes) 000578 1789546670.669110 punch from 0x03818e to 0x03a21d (0x2090 bytes) 000579 1789546670.673038 punch from 0x00f6bd to 0x010643 (0xf87 bytes) 000580 1789546670.678401 read 0x021f50 thru 0x02d27f (0xb330 bytes) 000581 1789546670.695707 fallocate 0x00dae9 thru 0x011db3 (0x42cb bytes) 000582 1789546670.701781 read 0x01e783 thru 0x02316c (0x49ea bytes) 000583 1789546670.711472 read 0x00333c thru 0x00ebcd (0xb892 bytes) 000584 1789546670.712429 read 0x037150 thru 0x03a21d (0x30ce bytes) 000585 1789546670.715594 mapwrite 0x002b35 thru 0x006b4a (0x4016 bytes) 000588 1789546670.730265 read 0x021a83 thru 0x02d22e (0xb7ac bytes) 000589 1789546670.731277 trunc from 0x03a21e to 0x033c3c (0x65e1 bytes) 000592 1789546670.752341 fallocate 0x03889e thru 0x03a768 (0x1ecb bytes) 000593 1789546670.760393 fallocate 0x00f023 thru 0x01deb9 (0xee97 bytes) 000594 1789546670.773113 punch from 0x0188f7 to 0x01b6d0 (0x2dda bytes) 000595 1789546670.779447 mapread 0x029d02 thru 0x03275e (0x8a5d bytes) 000596 1789546670.786631 mapwrite 0x01602b thru 0x01ace0 (0x4cb6 bytes) 000597 1789546670.825430 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33c3d): Mirror 13: 0x701b7183 (stale) Mirror 14: 0x701b7183 (stale) Mirror 15: 0x701b7183 (stale) Mirror 16: 0x701b7183 (stale) Mirror 17: 0x701b7183 (stale) Mirror 18: 0x701b7183 (stale) Mirror 19: 0x701b7183 (stale) Mirror 20: 0x701b7183 (stale) Mirror 21: 0xf97f19a8 (stale) Mirror 22: 0x99636133 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c3d): Mirror 13: 0x701b7183 (stale) Mirror 14: 0x701b7183 (stale) Mirror 15: 0x701b7183 (stale) Mirror 16: 0x701b7183 (stale) Mirror 17: 0x701b7183 (stale) Mirror 18: 0x701b7183 (stale) Mirror 19: 0x701b7183 (stale) Mirror 20: 0x701b7183 (stale) Mirror 21: 0xf97f19a8 (stale) Mirror 22: 0x99636133 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c3d: skipped 000598 1789546671.264279 trunc from 0x02fed6 to 0x033c3c (0x3d67 bytes) 000599 1789546671.268856 trunc from 0x033c3d to 0x026e12 (0xce2a bytes) 000600 1789546671.279577 trunc from 0x026e13 to 0x037e03 (0x10ff1 bytes) 000601 1789546671.287929 mapwrite 0x03b657 thru 0x03cf8c (0x1936 bytes) 000603 1789546671.315207 write 0x00f6e8 thru 0x01d8cc (0xe1e5 bytes) 000604 1789546671.325039 read 0x007f89 thru 0x009642 (0x16ba bytes) 000605 1789546671.328950 punch from 0x00bfee to 0x01b1fd (0xf210 bytes) 000606 1789546671.343176 mapread 0x0341cb thru 0x03cf8c (0x8dc2 bytes) 000607 1789546671.355842 write 0x017cd1 thru 0x01e4eb (0x681b bytes) 000610 1789546671.370660 mapwrite 0x0205e6 thru 0x0252c8 (0x4ce3 bytes) 000611 1789546671.403150 trunc from 0x03cf8d to 0x0062c5 (0x36cc7 bytes) 000612 1789546671.414217 mapwrite 0x009ecf thru 0x016fb8 (0xd0ea bytes) 000613 1789546671.478458 read 0x002f73 thru 0x00cfb8 (0xa046 bytes) 000614 1789546671.479129 mapread 0x0067be thru 0x010a2c (0xa26f bytes) 000615 1789546671.481051 fallocate 0x03a5b8 thru 0x03ffff (0x5a48 bytes) 000616 1789546671.485789 punch from 0x0166e7 to 0x016fb8 (0x8d2 bytes) 000617 1789546671.489097 trunc from 0x016fb9 to 0x00fea6 (0x7112 bytes) 000618 1789546671.499525 fallocate 0x02125c thru 0x02ed02 (0xdaa7 bytes) 000619 1789546671.506440 MIRROR_RESYNC 11 mirrors 000620 1789546672.974363 read 0x024011 thru 0x02b211 (0x7201 bytes) 000621 1789546672.979954 write 0x01c373 thru 0x02b944 (0xf5d2 bytes) 000622 1789546673.023149 trunc from 0x029c5e to 0x02ed02 (0x50a5 bytes) 000623 1789546673.037946 read 0x0167dd thru 0x01a4aa (0x3cce bytes) 000624 1789546673.048227 fallocate 0x039b96 thru 0x03ffff (0x646a bytes) 000625 1789546673.051818 write 0x025cee thru 0x033609 (0xd91c bytes) 000626 1789546673.053564 fallocate 0x022c8b thru 0x02340b (0x781 bytes) 000628 1789546673.064123 MIRROR_RESYNC 11 mirrors 000629 1789546674.586218 trunc from 0x03360a to 0x01799a (0x1bc6f bytes) 000630 1789546674.617078 read 0x003099 thru 0x003c71 (0xbd9 bytes) 000631 1789546674.621039 trunc from 0x01799b to 0x018cb3 (0x1319 bytes) 000632 1789546674.635053 trunc from 0x00f987 to 0x0150ad (0x5727 bytes) 000633 1789546674.639218 punch from 0x0011c7 to 0x00746e (0x62a8 bytes) 000634 1789546674.642808 punch from 0x00d3e5 to 0x00d9c1 (0x5dd bytes) 000635 1789546674.646980 trunc from 0x01873b to 0x018cb3 (0x579 bytes) 000636 1789546674.652371 read 0x018378 thru 0x018cb3 (0x93c bytes) 000637 1789546674.662096 trunc from 0x017978 to 0x018cb3 (0x133c bytes) 000638 1789546674.666167 fallocate 0x01560b thru 0x021cea (0xc6e0 bytes) 000639 1789546674.670267 trunc from 0x0075fb to 0x014e87 (0xd88d bytes) 000641 1789546674.679389 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21ceb): Mirror 13: 0xc0b611c9 (stale) Mirror 14: 0xc0b611c9 (stale) Mirror 15: 0xc0b611c9 (stale) Mirror 16: 0xc0b611c9 (stale) Mirror 17: 0xc0b611c9 (stale) Mirror 18: 0xc0b611c9 (stale) Mirror 19: 0xc0b611c9 (stale) Mirror 21: 0xc0b611c9 (stale) Mirror 22: 0xc0b611c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ceb): Mirror 13: 0xc0b611c9 (stale) Mirror 14: 0xc0b611c9 (stale) Mirror 15: 0xc0b611c9 (stale) Mirror 16: 0xc0b611c9 (stale) Mirror 17: 0xc0b611c9 (stale) Mirror 18: 0xc0b611c9 (stale) Mirror 19: 0xc0b611c9 (stale) Mirror 21: 0xc0b611c9 (stale) Mirror 22: 0xc0b611c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ceb: skipped 000642 1789546675.110675 mapwrite 0x03d8e4 thru 0x03ffff (0x271c bytes) 000643 1789546675.170463 fallocate 0x0197e0 thru 0x023244 (0x9a65 bytes) 000646 1789546675.180824 read 0x032d1f thru 0x032efa (0x1dc bytes) 000647 1789546675.188888 fallocate 0x02d53c thru 0x031f3a (0x49ff bytes) 000648 1789546675.196620 write 0x030835 thru 0x03aa05 (0xa1d1 bytes) 000649 1789546675.206905 mapwrite 0x0222f3 thru 0x0257fa (0x3508 bytes) 000650 1789546675.247363 write 0x033fcc thru 0x03cb5b (0x8b90 bytes) 000651 1789546675.249209 trunc from 0x040000 to 0x0373d5 (0x8c2a bytes) 000653 1789546675.266582 read 0x00ba25 thru 0x01a534 (0xeb10 bytes) 000654 1789546675.278607 mapread 0x02fb72 thru 0x0373d5 (0x7864 bytes) 000655 1789546675.280163 write 0x00504c thru 0x00dcf6 (0x8cab bytes) 000656 1789546675.282316 mapread 0x001bbd thru 0x002244 (0x688 bytes) 000660 1789546675.288129 trunc from 0x0373d6 to 0x037cbb (0x8e6 bytes) 000662 1789546675.305174 trunc from 0x003518 to 0x00e727 (0xb210 bytes) 000663 1789546675.324848 write 0x03462f thru 0x036af7 (0x24c9 bytes) 000664 1789546675.326144 mapread 0x0204ce thru 0x02cd94 (0xc8c7 bytes) 000665 1789546675.332229 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x37cbc): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37cbc): Mirror 13: 0x88665846 (stale) Mirror 14: 0x88665846 (stale) Mirror 15: 0x88665846 (stale) Mirror 16: 0x88665846 (stale) Mirror 17: 0x88665846 (stale) Mirror 18: 0x88665846 (stale) Mirror 19: 0x88665846 (stale) Mirror 21: 0x88665846 (stale) Mirror 22: 0x88665846 (stale) CRC-32 checksum value for chunk [0, 0x37cbc): Mirror 1: 0xe2e7fe23 Mirror 23: 0xe2e7fe23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37cbc: skipped 000666 1789546676.303294 mapwrite 0x02003d thru 0x02b34d (0xb311 bytes) 000667 1789546676.388049 write 0x003fad thru 0x00f606 (0xb65a bytes) 000668 1789546676.416240 mapwrite 0x029cca thru 0x032b00 (0x8e37 bytes) 000669 1789546676.457968 MIRROR_RESYNC 11 mirrors 000671 1789546678.235120 trunc from 0x01ed0b to 0x029d7d (0xb073 bytes) 000672 1789546678.265445 fallocate 0x025d65 thru 0x02ded8 (0x8174 bytes) 000673 1789546678.276320 MIRROR_RESYNC 11 mirrors 000674 1789546679.948048 trunc from 0x02155c to 0x02863b (0x70e0 bytes) 000675 1789546679.979353 trunc from 0x01d05b to 0x020acc (0x3a72 bytes) 000676 1789546679.983682 read 0x021140 thru 0x02ccaa (0xbb6b bytes) 000677 1789546679.993861 punch from 0x02771d to 0x02d92c (0x6210 bytes) 000678 1789546680.002160 punch from 0x008f87 to 0x0106e8 (0x7762 bytes) 000679 1789546680.017247 trunc from 0x025c21 to 0x02bceb (0x60cb bytes) 000681 1789546680.024664 trunc from 0x02d258 to 0x02f63b (0x23e4 bytes) 000682 1789546680.030002 write 0x0031cd thru 0x010edc (0xdd10 bytes) 000683 1789546680.042522 punch from 0x00b74f to 0x00bfae (0x860 bytes) 000684 1789546680.061049 punch from 0x03256d to 0x037cbb (0x574f bytes) 000685 1789546680.066435 read 0x00d8cd thru 0x017c53 (0xa387 bytes) 000686 1789546680.078914 punch from 0x00fca8 to 0x01f0f7 (0xf450 bytes) 000688 1789546680.082930 fallocate 0x0298ab thru 0x02ee67 (0x55bd bytes) 000689 1789546680.091000 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x37cbc): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37cbc): Mirror 13: 0x7a7b6a99 (stale) Mirror 14: 0x7a7b6a99 (stale) Mirror 15: 0x7a7b6a99 (stale) Mirror 16: 0x7a7b6a99 (stale) Mirror 17: 0x7a7b6a99 (stale) Mirror 18: 0x7a7b6a99 (stale) Mirror 19: 0x7a7b6a99 (stale) Mirror 21: 0x7a7b6a99 (stale) Mirror 22: 0x7a7b6a99 (stale) Mirror 23: 0x7a7b6a99 (stale) CRC-32 checksum value for chunk [0, 0x37cbc): Mirror 1: 0xa323bfe4 Mirror 24: 0xa323bfe4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37cbc: skipped 000690 1789546680.849949 punch from 0x0251f3 to 0x033eff (0xed0d bytes) 000691 1789546680.901936 mapwrite 0x036a3c thru 0x03ffff (0x95c4 bytes) 000692 1789546680.995758 trunc from 0x040000 to 0x009832 (0x367cd bytes) 000693 1789546681.009041 trunc from 0x008a58 to 0x009832 (0xddb bytes) 000694 1789546681.018007 MIRROR_RESYNC 12 mirrors 000695 1789546682.303101 trunc from 0x009833 to 0x00b9f1 (0x21bf bytes) 000696 1789546682.350940 fallocate 0x02f0d3 thru 0x03c328 (0xd256 bytes) 000697 1789546682.355053 punch from 0x031c71 to 0x03577c (0x3b0c bytes) 000698 1789546682.358344 mapread 0x02742c thru 0x0313a4 (0x9f79 bytes) 000699 1789546682.383650 read 0x029794 thru 0x02e046 (0x48b3 bytes) 000700 1789546682.384226 mapwrite 0x00c50c thru 0x01675d (0xa252 bytes) 000701 1789546682.423886 mapwrite 0x039edd thru 0x03ffff (0x6123 bytes) 000702 1789546682.466924 punch from 0x02e3e0 to 0x02fbc5 (0x17e6 bytes) 000703 1789546682.471002 punch from 0x003574 to 0x00ded1 (0xa95e bytes) 000704 1789546682.479880 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x1e0d7459 (stale) Mirror 15: 0x1e0d7459 (stale) Mirror 16: 0x1e0d7459 (stale) Mirror 17: 0x1e0d7459 (stale) Mirror 18: 0x1e0d7459 (stale) Mirror 19: 0x1e0d7459 (stale) Mirror 21: 0x1e0d7459 (stale) Mirror 22: 0x1e0d7459 (stale) Mirror 23: 0x1e0d7459 (stale) Mirror 24: 0x1e0d7459 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x1e0d7459 (stale) Mirror 15: 0x1e0d7459 (stale) Mirror 16: 0x1e0d7459 (stale) Mirror 17: 0x1e0d7459 (stale) Mirror 18: 0x1e0d7459 (stale) Mirror 19: 0x1e0d7459 (stale) Mirror 21: 0x1e0d7459 (stale) Mirror 22: 0x1e0d7459 (stale) Mirror 23: 0x1e0d7459 (stale) Mirror 24: 0x1e0d7459 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000705 1789546682.991656 mapwrite 0x027027 thru 0x030bcf (0x9ba9 bytes) 000706 1789546683.023061 fallocate 0x01bf9a thru 0x01d7fe (0x1865 bytes) 000707 1789546683.033855 trunc from 0x032219 to 0x032397 (0x17f bytes) 000708 1789546683.037597 trunc from 0x040000 to 0x00f440 (0x30bbf bytes) 000709 1789546683.056635 mapread 0x0031f5 thru 0x006386 (0x3192 bytes) 000711 1789546683.074739 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf441): Mirror 14: 0x7ecba064 (stale) Mirror 15: 0x7ecba064 (stale) Mirror 17: 0x7ecba064 (stale) Mirror 18: 0x7ecba064 (stale) Mirror 19: 0x7ecba064 (stale) Mirror 21: 0x7ecba064 (stale) Mirror 22: 0x7ecba064 (stale) Mirror 23: 0x7ecba064 (stale) Mirror 24: 0x7ecba064 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf441): Mirror 14: 0x7ecba064 (stale) Mirror 15: 0x7ecba064 (stale) Mirror 17: 0x7ecba064 (stale) Mirror 18: 0x7ecba064 (stale) Mirror 19: 0x7ecba064 (stale) Mirror 21: 0x7ecba064 (stale) Mirror 22: 0x7ecba064 (stale) Mirror 23: 0x7ecba064 (stale) Mirror 24: 0x7ecba064 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf441: skipped 000712 1789546683.499529 write 0x024b08 thru 0x02ba73 (0x6f6c bytes) 000713 1789546683.504676 mapread 0x0146ba thru 0x016c09 (0x2550 bytes) 000714 1789546683.530044 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ba74): Mirror 15: 0x2eed99a3 (stale) Mirror 17: 0x2eed99a3 (stale) Mirror 18: 0x2eed99a3 (stale) Mirror 19: 0x2eed99a3 (stale) Mirror 21: 0x2eed99a3 (stale) Mirror 22: 0x2eed99a3 (stale) Mirror 23: 0x2eed99a3 (stale) Mirror 24: 0x2eed99a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba74): Mirror 15: 0x2eed99a3 (stale) Mirror 17: 0x2eed99a3 (stale) Mirror 18: 0x2eed99a3 (stale) Mirror 19: 0x2eed99a3 (stale) Mirror 21: 0x2eed99a3 (stale) Mirror 22: 0x2eed99a3 (stale) Mirror 23: 0x2eed99a3 (stale) Mirror 24: 0x2eed99a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba74: skipped 000715 1789546684.026174 punch from 0x004c1d to 0x00f44f (0xa833 bytes) 000716 1789546684.032723 mapread 0x007050 thru 0x00ead8 (0x7a89 bytes) 000717 1789546684.049798 trunc from 0x02ba74 to 0x0281ce (0x38a5 bytes) 000718 1789546684.068445 mapwrite 0x025ec7 thru 0x02a24a (0x4384 bytes) 000719 1789546684.119271 punch from 0x00a281 to 0x00f2e7 (0x5067 bytes) 000720 1789546684.126740 read 0x00c9a9 thru 0x01968b (0xcce3 bytes) 000721 1789546684.143881 mapwrite 0x004ae0 thru 0x0110ea (0xc60b bytes) 000722 1789546684.172730 mapwrite 0x038777 thru 0x03ffff (0x7889 bytes) 000723 1789546684.261279 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x1e0d7459 (stale) Mirror 17: 0x1e0d7459 (stale) Mirror 18: 0x1e0d7459 (stale) Mirror 19: 0x1e0d7459 (stale) Mirror 21: 0x1e0d7459 (stale) Mirror 22: 0x1e0d7459 (stale) Mirror 23: 0x1e0d7459 (stale) Mirror 24: 0x1e0d7459 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x633ab50d Mirror 25: 0x633ab50d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000724 1789546684.969246 trunc from 0x040000 to 0x021264 (0x1ed9b bytes) 000725 1789546685.031710 read 0x00b65d thru 0x0174b9 (0xbe5d bytes) 000726 1789546685.038649 trunc from 0x021265 to 0x014201 (0xd063 bytes) 000727 1789546685.069247 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x14202): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14202): Mirror 15: 0xceca1a12 (stale) Mirror 17: 0xceca1a12 (stale) Mirror 18: 0xceca1a12 (stale) Mirror 19: 0xceca1a12 (stale) Mirror 21: 0xceca1a12 (stale) Mirror 22: 0xceca1a12 (stale) Mirror 23: 0xceca1a12 (stale) Mirror 24: 0xceca1a12 (stale) Mirror 25: 0xf7fbe0a (stale) CRC-32 checksum value for chunk [0, 0x14202): Mirror 1: 0xf7fbe0a Mirror 26: 0xf7fbe0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14202: skipped 000728 1789546685.787088 mapwrite 0x0074f9 thru 0x012088 (0xab90 bytes) 000729 1789546685.857101 mapwrite 0x002a38 thru 0x00a0a6 (0x766f bytes) 000730 1789546685.866539 fallocate 0x020874 thru 0x027855 (0x6fe2 bytes) 000731 1789546685.869952 write 0x017915 thru 0x01c1ee (0x48da bytes) 000732 1789546685.875197 fallocate 0x027b21 thru 0x033f0a (0xc3ea bytes) 000733 1789546685.880192 read 0x00eca6 thru 0x0190f1 (0xa44c bytes) 000734 1789546685.888607 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x33f0b): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f0b): Mirror 15: 0xe19bf37c (stale) Mirror 17: 0xe19bf37c (stale) Mirror 18: 0xe19bf37c (stale) Mirror 19: 0xe19bf37c (stale) Mirror 21: 0xe19bf37c (stale) Mirror 22: 0xe19bf37c (stale) Mirror 23: 0xe19bf37c (stale) Mirror 24: 0xe19bf37c (stale) Mirror 25: 0x509ef89d (stale) Mirror 26: 0xb7c41388 (stale) CRC-32 checksum value for chunk [0, 0x33f0b): Mirror 1: 0x789da433 Mirror 27: 0x789da433 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f0b: skipped 000735 1789546686.688376 fallocate 0x026135 thru 0x02d5a7 (0x7473 bytes) 000736 1789546686.739603 MIRROR_RESYNC 12 mirrors 000737 1789546688.388958 write 0x0259d5 thru 0x02ab1e (0x514a bytes) 000738 1789546688.416918 write 0x02f986 thru 0x030996 (0x1011 bytes) 000739 1789546688.423858 trunc from 0x007085 to 0x01080a (0x9786 bytes) 000740 1789546688.427204 punch from 0x02e591 to 0x03311a (0x4b8a bytes) 000741 1789546688.437523 trunc from 0x0041e8 to 0x005e40 (0x1c59 bytes) 000742 1789546688.441527 write 0x00cf7f thru 0x016f0c (0x9f8e bytes) 000743 1789546688.449059 write 0x0357f6 thru 0x036051 (0x85c bytes) 000744 1789546688.450042 read 0x020c96 thru 0x024089 (0x33f4 bytes) 000747 1789546688.472019 read 0x03184f thru 0x036051 (0x4803 bytes) 000748 1789546688.489104 mapread 0x019748 thru 0x01e217 (0x4ad0 bytes) 000749 1789546688.490051 write 0x0344b2 thru 0x035507 (0x1056 bytes) 000750 1789546688.496115 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36052): Mirror 15: 0x3afa49c0 (stale) Mirror 17: 0x3afa49c0 (stale) Mirror 18: 0x3afa49c0 (stale) Mirror 19: 0x3afa49c0 (stale) Mirror 22: 0x3afa49c0 (stale) Mirror 23: 0x3afa49c0 (stale) Mirror 24: 0x3afa49c0 (stale) Mirror 25: 0x3afa49c0 (stale) Mirror 26: 0x3afa49c0 (stale) Mirror 27: 0x3afa49c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36052): Mirror 15: 0x3afa49c0 (stale) Mirror 17: 0x3afa49c0 (stale) Mirror 18: 0x3afa49c0 (stale) Mirror 19: 0x3afa49c0 (stale) Mirror 22: 0x3afa49c0 (stale) Mirror 23: 0x3afa49c0 (stale) Mirror 24: 0x3afa49c0 (stale) Mirror 25: 0x3afa49c0 (stale) Mirror 26: 0x3afa49c0 (stale) Mirror 27: 0x3afa49c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36052: skipped 000751 1789546688.873664 fallocate 0x03fe77 thru 0x03ffff (0x189 bytes) 000752 1789546688.879003 mapwrite 0x01d70d thru 0x01f222 (0x1b16 bytes) 000753 1789546688.908393 mapread 0x01764e thru 0x022e05 (0xb7b8 bytes) 000754 1789546688.920278 mapwrite 0x0327e8 thru 0x03ffff (0xd818 bytes) 000755 1789546688.996754 read 0x000503 thru 0x001c6b (0x1769 bytes) 000756 1789546689.007011 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xd4dc6e14 (stale) Mirror 18: 0xd4dc6e14 (stale) Mirror 19: 0xd4dc6e14 (stale) Mirror 22: 0xd4dc6e14 (stale) Mirror 23: 0xd4dc6e14 (stale) Mirror 24: 0xd4dc6e14 (stale) Mirror 25: 0xd4dc6e14 (stale) Mirror 26: 0xd4dc6e14 (stale) Mirror 27: 0xd4dc6e14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xd4dc6e14 (stale) Mirror 18: 0xd4dc6e14 (stale) Mirror 19: 0xd4dc6e14 (stale) Mirror 22: 0xd4dc6e14 (stale) Mirror 23: 0xd4dc6e14 (stale) Mirror 24: 0xd4dc6e14 (stale) Mirror 25: 0xd4dc6e14 (stale) Mirror 26: 0xd4dc6e14 (stale) Mirror 27: 0xd4dc6e14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000757 1789546689.383584 mapread 0x03137a thru 0x035cf0 (0x4977 bytes) 000758 1789546689.407158 trunc from 0x040000 to 0x004c5f (0x3b3a0 bytes) 000759 1789546689.432407 mapread 0x002571 thru 0x004c5f (0x26ef bytes) 000760 1789546689.437819 mapwrite 0x02a20a thru 0x034539 (0xa330 bytes) 000762 1789546689.503067 write 0x01a817 thru 0x01bd9b (0x1585 bytes) 000763 1789546689.509710 fallocate 0x001eb4 thru 0x009b27 (0x7c74 bytes) 000764 1789546689.516633 trunc from 0x01f1e4 to 0x02791a (0x8737 bytes) 000765 1789546689.521966 fallocate 0x03f499 thru 0x03ffff (0xb67 bytes) 000766 1789546689.527246 write 0x0231bd thru 0x02bb43 (0x8987 bytes) 000767 1789546689.533989 fallocate 0x02bcf9 thru 0x035942 (0x9c4a bytes) 000770 1789546689.538661 mapread 0x02064a thru 0x020bce (0x585 bytes) 000771 1789546689.550916 punch from 0x005e9d to 0x011fbe (0xc122 bytes) 000772 1789546689.554676 trunc from 0x035943 to 0x0007ac (0x35196 bytes) 000773 1789546689.565682 write 0x016bed thru 0x01caff (0x5f13 bytes) 000774 1789546689.566995 trunc from 0x01cb00 to 0x01d2f8 (0x7f9 bytes) 000775 1789546689.575947 trunc from 0x00d3fa to 0x0136dd (0x62e4 bytes) 000776 1789546689.579697 fallocate 0x02a71d thru 0x038e3e (0xe722 bytes) 000777 1789546689.582375 read 0x0096f7 thru 0x00f571 (0x5e7b bytes) 000778 1789546689.588769 mapwrite 0x03e991 thru 0x03ffff (0x166f bytes) 000779 1789546689.618180 punch from 0x03af54 to 0x03ffff (0x50ac bytes) 000780 1789546689.621930 read 0x00a611 thru 0x0193f7 (0xede7 bytes) 000781 1789546689.628266 punch from 0x009327 to 0x011f75 (0x8c4f bytes) 000782 1789546689.634572 trunc from 0x040000 to 0x029557 (0x16aa8 bytes) 000783 1789546689.652024 fallocate 0x00e990 thru 0x012274 (0x38e5 bytes) 000784 1789546689.660973 mapread 0x028f03 thru 0x029557 (0x655 bytes) 000785 1789546689.663195 trunc from 0x029558 to 0x0248a5 (0x4cb2 bytes) 000786 1789546689.682260 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x248a6): Mirror 17: 0xb24f2e77 (stale) Mirror 18: 0xb24f2e77 (stale) Mirror 19: 0xb24f2e77 (stale) Mirror 22: 0xb24f2e77 (stale) Mirror 24: 0xb24f2e77 (stale) Mirror 25: 0xb24f2e77 (stale) Mirror 26: 0xb24f2e77 (stale) Mirror 27: 0xb24f2e77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x248a6): Mirror 17: 0xb24f2e77 (stale) Mirror 18: 0xb24f2e77 (stale) Mirror 19: 0xb24f2e77 (stale) Mirror 22: 0xb24f2e77 (stale) Mirror 24: 0xb24f2e77 (stale) Mirror 25: 0xb24f2e77 (stale) Mirror 26: 0xb24f2e77 (stale) Mirror 27: 0xb24f2e77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x248a6: skipped 000787 1789546690.026980 fallocate 0x027fec thru 0x032026 (0xa03b bytes) 000788 1789546690.031054 punch from 0x00f469 to 0x01406d (0x4c05 bytes) 000789 1789546690.034558 mapread 0x012836 thru 0x0136ac (0xe77 bytes) 000790 1789546690.041649 mapwrite 0x0182c2 thru 0x01aef3 (0x2c32 bytes) 000791 1789546690.062695 mapread 0x0000e4 thru 0x009f42 (0x9e5f bytes) 000792 1789546690.070913 trunc from 0x0248a6 to 0x016839 (0xe06c bytes) 000793 1789546690.090732 MIRROR_RESYNC 9 mirrors 000794 1789546691.017986 mapread 0x00c1e7 thru 0x016839 (0xa653 bytes) 000796 1789546691.037695 MIRROR_EXTEND to 10 mirrors 000798 1789546691.592086 trunc from 0x0099d0 to 0x016839 (0xce6a bytes) 000799 1789546691.631760 trunc from 0x01683a to 0x007ba3 (0xec96 bytes) 000800 1789546691.650121 mapread 0x0058e5 thru 0x007ba3 (0x22bf bytes) 000801 1789546691.661208 trunc from 0x007ba4 to 0x007b73 (0x30 bytes) 000802 1789546691.676419 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x7b74): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7b74): Mirror 17: 0xa0a324a (stale) Mirror 18: 0xa0a324a (stale) Mirror 19: 0xa0a324a (stale) Mirror 22: 0xa0a324a (stale) Mirror 24: 0xa0a324a (stale) Mirror 25: 0xa0a324a (stale) Mirror 26: 0xa0a324a (stale) Mirror 27: 0xa0a324a (stale) Mirror 28: 0xa0a324a (stale) CRC-32 checksum value for chunk [0, 0x7b74): Mirror 1: 0xa0a324a Mirror 29: 0xa0a324a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7b74: skipped 000805 1789546692.391526 punch from 0x004ccf to 0x007b73 (0x2ea5 bytes) 000806 1789546692.438554 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7b74): Mirror 17: 0xa0a324a (stale) Mirror 18: 0xa0a324a (stale) Mirror 19: 0xa0a324a (stale) Mirror 22: 0xa0a324a (stale) Mirror 24: 0xa0a324a (stale) Mirror 25: 0xa0a324a (stale) Mirror 26: 0xa0a324a (stale) Mirror 27: 0xa0a324a (stale) Mirror 28: 0xa0a324a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7b74): Mirror 17: 0xa0a324a (stale) Mirror 18: 0xa0a324a (stale) Mirror 19: 0xa0a324a (stale) Mirror 22: 0xa0a324a (stale) Mirror 24: 0xa0a324a (stale) Mirror 25: 0xa0a324a (stale) Mirror 26: 0xa0a324a (stale) Mirror 27: 0xa0a324a (stale) Mirror 28: 0xa0a324a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7b74: skipped 000807 1789546692.979895 MIRROR_RESYNC 10 mirrors 000809 1789546694.001711 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) 000810 1789546694.247528 mapwrite 0x030e84 thru 0x03b4e3 (0xa660 bytes) 000811 1789546694.325976 trunc from 0x03b4e4 to 0x01c7cb (0x1ed18 bytes) 000812 1789546694.352433 trunc from 0x01588f to 0x01c7cb (0x6f3d bytes) 000813 1789546694.356537 punch from 0x0053c6 to 0x01173b (0xc376 bytes) 000814 1789546694.362616 mapread 0x00435f thru 0x013963 (0xf605 bytes) 000815 1789546694.379295 mapwrite 0x01dbe8 thru 0x02d705 (0xfb1e bytes) 000816 1789546694.412444 trunc from 0x02d706 to 0x003ea1 (0x29864 bytes) 000817 1789546694.422372 trunc from 0x003ea2 to 0x00d3f5 (0x9554 bytes) 000818 1789546694.442562 punch from 0x009c19 to 0x00d3f5 (0x37dd bytes) 000819 1789546694.448834 mapread 0x00cea2 thru 0x00d3f5 (0x554 bytes) 000821 1789546694.459206 trunc from 0x003a14 to 0x00d3f5 (0x99e2 bytes) 000823 1789546694.464447 punch from 0x00bd28 to 0x00d3f5 (0x16ce bytes) 000824 1789546694.469966 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xd3f6): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd3f6): Mirror 17: 0x9da323c0 (stale) Mirror 18: 0x9da323c0 (stale) Mirror 19: 0x9da323c0 (stale) Mirror 22: 0x9da323c0 (stale) Mirror 24: 0x9da323c0 (stale) Mirror 25: 0x9da323c0 (stale) Mirror 26: 0x9da323c0 (stale) Mirror 27: 0x9da323c0 (stale) CRC-32 checksum value for chunk [0, 0xd3f6): Mirror 1: 0x9da323c0 Mirror 28: 0x9da323c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd3f6: skipped 000825 1789546695.064001 mapread 0x0021ee thru 0x00d3f5 (0xb208 bytes) 000826 1789546695.132783 MIRROR_RESYNC 10 mirrors 000827 1789546696.047723 write 0x024d2f thru 0x02b76d (0x6a3f bytes) 000828 1789546696.067348 read 0x0203e5 thru 0x02b76d (0xb389 bytes) 000829 1789546696.070029 punch from 0x00ec62 to 0x018cab (0xa04a bytes) 000830 1789546696.072091 trunc from 0x02b76e to 0x017836 (0x13f37 bytes) 000832 1789546696.092394 punch from 0x004df1 to 0x011594 (0xc7a4 bytes) 000833 1789546696.096330 write 0x001f78 thru 0x006793 (0x481c bytes) 000834 1789546696.102056 trunc from 0x01432e to 0x017836 (0x3509 bytes) 000835 1789546696.106002 punch from 0x011b5a to 0x017836 (0x5cdd bytes) 000836 1789546696.109992 trunc from 0x017837 to 0x021bf4 (0xa3be bytes) 000837 1789546696.123446 punch from 0x00a173 to 0x010d17 (0x6ba5 bytes) 000838 1789546696.128130 write 0x0200a1 thru 0x0267ab (0x670b bytes) 000839 1789546696.133235 trunc from 0x020bda to 0x0267ab (0x5bd2 bytes) 000841 1789546696.149796 trunc from 0x0267ac to 0x031c10 (0xb465 bytes) 000842 1789546696.167868 fallocate 0x031f82 thru 0x03ffff (0xe07e bytes) 000843 1789546696.172872 read 0x000fcd thru 0x010042 (0xf076 bytes) 000844 1789546696.182883 read 0x00d49f thru 0x01a13e (0xcca0 bytes) 000848 1789546696.185086 write 0x00625d thru 0x0161fc (0xffa0 bytes) 000849 1789546696.188683 mapread 0x03ab61 thru 0x03ffff (0x549f bytes) 000850 1789546696.189561 read 0x019a6a thru 0x0211e0 (0x7777 bytes) 000851 1789546696.190077 fallocate 0x038e91 thru 0x03b81d (0x298d bytes) 000852 1789546696.197057 mapwrite 0x014fd9 thru 0x01c830 (0x7858 bytes) 000854 1789546696.223082 punch from 0x02a176 to 0x0385b3 (0xe43e bytes) 000855 1789546696.227370 trunc from 0x040000 to 0x01e567 (0x21a98 bytes) 000856 1789546696.243388 mapwrite 0x017003 thru 0x01fd35 (0x8d33 bytes) 000857 1789546696.291819 read 0x01902d thru 0x01af7a (0x1f4e bytes) 000858 1789546696.294913 punch from 0x0025fd to 0x00442d (0x1e31 bytes) 000859 1789546696.299922 read 0x019d40 thru 0x01b7f8 (0x1ab9 bytes) 000860 1789546696.300794 mapwrite 0x01469f thru 0x01db3c (0x949e bytes) 000861 1789546696.315696 fallocate 0x024bc0 thru 0x0260eb (0x152c bytes) 000862 1789546696.319601 mapread 0x00bf69 thru 0x016a1d (0xaab5 bytes) 000863 1789546696.321439 mapread 0x00e879 thru 0x01409e (0x5826 bytes) 000864 1789546696.322502 mapread 0x01c48a thru 0x022b46 (0x66bd bytes) 000865 1789546696.332957 mapwrite 0x016600 thru 0x023827 (0xd228 bytes) 000868 1789546696.352731 write 0x0109fd thru 0x01ba78 (0xb07c bytes) 000869 1789546696.354031 write 0x011120 thru 0x01bb1f (0xaa00 bytes) 000870 1789546696.359424 trunc from 0x0260ec to 0x031abb (0xb9d0 bytes) 000871 1789546696.372017 mapwrite 0x02063d thru 0x02e5a5 (0xdf69 bytes) 000872 1789546696.396692 mapwrite 0x00d9b1 thru 0x013fd0 (0x6620 bytes) 000873 1789546696.412608 trunc from 0x017b58 to 0x0219fc (0x9ea5 bytes) 000874 1789546696.424162 mapwrite 0x015c85 thru 0x020939 (0xacb5 bytes) 000875 1789546696.469952 mapwrite 0x00ca7c thru 0x011391 (0x4916 bytes) 000876 1789546696.483298 write 0x002145 thru 0x00d510 (0xb3cc bytes) 000877 1789546696.487311 fallocate 0x033be1 thru 0x038274 (0x4694 bytes) 000878 1789546696.497464 trunc from 0x0015ae to 0x004472 (0x2ec5 bytes) 000879 1789546696.518634 write 0x033da4 thru 0x03541e (0x167b bytes) 000880 1789546696.530551 punch from 0x0005d9 to 0x00513f (0x4b67 bytes) 000883 1789546696.537593 mapwrite 0x02b4fa thru 0x02fe7f (0x4986 bytes) 000884 1789546696.557986 punch from 0x021576 to 0x022e83 (0x190e bytes) 000885 1789546696.562955 trunc from 0x01a9fc to 0x024e57 (0xa45c bytes) 000886 1789546696.574347 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38275): Mirror 17: 0x14e63a8f (stale) Mirror 18: 0x14e63a8f (stale) Mirror 19: 0x14e63a8f (stale) Mirror 22: 0x14e63a8f (stale) Mirror 24: 0x14e63a8f (stale) Mirror 25: 0x14e63a8f (stale) Mirror 27: 0x14e63a8f (stale) Mirror 28: 0x14e63a8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38275): Mirror 17: 0x14e63a8f (stale) Mirror 18: 0x14e63a8f (stale) Mirror 19: 0x14e63a8f (stale) Mirror 22: 0x14e63a8f (stale) Mirror 24: 0x14e63a8f (stale) Mirror 25: 0x14e63a8f (stale) Mirror 27: 0x14e63a8f (stale) Mirror 28: 0x14e63a8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38275: skipped 000888 1789546696.955174 fallocate 0x011065 thru 0x015e44 (0x4de0 bytes) 000889 1789546696.959564 trunc from 0x017949 to 0x027722 (0xfdda bytes) 000891 1789546696.963854 fallocate 0x006260 thru 0x015318 (0xf0b9 bytes) 000892 1789546696.971647 punch from 0x01e52a to 0x027bc0 (0x9697 bytes) 000893 1789546696.977048 fallocate 0x01c1dc thru 0x01f619 (0x343e bytes) 000894 1789546696.983434 mapread 0x01531f thru 0x01db7d (0x885f bytes) 000896 1789546697.005634 trunc from 0x02a8ce to 0x038274 (0xd9a7 bytes) 000897 1789546697.012789 punch from 0x02dcda to 0x033f8a (0x62b1 bytes) 000898 1789546697.017158 mapread 0x02b25f thru 0x03425f (0x9001 bytes) 000899 1789546697.039827 write 0x023721 thru 0x028915 (0x51f5 bytes) 000900 1789546697.049948 punch from 0x028d89 to 0x02ce68 (0x40e0 bytes) 000901 1789546697.066969 punch from 0x019bd2 to 0x02013f (0x656e bytes) 000902 1789546697.075394 trunc from 0x038275 to 0x00c66c (0x2bc08 bytes) 000903 1789546697.089342 read 0x00af4e thru 0x00c66c (0x171f bytes) 000904 1789546697.093584 trunc from 0x006434 to 0x00c66c (0x6239 bytes) 000905 1789546697.099913 fallocate 0x00eec3 thru 0x012db1 (0x3eef bytes) 000906 1789546697.103456 write 0x014e7e thru 0x01845e (0x35e1 bytes) 000907 1789546697.104386 trunc from 0x005561 to 0x00ab28 (0x55c8 bytes) 000909 1789546697.109118 write 0x0334c4 thru 0x03d5d9 (0xa116 bytes) 000910 1789546697.110407 write 0x00a04d thru 0x013362 (0x9316 bytes) 000911 1789546697.119678 fallocate 0x0001db thru 0x005a1d (0x5843 bytes) 000912 1789546697.124205 punch from 0x01a4ed to 0x022ca6 (0x87ba bytes) 000913 1789546697.127695 fallocate 0x03c9ba thru 0x03ffff (0x3646 bytes) 000916 1789546697.131721 trunc from 0x03d5da to 0x03a21a (0x33bf bytes) 000917 1789546697.141438 write 0x01be12 thru 0x0284c2 (0xc6b1 bytes) 000918 1789546697.150592 read 0x00729a thru 0x00e3fa (0x7161 bytes) 000919 1789546697.159461 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3a21b): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a21b): Mirror 17: 0x3f4058f7 (stale) Mirror 18: 0x3f4058f7 (stale) Mirror 19: 0x3f4058f7 (stale) Mirror 22: 0x3f4058f7 (stale) Mirror 24: 0x3f4058f7 (stale) Mirror 25: 0x3f4058f7 (stale) Mirror 27: 0x3f4058f7 (stale) Mirror 28: 0x3f4058f7 (stale) CRC-32 checksum value for chunk [0, 0x3a21b): Mirror 1: 0x6502d543 Mirror 29: 0x6502d543 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a21b: skipped 000920 1789546697.832163 write 0x005d99 thru 0x00b6c1 (0x5929 bytes) 000921 1789546697.866690 trunc from 0x03a21b to 0x01f0d2 (0x1b148 bytes) 000922 1789546697.895286 trunc from 0x01138b to 0x016d5c (0x59d2 bytes) 000924 1789546697.900052 mapread 0x00d593 thru 0x0122f7 (0x4d65 bytes) 000925 1789546697.914052 trunc from 0x0045ed to 0x010c7d (0xc691 bytes) 000926 1789546697.925709 punch from 0x00e22e to 0x01d632 (0xf405 bytes) 000927 1789546697.937780 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f0d3): Mirror 18: 0x9ea95db7 (stale) Mirror 19: 0x9ea95db7 (stale) Mirror 22: 0x9ea95db7 (stale) Mirror 24: 0x9ea95db7 (stale) Mirror 25: 0x9ea95db7 (stale) Mirror 27: 0x9ea95db7 (stale) Mirror 28: 0x9ea95db7 (stale) Mirror 29: 0x6503b6e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f0d3): Mirror 18: 0x9ea95db7 (stale) Mirror 19: 0x9ea95db7 (stale) Mirror 22: 0x9ea95db7 (stale) Mirror 24: 0x9ea95db7 (stale) Mirror 25: 0x9ea95db7 (stale) Mirror 27: 0x9ea95db7 (stale) Mirror 28: 0x9ea95db7 (stale) Mirror 29: 0x6503b6e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f0d3: skipped 000928 1789546698.309243 write 0x001178 thru 0x009f1d (0x8da6 bytes) 000929 1789546698.321506 mapwrite 0x028c9c thru 0x02aadf (0x1e44 bytes) 000930 1789546698.376129 trunc from 0x01e0e8 to 0x027b49 (0x9a62 bytes) 000931 1789546698.380977 trunc from 0x02aae0 to 0x004941 (0x2619e bytes) 000933 1789546698.399110 write 0x0287c3 thru 0x035fd6 (0xd814 bytes) 000934 1789546698.406322 mapread 0x0257c6 thru 0x026f1c (0x1757 bytes) 000936 1789546698.415270 trunc from 0x014a46 to 0x01fd2a (0xb2e5 bytes) 000937 1789546698.421948 write 0x02bcaa thru 0x036dc1 (0xb118 bytes) 000938 1789546698.423738 read 0x027f50 thru 0x02c0c4 (0x4175 bytes) 000939 1789546698.424652 mapwrite 0x02d69f thru 0x03d37f (0xfce1 bytes) 000940 1789546698.469730 punch from 0x02ea97 to 0x02ead0 (0x3a bytes) 000941 1789546698.474151 write 0x01d2f5 thru 0x0248f0 (0x75fc bytes) 000942 1789546698.478718 trunc from 0x03d380 to 0x01433b (0x29044 bytes) 000943 1789546698.489028 write 0x02e6e6 thru 0x03121b (0x2b36 bytes) 000944 1789546698.490928 trunc from 0x028e59 to 0x03121b (0x83c3 bytes) 000945 1789546698.502643 mapread 0x0048f3 thru 0x01205b (0xd769 bytes) 000946 1789546698.521998 trunc from 0x03121c to 0x00705d (0x2a1be bytes) 000947 1789546698.539742 write 0x03f302 thru 0x03ffff (0xcfe bytes) 000949 1789546698.541959 read 0x011eaa thru 0x021858 (0xf9af bytes) 000950 1789546698.553170 write 0x00b0a3 thru 0x01864f (0xd5ad bytes) 000951 1789546698.556256 mapwrite 0x0159ae thru 0x0240c8 (0xe71b bytes) 000952 1789546698.572369 punch from 0x001e2f to 0x008516 (0x66e8 bytes) 000953 1789546698.575930 read 0x0085b8 thru 0x015741 (0xd18a bytes) 000954 1789546698.582658 mapread 0x02022e thru 0x0292e8 (0x90bb bytes) 000955 1789546698.583953 mapread 0x03f2b2 thru 0x03ffff (0xd4e bytes) 000956 1789546698.584976 read 0x00a937 thru 0x018986 (0xe050 bytes) 000957 1789546698.586083 write 0x001d9b thru 0x00d8f1 (0xbb57 bytes) 000959 1789546698.591157 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 28: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3c98c5ad Mirror 30: 0x3c98c5ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000960 1789546699.192326 mapread 0x004ee8 thru 0x006a0d (0x1b26 bytes) 000961 1789546699.210944 fallocate 0x02d203 thru 0x0359c1 (0x87bf bytes) 000962 1789546699.238610 trunc from 0x01771f to 0x0219e4 (0xa2c6 bytes) 000963 1789546699.244466 trunc from 0x002e3e to 0x00bd18 (0x8edb bytes) 000964 1789546699.248956 mapread 0x014aa4 thru 0x0180d6 (0x3633 bytes) 000965 1789546699.263961 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 28: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe4b4412b Mirror 31: 0xe4b4412b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000966 1789546699.990236 punch from 0x007457 to 0x00cd93 (0x593d bytes) 000967 1789546700.024901 fallocate 0x0289a5 thru 0x02b83b (0x2e97 bytes) 000968 1789546700.036057 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 28: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 28: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000969 1789546700.389897 trunc from 0x006412 to 0x0106b0 (0xa29f bytes) 000970 1789546700.394091 trunc from 0x000844 to 0x007303 (0x6ac0 bytes) 000971 1789546700.398880 read 0x016f02 thru 0x02249f (0xb59e bytes) 000972 1789546700.404473 read 0x0317a7 thru 0x03dabb (0xc315 bytes) 000973 1789546700.419652 read 0x01ed71 thru 0x0233d2 (0x4662 bytes) 000974 1789546700.420059 write 0x016848 thru 0x01da6e (0x7227 bytes) 000976 1789546700.424126 read 0x03d57b thru 0x03ffff (0x2a85 bytes) 000977 1789546700.428116 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 28: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 28: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000978 1789546700.911218 write 0x0394fa thru 0x03ffff (0x6b06 bytes) 000979 1789546700.919836 trunc from 0x010795 to 0x01e9e4 (0xe250 bytes) 000980 1789546700.924056 fallocate 0x018464 thru 0x0208a9 (0x8446 bytes) 000981 1789546700.928072 punch from 0x03bd40 to 0x03c5bb (0x87c bytes) 000982 1789546700.953447 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 27: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000983 1789546701.344710 mapread 0x038848 thru 0x03ffff (0x77b8 bytes) 000984 1789546701.362635 write 0x029718 thru 0x02b04f (0x1938 bytes) 000985 1789546701.375130 punch from 0x02184d to 0x02379d (0x1f51 bytes) 000986 1789546701.383345 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1789546701.704929 fallocate 0x023ac7 thru 0x02d6c1 (0x9bfb bytes) 000988 1789546701.713318 read 0x039493 thru 0x03ffff (0x6b6d bytes) 000989 1789546701.721859 mapwrite 0x018411 thru 0x0205ae (0x819e bytes) 000990 1789546701.755063 punch from 0x00b5af to 0x017b16 (0xc568 bytes) 000991 1789546701.759454 trunc from 0x010356 to 0x015eb4 (0x5b5f bytes) 000993 1789546701.764187 read 0x0303ba thru 0x03b0bb (0xad02 bytes) 000994 1789546701.790303 mapwrite 0x00a7e4 thru 0x0166ca (0xbee7 bytes) 000995 1789546701.810889 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x29d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000996 1789546702.171568 read 0x037140 thru 0x038522 (0x13e3 bytes) 000998 1789546702.183706 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xe20eea22 (stale) Mirror 25: 0xe20eea22 (stale) Mirror 29: 0xc9faeaf5 (stale) Mirror 30: 0x3c98c5ad (stale) Mirror 31: 0xe4b4412b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9ca73b77 Mirror 32: 0x9ca73b77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000999 1789546703.044890 mapread 0x034fe6 thru 0x037c5d (0x2c78 bytes) 001000 1789546703.086438 trunc from 0x040000 to 0x00cdc5 (0x3323a bytes) All operations completed A-OK! PASS 70a (82s) == sanity-flr test 71: check mirror extend parent fid ==== 04:18:26 (1789546706) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x2e2:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 04:18:33 (1789546713) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137503 s, 15.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765349 s, 13.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 04:18:39 (1789546719) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6891968 .. Extending file size to 5781824 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7748032 .. failed Extending file size to 8944832 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8612992 .. Extending file size to 8921536 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9162272 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62056 62057 62058 62061 62062 done 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) 192.168.201.154@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:19:54 (1789546794) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6984832 .. done Extending file size to 7567584 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7617056 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7998624 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9169472 .. done Extending file size to 7207072 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8390080 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8394080 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 63828 63829 63832 63833 63836 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) 192.168.201.154@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (72s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:21:07 (1789546867) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2484 1268820 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17896 3589124 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25732 3581288 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43628 7170412 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2484 1268820 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17896 3589124 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25732 3581288 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43628 7170412 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0699167 s, 30.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0970868 s, 21.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd42f07ff (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) PASS 200c (12s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:21:19 (1789546879) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0781952 s, 26.8 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 04:21:24 (1789546884) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:21:28 (1789546888) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0546037 s, 38.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3bb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:21:33 (1789546893) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0553093 s, 37.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:21:38 (1789546898) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0796378 s, 26.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3bc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3bd:0x0] } PASS 204c (28s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:22:06 (1789546926) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg154-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559049 s, 18.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3ba:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:22:50 (1789546970) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg154-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0607568 s, 34.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:23:34 (1789547014) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg154-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg154-server: Pool lustre.test_204f created Adding targets to pool oleg154-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg154-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.086533 s, 24.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg154-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg154-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg154-server: Pool lustre.test_204f destroyed PASS 204f (57s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:24:31 (1789547071) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:24:36 (1789547076) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:24:40 (1789547080) oleg154-server: Pool lustre.test_206 created Adding targets to pool oleg154-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg154-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg154-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg154-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg154-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:25:01 (1789547101) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344649 s, 30.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247062 s, 42.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8450f621 Mirror 3: 0x8450f621 PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:25:07 (1789547107) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:25:09 (1789547109) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:25:11 (1789547111) Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0340319 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x890000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x890000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x890000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xa50000 Read this (4 bytes): BBBB 192.168.201.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg154-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:25:35 (1789547135) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00778828 s, 1.1 MB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:25:39 (1789547139) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223441 s, 46.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7d lmm_fid: [0x200000405:0x7d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7d lmm_fid: [0x200000405:0x7d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611261 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 04:25:40.000000000 -0400 Modify: 2026-09-16 04:25:40.000000000 -0400 Change: 2026-09-16 04:25:41.000000000 -0400 Birth: 2026-09-16 04:25:41.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:25:44 (1789547144) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250528 s, 41.9 MB/s oleg154-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg154-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg154-server: O/280000401/d28/13c: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:26:00 (1789547160) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.135337 s, 303 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0101213 s, 405 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 16 04:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x82:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 16 04:26 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:26:04 (1789547164) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:26:08 (1789547168) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1484 sec =========== 04:26:12 (1789547172) === sanity-flr: start cleanup 04:26:13 (1789547173) === === sanity-flr: finish cleanup 04:26:18 (1789547178) ===