-----============= acceptance-small: sanity-flr ============----- Tue Sep 8 21:48:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg308-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:49:16 (1788918556) === oleg308-client.virtnet: executing check_config_client /mnt/lustre oleg308-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg308-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93b2c2c92000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93b2c2c92000.idle_timeout=debug disable quota as required oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:49:50 (1788918590) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7463] == sanity-flr test 0a: lfs mirror create with -N option == 21:49:52 (1788918592) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:50:07 (1788918607) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:50:11 (1788918611) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:50:14 (1788918614) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:50:41 (1788918641) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:50:44 (1788918644) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 21:50:47 (1788918647) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93b2c2c92000.stats=clear osc.lustre-OST0001-osc-ffff93b2c2c92000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:50:55 (1788918655) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:51:06 (1788918666) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:51:15 (1788918675) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:51:19 (1788918679) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:51:27 (1788918687) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 21:51:37 (1788918697) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:51:45 (1788918705) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:51:49 (1788918709) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:51:58 (1788918718) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:52:11 (1788918731) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:52:17 (1788918737) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.58962 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809559 s, 13.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:52:27 (1788918747) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.59209 s, 31.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:52:36 (1788918756) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:52:47 (1788918767) Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93b2c2c92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93b2c2c92000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg308-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93b2c2c92000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93b2c2c92000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (139s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:55:05 (1788918905) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.370781 s, 11.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93b2c2c92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93b2c2c92000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:55:57 (1788918957) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93b2c2c92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93b2c2c92000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93b2c2c92000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93b2c2c92000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:57:20 (1788919040) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.40261 s, 15.4 MB/s -rw-r--r-- 1 root root 52428800 Sep 8 21:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.105s user 0m0.001s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93b2c2c92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93b2c2c92000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.915s user 0m0.001s sys 0m0.163s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg308-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg308-server oleg308-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93b2c2c92000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93b2c2c92000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.142s user 0m0.002s sys 0m0.182s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg308-server: oleg308-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg308-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg308-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93b2c2c92000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93b2c2c92000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:58:56 (1788919136) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:58:59 (1788919139) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:59:02 (1788919142) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:59:05 (1788919145) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 21:59:12 (1788919152) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.569594 s, 22.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:59:24 (1788919164) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643457 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353101 s, 29.7 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:59:30 (1788919170) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:59:38 (1788919178) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000943933 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000967352 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00153341 s, 668 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138781 s, 738 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 21:59:46 (1788919186) create test files with size 15741087 .. e0c06914f42f922749a43dacd3f7ade0 - 6f2ee8b6c4410c6039fde2cacba5e6b9 - d7cb72495a2d13f8dfdd320be3a10f2b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 22:00:06 (1788919206) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:00:19 (1788919219) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:00:25 (1788919225) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910202 s, 11.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296285 s, 35.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (6s) == sanity-flr test 41: lfs mirror resync check =========== 22:00:31 (1788919231) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0949882 s, 22.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.290118 s, 14.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147429 s, 14.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 22:00:43 (1788919243) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:00:46 (1788919246) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:00:49 (1788919249) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521898 s, 20.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 22:00:55 (1788919255) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:00:57 (1788919257) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:01:04 (1788919264) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.212797 s, 49.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:01:12 (1788919272) llite.lustre-ffff93b2c77b9000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.231531 s, 45.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg308-client.virtnet /mnt/lustre (opts:) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff93b2d0f0c000.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:01:21 (1788919281) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (24s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:01:45 (1788919305) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (23s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:02:08 (1788919328) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 22:02:10 (1788919330) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.137341 s, 22.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0768176 s, 41.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa3f5724 (nosync,stale) Mirror 2: 0xa3f5724 (nosync,stale) ** verify mirror 2 doesn't change original checksum: cd3e26ff3d0f871cee372b48fdf5f1bf - mirror 1 checksum: c94b7f85f2d96bb190d22b6308b8747f - mirror 2 checksum: cd3e26ff3d0f871cee372b48fdf5f1bf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T22:02:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-08T22:02:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cd3e26ff3d0f871cee372b48fdf5f1bf - mirror 1 checksum: c94b7f85f2d96bb190d22b6308b8747f - mirror 2 checksum: c94b7f85f2d96bb190d22b6308b8747f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T22:02:11-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-08T22:02:11-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:02:15 (1788919335) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258457 s, 40.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:02:19 (1788919339) Failing mds1 on oleg308-server Stopping /mnt/lustre-mds1 (opts:) on oleg308-server 22:02:28 (1788919348) shut down facet: mds1 facet_host: oleg308-server facet_failover_host: oleg308-server Failover mds1 to oleg308-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 22:02:45 (1788919365) targets are mounted 22:02:45 (1788919365) facet_failover done oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196496 s, 53.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x17 lmm_fid: [0x200000404:0x17:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } PASS 50A (41s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:03:00 (1788919380) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0205491 s, 997 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0100149 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:03:05 (1788919385) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0298318 s, 687 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00738726 s, 2.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:03:10 (1788919390) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00521081 s, 3.1 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:03:13 (1788919393) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0304728 s, 672 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:03:18 (1788919398) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00466421 s, 878 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0304245 s, 135 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:03:22 (1788919402) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:03:34 (1788919414) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:03:55 (1788919435) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x61:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:04:15 (1788919455) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:04:35 (1788919475) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 22:04:39 (1788919479) mirror create pid 55516 mirror split pid 55517 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 55516 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 55517 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 22:05:42 (1788919542) Seed set to 9543 skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read 000008 1788919543.686057 write 0x0170df thru 0x0245ce (0xd4f0 bytes) fallocating to largest ever: 0x3d258 000009 1788919543.688718 fallocate 0x034ab6 thru 0x03d257 (0x87a2 bytes) 000010 1788919543.694059 trunc from 0x01fe2a to 0x0245ce (0x47a5 bytes) fallocating to largest ever: 0x40000 000013 1788919543.704777 fallocate 0x0398f1 thru 0x03ffff (0x670f bytes) 000014 1788919543.707832 punch from 0x02b3b4 to 0x02cf64 (0x1bb1 bytes) 000015 1788919543.712707 fallocate 0x01e3eb thru 0x02195d (0x3573 bytes) 000016 1788919543.716096 fallocate 0x014845 thru 0x018f52 (0x470e bytes) 000017 1788919543.719650 fallocate 0x013e51 thru 0x01daf3 (0x9ca3 bytes) 000018 1788919543.722474 trunc from 0x033511 to 0x0348b6 (0x13a6 bytes) 000019 1788919543.726127 mapread 0x012950 thru 0x02139f (0xea50 bytes) 000020 1788919543.742415 read 0x02e91d thru 0x036700 (0x7de4 bytes) 000021 1788919543.748462 read 0x00757f thru 0x0104dc (0x8f5e bytes) 000022 1788919543.757556 mapread 0x01bc1a thru 0x01ffde (0x43c5 bytes) 000023 1788919543.758801 mapwrite 0x01d15e thru 0x02560d (0x84b0 bytes) 000024 1788919543.774055 write 0x010717 thru 0x016838 (0x6122 bytes) 000025 1788919543.775023 mapread 0x02f5e9 thru 0x033d5a (0x4772 bytes) 000026 1788919543.775829 mapwrite 0x03736c thru 0x03e6bf (0x7354 bytes) 000027 1788919543.787661 punch from 0x00f7f2 to 0x017c8c (0x849b bytes) 000028 1788919543.791226 punch from 0x03dcec to 0x03ffff (0x2314 bytes) 000030 1788919543.794770 punch from 0x036e22 to 0x03ffff (0x91de bytes) 000031 1788919543.798098 write 0x000c04 thru 0x008c28 (0x8025 bytes) 000032 1788919543.799477 read 0x031188 thru 0x038c66 (0x7adf bytes) 000033 1788919543.810154 trunc from 0x040000 to 0x014ed4 (0x2b12b bytes) 000034 1788919543.820142 mapread 0x0085fb thru 0x00a7a7 (0x21ad bytes) 000035 1788919543.821097 write 0x034da4 thru 0x03ffff (0xb25c bytes) 000036 1788919543.822943 punch from 0x00b770 to 0x01a856 (0xf0e7 bytes) 000037 1788919543.829390 fallocate 0x0270d0 thru 0x02f85f (0x8790 bytes) 000038 1788919543.833909 mapwrite 0x033988 thru 0x03ffff (0xc678 bytes) 000039 1788919543.848439 fallocate 0x017d01 thru 0x01b314 (0x3614 bytes) 000040 1788919543.854138 trunc from 0x040000 to 0x02319b (0x1ce64 bytes) 000041 1788919543.861642 write 0x03af49 thru 0x03ffff (0x50b7 bytes) 000042 1788919543.862769 mapread 0x00c11b thru 0x016b6b (0xaa51 bytes) 000043 1788919543.890094 mapwrite 0x016106 thru 0x0246a6 (0xe5a1 bytes) 000044 1788919543.956140 read 0x010958 thru 0x01c92b (0xbfd4 bytes) 000046 1788919543.958856 mapwrite 0x00574f thru 0x00dc18 (0x84ca bytes) 000047 1788919543.981726 punch from 0x01cc77 to 0x022d9d (0x6127 bytes) 000048 1788919543.989752 punch from 0x020b6a to 0x02237d (0x1814 bytes) 000049 1788919543.994482 punch from 0x0093ea to 0x015ebb (0xcad2 bytes) 000050 1788919543.998712 trunc from 0x040000 to 0x0314b1 (0xeb4e bytes) 000051 1788919544.009163 mapread 0x00643b thru 0x00fc7d (0x9843 bytes) 000052 1788919544.044583 mapread 0x001177 thru 0x00210d (0xf97 bytes) 000054 1788919544.046539 mapwrite 0x034bcc thru 0x03aa94 (0x5ec9 bytes) 000055 1788919544.094025 trunc from 0x03aa95 to 0x02ee59 (0xbc3b bytes) 000057 1788919544.105848 trunc from 0x02ee5a to 0x012f9f (0x1beba bytes) 000059 1788919544.117378 read 0x0091e6 thru 0x012f9f (0x9dba bytes) 000060 1788919544.120783 read 0x00866b thru 0x00b0a3 (0x2a39 bytes) 000063 1788919544.122077 mapread 0x00f107 thru 0x012f9f (0x3e99 bytes) 000064 1788919544.122788 punch from 0x00cdfb to 0x00f6d6 (0x28dc bytes) 000065 1788919544.125845 write 0x0016f2 thru 0x00af9f (0x98ae bytes) 000066 1788919544.127104 write 0x003e2e thru 0x010ab7 (0xcc8a bytes) 000067 1788919544.128448 punch from 0x002ef0 to 0x005b1f (0x2c30 bytes) 000069 1788919544.141170 write 0x018953 thru 0x019e53 (0x1501 bytes) 000070 1788919544.142645 mapread 0x015911 thru 0x015a11 (0x101 bytes) 000071 1788919544.147589 trunc from 0x019e54 to 0x01b1f9 (0x13a6 bytes) 000072 1788919544.154771 fallocate 0x019873 thru 0x01bb89 (0x2317 bytes) 000073 1788919544.159581 write 0x014878 thru 0x017d38 (0x34c1 bytes) 000074 1788919544.160500 trunc from 0x014749 to 0x01b1f9 (0x6ab1 bytes) 000075 1788919544.168633 punch from 0x01656c to 0x01b1f9 (0x4c8e bytes) 000076 1788919544.172555 punch from 0x017b28 to 0x01b1f9 (0x36d2 bytes) 000077 1788919544.179268 MIRROR_EXTEND to 2 mirrors 000078 1788919544.571783 fallocate 0x0215d9 thru 0x02a4dd (0x8f05 bytes) 000079 1788919544.592957 trunc from 0x02a4de to 0x01ac09 (0xf8d4 bytes) 000080 1788919544.619116 mapread 0x009116 thru 0x00ced0 (0x3dbb bytes) 000081 1788919544.640761 fallocate 0x016fda thru 0x01808f (0x10b6 bytes) 000082 1788919544.647417 mapread 0x00567c thru 0x00efce (0x9953 bytes) 000083 1788919544.651969 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) 000084 1788919544.762320 mapwrite 0x01d0c5 thru 0x0280ca (0xb006 bytes) 000086 1788919544.824896 read 0x026892 thru 0x0280ca (0x1839 bytes) 000090 1788919544.828639 punch from 0x006e0e to 0x009e99 (0x308c bytes) 000091 1788919544.837443 mapwrite 0x026430 thru 0x033dae (0xd97f bytes) 000093 1788919544.885700 mapread 0x00b234 thru 0x010b0d (0x58da bytes) 000094 1788919544.896555 mapwrite 0x02a6a5 thru 0x0320ad (0x7a09 bytes) 000095 1788919544.911495 trunc from 0x033daf to 0x01a775 (0x19639 bytes) 000097 1788919544.926151 mapread 0x015de2 thru 0x01a775 (0x4994 bytes) 000098 1788919544.932047 trunc from 0x00f258 to 0x015971 (0x671a bytes) 000099 1788919544.941996 punch from 0x007107 to 0x0111db (0xa0d5 bytes) 000101 1788919544.949112 mapwrite 0x01d7e2 thru 0x021a2a (0x4249 bytes) 000102 1788919544.977891 punch from 0x0080a5 to 0x00ab0a (0x2a66 bytes) 000103 1788919544.983761 punch from 0x00c368 to 0x019a03 (0xd69c bytes) 000104 1788919544.988991 fallocate 0x03047f thru 0x03f828 (0xf3aa bytes) 000105 1788919544.994620 mapread 0x01db2c thru 0x021a2a (0x3eff bytes) 000107 1788919544.995963 mapread 0x0176ab thru 0x021a2a (0xa380 bytes) 000110 1788919545.005852 read 0x0193e7 thru 0x021a2a (0x8644 bytes) 000111 1788919545.006535 mapread 0x017b1d thru 0x021a2a (0x9f0e bytes) 000112 1788919545.009658 mapread 0x01fc62 thru 0x021a2a (0x1dc9 bytes) 000113 1788919545.010429 fallocate 0x0071c3 thru 0x008c46 (0x1a84 bytes) 000114 1788919545.015129 mapread 0x00abc1 thru 0x0127b6 (0x7bf6 bytes) 000115 1788919545.023630 punch from 0x01ad6f to 0x021a2a (0x6cbc bytes) 000116 1788919545.030034 write 0x02c90f thru 0x032c31 (0x6323 bytes) 000118 1788919545.032513 read 0x02e406 thru 0x0329cf (0x45ca bytes) 000119 1788919545.033260 punch from 0x02c7fd to 0x02eef4 (0x26f8 bytes) 000120 1788919545.046959 write 0x02e6fa thru 0x030838 (0x213f bytes) 000121 1788919545.053547 write 0x01d835 thru 0x02836a (0xab36 bytes) 000122 1788919545.054757 trunc from 0x01c246 to 0x0232a8 (0x7063 bytes) 000123 1788919545.063613 fallocate 0x032ca6 thru 0x03ffff (0xd35a bytes) 000124 1788919545.066419 fallocate 0x02c8ef thru 0x02dc5f (0x1371 bytes) 000126 1788919545.069431 trunc from 0x0039eb to 0x00d192 (0x97a8 bytes) 000127 1788919545.073130 punch from 0x025184 to 0x02a87d (0x56fa bytes) 000128 1788919545.077490 trunc from 0x032c32 to 0x02e4d9 (0x4758 bytes) 000131 1788919545.089072 read 0x0040f6 thru 0x00682b (0x2736 bytes) 000133 1788919545.098668 fallocate 0x015195 thru 0x01a3d9 (0x5245 bytes) 000135 1788919545.103602 trunc from 0x02e4da to 0x02c4e5 (0x1ff4 bytes) 000136 1788919545.119411 MIRROR_EXTEND to 2 mirrors 000137 1788919545.441640 mapread 0x0126dd thru 0x018df6 (0x671a bytes) 000140 1788919545.485952 trunc from 0x00cc41 to 0x01a506 (0xd8c6 bytes) 000141 1788919545.515024 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c4e6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4e6): Mirror 2: 0x15e3a05f (stale) CRC-32 checksum value for chunk [0, 0x2c4e6): Mirror 1: 0x15e3a05f Mirror 3: 0x15e3a05f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4e6: skipped 000142 1788919545.972292 trunc from 0x02c4e6 to 0x006a4f (0x25a96 bytes) 000143 1788919546.015551 read 0x001da5 thru 0x006a4f (0x4cab bytes) 000144 1788919546.030310 mapread 0x003824 thru 0x006a4f (0x322c bytes) 000145 1788919546.043190 trunc from 0x0013cd to 0x006a4f (0x5683 bytes) 000146 1788919546.051723 trunc from 0x006a50 to 0x03741d (0x309ce bytes) 000147 1788919546.065889 trunc from 0x0082e7 to 0x010170 (0x7e8a bytes) 000148 1788919546.071152 read 0x02ccd6 thru 0x033f7f (0x72aa bytes) 000149 1788919546.078862 trunc from 0x02ab4f to 0x03741d (0xc8cf bytes) 000150 1788919546.082187 mapwrite 0x0291ea thru 0x036e0a (0xdc21 bytes) 000151 1788919546.156761 write 0x0337f2 thru 0x03e4ef (0xacfe bytes) 000152 1788919546.159488 trunc from 0x03e4f0 to 0x02b0e7 (0x13408 bytes) 000153 1788919546.169015 fallocate 0x03ecc1 thru 0x03ffff (0x133f bytes) 000154 1788919546.175953 MIRROR_RESYNC 3 mirrors 000155 1788919546.607750 trunc from 0x0113f9 to 0x016ff8 (0x5c00 bytes) 000156 1788919546.633079 fallocate 0x00fd26 thru 0x01aca7 (0xaf82 bytes) 000157 1788919546.637152 write 0x0213ca thru 0x0277c6 (0x63fd bytes) 000158 1788919546.644898 mapwrite 0x03e20b thru 0x03ffff (0x1df5 bytes) 000159 1788919546.665949 trunc from 0x010442 to 0x018157 (0x7d16 bytes) 000160 1788919546.670093 punch from 0x01cce5 to 0x026b0d (0x9e29 bytes) 000161 1788919546.676024 trunc from 0x040000 to 0x0133eb (0x2cc14 bytes) 000163 1788919546.710117 write 0x00909b thru 0x00ae5c (0x1dc2 bytes) 000164 1788919546.718956 mapwrite 0x02043d thru 0x021067 (0xc2b bytes) 000165 1788919546.752038 mapwrite 0x032876 thru 0x03f0a6 (0xc831 bytes) 000166 1788919546.803046 fallocate 0x000ac9 thru 0x00c683 (0xbbbb bytes) 000167 1788919546.808576 punch from 0x028dde to 0x02eb16 (0x5d39 bytes) 000168 1788919546.817459 fallocate 0x00f1ae thru 0x015ce7 (0x6b3a bytes) 000170 1788919546.823382 mapwrite 0x03e11d thru 0x03ffff (0x1ee3 bytes) 000171 1788919546.852840 trunc from 0x040000 to 0x0342b9 (0xbd46 bytes) 000172 1788919546.865463 trunc from 0x0342ba to 0x036f44 (0x2c8b bytes) 000173 1788919546.883730 write 0x0365de thru 0x038420 (0x1e43 bytes) 000174 1788919546.889540 mapwrite 0x00dc97 thru 0x0168cd (0x8c37 bytes) 000175 1788919546.922043 read 0x028bed thru 0x02d1db (0x45ef bytes) 000176 1788919546.927800 write 0x03424f thru 0x039a47 (0x57f9 bytes) 000177 1788919546.928691 punch from 0x035cc1 to 0x038ef6 (0x3236 bytes) 000178 1788919546.938285 trunc from 0x011e97 to 0x01b82f (0x9999 bytes) 000179 1788919546.942311 trunc from 0x026c7c to 0x0332dd (0xc662 bytes) 000180 1788919546.946575 mapread 0x01f03f thru 0x02cf1f (0xdee1 bytes) 000181 1788919546.963128 punch from 0x0347e2 to 0x03552c (0xd4b bytes) 000182 1788919546.966723 mapread 0x0025b4 thru 0x002938 (0x385 bytes) 000183 1788919546.976059 MIRROR_RESYNC 3 mirrors 000184 1788919547.532721 write 0x03384c thru 0x0384cd (0x4c82 bytes) 000185 1788919547.557847 punch from 0x015c04 to 0x01bb48 (0x5f45 bytes) 000186 1788919547.561977 fallocate 0x0288a5 thru 0x032434 (0x9b90 bytes) 000187 1788919547.565648 read 0x0084c1 thru 0x00fe76 (0x79b6 bytes) 000188 1788919547.571805 punch from 0x001bed to 0x001d18 (0x12c bytes) 000189 1788919547.575680 trunc from 0x039a48 to 0x006229 (0x3381e bytes) 000190 1788919547.589738 mapread 0x00078b thru 0x006229 (0x5a9f bytes) 000191 1788919547.604419 mapwrite 0x004014 thru 0x0061ab (0x2198 bytes) 000192 1788919547.621316 trunc from 0x00622a to 0x00ad74 (0x4b4b bytes) 000193 1788919547.630513 punch from 0x001d58 to 0x00ad74 (0x901d bytes) 000194 1788919547.638332 punch from 0x000b5b to 0x00ad74 (0xa21a bytes) 000197 1788919547.643035 read 0x0010eb thru 0x006ea3 (0x5db9 bytes) 000198 1788919547.652000 write 0x02da6f thru 0x03ae33 (0xd3c5 bytes) 000199 1788919547.656485 trunc from 0x006ed2 to 0x00b6e5 (0x4814 bytes) 000200 1788919547.661005 trunc from 0x02bff5 to 0x0346f6 (0x8702 bytes) 000201 1788919547.669364 mapwrite 0x03d01f thru 0x03ffff (0x2fe1 bytes) 000202 1788919547.695984 punch from 0x03376a to 0x0378e7 (0x417e bytes) 000203 1788919547.699174 mapread 0x02e0ca thru 0x03a822 (0xc759 bytes) 000204 1788919547.718482 MIRROR_RESYNC 3 mirrors 000205 1788919548.155864 trunc from 0x02fa58 to 0x039968 (0x9f11 bytes) 000206 1788919548.182046 write 0x03e89a thru 0x03ffff (0x1766 bytes) 000207 1788919548.186103 write 0x037a98 thru 0x03ffff (0x8568 bytes) 000208 1788919548.190791 write 0x00f248 thru 0x012ba3 (0x395c bytes) 000209 1788919548.201376 mapread 0x01f886 thru 0x02633b (0x6ab6 bytes) 000210 1788919548.216555 punch from 0x0350a7 to 0x03b23a (0x6194 bytes) 000211 1788919548.234119 read 0x022c42 thru 0x02d3cc (0xa78b bytes) 000213 1788919548.241156 punch from 0x0133fe to 0x01d53b (0xa13e bytes) 000214 1788919548.247383 mapread 0x014577 thru 0x01a721 (0x61ab bytes) 000216 1788919548.256141 mapread 0x0388ea thru 0x03ffff (0x7716 bytes) 000217 1788919548.267542 mapread 0x02673f thru 0x02efda (0x889c bytes) 000218 1788919548.275046 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x25c0db62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x25c0db62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000219 1788919548.473876 trunc from 0x040000 to 0x02bc62 (0x1439d bytes) 000220 1788919548.497241 read 0x02346b thru 0x028603 (0x5199 bytes) 000221 1788919548.502644 mapwrite 0x03062b thru 0x03e81e (0xe1f4 bytes) 000222 1788919548.549570 read 0x032b01 thru 0x03456f (0x1a6f bytes) 000223 1788919548.555747 MIRROR_RESYNC 2 mirrors 000224 1788919548.928359 mapwrite 0x017188 thru 0x018bdb (0x1a54 bytes) 000225 1788919549.013424 punch from 0x000312 to 0x008409 (0x80f8 bytes) 000226 1788919549.018012 mapwrite 0x004ac2 thru 0x00dbbb (0x90fa bytes) 000227 1788919549.039924 fallocate 0x00ed38 thru 0x01deef (0xf1b8 bytes) 000229 1788919549.047436 mapwrite 0x00bcbf thru 0x015289 (0x95cb bytes) 000230 1788919549.068447 MIRROR_RESYNC 2 mirrors 000231 1788919549.502800 punch from 0x023fe2 to 0x0266ad (0x26cc bytes) 000233 1788919549.523633 mapread 0x02af05 thru 0x02ecee (0x3dea bytes) 000234 1788919549.542416 read 0x011345 thru 0x02117f (0xfe3b bytes) 000236 1788919549.549421 mapwrite 0x026daa thru 0x02cf34 (0x618b bytes) 000238 1788919549.561860 trunc from 0x03e81f to 0x035e1e (0x8a00 bytes) 000239 1788919549.590271 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) 000240 1788919549.717136 trunc from 0x0027a6 to 0x011d04 (0xf55f bytes) 000242 1788919549.725780 mapread 0x01e3ee thru 0x021513 (0x3126 bytes) 000243 1788919549.743682 write 0x017ff5 thru 0x021ba4 (0x9bb0 bytes) 000245 1788919549.762704 fallocate 0x0272ce thru 0x02d87b (0x65ae bytes) 000246 1788919549.772590 fallocate 0x002b23 thru 0x00aee0 (0x83be bytes) 000247 1788919549.781130 write 0x01e152 thru 0x0229c0 (0x486f bytes) 000249 1788919549.783388 mapread 0x027a68 thru 0x0318ab (0x9e44 bytes) 000250 1788919549.793517 mapread 0x02b836 thru 0x033a06 (0x81d1 bytes) 000251 1788919549.805813 mapread 0x01b7f9 thru 0x0235f3 (0x7dfb bytes) 000252 1788919549.806772 mapwrite 0x02dd1f thru 0x03d6f0 (0xf9d2 bytes) 000253 1788919549.851823 mapwrite 0x026c97 thru 0x032d8f (0xc0f9 bytes) 000254 1788919549.868931 fallocate 0x024d18 thru 0x0305ed (0xb8d6 bytes) 000255 1788919549.875654 MIRROR_EXTEND to 2 mirrors 000256 1788919550.388297 read 0x030694 thru 0x03d6f0 (0xd05d bytes) 000257 1788919550.408688 punch from 0x01c52b to 0x02979e (0xd274 bytes) 000259 1788919550.444660 write 0x017a83 thru 0x01d9f7 (0x5f75 bytes) 000260 1788919550.460178 read 0x01b25f thru 0x01cc0a (0x19ac bytes) 000261 1788919550.460757 mapread 0x02c26f thru 0x038aee (0xc880 bytes) 000262 1788919550.469739 write 0x038d0c thru 0x03ffff (0x72f4 bytes) 000263 1788919550.471045 mapread 0x01f1d5 thru 0x025cfa (0x6b26 bytes) 000264 1788919550.472128 trunc from 0x01f05e to 0x023b81 (0x4b24 bytes) 000265 1788919550.476124 punch from 0x0374a3 to 0x03f83e (0x839c bytes) 000266 1788919550.487140 mapread 0x026f4c thru 0x031acd (0xab82 bytes) 000268 1788919550.488746 trunc from 0x008147 to 0x00b5e3 (0x349d bytes) 000269 1788919550.495999 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) 000270 1788919550.569002 punch from 0x007d2d to 0x009e0f (0x20e3 bytes) 000271 1788919550.579109 MIRROR_EXTEND to 2 mirrors 000272 1788919551.167107 fallocate 0x006c8c thru 0x011937 (0xacac bytes) 000273 1788919551.186974 mapwrite 0x03e650 thru 0x03f62b (0xfdc bytes) 000274 1788919551.204107 fallocate 0x00fa8a thru 0x00fb12 (0x89 bytes) 000276 1788919551.207699 write 0x03c12c thru 0x03ffff (0x3ed4 bytes) 000277 1788919551.220495 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x90d52643 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1ddfb395 Mirror 3: 0x1ddfb395 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000278 1788919551.882084 read 0x011078 thru 0x01d9db (0xc964 bytes) 000279 1788919551.891798 trunc from 0x03ead1 to 0x03f8e7 (0xe17 bytes) 000280 1788919551.924005 write 0x00faf2 thru 0x01b0b3 (0xb5c2 bytes) 000281 1788919551.936885 read 0x02e447 thru 0x0319a8 (0x3562 bytes) 000282 1788919551.948954 trunc from 0x00c0f9 to 0x0178b6 (0xb7be bytes) 000283 1788919551.959054 trunc from 0x040000 to 0x00703a (0x38fc5 bytes) 000284 1788919552.010656 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x703b): Mirror 3: 0xe64eeaaf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x703b): Mirror 3: 0xe64eeaaf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x703b: skipped 000285 1788919552.256632 read 0x0018c7 thru 0x00703a (0x5774 bytes) 000286 1788919552.266141 mapwrite 0x02b576 thru 0x037074 (0xbaff bytes) 000287 1788919552.328747 mapread 0x02f79f thru 0x02f9ed (0x24f bytes) 000288 1788919552.339099 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37075): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37075): Mirror 3: 0x320a37f5 (stale) CRC-32 checksum value for chunk [0, 0x37075): Mirror 1: 0x4409dd14 Mirror 4: 0x4409dd14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37075: skipped 000289 1788919552.921506 fallocate 0x02f2b2 thru 0x03df01 (0xec50 bytes) 000290 1788919552.959271 fallocate 0x02ae81 thru 0x033e0c (0x8f8c bytes) 000292 1788919552.966922 write 0x001103 thru 0x00cd8b (0xbc89 bytes) 000293 1788919552.981906 trunc from 0x03df02 to 0x037944 (0x65bd bytes) 000294 1788919553.028442 mapwrite 0x03abfb thru 0x03ffff (0x5405 bytes) 000295 1788919553.088921 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1ddfb395 (stale) Mirror 4: 0xcc7c06e8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee9fe800 Mirror 5: 0xee9fe800 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000296 1788919553.744679 punch from 0x018422 to 0x0232c1 (0xaea0 bytes) 000297 1788919553.805633 read 0x0033c3 thru 0x00ec25 (0xb863 bytes) 000298 1788919553.822049 trunc from 0x019189 to 0x01e1c1 (0x5039 bytes) 000299 1788919553.830989 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xcc7c06e8 (stale) Mirror 5: 0xee9fe800 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xcc7c06e8 (stale) Mirror 5: 0xee9fe800 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000300 1788919554.060649 write 0x02a6dd thru 0x03448a (0x9dae bytes) 000301 1788919554.076483 mapwrite 0x01127a thru 0x01e60f (0xd396 bytes) 000302 1788919554.140568 punch from 0x038d1a to 0x03ffff (0x72e6 bytes) 000303 1788919554.144767 mapwrite 0x0302e0 thru 0x03ffff (0xfd20 bytes) 000305 1788919554.172695 mapwrite 0x01454e thru 0x01fab7 (0xb56a bytes) 000306 1788919554.196071 trunc from 0x040000 to 0x032afc (0xd503 bytes) 000307 1788919554.246873 trunc from 0x032afd to 0x03511c (0x2620 bytes) 000308 1788919554.269368 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3511d): Mirror 5: 0xf2857264 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3511d): Mirror 5: 0xf2857264 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3511d: skipped 000309 1788919554.522574 read 0x00b531 thru 0x00e6fc (0x31cc bytes) 000310 1788919554.529940 trunc from 0x02b572 to 0x033e7b (0x890a bytes) 000311 1788919554.535586 write 0x01ff74 thru 0x0220a7 (0x2134 bytes) 000312 1788919554.561962 read 0x02a589 thru 0x03511c (0xab94 bytes) 000313 1788919554.574935 trunc from 0x032146 to 0x03511c (0x2fd7 bytes) 000315 1788919554.582499 mapwrite 0x009c9b thru 0x00f773 (0x5ad9 bytes) 000316 1788919554.609618 write 0x01ed6c thru 0x023b93 (0x4e28 bytes) 000317 1788919554.610980 fallocate 0x01830d thru 0x01ebc4 (0x68b8 bytes) 000318 1788919554.617563 trunc from 0x026b79 to 0x02cca3 (0x612b bytes) 000319 1788919554.623164 trunc from 0x02c9fa to 0x03511c (0x8723 bytes) 000320 1788919554.629002 read 0x0022cc thru 0x01065a (0xe38f bytes) 000321 1788919554.629670 write 0x013f8b thru 0x01c85a (0x88d0 bytes) 000322 1788919554.630937 write 0x0247da thru 0x026501 (0x1d28 bytes) 000323 1788919554.638700 fallocate 0x00084d thru 0x001a3b (0x11ef bytes) 000324 1788919554.644049 trunc from 0x00ed96 to 0x017ce6 (0x8f51 bytes) 000325 1788919554.654641 read 0x0062a8 thru 0x00bc87 (0x59e0 bytes) 000326 1788919554.655314 punch from 0x01bffa to 0x025317 (0x931e bytes) 000327 1788919554.658983 write 0x00de09 thru 0x0193ed (0xb5e5 bytes) 000328 1788919554.660296 read 0x014caf thru 0x0182e7 (0x3639 bytes) 000329 1788919554.660667 mapwrite 0x03080b thru 0x037dd2 (0x75c8 bytes) 000330 1788919554.695291 mapwrite 0x03fd7c thru 0x03ffff (0x284 bytes) 000332 1788919554.735587 punch from 0x0012e2 to 0x00ead4 (0xd7f3 bytes) 000333 1788919554.739657 fallocate 0x03c0e8 thru 0x03ffff (0x3f18 bytes) 000334 1788919554.743320 mapwrite 0x01431a thru 0x01e7b8 (0xa49f bytes) 000335 1788919554.762120 trunc from 0x040000 to 0x02eb76 (0x11489 bytes) 000336 1788919554.771695 read 0x009fa4 thru 0x00b326 (0x1383 bytes) 000337 1788919554.779051 punch from 0x019f24 to 0x027410 (0xd4ed bytes) 000338 1788919554.788019 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2eb77): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb77): Mirror 5: 0x96d8d26b (stale) CRC-32 checksum value for chunk [0, 0x2eb77): Mirror 1: 0x40876304 Mirror 6: 0x40876304 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb77: skipped 000339 1788919555.433546 trunc from 0x02eb77 to 0x02c63c (0x253a bytes) 000340 1788919555.472839 trunc from 0x02c63d to 0x01ab62 (0x11ada bytes) 000341 1788919555.490068 write 0x0269eb thru 0x035761 (0xed77 bytes) 000342 1788919555.495481 mapread 0x031f7f thru 0x035761 (0x37e3 bytes) 000344 1788919555.500707 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35762): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35762): Mirror 5: 0x525c8c7 (stale) Mirror 6: 0xe0442b75 (stale) CRC-32 checksum value for chunk [0, 0x35762): Mirror 1: 0x3741d17b Mirror 7: 0x3741d17b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35762: skipped 000345 1788919556.109950 punch from 0x00f7fb to 0x010df4 (0x15fa bytes) 000346 1788919556.135118 trunc from 0x013b12 to 0x014642 (0xb31 bytes) 000347 1788919556.139303 punch from 0x02dc16 to 0x02dd1b (0x106 bytes) 000348 1788919556.144037 punch from 0x01fc5f to 0x02cbd6 (0xcf78 bytes) 000349 1788919556.149474 trunc from 0x035762 to 0x014e29 (0x20938 bytes) 000350 1788919556.183297 read 0x00954a thru 0x013842 (0xa2f9 bytes) 000351 1788919556.188035 mapread 0x013c0d thru 0x014e29 (0x121d bytes) 000352 1788919556.194486 mapread 0x00f420 thru 0x00fa89 (0x66a bytes) 000353 1788919556.195260 punch from 0x013291 to 0x014e29 (0x1b99 bytes) 000355 1788919556.204680 read 0x0102eb thru 0x014e29 (0x4b3f bytes) 000356 1788919556.225187 fallocate 0x02e238 thru 0x03d312 (0xf0db bytes) 000357 1788919556.231795 mapread 0x00d6bd thru 0x014e29 (0x776d bytes) 000358 1788919556.237563 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14e2a): Mirror 6: 0xcab54d58 (stale) Mirror 7: 0xcab54d58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14e2a): Mirror 6: 0xcab54d58 (stale) Mirror 7: 0xcab54d58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14e2a: skipped 000359 1788919556.452814 trunc from 0x014e2a to 0x038791 (0x23968 bytes) 000360 1788919556.475817 punch from 0x033a5d to 0x03838d (0x4931 bytes) 000361 1788919556.481004 write 0x03acf4 thru 0x03ffff (0x530c bytes) 000362 1788919556.482871 punch from 0x00e3a7 to 0x018632 (0xa28c bytes) 000363 1788919556.491743 fallocate 0x039e18 thru 0x03ffff (0x61e8 bytes) 000364 1788919556.499608 trunc from 0x040000 to 0x00ec91 (0x3136e bytes) 000365 1788919556.521044 write 0x01100b thru 0x01685c (0x5852 bytes) 000366 1788919556.522948 read 0x007631 thru 0x015a9f (0xe46f bytes) 000367 1788919556.529898 mapwrite 0x00770b thru 0x00f08d (0x7983 bytes) 000369 1788919556.550283 read 0x006224 thru 0x00aade (0x48bb bytes) 000370 1788919556.561666 read 0x00adf6 thru 0x00b3dc (0x5e7 bytes) 000371 1788919556.562292 trunc from 0x0149c9 to 0x01685c (0x1e94 bytes) 000372 1788919556.570492 MIRROR_RESYNC 3 mirrors 000374 1788919557.199649 write 0x018772 thru 0x01e52c (0x5dbb bytes) 000375 1788919557.222669 write 0x002fa4 thru 0x007994 (0x49f1 bytes) 000376 1788919557.232774 write 0x0359a2 thru 0x03ffff (0xa65e bytes) 000377 1788919557.234785 trunc from 0x040000 to 0x0012a6 (0x3ed59 bytes) 000378 1788919557.249369 mapwrite 0x0307a5 thru 0x035988 (0x51e4 bytes) 000379 1788919557.289252 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35989): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35989): Mirror 6: 0xacaa38fe (stale) Mirror 7: 0xacaa38fe (stale) CRC-32 checksum value for chunk [0, 0x35989): Mirror 1: 0xa23fe4c1 Mirror 8: 0xa23fe4c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35989: skipped 000381 1788919557.812034 trunc from 0x02ca6f to 0x035988 (0x8f1a bytes) 000382 1788919557.838932 fallocate 0x0352ff thru 0x03ffff (0xad01 bytes) 000383 1788919557.844987 mapwrite 0x00a1d7 thru 0x00d4b6 (0x32e0 bytes) 000384 1788919557.874258 punch from 0x007658 to 0x00d149 (0x5af2 bytes) 000386 1788919557.879894 trunc from 0x035989 to 0x007583 (0x2e405 bytes) 000387 1788919557.915410 mapread 0x0022b2 thru 0x007583 (0x52d2 bytes) 000388 1788919557.933621 mapwrite 0x02c71d thru 0x03a788 (0xe06c bytes) 000389 1788919557.978409 write 0x00799a thru 0x016566 (0xebcd bytes) 000390 1788919557.988684 punch from 0x02ecd6 to 0x034e12 (0x613d bytes) 000391 1788919558.003060 read 0x00aea1 thru 0x01872b (0xd88b bytes) 000392 1788919558.007774 write 0x037a05 thru 0x03ffff (0x85fb bytes) 000393 1788919558.009146 fallocate 0x0045ce thru 0x0108dc (0xc30f bytes) 000394 1788919558.015948 punch from 0x02e2e5 to 0x036fdb (0x8cf7 bytes) 000395 1788919558.022513 mapwrite 0x0232ef thru 0x029ab0 (0x67c2 bytes) 000396 1788919558.052735 trunc from 0x040000 to 0x02ca1c (0x135e3 bytes) 000397 1788919558.072144 fallocate 0x02f076 thru 0x035c81 (0x6c0c bytes) 000398 1788919558.081386 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35c82): Mirror 6: 0x594ad2eb (stale) Mirror 7: 0x594ad2eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c82): Mirror 6: 0x594ad2eb (stale) Mirror 7: 0x594ad2eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c82: skipped 000399 1788919558.297608 mapread 0x0044a1 thru 0x00d057 (0x8bb7 bytes) 000400 1788919558.318693 trunc from 0x024dfa to 0x02b1d4 (0x63db bytes) 000401 1788919558.325378 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35c82): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c82): Mirror 6: 0x594ad2eb (stale) Mirror 7: 0x594ad2eb (stale) CRC-32 checksum value for chunk [0, 0x35c82): Mirror 1: 0x78d6b37d Mirror 8: 0x78d6b37d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c82: skipped 000402 1788919558.924842 punch from 0x01ad0b to 0x023f03 (0x91f9 bytes) 000403 1788919558.958189 mapwrite 0x01f9db thru 0x02a186 (0xa7ac bytes) 000406 1788919558.995363 mapread 0x0202a3 thru 0x0225f7 (0x2355 bytes) 000407 1788919558.996332 mapread 0x005d2f thru 0x00653c (0x80e bytes) 000408 1788919559.002855 mapread 0x0353c9 thru 0x035c81 (0x8b9 bytes) 000409 1788919559.007775 punch from 0x034a6b to 0x035c81 (0x1217 bytes) 000410 1788919559.016257 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35c82): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c82): Mirror 6: 0x594ad2eb (stale) Mirror 7: 0x594ad2eb (stale) Mirror 8: 0x78d6b37d (stale) CRC-32 checksum value for chunk [0, 0x35c82): Mirror 1: 0x405cdc76 Mirror 9: 0x405cdc76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c82: skipped 000411 1788919559.582655 MIRROR_RESYNC 5 mirrors 000413 1788919560.259249 write 0x0128dc thru 0x0134b8 (0xbdd bytes) 000414 1788919560.292410 trunc from 0x02c7ed to 0x02e7be (0x1fd2 bytes) 000415 1788919560.296059 trunc from 0x035c82 to 0x0207a1 (0x154e0 bytes) 000417 1788919560.325560 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x207a2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x207a2): Mirror 6: 0xcde5dd0e (stale) Mirror 7: 0xcde5dd0e (stale) Mirror 8: 0xcde5dd0e (stale) Mirror 9: 0xcde5dd0e (stale) CRC-32 checksum value for chunk [0, 0x207a2): Mirror 1: 0x258642dc Mirror 10: 0x258642dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x207a2: skipped 000419 1788919560.963641 punch from 0x01af1b to 0x0207a1 (0x5887 bytes) 000420 1788919560.993077 mapwrite 0x007b74 thru 0x0156b4 (0xdb41 bytes) 000421 1788919561.056185 punch from 0x015716 to 0x0207a1 (0xb08c bytes) 000422 1788919561.060519 trunc from 0x0207a2 to 0x01a622 (0x617f bytes) 000423 1788919561.082667 MIRROR_RESYNC 6 mirrors 000424 1788919561.785915 write 0x03343d thru 0x0345e1 (0x11a5 bytes) 000425 1788919561.809561 write 0x0132b3 thru 0x017710 (0x445e bytes) 000426 1788919561.818073 mapwrite 0x03fc0b thru 0x03ffff (0x3f5 bytes) 000427 1788919561.856852 mapwrite 0x019ed5 thru 0x0264ce (0xc5fa bytes) 000428 1788919561.894421 punch from 0x00fa67 to 0x012f50 (0x34ea bytes) 000430 1788919561.901570 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe8cfd9e2 (stale) Mirror 7: 0xe8cfd9e2 (stale) Mirror 8: 0xe8cfd9e2 (stale) Mirror 9: 0xe8cfd9e2 (stale) Mirror 10: 0xe8cfd9e2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x99ebb47f Mirror 11: 0x99ebb47f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000431 1788919562.538994 fallocate 0x02753e thru 0x03539c (0xde5f bytes) 000432 1788919562.563495 trunc from 0x040000 to 0x00657e (0x39a81 bytes) 000434 1788919562.584359 trunc from 0x001732 to 0x00657e (0x4e4d bytes) 000435 1788919562.589032 trunc from 0x004ed2 to 0x00657e (0x16ad bytes) 000436 1788919562.592461 punch from 0x001315 to 0x00657e (0x526a bytes) 000437 1788919562.595902 mapread 0x004c3c thru 0x00657e (0x1943 bytes) 000438 1788919562.608863 mapread 0x002ed4 thru 0x003960 (0xa8d bytes) 000439 1788919562.613631 MIRROR_RESYNC 7 mirrors 000440 1788919563.259735 trunc from 0x002d80 to 0x00657e (0x37ff bytes) 000441 1788919563.282877 trunc from 0x00096c to 0x00657e (0x5c13 bytes) 000442 1788919563.287406 trunc from 0x00657f to 0x032ee1 (0x2c963 bytes) 000443 1788919563.303468 trunc from 0x01d1fb to 0x01ebe3 (0x19e9 bytes) 000444 1788919563.306590 fallocate 0x01fbd7 thru 0x028dbe (0x91e8 bytes) 000445 1788919563.311658 read 0x01675e thru 0x01ee68 (0x870b bytes) 000446 1788919563.318603 trunc from 0x032ee2 to 0x03f22d (0xc34c bytes) 000447 1788919563.330012 fallocate 0x01b7e2 thru 0x01ec3a (0x3459 bytes) 000448 1788919563.333047 trunc from 0x00ec3d to 0x013761 (0x4b25 bytes) 000449 1788919563.335742 mapread 0x016f39 thru 0x02037e (0x9446 bytes) 000450 1788919563.345394 read 0x01eb6c thru 0x021edc (0x3371 bytes) 000451 1788919563.360018 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f22e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f22e): Mirror 6: 0xfa333ecc (stale) Mirror 7: 0xfa333ecc (stale) Mirror 8: 0xfa333ecc (stale) Mirror 9: 0xfa333ecc (stale) Mirror 10: 0xfa333ecc (stale) Mirror 11: 0xfa333ecc (stale) CRC-32 checksum value for chunk [0, 0x3f22e): Mirror 1: 0x96fdad9c Mirror 12: 0x96fdad9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f22e: skipped 000452 1788919563.825062 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f22e): Mirror 6: 0xfa333ecc (stale) Mirror 7: 0xfa333ecc (stale) Mirror 8: 0xfa333ecc (stale) Mirror 9: 0xfa333ecc (stale) Mirror 11: 0xfa333ecc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f22e): Mirror 6: 0xfa333ecc (stale) Mirror 7: 0xfa333ecc (stale) Mirror 8: 0xfa333ecc (stale) Mirror 9: 0xfa333ecc (stale) Mirror 11: 0xfa333ecc (stale) CRC-32 checksum value for chunk [0, 0x3f22e): Mirror 1: 0x96fdad9c Mirror 12: 0x96fdad9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f22e: skipped 000453 1788919564.270256 trunc from 0x007865 to 0x01723d (0xf9d9 bytes) 000454 1788919564.293466 mapread 0x00bcf5 thru 0x018b1e (0xce2a bytes) 000456 1788919564.312466 fallocate 0x02c911 thru 0x037069 (0xa759 bytes) 000457 1788919564.318372 read 0x02cdfc thru 0x032b83 (0x5d88 bytes) 000458 1788919564.327017 write 0x039a11 thru 0x03bf64 (0x2554 bytes) 000459 1788919564.336028 trunc from 0x03f22e to 0x00b3e6 (0x33e47 bytes) 000460 1788919564.358847 mapread 0x0098e8 thru 0x00b3e6 (0x1aff bytes) 000461 1788919564.359774 mapwrite 0x03c32c thru 0x03ffff (0x3cd4 bytes) 000462 1788919564.387171 trunc from 0x040000 to 0x027b18 (0x184e7 bytes) 000463 1788919564.399272 trunc from 0x014e10 to 0x01d9b4 (0x8ba5 bytes) 000464 1788919564.405465 write 0x009ebb thru 0x00ff5c (0x60a2 bytes) 000465 1788919564.409341 trunc from 0x009702 to 0x011b5c (0x845b bytes) 000466 1788919564.420424 trunc from 0x00f7cb to 0x012ca5 (0x34db bytes) 000467 1788919564.424129 fallocate 0x00475a thru 0x00d6fb (0x8fa2 bytes) 000468 1788919564.428958 trunc from 0x027b19 to 0x021841 (0x62d7 bytes) 000470 1788919564.440865 write 0x02bae7 thru 0x02cfc8 (0x14e2 bytes) 000471 1788919564.441856 trunc from 0x02cfc9 to 0x0207f6 (0xc7d2 bytes) 000472 1788919564.453308 punch from 0x00fdc0 to 0x012aef (0x2d30 bytes) 000473 1788919564.456671 mapwrite 0x007186 thru 0x00cd09 (0x5b84 bytes) 000474 1788919564.475994 mapread 0x01f309 thru 0x0207f6 (0x14ee bytes) 000475 1788919564.482604 trunc from 0x0207f7 to 0x038078 (0x17882 bytes) 000477 1788919564.494045 fallocate 0x0223af thru 0x02c084 (0x9cd6 bytes) 000478 1788919564.498059 trunc from 0x038079 to 0x01fe36 (0x18242 bytes) 000479 1788919564.506949 trunc from 0x012e88 to 0x01873f (0x58b8 bytes) 000480 1788919564.511304 fallocate 0x002fc4 thru 0x004f71 (0x1fae bytes) 000481 1788919564.519577 MIRROR_RESYNC 7 mirrors 000482 1788919565.360526 MIRROR_EXTEND to 8 mirrors 000483 1788919565.797560 fallocate 0x03eee2 thru 0x03ffff (0x111e bytes) 000484 1788919565.827301 mapwrite 0x032bf6 thru 0x039856 (0x6c61 bytes) 000485 1788919565.872117 write 0x01da1c thru 0x02457e (0x6b63 bytes) 000486 1788919565.878334 fallocate 0x026872 thru 0x029111 (0x28a0 bytes) 000487 1788919565.882122 trunc from 0x039857 to 0x011a18 (0x27e3e bytes) 000489 1788919565.892213 trunc from 0x00b229 to 0x011a18 (0x67f0 bytes) 000490 1788919565.894709 fallocate 0x02ddbf thru 0x030194 (0x23d6 bytes) 000491 1788919565.897121 punch from 0x017403 to 0x022f93 (0xbb91 bytes) 000492 1788919565.899085 fallocate 0x03c8e4 thru 0x03f8be (0x2fdb bytes) 000493 1788919565.903610 punch from 0x008a2c to 0x017844 (0xee19 bytes) 000494 1788919565.906382 punch from 0x02272f to 0x02b167 (0x8a39 bytes) 000496 1788919565.910798 punch from 0x03c1d2 to 0x03f8be (0x36ed bytes) 000497 1788919565.913280 write 0x039685 thru 0x03ffff (0x697b bytes) 000498 1788919565.917443 punch from 0x02f878 to 0x0385ef (0x8d78 bytes) 000499 1788919565.921951 fallocate 0x024f22 thru 0x0309f8 (0xbad7 bytes) 000500 1788919565.926488 read 0x01cf6c thru 0x028608 (0xb69d bytes) 000501 1788919565.937952 read 0x00a137 thru 0x01864e (0xe518 bytes) 000502 1788919565.947394 mapwrite 0x03dd0b thru 0x03ffff (0x22f5 bytes) 000503 1788919565.962826 punch from 0x030454 to 0x030e36 (0x9e3 bytes) 000504 1788919565.966524 read 0x007d2c thru 0x00c86d (0x4b42 bytes) 000505 1788919565.969842 MIRROR_RESYNC 8 mirrors 000506 1788919566.880267 read 0x03455f thru 0x03982e (0x52d0 bytes) 000507 1788919566.885989 punch from 0x023da0 to 0x02406e (0x2cf bytes) 000508 1788919566.909759 read 0x017d7f thru 0x026273 (0xe4f5 bytes) 000512 1788919566.918726 mapwrite 0x01d1fa thru 0x02cec5 (0xfccc bytes) 000513 1788919566.932656 trunc from 0x040000 to 0x004fda (0x3b025 bytes) 000514 1788919566.952383 write 0x03f2af thru 0x03ffff (0xd51 bytes) 000515 1788919566.953704 trunc from 0x040000 to 0x026850 (0x197af bytes) 000516 1788919566.966521 punch from 0x011048 to 0x01d525 (0xc4de bytes) 000517 1788919566.970495 fallocate 0x0179e8 thru 0x024629 (0xcc42 bytes) 000518 1788919566.975254 mapwrite 0x036792 thru 0x03a3c0 (0x3c2f bytes) 000519 1788919567.015736 mapread 0x014efe thru 0x01e5c9 (0x96cc bytes) 000520 1788919567.045664 read 0x012046 thru 0x01e4af (0xc46a bytes) 000521 1788919567.053165 fallocate 0x023aed thru 0x0287fd (0x4d11 bytes) 000522 1788919567.059480 trunc from 0x03a3c1 to 0x03a0c4 (0x2fc bytes) 000523 1788919567.067879 trunc from 0x02fa88 to 0x02feff (0x478 bytes) 000525 1788919567.072954 fallocate 0x03a508 thru 0x03baf3 (0x15ec bytes) 000526 1788919567.078365 punch from 0x015235 to 0x019c06 (0x49d2 bytes) 000527 1788919567.083452 punch from 0x006295 to 0x01503c (0xeda8 bytes) 000528 1788919567.087459 fallocate 0x021aab thru 0x030cb2 (0xf208 bytes) 000529 1788919567.092357 mapwrite 0x0205d3 thru 0x028eee (0x891c bytes) 000530 1788919567.107915 read 0x036bc0 thru 0x03baf3 (0x4f34 bytes) 000531 1788919567.124106 MIRROR_RESYNC 8 mirrors 000532 1788919568.090041 MIRROR_RESYNC 8 mirrors 000533 1788919568.350922 read 0x007d4f thru 0x00facc (0x7d7e bytes) 000534 1788919568.357967 fallocate 0x02b5c9 thru 0x02de77 (0x28af bytes) 000535 1788919568.381404 MIRROR_RESYNC 8 mirrors 000536 1788919569.260160 mapwrite 0x030b20 thru 0x036eab (0x638c bytes) 000537 1788919569.319694 write 0x0374a1 thru 0x038349 (0xea9 bytes) 000538 1788919569.320713 punch from 0x017185 to 0x018377 (0x11f3 bytes) 000539 1788919569.326910 fallocate 0x02a6c1 thru 0x02b5d3 (0xf13 bytes) 000540 1788919569.332260 fallocate 0x0202cb thru 0x0225a6 (0x22dc bytes) 000541 1788919569.336230 trunc from 0x03baf4 to 0x0195d5 (0x2251e bytes) 000542 1788919569.346728 mapwrite 0x014bcd thru 0x0182dd (0x3711 bytes) 000543 1788919569.361793 mapread 0x019468 thru 0x0195d5 (0x16e bytes) 000544 1788919569.362645 trunc from 0x0195d6 to 0x01ec29 (0x5654 bytes) 000545 1788919569.376810 mapread 0x00f196 thru 0x018d94 (0x9bff bytes) 000546 1788919569.377894 mapwrite 0x0036d6 thru 0x010277 (0xcba2 bytes) 000547 1788919569.398237 write 0x03dd74 thru 0x03ffff (0x228c bytes) 000548 1788919569.399293 mapread 0x029e77 thru 0x02fb01 (0x5c8b bytes) 000549 1788919569.405707 mapwrite 0x00f32e thru 0x014888 (0x555b bytes) 000550 1788919569.416502 trunc from 0x040000 to 0x01f46c (0x20b93 bytes) 000551 1788919569.426905 write 0x017f3b thru 0x0203ab (0x8471 bytes) 000552 1788919569.429321 write 0x01e73e thru 0x025c82 (0x7545 bytes) 000553 1788919569.430949 mapwrite 0x032c41 thru 0x03702a (0x43ea bytes) 000554 1788919569.462438 punch from 0x016ca6 to 0x0186fc (0x1a57 bytes) 000556 1788919569.466371 trunc from 0x03702b to 0x01088c (0x2679e bytes) 000557 1788919569.475560 write 0x03313c thru 0x03e864 (0xb729 bytes) 000558 1788919569.477509 trunc from 0x03e865 to 0x0190fd (0x25767 bytes) 000559 1788919569.489491 MIRROR_RESYNC 8 mirrors 000560 1788919570.213655 trunc from 0x00838d to 0x016956 (0xe5ca bytes) 000561 1788919570.230719 mapread 0x00d08f thru 0x011e37 (0x4da9 bytes) 000562 1788919570.238225 punch from 0x00475a to 0x00bb7e (0x7425 bytes) 000563 1788919570.240955 punch from 0x00159d to 0x0086a1 (0x7105 bytes) 000564 1788919570.243616 mapwrite 0x00898d thru 0x008ba4 (0x218 bytes) 000565 1788919570.259932 mapwrite 0x0081dc thru 0x00eea5 (0x6cca bytes) 000566 1788919570.276906 fallocate 0x024bb6 thru 0x033647 (0xea92 bytes) 000567 1788919570.285521 MIRROR_RESYNC 8 mirrors 000568 1788919571.017308 mapwrite 0x032cdf thru 0x03b6b1 (0x89d3 bytes) 000569 1788919571.074438 trunc from 0x035244 to 0x0353a7 (0x164 bytes) 000570 1788919571.077269 read 0x034754 thru 0x03b6b1 (0x6f5e bytes) 000572 1788919571.083020 trunc from 0x01b264 to 0x01e0e3 (0x2e80 bytes) 000573 1788919571.092157 punch from 0x0231c4 to 0x028232 (0x506f bytes) 000574 1788919571.096177 trunc from 0x007cb3 to 0x00af82 (0x32d0 bytes) 000575 1788919571.106152 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b6b2): Mirror 7: 0x190b495d (stale) Mirror 8: 0x190b495d (stale) Mirror 9: 0x190b495d (stale) Mirror 11: 0x190b495d (stale) Mirror 12: 0x190b495d (stale) Mirror 13: 0x190b495d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6b2): Mirror 7: 0x190b495d (stale) Mirror 8: 0x190b495d (stale) Mirror 9: 0x190b495d (stale) Mirror 11: 0x190b495d (stale) Mirror 12: 0x190b495d (stale) Mirror 13: 0x190b495d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6b2: skipped 000576 1788919571.332559 read 0x00dbbd thru 0x017cc7 (0xa10b bytes) 000578 1788919571.338410 trunc from 0x03b6b2 to 0x018ba6 (0x22b0b bytes) 000579 1788919571.347377 mapread 0x0032eb thru 0x0074ac (0x41c2 bytes) 000580 1788919571.355454 fallocate 0x00d7e1 thru 0x01500f (0x782f bytes) 000581 1788919571.358254 fallocate 0x02d390 thru 0x0355d5 (0x8246 bytes) 000582 1788919571.361163 mapwrite 0x006988 thru 0x00fa1e (0x9097 bytes) 000583 1788919571.372099 write 0x0009c7 thru 0x00d2c1 (0xc8fb bytes) 000584 1788919571.375688 write 0x016a60 thru 0x02620e (0xf7af bytes) 000585 1788919571.377094 fallocate 0x014369 thru 0x01537c (0x1014 bytes) 000586 1788919571.380087 read 0x00d8a9 thru 0x00e7eb (0xf43 bytes) 000587 1788919571.380806 write 0x031ca5 thru 0x0323df (0x73b bytes) 000589 1788919571.385398 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x323e0): Mirror 8: 0x164c7708 (stale) Mirror 9: 0x164c7708 (stale) Mirror 11: 0x164c7708 (stale) Mirror 12: 0x164c7708 (stale) Mirror 13: 0x164c7708 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323e0): Mirror 8: 0x164c7708 (stale) Mirror 9: 0x164c7708 (stale) Mirror 11: 0x164c7708 (stale) Mirror 12: 0x164c7708 (stale) Mirror 13: 0x164c7708 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323e0: skipped 000590 1788919571.613535 trunc from 0x02dd59 to 0x0323df (0x4687 bytes) 000591 1788919571.616109 trunc from 0x00c7ff to 0x0161a4 (0x99a6 bytes) 000592 1788919571.618954 mapwrite 0x03b9cd thru 0x03ffff (0x4633 bytes) 000593 1788919571.650890 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xced46703 (stale) Mirror 9: 0xced46703 (stale) Mirror 11: 0xced46703 (stale) Mirror 12: 0xced46703 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xced46703 (stale) Mirror 9: 0xced46703 (stale) Mirror 11: 0xced46703 (stale) Mirror 12: 0xced46703 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000595 1788919571.878011 trunc from 0x03481b to 0x03e771 (0x9f57 bytes) 000596 1788919571.882869 write 0x02b75a thru 0x02b9dc (0x283 bytes) 000597 1788919571.893994 punch from 0x00201c to 0x009ef5 (0x7eda bytes) 000598 1788919571.902499 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xced46703 (stale) Mirror 11: 0xced46703 (stale) Mirror 12: 0xced46703 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xced46703 (stale) Mirror 11: 0xced46703 (stale) Mirror 12: 0xced46703 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000599 1788919572.088288 trunc from 0x02e5e0 to 0x03ca1d (0xe43e bytes) 000600 1788919572.090895 write 0x0278d5 thru 0x02947e (0x1baa bytes) 000601 1788919572.101651 fallocate 0x02b951 thru 0x03695a (0xb00a bytes) 000603 1788919572.105412 trunc from 0x040000 to 0x015b93 (0x2a46c bytes) 000605 1788919572.114897 read 0x000399 thru 0x00a2ae (0x9f16 bytes) 000606 1788919572.120143 fallocate 0x002740 thru 0x00e7f9 (0xc0ba bytes) 000607 1788919572.124973 punch from 0x00a517 to 0x015b93 (0xb67d bytes) 000608 1788919572.128460 punch from 0x0021c9 to 0x0092cf (0x7107 bytes) 000609 1788919572.132593 punch from 0x013a1c to 0x015b93 (0x2178 bytes) 000610 1788919572.138179 read 0x01526f thru 0x015b93 (0x925 bytes) 000611 1788919572.147652 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x15b94): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15b94): Mirror 8: 0xcb6a3758 (stale) Mirror 11: 0xcb6a3758 (stale) Mirror 12: 0xcb6a3758 (stale) CRC-32 checksum value for chunk [0, 0x15b94): Mirror 1: 0x915fd8d2 Mirror 13: 0x915fd8d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15b94: skipped 000612 1788919572.607145 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15b94): Mirror 8: 0xcb6a3758 (stale) Mirror 11: 0xcb6a3758 (stale) Mirror 12: 0xcb6a3758 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15b94): Mirror 8: 0xcb6a3758 (stale) Mirror 11: 0xcb6a3758 (stale) Mirror 12: 0xcb6a3758 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15b94: skipped 000613 1788919572.798426 read 0x009a9b thru 0x00bad0 (0x2036 bytes) 000614 1788919572.803093 trunc from 0x015b94 to 0x03d414 (0x27881 bytes) 000615 1788919572.833022 MIRROR_RESYNC 4 mirrors 000619 1788919573.383305 trunc from 0x02db5f to 0x035189 (0x762b bytes) 000620 1788919573.404925 mapread 0x00c65b thru 0x01c487 (0xfe2d bytes) 000622 1788919573.426538 fallocate 0x029211 thru 0x031944 (0x8734 bytes) 000623 1788919573.434534 write 0x03b4fe thru 0x03ffff (0x4b02 bytes) 000624 1788919573.438605 mapwrite 0x0297b6 thru 0x02f0cd (0x5918 bytes) 000625 1788919573.460410 read 0x004046 thru 0x00e94f (0xa90a bytes) 000626 1788919573.466493 mapwrite 0x02deaf thru 0x0317c0 (0x3912 bytes) 000627 1788919573.477382 mapwrite 0x0102ba thru 0x01df4e (0xdc95 bytes) 000628 1788919573.489865 mapread 0x0250c9 thru 0x030569 (0xb4a1 bytes) 000631 1788919573.493133 fallocate 0x02ff4a thru 0x036aea (0x6ba1 bytes) 000632 1788919573.504989 MIRROR_RESYNC 4 mirrors 000633 1788919574.278631 fallocate 0x022785 thru 0x02a10f (0x798b bytes) 000634 1788919574.300471 write 0x00c95a thru 0x01092d (0x3fd4 bytes) 000635 1788919574.311446 mapwrite 0x01ad64 thru 0x021c6d (0x6f0a bytes) 000636 1788919574.347155 mapread 0x03b133 thru 0x03ffff (0x4ecd bytes) 000637 1788919574.359167 punch from 0x011c9d to 0x0142ac (0x2610 bytes) 000638 1788919574.364658 trunc from 0x040000 to 0x00b5cb (0x34a34 bytes) 000639 1788919574.379150 read 0x0002dc thru 0x00b5cb (0xb2f0 bytes) 000640 1788919574.386583 trunc from 0x003487 to 0x00b5cb (0x8145 bytes) 000641 1788919574.390635 read 0x001057 thru 0x0020cf (0x1079 bytes) 000642 1788919574.391289 punch from 0x007a87 to 0x00b5cb (0x3b45 bytes) 000644 1788919574.399907 trunc from 0x00951a to 0x00b5cb (0x20b2 bytes) 000645 1788919574.405674 write 0x0383a6 thru 0x03ffff (0x7c5a bytes) 000646 1788919574.407079 trunc from 0x006bb9 to 0x00d341 (0x6789 bytes) 000647 1788919574.410904 read 0x01ff71 thru 0x021f02 (0x1f92 bytes) 000648 1788919574.420801 read 0x00d6bd thru 0x013527 (0x5e6b bytes) 000649 1788919574.421740 trunc from 0x03f855 to 0x03ffff (0x7ab bytes) 000650 1788919574.434274 trunc from 0x008861 to 0x011f21 (0x96c1 bytes) 000651 1788919574.440046 trunc from 0x00466c to 0x00533a (0xccf bytes) 000652 1788919574.445868 mapwrite 0x032db1 thru 0x038b3d (0x5d8d bytes) 000653 1788919574.464135 trunc from 0x01e3ec to 0x02d881 (0xf496 bytes) 000655 1788919574.468624 trunc from 0x040000 to 0x02426d (0x1bd92 bytes) 000656 1788919574.486505 mapread 0x00b7cc thru 0x0176c4 (0xbef9 bytes) 000657 1788919574.494941 fallocate 0x013259 thru 0x01db77 (0xa91f bytes) 000658 1788919574.499178 mapwrite 0x025d37 thru 0x02b9da (0x5ca4 bytes) 000659 1788919574.526743 read 0x026954 thru 0x02909c (0x2749 bytes) 000660 1788919574.527402 trunc from 0x02b9db to 0x034200 (0x8826 bytes) 000661 1788919574.538274 mapread 0x00ed74 thru 0x014f48 (0x61d5 bytes) 000662 1788919574.539183 trunc from 0x034201 to 0x00f365 (0x24e9b bytes) 000663 1788919574.551752 trunc from 0x008724 to 0x00f365 (0x6c42 bytes) 000665 1788919574.554942 mapread 0x0059a1 thru 0x00d895 (0x7ef5 bytes) 000666 1788919574.560929 punch from 0x0070e9 to 0x00f365 (0x827d bytes) 000667 1788919574.564020 trunc from 0x00f366 to 0x010830 (0x14cb bytes) 000669 1788919574.578376 mapread 0x009845 thru 0x00eef6 (0x56b2 bytes) 000670 1788919574.609525 punch from 0x00b861 to 0x010830 (0x4fd0 bytes) 000671 1788919574.617635 read 0x0041e4 thru 0x00e736 (0xa553 bytes) 000672 1788919574.623922 trunc from 0x010831 to 0x03e6dd (0x2dead bytes) 000673 1788919574.638462 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e6de): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6de): Mirror 8: 0x4835587e (stale) Mirror 11: 0x4835587e (stale) Mirror 12: 0x4835587e (stale) CRC-32 checksum value for chunk [0, 0x3e6de): Mirror 1: 0xa8e669cd Mirror 13: 0xa8e669cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6de: skipped 000674 1788919575.067443 write 0x006e2f thru 0x00a80b (0x39dd bytes) 000675 1788919575.105900 read 0x009b38 thru 0x010828 (0x6cf1 bytes) 000676 1788919575.114956 read 0x00367e thru 0x00640a (0x2d8d bytes) 000677 1788919575.122952 trunc from 0x03e6de to 0x02ae07 (0x138d6 bytes) 000679 1788919575.146935 mapread 0x022ece thru 0x02ae07 (0x7f3a bytes) 000680 1788919575.147671 punch from 0x001901 to 0x00c771 (0xae71 bytes) 000681 1788919575.159097 mapwrite 0x02ad07 thru 0x03339a (0x8694 bytes) 000682 1788919575.213346 punch from 0x01ba04 to 0x01d29f (0x189c bytes) 000683 1788919575.217765 punch from 0x014a12 to 0x016c88 (0x2277 bytes) 000684 1788919575.223373 trunc from 0x03339b to 0x00dba2 (0x257f8 bytes) 000685 1788919575.232496 write 0x00f8e3 thru 0x0106d9 (0xdf7 bytes) 000686 1788919575.233899 mapread 0x0082f6 thru 0x0106d9 (0x83e4 bytes) 000687 1788919575.258971 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x106da): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x106da): Mirror 8: 0xabbe87b4 (stale) Mirror 11: 0xabbe87b4 (stale) Mirror 12: 0xabbe87b4 (stale) Mirror 13: 0x76ce1c7f (stale) CRC-32 checksum value for chunk [0, 0x106da): Mirror 1: 0x7b7e5ee Mirror 14: 0x7b7e5ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x106da: skipped 000688 1788919575.626875 punch from 0x0042f4 to 0x00d50d (0x921a bytes) 000689 1788919575.644684 mapwrite 0x021be7 thru 0x02d904 (0xbd1e bytes) 000691 1788919575.694584 punch from 0x00e41e to 0x00ea31 (0x614 bytes) 000692 1788919575.699540 punch from 0x01e4d1 to 0x02b885 (0xd3b5 bytes) 000693 1788919575.706533 punch from 0x00eeef to 0x01cf1a (0xe02c bytes) 000694 1788919575.711834 mapwrite 0x03a485 thru 0x03ffff (0x5b7b bytes) 000695 1788919575.748015 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xa1145b0d (stale) Mirror 12: 0xa1145b0d (stale) Mirror 13: 0x4ee23bbc (stale) Mirror 14: 0x1b959c16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xa1145b0d (stale) Mirror 12: 0xa1145b0d (stale) Mirror 13: 0x4ee23bbc (stale) Mirror 14: 0x1b959c16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000696 1788919575.944430 read 0x0340fa thru 0x036c3c (0x2b43 bytes) 000697 1788919575.947453 fallocate 0x015e45 thru 0x01e64e (0x880a bytes) 000698 1788919575.950561 punch from 0x023221 to 0x02e535 (0xb315 bytes) 000699 1788919575.953245 trunc from 0x0170c1 to 0x0237c9 (0xc709 bytes) 000700 1788919575.956299 mapread 0x024020 thru 0x02cfb9 (0x8f9a bytes) 000701 1788919575.971899 read 0x01eda4 thru 0x023b19 (0x4d76 bytes) 000702 1788919575.978480 read 0x030223 thru 0x03f9ce (0xf7ac bytes) 000703 1788919575.979059 trunc from 0x040000 to 0x01372c (0x2c8d3 bytes) 000704 1788919575.989397 punch from 0x00ad09 to 0x01372c (0x8a24 bytes) 000705 1788919575.992245 fallocate 0x000da9 thru 0x00e4c2 (0xd71a bytes) 000706 1788919575.995523 write 0x02448c thru 0x0265be (0x2133 bytes) 000707 1788919575.996552 write 0x007b5b thru 0x014530 (0xc9d6 bytes) 000708 1788919576.007479 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x265bf): Mirror 12: 0xa04e23a3 (stale) Mirror 13: 0x88d85c3c (stale) Mirror 14: 0xd1979bf7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x265bf): Mirror 12: 0xa04e23a3 (stale) Mirror 13: 0x88d85c3c (stale) Mirror 14: 0xd1979bf7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x265bf: skipped 000709 1788919576.192323 punch from 0x024dad to 0x0265be (0x1812 bytes) 000710 1788919576.195176 trunc from 0x023052 to 0x0265be (0x356d bytes) 000711 1788919576.199399 mapread 0x004499 thru 0x013167 (0xeccf bytes) 000712 1788919576.215518 fallocate 0x011a9e thru 0x011e24 (0x387 bytes) 000713 1788919576.218626 punch from 0x0261b8 to 0x0265be (0x407 bytes) 000715 1788919576.224399 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x265bf): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x265bf): Mirror 12: 0xa04e23a3 (stale) Mirror 13: 0x88d85c3c (stale) Mirror 14: 0xd1979bf7 (stale) CRC-32 checksum value for chunk [0, 0x265bf): Mirror 1: 0xa7677838 Mirror 15: 0xa7677838 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x265bf: skipped 000716 1788919576.638508 read 0x0121aa thru 0x018f24 (0x6d7b bytes) 000717 1788919576.649104 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x265bf): Mirror 12: 0xa04e23a3 (stale) Mirror 13: 0x88d85c3c (stale) Mirror 14: 0xd1979bf7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x265bf): Mirror 12: 0xa04e23a3 (stale) Mirror 13: 0x88d85c3c (stale) Mirror 14: 0xd1979bf7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x265bf: skipped 000718 1788919576.834337 write 0x032f89 thru 0x03ffff (0xd077 bytes) 000719 1788919576.843938 read 0x02bf2e thru 0x036a38 (0xab0b bytes) 000720 1788919576.852985 mapwrite 0x02eee7 thru 0x03b19a (0xc2b4 bytes) 000723 1788919576.867491 punch from 0x002b81 to 0x0119bf (0xee3f bytes) 000724 1788919576.870377 punch from 0x038d75 to 0x03ffff (0x728b bytes) 000725 1788919576.874648 mapwrite 0x02750a thru 0x02e4f8 (0x6fef bytes) 000726 1788919576.899588 mapwrite 0x004667 thru 0x00ea86 (0xa420 bytes) 000727 1788919576.916872 mapread 0x0139d4 thru 0x0209bb (0xcfe8 bytes) 000728 1788919576.927193 write 0x02f458 thru 0x032658 (0x3201 bytes) 000729 1788919576.930835 MIRROR_RESYNC 4 mirrors 000730 1788919577.554581 trunc from 0x040000 to 0x03af9b (0x5064 bytes) 000731 1788919577.577298 read 0x008e90 thru 0x0172f2 (0xe463 bytes) 000732 1788919577.581981 read 0x034820 thru 0x03af9b (0x677c bytes) 000733 1788919577.587977 read 0x0043b4 thru 0x0133f6 (0xf043 bytes) 000734 1788919577.588646 punch from 0x039975 to 0x03af9b (0x1627 bytes) 000736 1788919577.592219 mapwrite 0x02a74d thru 0x02d1b2 (0x2a66 bytes) 000737 1788919577.600930 read 0x02465e thru 0x02cefd (0x88a0 bytes) 000738 1788919577.601594 fallocate 0x034182 thru 0x03a16f (0x5fee bytes) 000739 1788919577.604953 write 0x022b8c thru 0x031b64 (0xefd9 bytes) 000740 1788919577.606334 read 0x00006b thru 0x00f3d2 (0xf368 bytes) 000741 1788919577.606893 mapwrite 0x03fcdd thru 0x03ffff (0x323 bytes) 000742 1788919577.639579 fallocate 0x00cc5a thru 0x0112d6 (0x467d bytes) 000743 1788919577.643136 punch from 0x01df8c to 0x01fadd (0x1b52 bytes) 000744 1788919577.648306 punch from 0x02c04d to 0x034f75 (0x8f29 bytes) 000745 1788919577.652104 write 0x001e13 thru 0x003b47 (0x1d35 bytes) 000746 1788919577.655951 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa55a571b (stale) Mirror 13: 0xa55a571b (stale) Mirror 14: 0xa55a571b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4d56b14e Mirror 15: 0x4d56b14e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000747 1788919578.068926 trunc from 0x00a7a1 to 0x00e94e (0x41ae bytes) 000748 1788919578.100505 mapwrite 0x01dc6e thru 0x01de40 (0x1d3 bytes) 000749 1788919578.112579 mapread 0x003ef7 thru 0x0067f6 (0x2900 bytes) 000750 1788919578.122191 punch from 0x019915 to 0x01dbbd (0x42a9 bytes) 000751 1788919578.126993 mapwrite 0x034673 thru 0x0399ae (0x533c bytes) 000752 1788919578.144446 trunc from 0x040000 to 0x019c61 (0x2639e bytes) 000754 1788919578.166655 punch from 0x00fe70 to 0x019c61 (0x9df2 bytes) 000755 1788919578.169654 punch from 0x006df8 to 0x015d2c (0xef35 bytes) 000756 1788919578.172894 punch from 0x001a78 to 0x00a5a2 (0x8b2b bytes) 000757 1788919578.176560 trunc from 0x00cdca to 0x019c61 (0xce98 bytes) 000758 1788919578.181067 mapread 0x016de9 thru 0x019c61 (0x2e79 bytes) 000759 1788919578.190645 read 0x013dfb thru 0x017e62 (0x4068 bytes) 000760 1788919578.191435 write 0x02952a thru 0x02d81b (0x42f2 bytes) 000761 1788919578.193512 punch from 0x0013e1 to 0x00f8db (0xe4fb bytes) 000762 1788919578.199349 read 0x0090f3 thru 0x015711 (0xc61f bytes) 000763 1788919578.205902 trunc from 0x02d81c to 0x018256 (0x155c5 bytes) 000764 1788919578.229634 read 0x01460f thru 0x015c16 (0x1608 bytes) 000765 1788919578.230361 read 0x010eb9 thru 0x018256 (0x739e bytes) 000766 1788919578.230893 trunc from 0x001be1 to 0x00a057 (0x8477 bytes) 000767 1788919578.238152 mapread 0x00ad66 thru 0x00b2dd (0x578 bytes) 000768 1788919578.244887 trunc from 0x018257 to 0x03889b (0x20645 bytes) 000769 1788919578.255443 read 0x0178fd thru 0x01b8bf (0x3fc3 bytes) 000770 1788919578.264137 mapwrite 0x0342b4 thru 0x037085 (0x2dd2 bytes) 000771 1788919578.276178 mapread 0x00dc10 thru 0x01c051 (0xe442 bytes) 000772 1788919578.277146 trunc from 0x03889c to 0x025ab1 (0x12dea bytes) 000773 1788919578.287768 mapwrite 0x03dc1d thru 0x03ffff (0x23e3 bytes) 000774 1788919578.320600 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa55a571b (stale) Mirror 14: 0xa55a571b (stale) Mirror 15: 0x4d56b14e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa55a571b (stale) Mirror 14: 0xa55a571b (stale) Mirror 15: 0x4d56b14e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000776 1788919578.513025 read 0x009ef9 thru 0x00c751 (0x2859 bytes) 000777 1788919578.516080 mapwrite 0x021bc5 thru 0x0301dd (0xe619 bytes) 000778 1788919578.546027 fallocate 0x032a56 thru 0x038a94 (0x603f bytes) 000779 1788919578.551357 trunc from 0x040000 to 0x023fb7 (0x1c048 bytes) 000780 1788919578.563259 mapread 0x0228aa thru 0x023fb7 (0x170e bytes) 000781 1788919578.566319 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23fb8): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fb8): Mirror 12: 0xf2445178 (stale) Mirror 14: 0xf2445178 (stale) Mirror 15: 0xcda0e420 (stale) CRC-32 checksum value for chunk [0, 0x23fb8): Mirror 1: 0x57a60b5b Mirror 16: 0x57a60b5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fb8: skipped 000782 1788919578.947627 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23fb8): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fb8): Mirror 12: 0xf2445178 (stale) Mirror 14: 0xf2445178 (stale) Mirror 15: 0xcda0e420 (stale) CRC-32 checksum value for chunk [0, 0x23fb8): Mirror 1: 0x57a60b5b Mirror 16: 0x57a60b5b Mirror 17: 0x57a60b5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fb8: skipped 000783 1788919579.350572 MIRROR_RESYNC 6 mirrors 000784 1788919579.797572 mapwrite 0x03b4f1 thru 0x03bc68 (0x778 bytes) 000785 1788919579.839712 read 0x017891 thru 0x0237bf (0xbf2f bytes) 000786 1788919579.847619 fallocate 0x01a18c thru 0x0257cc (0xb641 bytes) 000787 1788919579.852024 punch from 0x00cce9 to 0x0143d4 (0x76ec bytes) 000788 1788919579.855023 read 0x027141 thru 0x033e56 (0xcd16 bytes) 000789 1788919579.862080 trunc from 0x003ced to 0x00ecaf (0xafc3 bytes) 000790 1788919579.866033 mapwrite 0x029cd5 thru 0x031c70 (0x7f9c bytes) 000791 1788919579.893116 mapwrite 0x035223 thru 0x03ffff (0xaddd bytes) 000793 1788919579.927437 fallocate 0x03f0cf thru 0x03f6d7 (0x609 bytes) 000795 1788919579.931513 write 0x03703a thru 0x03c33b (0x5302 bytes) 000797 1788919579.933060 write 0x00f4e9 thru 0x01e9f8 (0xf510 bytes) 000798 1788919579.934520 trunc from 0x040000 to 0x000b31 (0x3f4ce bytes) 000799 1788919579.947599 mapread 0x0004a4 thru 0x000b31 (0x68e bytes) 000800 1788919579.953397 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xb32): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb32): Mirror 12: 0x71e9ad16 (stale) Mirror 14: 0x71e9ad16 (stale) Mirror 15: 0x71e9ad16 (stale) Mirror 16: 0x71e9ad16 (stale) Mirror 17: 0x71e9ad16 (stale) CRC-32 checksum value for chunk [0, 0xb32): Mirror 1: 0x71e9ad16 Mirror 18: 0x71e9ad16 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb32: skipped 000801 1788919580.351413 punch from 0x00014c to 0x000b31 (0x9e6 bytes) 000802 1788919580.376906 mapwrite 0x00a04c thru 0x01326f (0x9224 bytes) 000803 1788919580.422190 trunc from 0x013270 to 0x038667 (0x253f8 bytes) 000805 1788919580.441960 trunc from 0x038668 to 0x021cce (0x16999 bytes) 000806 1788919580.455955 punch from 0x01e436 to 0x021cce (0x3899 bytes) 000807 1788919580.460486 write 0x027214 thru 0x02c2e6 (0x50d3 bytes) 000808 1788919580.461760 trunc from 0x02c2e7 to 0x03e45f (0x12179 bytes) 000809 1788919580.475616 trunc from 0x014fab to 0x01ab12 (0x5b68 bytes) 000810 1788919580.481877 MIRROR_RESYNC 7 mirrors 000811 1788919581.454652 trunc from 0x03e460 to 0x02053a (0x1df25 bytes) 000813 1788919581.491287 trunc from 0x02053b to 0x00a794 (0x15da6 bytes) 000814 1788919581.503656 fallocate 0x029126 thru 0x02acfc (0x1bd7 bytes) 000815 1788919581.507069 read 0x0077f2 thru 0x007f38 (0x747 bytes) 000816 1788919581.511363 trunc from 0x00a795 to 0x027488 (0x1ccf4 bytes) 000817 1788919581.525891 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x27489): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27489): Mirror 12: 0x73f4b181 (stale) Mirror 14: 0x73f4b181 (stale) Mirror 15: 0x73f4b181 (stale) Mirror 16: 0x73f4b181 (stale) Mirror 17: 0x73f4b181 (stale) Mirror 18: 0x73f4b181 (stale) CRC-32 checksum value for chunk [0, 0x27489): Mirror 1: 0xdf601ec Mirror 19: 0xdf601ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27489: skipped 000818 1788919582.014749 MIRROR_RESYNC 8 mirrors 000819 1788919582.698058 mapread 0x0046c0 thru 0x011a8b (0xd3cc bytes) 000820 1788919582.710770 trunc from 0x027489 to 0x02879d (0x1315 bytes) 000821 1788919582.742405 punch from 0x010261 to 0x01f81d (0xf5bd bytes) 000822 1788919582.745943 read 0x01ff46 thru 0x02879d (0x8858 bytes) 000823 1788919582.751700 mapread 0x028713 thru 0x02879d (0x8b bytes) 000824 1788919582.752613 mapwrite 0x0023a4 thru 0x00fcba (0xd917 bytes) 000826 1788919582.776060 punch from 0x01bf28 to 0x023ca9 (0x7d82 bytes) 000827 1788919582.779446 punch from 0x00b0c7 to 0x01631a (0xb254 bytes) 000828 1788919582.782680 trunc from 0x02879e to 0x0264bb (0x22e2 bytes) 000829 1788919582.793290 punch from 0x00a0fc to 0x012d9d (0x8ca2 bytes) 000830 1788919582.797778 mapread 0x01a221 thru 0x0245d5 (0xa3b5 bytes) 000832 1788919582.815489 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x264bc): Mirror 12: 0xecb682e (stale) Mirror 14: 0xecb682e (stale) Mirror 15: 0xecb682e (stale) Mirror 16: 0xecb682e (stale) Mirror 17: 0xecb682e (stale) Mirror 18: 0xecb682e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x264bc): Mirror 12: 0xecb682e (stale) Mirror 14: 0xecb682e (stale) Mirror 15: 0xecb682e (stale) Mirror 16: 0xecb682e (stale) Mirror 17: 0xecb682e (stale) Mirror 18: 0xecb682e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x264bc: skipped 000833 1788919583.081629 punch from 0x00a5af to 0x013e7d (0x98cf bytes) 000835 1788919583.090110 mapread 0x008f6b thru 0x01258a (0x9620 bytes) 000836 1788919583.101405 mapwrite 0x016225 thru 0x0236bf (0xd49b bytes) 000837 1788919583.129483 write 0x02e71a thru 0x02fdba (0x16a1 bytes) 000838 1788919583.133360 read 0x004bef thru 0x007458 (0x286a bytes) 000839 1788919583.133897 trunc from 0x02f48e to 0x02fdba (0x92d bytes) 000840 1788919583.140918 mapwrite 0x011fed thru 0x014b99 (0x2bad bytes) 000841 1788919583.151576 trunc from 0x013b03 to 0x020714 (0xcc12 bytes) 000842 1788919583.154681 read 0x01478f thru 0x01d6db (0x8f4d bytes) 000843 1788919583.160063 trunc from 0x005840 to 0x007adf (0x22a0 bytes) 000844 1788919583.162836 write 0x015cf7 thru 0x022fc5 (0xd2cf bytes) 000845 1788919583.164076 trunc from 0x01cc74 to 0x02a51e (0xd8ab bytes) 000846 1788919583.173425 read 0x026a38 thru 0x02fdba (0x9383 bytes) 000847 1788919583.180898 trunc from 0x027784 to 0x02ed21 (0x759e bytes) 000848 1788919583.185296 fallocate 0x00a0ba thru 0x018e9e (0xede5 bytes) 000850 1788919583.190127 punch from 0x02c53a to 0x02fdba (0x3881 bytes) 000851 1788919583.195132 read 0x02b27f thru 0x02fdba (0x4b3c bytes) 000852 1788919583.199893 mapread 0x01fa13 thru 0x024224 (0x4812 bytes) 000853 1788919583.200815 punch from 0x0178ea to 0x01f198 (0x78af bytes) 000854 1788919583.204933 trunc from 0x02fdbb to 0x02c728 (0x3692 bytes) 000855 1788919583.218474 read 0x00b929 thru 0x01b4e9 (0xfbc1 bytes) 000856 1788919583.223568 trunc from 0x017374 to 0x01d2bc (0x5f49 bytes) 000858 1788919583.228466 write 0x0062f1 thru 0x0118f2 (0xb602 bytes) 000859 1788919583.229827 write 0x034921 thru 0x03ffff (0xb6df bytes) 000860 1788919583.231104 mapwrite 0x029c6e thru 0x036651 (0xc9e4 bytes) 000861 1788919583.260241 MIRROR_RESYNC 7 mirrors 000863 1788919584.139084 MIRROR_EXTEND to 8 mirrors 000864 1788919584.536187 trunc from 0x03bfac to 0x03ffff (0x4054 bytes) 000865 1788919584.568828 trunc from 0x018e4a to 0x02797d (0xeb34 bytes) 000866 1788919584.572479 mapwrite 0x00de50 thru 0x016847 (0x89f8 bytes) 000867 1788919584.599638 write 0x018c4a thru 0x026ad2 (0xde89 bytes) 000868 1788919584.614463 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x40768b3c (stale) Mirror 14: 0x40768b3c (stale) Mirror 15: 0x40768b3c (stale) Mirror 16: 0x40768b3c (stale) Mirror 17: 0x40768b3c (stale) Mirror 18: 0x40768b3c (stale) Mirror 19: 0x40768b3c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x86aaf954 Mirror 20: 0x86aaf954 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000869 1788919585.123551 MIRROR_RESYNC 9 mirrors 000870 1788919586.095084 punch from 0x038a3f to 0x03ffff (0x75c1 bytes) 000871 1788919586.127959 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x86aaf954 (stale) Mirror 15: 0x86aaf954 (stale) Mirror 16: 0x86aaf954 (stale) Mirror 17: 0x86aaf954 (stale) Mirror 18: 0x86aaf954 (stale) Mirror 19: 0x86aaf954 (stale) Mirror 20: 0x86aaf954 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x86aaf954 (stale) Mirror 15: 0x86aaf954 (stale) Mirror 16: 0x86aaf954 (stale) Mirror 17: 0x86aaf954 (stale) Mirror 18: 0x86aaf954 (stale) Mirror 19: 0x86aaf954 (stale) Mirror 20: 0x86aaf954 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000872 1788919586.499604 mapwrite 0x01d15d thru 0x01fbaa (0x2a4e bytes) 000873 1788919586.522893 trunc from 0x040000 to 0x0275c3 (0x18a3c bytes) 000874 1788919586.549411 write 0x02e020 thru 0x03d35e (0xf33f bytes) 000875 1788919586.551627 fallocate 0x0151bb thru 0x019401 (0x4247 bytes) 000876 1788919586.555682 write 0x02c39f thru 0x036b70 (0xa7d2 bytes) 000877 1788919586.559990 trunc from 0x03d35f to 0x037638 (0x5d26 bytes) 000878 1788919586.576274 MIRROR_RESYNC 8 mirrors 000879 1788919587.607108 fallocate 0x00b145 thru 0x0194c8 (0xe384 bytes) 000880 1788919587.643551 trunc from 0x0163df to 0x01ee21 (0x8a43 bytes) 000881 1788919587.652522 MIRROR_RESYNC 8 mirrors 000882 1788919588.941455 MIRROR_RESYNC 8 mirrors 000883 1788919589.215060 fallocate 0x02c699 thru 0x03426e (0x7bd6 bytes) 000884 1788919589.231001 read 0x001ebe thru 0x00c03d (0xa180 bytes) 000885 1788919589.236775 mapwrite 0x0280c6 thru 0x028783 (0x6be bytes) 000886 1788919589.254304 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37639): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37639): Mirror 14: 0x907ec616 (stale) Mirror 15: 0x907ec616 (stale) Mirror 16: 0x907ec616 (stale) Mirror 17: 0x907ec616 (stale) Mirror 18: 0x907ec616 (stale) Mirror 19: 0x907ec616 (stale) Mirror 20: 0x907ec616 (stale) CRC-32 checksum value for chunk [0, 0x37639): Mirror 1: 0x9ab6ca8d Mirror 21: 0x9ab6ca8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37639: skipped 000887 1788919589.778315 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37639): Mirror 14: 0x907ec616 (stale) Mirror 15: 0x907ec616 (stale) Mirror 16: 0x907ec616 (stale) Mirror 17: 0x907ec616 (stale) Mirror 18: 0x907ec616 (stale) Mirror 19: 0x907ec616 (stale) Mirror 20: 0x907ec616 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37639): Mirror 14: 0x907ec616 (stale) Mirror 15: 0x907ec616 (stale) Mirror 16: 0x907ec616 (stale) Mirror 17: 0x907ec616 (stale) Mirror 18: 0x907ec616 (stale) Mirror 19: 0x907ec616 (stale) Mirror 20: 0x907ec616 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37639: skipped 000888 1788919590.060046 write 0x012848 thru 0x015db1 (0x356a bytes) 000889 1788919590.073539 read 0x015691 thru 0x01d239 (0x7ba9 bytes) 000890 1788919590.080178 write 0x02827d thru 0x02f4a6 (0x722a bytes) 000891 1788919590.087866 mapread 0x035ef4 thru 0x037638 (0x1745 bytes) 000894 1788919590.108033 read 0x02a465 thru 0x02f0a4 (0x4c40 bytes) 000895 1788919590.108860 mapwrite 0x03f25e thru 0x03ffff (0xda2 bytes) 000896 1788919590.174689 trunc from 0x01fb77 to 0x028e76 (0x9300 bytes) 000897 1788919590.188980 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xa6cdc33b (stale) Mirror 15: 0xa6cdc33b (stale) Mirror 16: 0xa6cdc33b (stale) Mirror 17: 0xa6cdc33b (stale) Mirror 18: 0xa6cdc33b (stale) Mirror 19: 0xa6cdc33b (stale) Mirror 20: 0xa6cdc33b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x904c3d34 Mirror 21: 0x904c3d34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000898 1788919590.821747 write 0x03ef5f thru 0x03ffff (0x10a1 bytes) 000899 1788919590.864620 mapread 0x000249 thru 0x002e61 (0x2c19 bytes) 000900 1788919590.881050 trunc from 0x016036 to 0x020653 (0xa61e bytes) 000901 1788919590.886508 mapwrite 0x02d4ba thru 0x031878 (0x43bf bytes) 000902 1788919590.921996 fallocate 0x021d65 thru 0x02310a (0x13a6 bytes) 000903 1788919590.934404 mapwrite 0x019af6 thru 0x022927 (0x8e32 bytes) 000904 1788919590.967875 read 0x0372ba thru 0x03c3c6 (0x510d bytes) 000905 1788919590.978402 mapread 0x025d22 thru 0x028512 (0x27f1 bytes) 000906 1788919590.979349 read 0x0127fa thru 0x015be9 (0x33f0 bytes) 000907 1788919590.979800 fallocate 0x030234 thru 0x038c5b (0x8a28 bytes) 000908 1788919590.983612 read 0x01d2f6 thru 0x01d764 (0x46f bytes) 000909 1788919590.984623 mapread 0x00c48b thru 0x00eb2b (0x26a1 bytes) 000910 1788919590.995964 read 0x0323ac thru 0x03a3bc (0x8011 bytes) 000911 1788919590.996638 read 0x02934f thru 0x02b81b (0x24cd bytes) 000912 1788919590.997173 read 0x016a60 thru 0x019db5 (0x3356 bytes) 000913 1788919591.006946 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xa6cdc33b (stale) Mirror 16: 0xa6cdc33b (stale) Mirror 17: 0xa6cdc33b (stale) Mirror 18: 0xa6cdc33b (stale) Mirror 19: 0xa6cdc33b (stale) Mirror 20: 0xa6cdc33b (stale) Mirror 21: 0x904c3d34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xa6cdc33b (stale) Mirror 16: 0xa6cdc33b (stale) Mirror 17: 0xa6cdc33b (stale) Mirror 18: 0xa6cdc33b (stale) Mirror 19: 0xa6cdc33b (stale) Mirror 20: 0xa6cdc33b (stale) Mirror 21: 0x904c3d34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000914 1788919591.350414 mapwrite 0x0081f9 thru 0x00b58b (0x3393 bytes) 000915 1788919591.400090 punch from 0x010bf1 to 0x01571b (0x4b2b bytes) 000916 1788919591.405458 mapwrite 0x0314f4 thru 0x034a2e (0x353b bytes) 000917 1788919591.436663 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xa6cdc33b (stale) Mirror 16: 0xa6cdc33b (stale) Mirror 17: 0xa6cdc33b (stale) Mirror 18: 0xa6cdc33b (stale) Mirror 19: 0xa6cdc33b (stale) Mirror 20: 0xa6cdc33b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xa6cdc33b (stale) Mirror 16: 0xa6cdc33b (stale) Mirror 17: 0xa6cdc33b (stale) Mirror 18: 0xa6cdc33b (stale) Mirror 19: 0xa6cdc33b (stale) Mirror 20: 0xa6cdc33b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000918 1788919591.734011 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xa6cdc33b (stale) Mirror 16: 0xa6cdc33b (stale) Mirror 17: 0xa6cdc33b (stale) Mirror 18: 0xa6cdc33b (stale) Mirror 20: 0xa6cdc33b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xa6cdc33b (stale) Mirror 16: 0xa6cdc33b (stale) Mirror 17: 0xa6cdc33b (stale) Mirror 18: 0xa6cdc33b (stale) Mirror 20: 0xa6cdc33b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000919 1788919592.013687 mapread 0x00e15e thru 0x018336 (0xa1d9 bytes) 000920 1788919592.032733 trunc from 0x03f6b8 to 0x03ffff (0x948 bytes) 000921 1788919592.035666 write 0x0084e7 thru 0x008ef9 (0xa13 bytes) 000922 1788919592.041687 trunc from 0x024188 to 0x028dc4 (0x4c3d bytes) 000923 1788919592.044999 trunc from 0x0388eb to 0x03ffff (0x7715 bytes) 000924 1788919592.048492 punch from 0x03bf25 to 0x03ffff (0x40db bytes) 000925 1788919592.051463 fallocate 0x016282 thru 0x01a919 (0x4698 bytes) 000926 1788919592.055667 mapwrite 0x03a6fd thru 0x03ffff (0x5903 bytes) 000927 1788919592.077041 fallocate 0x02c7db thru 0x03218e (0x59b4 bytes) 000928 1788919592.081542 trunc from 0x040000 to 0x008613 (0x379ec bytes) 000929 1788919592.109480 punch from 0x006f26 to 0x008613 (0x16ee bytes) 000930 1788919592.113641 trunc from 0x000c12 to 0x008613 (0x7a02 bytes) 000931 1788919592.119521 fallocate 0x00286a thru 0x00ca0d (0xa1a4 bytes) 000932 1788919592.123720 mapread 0x00c6ba thru 0x00ca0d (0x354 bytes) 000933 1788919592.134320 trunc from 0x004517 to 0x00ca0d (0x84f7 bytes) 000934 1788919592.138451 trunc from 0x00963d to 0x00bd26 (0x26ea bytes) 000935 1788919592.142326 fallocate 0x010fc4 thru 0x017a1e (0x6a5b bytes) 000936 1788919592.145957 fallocate 0x021ba8 thru 0x02f65f (0xdab8 bytes) 000937 1788919592.154136 trunc from 0x02b792 to 0x02f65f (0x3ece bytes) 000938 1788919592.158299 mapwrite 0x01f976 thru 0x02e1c1 (0xe84c bytes) 000939 1788919592.187022 fallocate 0x014b2b thru 0x0246d9 (0xfbaf bytes) 000940 1788919592.195637 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f660): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f660): Mirror 15: 0x3791c903 (stale) Mirror 16: 0x3791c903 (stale) Mirror 17: 0x3791c903 (stale) Mirror 18: 0x3791c903 (stale) Mirror 20: 0x3791c903 (stale) CRC-32 checksum value for chunk [0, 0x2f660): Mirror 1: 0x95ed8e2a Mirror 21: 0x95ed8e2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f660: skipped 000941 1788919592.669744 read 0x007171 thru 0x00b746 (0x45d6 bytes) 000942 1788919592.679503 trunc from 0x00a8a1 to 0x0124b9 (0x7c19 bytes) 000943 1788919592.700854 trunc from 0x02f660 to 0x00adbd (0x248a2 bytes) 000944 1788919592.722306 mapwrite 0x0037ed thru 0x003bbe (0x3d2 bytes) 000945 1788919592.735949 mapread 0x001f38 thru 0x00adbd (0x8e86 bytes) 000946 1788919592.747078 trunc from 0x00adbe to 0x00bc6f (0xeb2 bytes) 000947 1788919592.760528 mapwrite 0x02a900 thru 0x02e2c7 (0x39c8 bytes) 000948 1788919592.790431 punch from 0x01ab95 to 0x0209ac (0x5e18 bytes) 000949 1788919592.793342 write 0x00698a thru 0x015d6a (0xf3e1 bytes) 000950 1788919592.798362 read 0x0010ba thru 0x002616 (0x155d bytes) 000951 1788919592.799022 trunc from 0x007118 to 0x011553 (0xa43c bytes) 000952 1788919592.808135 mapread 0x02a5b3 thru 0x02e2c7 (0x3d15 bytes) 000953 1788919592.809116 fallocate 0x03bf91 thru 0x03f600 (0x3670 bytes) 000954 1788919592.812652 punch from 0x020b0a to 0x021c25 (0x111c bytes) 000955 1788919592.815974 mapread 0x02ddf0 thru 0x02e2c7 (0x4d8 bytes) 000956 1788919592.816805 read 0x0043ef thru 0x007a28 (0x363a bytes) 000957 1788919592.825846 trunc from 0x02e2c8 to 0x02ff25 (0x1c5e bytes) 000958 1788919592.840002 MIRROR_RESYNC 7 mirrors 000959 1788919593.824738 mapread 0x009e43 thru 0x00ea79 (0x4c37 bytes) 000960 1788919593.837794 write 0x02ad11 thru 0x0335d6 (0x88c6 bytes) 000961 1788919593.858607 mapwrite 0x016c2f thru 0x01e856 (0x7c28 bytes) 000962 1788919593.876718 MIRROR_RESYNC 7 mirrors 000963 1788919594.822310 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) 000964 1788919595.038775 mapwrite 0x00f97f thru 0x01963d (0x9cbf bytes) 000965 1788919595.130915 read 0x01d780 thru 0x01e9b7 (0x1238 bytes) 000966 1788919595.131669 fallocate 0x028fff thru 0x0320df (0x90e1 bytes) 000968 1788919595.139438 punch from 0x028f8a to 0x0335d6 (0xa64d bytes) 000969 1788919595.146651 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x335d7): Mirror 17: 0xb0abf6bc (stale) Mirror 18: 0xb0abf6bc (stale) Mirror 20: 0xb0abf6bc (stale) Mirror 21: 0xb0abf6bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335d7): Mirror 17: 0xb0abf6bc (stale) Mirror 18: 0xb0abf6bc (stale) Mirror 20: 0xb0abf6bc (stale) Mirror 21: 0xb0abf6bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335d7: skipped 000972 1788919595.390623 mapwrite 0x03df05 thru 0x03ffff (0x20fb bytes) 000974 1788919595.445047 trunc from 0x03d05b to 0x03ffff (0x2fa5 bytes) 000975 1788919595.449970 mapwrite 0x00c645 thru 0x00f7a2 (0x315e bytes) 000977 1788919595.470700 trunc from 0x040000 to 0x004491 (0x3bb6e bytes) 000978 1788919595.479713 read 0x000d11 thru 0x004491 (0x3781 bytes) 000980 1788919595.489108 write 0x03ebbc thru 0x03ffff (0x1444 bytes) 000982 1788919595.490708 write 0x0317e4 thru 0x03651c (0x4d39 bytes) 000983 1788919595.504378 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3d1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xb6fbcc66 (stale) Mirror 20: 0xb6fbcc66 (stale) Mirror 21: 0xb6fbcc66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xb6fbcc66 (stale) Mirror 20: 0xb6fbcc66 (stale) Mirror 21: 0xb6fbcc66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000984 1788919595.718964 trunc from 0x040000 to 0x025ed4 (0x1a12b bytes) 000985 1788919595.734790 mapread 0x00b05d thru 0x0136cc (0x8670 bytes) 000987 1788919595.756930 mapread 0x01c05b thru 0x02548a (0x9430 bytes) 000988 1788919595.768328 mapwrite 0x037048 thru 0x03ffff (0x8fb8 bytes) 000989 1788919595.795635 mapread 0x008bc1 thru 0x011a6a (0x8eaa bytes) 000990 1788919595.798014 trunc from 0x000109 to 0x002322 (0x221a bytes) 000991 1788919595.804502 mapwrite 0x02c19d thru 0x02dd91 (0x1bf5 bytes) 000992 1788919595.830698 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xb6fbcc66 (stale) Mirror 20: 0xb6fbcc66 (stale) Mirror 21: 0xb6fbcc66 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x40d77b95 Mirror 22: 0x40d77b95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1788919596.407996 mapwrite 0x02dc4a thru 0x032dfa (0x51b1 bytes) 000995 1788919596.491023 write 0x00b507 thru 0x00d010 (0x1b0a bytes) 000997 1788919596.500097 fallocate 0x00ae95 thru 0x016b53 (0xbcbf bytes) 000998 1788919596.504383 trunc from 0x040000 to 0x00c19d (0x33e62 bytes) 000999 1788919596.518431 read 0x0071cd thru 0x00bb57 (0x498b bytes) 001000 1788919596.523555 read 0x008139 thru 0x00c19d (0x4065 bytes) All operations completed A-OK! PASS 70a (57s) == sanity-flr test 71: check mirror extend parent fid ==== 22:06:39 (1788919599) ** lfs fid2path /mnt/lustre 0x200000404:0x412:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 72: check append on FLR file ========== 22:06:43 (1788919603) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122573 s, 17.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682248 s, 15.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ee:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ee:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 22:06:47 (1788919607) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8184896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1767936 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5996832 .. Extending file size to 8415840 .. failed Extending file size to 6401024 .. Extending file size to 7596160 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8062848 .. failed Extending file size to 9206208 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8677920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8694464 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8856128 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60447 60448 60450 60452 60454 done 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:07:57 (1788919677) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8576064 .. Extending file size to 7477312 .. failed Extending file size to 7749600 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7800480 .. failed Extending file size to 7836960 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8681888 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8951424 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8995264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9030272 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9176512 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 62779 62780 62781 62782 62784 failed 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) 192.168.203.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:09:04 (1788919744) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2796 1268508 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 16636 3590384 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23652 3583368 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 40288 7173752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2796 1268508 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3833116 16636 3590384 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23652 3583368 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 40288 7173752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0547782 s, 38.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.089157 s, 23.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6831abcb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:09:12 (1788919752) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0471938 s, 44.4 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 22:09:15 (1788919755) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x421:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x604:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x604:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x605:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x5e9:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:09:18 (1788919758) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0332411 s, 63.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x606:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:09:21 (1788919761) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0398942 s, 52.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x607:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x608:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x607:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x608:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ed:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:09:24 (1788919764) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0258412 s, 81.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x609:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x609:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ee:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ef:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:09:49 (1788919789) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299671 s, 35.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60b:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:10:25 (1788919825) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.04175 s, 50.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:11:00 (1788919860) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg308-server mds-ost sync done. Creating new pool test_204f oleg308-server: Pool lustre.test_204f created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0516525 s, 40.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg308-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (46s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:11:46 (1788919906) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:11:49 (1788919909) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:11:52 (1788919912) oleg308-server: Pool lustre.test_206 created Adding targets to pool oleg308-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg308-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg308-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg308-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:12:06 (1788919926) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299054 s, 35.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144778 s, 72.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x654d0a3a Mirror 3: 0x654d0a3a PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:12:10 (1788919930) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:12:11 (1788919931) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:12:12 (1788919932) Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0158067 s, 0.3 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x840000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x690000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x690000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x690000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xda0000 Read this (4 bytes): BBBB 192.168.203.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:12:36 (1788919956) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0129973 s, 630 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:12:42 (1788919962) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437937 s, 23.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x446 lmm_fid: [0x200000404:0x446:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x446 lmm_fid: [0x200000404:0x446:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835014 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 22:12:43.000000000 -0400 Modify: 2026-09-08 22:12:43.000000000 -0400 Change: 2026-09-08 22:12:44.000000000 -0400 Birth: 2026-09-08 22:12:43.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:12:48 (1788919968) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256999 s, 40.8 MB/s oleg308-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg308-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg308-server: O/240000400/d25/619: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:13:09 (1788919989) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.220938 s, 185 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0222735 s, 184 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 8 22:13 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x44b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 8 22:13 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:13:16 (1788919996) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:13:25 (1788920005) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1477 sec =========== 22:13:35 (1788920015) === sanity-flr: start cleanup 22:13:36 (1788920016) === === sanity-flr: finish cleanup 22:13:42 (1788920022) ===