-----============= acceptance-small: sanity-flr ============----- Mon Sep 14 12:11:41 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg306-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:12:00 (1789402320) === oleg306-client.virtnet: executing check_config_client /mnt/lustre oleg306-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg306-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff957ec8514800.idle_timeout=debug osc.lustre-OST0001-osc-ffff957ec8514800.idle_timeout=debug disable quota as required oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:12:42 (1789402362) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8211] == sanity-flr test 0a: lfs mirror create with -N option == 12:12:44 (1789402364) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:13:03 (1789402383) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:13:07 (1789402387) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:13:11 (1789402391) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:13:38 (1789402418) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:13:42 (1789402422) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 12:13:46 (1789402426) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff957ec8514800.stats=clear osc.lustre-OST0001-osc-ffff957ec8514800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:13:54 (1789402434) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:14:06 (1789402446) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:14:13 (1789402453) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:14:16 (1789402456) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:14:24 (1789402464) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 12:14:33 (1789402473) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:14:41 (1789402481) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:14:49 (1789402489) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:14:59 (1789402499) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:15:10 (1789402510) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:15:18 (1789402518) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.898736 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893768 s, 11.7 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:15:27 (1789402527) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.102343 s, 41.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:15:35 (1789402535) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:15:45 (1789402545) Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff957ec8514800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff957ec8514800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg306-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff957ec8514800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff957ec8514800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (131s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:17:56 (1789402676) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.658591 s, 14.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff957ec8514800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff957ec8514800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:18:41 (1789402721) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff957ec8514800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff957ec8514800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff957ec8514800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff957ec8514800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:19:59 (1789402799) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.94124 s, 17.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 14 12:20 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.133s user 0m0.000s sys 0m0.150s Stopping /mnt/lustre-ost1 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff957ec8514800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff957ec8514800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.077s user 0m0.001s sys 0m0.185s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg306-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg306-server oleg306-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff957ec8514800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff957ec8514800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.422s user 0m0.000s sys 0m0.207s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg306-server: oleg306-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg306-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff957ec8514800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff957ec8514800.ost_server_uuid in FULL state after 0 sec PASS 33b (95s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:21:33 (1789402893) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:21:37 (1789402897) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:21:40 (1789402900) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:21:44 (1789402904) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 12:21:53 (1789402913) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.632204 s, 19.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:22:08 (1789402928) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282962 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074545 s, 14.1 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:22:18 (1789402938) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:22:27 (1789402947) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00171191 s, 598 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000776281 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00251333 s, 407 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102937 s, 995 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 12:22:39 (1789402959) create test files with size 15756134 .. 285f2d41b5dfbe53a85c973e39485add - 36576cb9bba16fa5fff608ff4b1ebd72 - 7607f14317da034b7466314fbe5535d0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 12:23:06 (1789402986) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:23:20 (1789403000) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:23:27 (1789403007) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722762 s, 14.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433085 s, 24.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 12:23:36 (1789403016) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0968884 s, 21.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.349426 s, 12.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172317 s, 12.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 12:23:52 (1789403032) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 12:23:55 (1789403035) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:23:58 (1789403038) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057808 s, 18.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 12:24:07 (1789403047) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.437841 s, 7.2 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:24:19 (1789403059) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:24:29 (1789403069) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.311471 s, 33.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:24:41 (1789403081) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff957ed8c47000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.391005 s, 26.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg306-client.virtnet /mnt/lustre (opts:) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff957ee69bd000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:24:57 (1789403097) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (49s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:25:46 (1789403146) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:26:20 (1789403180) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:26:23 (1789403183) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.304538 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.199008 s, 15.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf437d005 (nosync,stale) Mirror 2: 0xf437d005 (nosync,stale) ** verify mirror 2 doesn't change original checksum: aa05ee2f4ff9d025a936572787a935d7 - mirror 1 checksum: 313350426d042049d618ca9c4706112a - mirror 2 checksum: aa05ee2f4ff9d025a936572787a935d7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-14T12:26:27-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-14T12:26:27-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: aa05ee2f4ff9d025a936572787a935d7 - mirror 1 checksum: 313350426d042049d618ca9c4706112a - mirror 2 checksum: 313350426d042049d618ca9c4706112a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-14T12:26:27-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-14T12:26:27-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:26:33 (1789403193) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029856 s, 35.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:26:41 (1789403201) Failing mds1 on oleg306-server Stopping /mnt/lustre-mds1 (opts:) on oleg306-server 12:26:52 (1789403212) shut down facet: mds1 facet_host: oleg306-server facet_failover_host: oleg306-server Failover mds1 to oleg306-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 12:27:25 (1789403245) targets are mounted 12:27:25 (1789403245) facet_failover done oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449124 s, 23.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:27:41 (1789403261) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0586392 s, 349 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0254058 s, 806 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:27:51 (1789403271) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0429829 s, 476 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0124713 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:28:00 (1789403280) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00599678 s, 2.7 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:28:07 (1789403287) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0671064 s, 305 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:28:15 (1789403295) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128763 s, 318 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.018664 s, 219 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:28:23 (1789403303) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:28:45 (1789403325) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:29:13 (1789403353) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:29:36 (1789403376) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:29:59 (1789403399) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 12:30:07 (1789403407) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57561 mirror split pid 57562 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57561 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57562 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 12:31:14 (1789403474) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3476 000002 1789403477.220701 write 0x00d551 thru 0x01488b (0x733b bytes) 000003 1789403477.238044 mapread 0x013905 thru 0x01488b (0xf87 bytes) 000004 1789403477.239075 mapwrite 0x0393f6 thru 0x03bccc (0x28d7 bytes) 000005 1789403477.318116 mapread 0x024c13 thru 0x02aa45 (0x5e33 bytes) 000006 1789403477.334628 mapwrite 0x03d656 thru 0x03ffff (0x29aa bytes) truncating to largest ever: 0xce90 000007 1789403477.371190 trunc from 0x040000 to 0x00ce8f (0x33170 bytes) fallocating to largest ever: 0x26bc1 000008 1789403477.400106 fallocate 0x01b203 thru 0x026bc0 (0xb9be bytes) 000010 1789403477.409719 trunc from 0x005f60 to 0x00ce8f (0x6f30 bytes) 000012 1789403477.417139 mapread 0x004d5b thru 0x00ce8f (0x8135 bytes) 000014 1789403477.432406 mapwrite 0x00bad8 thru 0x0176e3 (0xbc0c bytes) 000015 1789403477.494609 mapread 0x010f1c thru 0x0176e3 (0x67c8 bytes) 000016 1789403477.496359 read 0x007193 thru 0x011b73 (0xa9e1 bytes) 000017 1789403477.498415 trunc from 0x0176e4 to 0x01a208 (0x2b25 bytes) 000018 1789403477.514475 mapwrite 0x0004f7 thru 0x0027d7 (0x22e1 bytes) 000019 1789403477.533788 trunc from 0x01a209 to 0x0269b0 (0xc7a8 bytes) 000020 1789403477.552711 punch from 0x003406 to 0x00d802 (0xa3fd bytes) 000021 1789403477.559522 fallocate 0x0213e0 thru 0x025731 (0x4352 bytes) 000022 1789403477.567587 punch from 0x0163a8 to 0x01aaa0 (0x46f9 bytes) 000023 1789403477.574159 write 0x03a796 thru 0x03ca81 (0x22ec bytes) 000024 1789403477.576804 punch from 0x00d5fe to 0x011d4b (0x474e bytes) 000025 1789403477.581413 punch from 0x02e94d to 0x0334c8 (0x4b7c bytes) 000026 1789403477.587132 fallocate 0x01ad73 thru 0x01d123 (0x23b1 bytes) truncating to largest ever: 0x2e4ab 000027 1789403477.592566 trunc from 0x03ca82 to 0x02e4aa (0xe5d7 bytes) 000028 1789403477.607237 trunc from 0x026254 to 0x02e4aa (0x8257 bytes) 000029 1789403477.611718 read 0x01f76f thru 0x02e4aa (0xed3c bytes) 000030 1789403477.635778 mapwrite 0x013699 thru 0x01a80f (0x7177 bytes) 000031 1789403477.674061 trunc from 0x015eef to 0x023a78 (0xdb8a bytes) 000032 1789403477.689840 read 0x025e23 thru 0x026ed7 (0x10b5 bytes) truncating to largest ever: 0x3a7b7 000033 1789403477.701570 trunc from 0x02e4ab to 0x03a7b6 (0xc30c bytes) 000034 1789403477.735277 trunc from 0x0355d1 to 0x036073 (0xaa3 bytes) 000035 1789403477.753550 trunc from 0x0347ca to 0x034eea (0x721 bytes) 000036 1789403477.764758 punch from 0x00e5c4 to 0x015947 (0x7384 bytes) 000037 1789403477.779385 MIRROR_EXTEND to 2 mirrors 000038 1789403478.743283 punch from 0x012d9f to 0x021494 (0xe6f6 bytes) 000039 1789403478.805603 mapread 0x02a120 thru 0x03565b (0xb53c bytes) 000040 1789403478.878620 trunc from 0x022568 to 0x027815 (0x52ae bytes) 000041 1789403478.894571 read 0x01eccc thru 0x027d75 (0x90aa bytes) 000042 1789403478.909755 write 0x01410c thru 0x01aa00 (0x68f5 bytes) 000043 1789403478.927983 trunc from 0x032f00 to 0x0346f6 (0x17f7 bytes) 000044 1789403478.934304 trunc from 0x006eeb to 0x009b34 (0x2c4a bytes) 000045 1789403478.940275 mapwrite 0x005fa0 thru 0x007c37 (0x1c98 bytes) 000046 1789403478.976688 write 0x0387d8 thru 0x039160 (0x989 bytes) 000047 1789403478.978367 punch from 0x0346cc to 0x03a7b6 (0x60eb bytes) 000049 1789403478.987776 mapwrite 0x012dd2 thru 0x01f21e (0xc44d bytes) 000050 1789403479.022176 mapread 0x036cce thru 0x036ff0 (0x323 bytes) 000051 1789403479.038823 punch from 0x034573 to 0x0357d9 (0x1267 bytes) 000052 1789403479.060057 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a7b7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7b7): Mirror 2: 0xd430e2e3 (stale) CRC-32 checksum value for chunk [0, 0x3a7b7): Mirror 1: 0xa5553f5c Mirror 3: 0xa5553f5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7b7: skipped 000053 1789403480.559827 read 0x032121 thru 0x037656 (0x5536 bytes) 000054 1789403480.571121 trunc from 0x03a7b7 to 0x00a73d (0x30079 bytes) 000056 1789403480.648693 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xa73e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa73e): Mirror 2: 0x940ba61a (stale) Mirror 3: 0xb13a68da (stale) CRC-32 checksum value for chunk [0, 0xa73e): Mirror 1: 0xb13a68da Mirror 4: 0xb13a68da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa73e: skipped 000057 1789403481.524625 write 0x01eb83 thru 0x02c4e1 (0xd95f bytes) 000058 1789403481.576153 write 0x03009e thru 0x03a2f4 (0xa257 bytes) 000059 1789403481.577770 mapread 0x022770 thru 0x026a8d (0x431e bytes) 000060 1789403481.580172 read 0x012c06 thru 0x0165ce (0x39c9 bytes) 000061 1789403481.591138 mapwrite 0x02204f thru 0x023676 (0x1628 bytes) 000062 1789403481.631565 read 0x0334d7 thru 0x037854 (0x437e bytes) 000064 1789403481.632856 mapwrite 0x0021af thru 0x00afe1 (0x8e33 bytes) 000065 1789403481.708581 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a2f5): Mirror 2: 0xb4ce8aa6 (stale) Mirror 3: 0x5cf7e4aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2f5): Mirror 2: 0xb4ce8aa6 (stale) Mirror 3: 0x5cf7e4aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2f5: skipped truncating to largest ever: 0x3b2d6 000066 1789403482.140486 trunc from 0x03a2f5 to 0x03b2d5 (0xfe1 bytes) 000067 1789403482.165467 mapread 0x0365bf thru 0x03a360 (0x3da2 bytes) 000069 1789403482.183783 trunc from 0x02f4a2 to 0x037b3a (0x8699 bytes) 000070 1789403482.188603 fallocate 0x00f652 thru 0x0100a5 (0xa54 bytes) 000071 1789403482.193007 read 0x009828 thru 0x013c03 (0xa3dc bytes) 000074 1789403482.203143 fallocate 0x01e289 thru 0x029ccf (0xba47 bytes) 000075 1789403482.210095 punch from 0x01df6c to 0x02296d (0x4a02 bytes) 000076 1789403482.215382 read 0x034115 thru 0x03b2d5 (0x71c1 bytes) 000077 1789403482.225908 punch from 0x016214 to 0x01b1ef (0x4fdc bytes) 000078 1789403482.230840 trunc from 0x03b2d6 to 0x019fa0 (0x21335 bytes) 000080 1789403482.251067 trunc from 0x0131ac to 0x0186e9 (0x553e bytes) 000081 1789403482.260562 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19fa1): Mirror 3: 0x59c396b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19fa1): Mirror 3: 0x59c396b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19fa1: skipped 000082 1789403482.654674 MIRROR_RESYNC 2 mirrors 000083 1789403483.539089 write 0x02addb thru 0x032d87 (0x7fad bytes) 000084 1789403483.616741 punch from 0x020067 to 0x020d5f (0xcf9 bytes) 000085 1789403483.629100 trunc from 0x032d88 to 0x02a6b1 (0x86d6 bytes) 000086 1789403483.663166 read 0x01d70e thru 0x02a6b1 (0xcfa4 bytes) 000087 1789403483.676507 mapwrite 0x027944 thru 0x029f8d (0x264a bytes) 000088 1789403483.702328 MIRROR_RESYNC 2 mirrors 000089 1789403484.487314 read 0x01e36a thru 0x02803a (0x9cd1 bytes) 000090 1789403484.495458 mapread 0x0111c6 thru 0x0130c2 (0x1efd bytes) 000091 1789403484.515788 punch from 0x01fe64 to 0x022fd3 (0x3170 bytes) 000092 1789403484.548398 write 0x0105a6 thru 0x01e9ae (0xe409 bytes) 000093 1789403484.560127 fallocate 0x026e54 thru 0x02757b (0x728 bytes) 000094 1789403484.565025 write 0x030cc1 thru 0x03c6e6 (0xba26 bytes) 000096 1789403484.568438 mapread 0x007146 thru 0x0074a6 (0x361 bytes) zero_range to largest ever: 0x3c6e7 000097 1789403484.577630 trunc from 0x02e7f7 to 0x03c6e6 (0xdef0 bytes) 000098 1789403484.593920 mapread 0x0179ad thru 0x02408c (0xc6e0 bytes) 000099 1789403484.606314 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c6e7): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6e7): Mirror 3: 0x5c962e9a (stale) CRC-32 checksum value for chunk [0, 0x3c6e7): Mirror 1: 0xd888324 Mirror 4: 0xd888324 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6e7: skipped 000100 1789403485.559191 trunc from 0x02f729 to 0x037e28 (0x8700 bytes) 000102 1789403485.597300 mapread 0x0178bf thru 0x02241b (0xab5d bytes) 000103 1789403485.623429 read 0x02ab4a thru 0x0330fa (0x85b1 bytes) 000104 1789403485.635081 fallocate 0x009417 thru 0x0106eb (0x72d5 bytes) 000105 1789403485.642341 mapread 0x034c94 thru 0x03c6e6 (0x7a53 bytes) 000107 1789403485.645115 mapwrite 0x01e7d5 thru 0x022f48 (0x4774 bytes) fallocating to largest ever: 0x40000 000108 1789403485.661505 fallocate 0x03c296 thru 0x03ffff (0x3d6a bytes) 000109 1789403485.666838 mapwrite 0x025102 thru 0x02b3ce (0x62cd bytes) 000110 1789403485.683219 read 0x013c34 thru 0x01a57b (0x6948 bytes) 000111 1789403485.683856 mapwrite 0x003299 thru 0x01153f (0xe2a7 bytes) 000112 1789403485.715095 read 0x032e89 thru 0x03c6e6 (0x985e bytes) 000113 1789403485.715828 trunc from 0x02336a to 0x030b33 (0xd7ca bytes) 000114 1789403485.723535 mapread 0x006a6c thru 0x0128d3 (0xbe68 bytes) 000115 1789403485.724591 trunc from 0x03c6e7 to 0x005caa (0x36a3c bytes) 000116 1789403485.750331 punch from 0x001c68 to 0x004bcc (0x2f65 bytes) 000117 1789403485.756584 write 0x0051e2 thru 0x0069a6 (0x17c5 bytes) 000118 1789403485.757951 mapwrite 0x030a93 thru 0x031eee (0x145c bytes) 000119 1789403485.810008 read 0x012ac6 thru 0x012fd8 (0x513 bytes) 000120 1789403485.819115 mapread 0x015714 thru 0x024b42 (0xf42f bytes) 000121 1789403485.824599 MIRROR_RESYNC 3 mirrors 000123 1789403486.766514 mapwrite 0x01a5f3 thru 0x01dc5a (0x3668 bytes) 000124 1789403486.945696 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31eef): Mirror 4: 0xa5dacf84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31eef): Mirror 4: 0xa5dacf84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31eef: skipped 000125 1789403487.348608 trunc from 0x031eef to 0x017de9 (0x1a105 bytes) 000126 1789403487.388701 trunc from 0x00f649 to 0x014844 (0x51fc bytes) 000127 1789403487.404492 write 0x03b99f thru 0x03fb4c (0x41ae bytes) 000128 1789403487.406215 trunc from 0x03fb4d to 0x019101 (0x26a4b bytes) 000129 1789403487.451001 MIRROR_RESYNC 2 mirrors 000130 1789403488.281413 write 0x013e99 thru 0x01fb13 (0xbc7b bytes) 000131 1789403488.324366 punch from 0x00b8a3 to 0x018b0b (0xd269 bytes) 000132 1789403488.342452 fallocate 0x0026c8 thru 0x00ed83 (0xc6bc bytes) 000133 1789403488.347914 mapwrite 0x01952a thru 0x026081 (0xcb58 bytes) 000134 1789403488.415714 trunc from 0x022853 to 0x026081 (0x382f bytes) 000135 1789403488.422199 fallocate 0x0289b2 thru 0x034606 (0xbc55 bytes) 000136 1789403488.430334 fallocate 0x00cd17 thru 0x016593 (0x987d bytes) 000137 1789403488.436570 fallocate 0x02cac5 thru 0x0384ed (0xba29 bytes) 000138 1789403488.451861 trunc from 0x034607 to 0x00f0fc (0x2550a bytes) 000139 1789403488.480674 mapwrite 0x000eff thru 0x003e32 (0x2f34 bytes) 000140 1789403488.521983 write 0x0159c4 thru 0x0195a6 (0x3be3 bytes) 000141 1789403488.523318 write 0x02482b thru 0x02fd77 (0xb54d bytes) 000143 1789403488.528707 write 0x019962 thru 0x0235ef (0x9c8e bytes) 000144 1789403488.535746 mapread 0x001467 thru 0x00bb53 (0xa6ed bytes) 000145 1789403488.537454 write 0x00c457 thru 0x011798 (0x5342 bytes) 000146 1789403488.545026 punch from 0x018075 to 0x01832a (0x2b6 bytes) 000148 1789403488.563462 mapread 0x025bc0 thru 0x029be1 (0x4022 bytes) 000149 1789403488.564613 mapread 0x00e3b1 thru 0x01e2fe (0xff4e bytes) 000150 1789403488.594949 trunc from 0x00c216 to 0x017f8b (0xbd76 bytes) 000151 1789403488.601630 read 0x01f03f thru 0x02d38b (0xe34d bytes) 000152 1789403488.602352 trunc from 0x02fd78 to 0x02340a (0xc96d bytes) 000154 1789403488.620336 mapread 0x004771 thru 0x009570 (0x4e00 bytes) 000155 1789403488.621942 read 0x010ea6 thru 0x017a26 (0x6b81 bytes) 000156 1789403488.632162 write 0x007846 thru 0x00cfee (0x57a9 bytes) 000157 1789403488.633761 fallocate 0x013dd1 thru 0x02366c (0xf89c bytes) 000159 1789403488.641122 trunc from 0x00af07 to 0x00e9b7 (0x3ab1 bytes) 000160 1789403488.656472 mapwrite 0x023d39 thru 0x02b6dc (0x79a4 bytes) 000161 1789403488.701728 trunc from 0x02b6dd to 0x025c36 (0x5aa6 bytes) 000162 1789403488.720330 fallocate 0x0325f5 thru 0x0368eb (0x42f7 bytes) 000163 1789403488.726766 mapread 0x01b5df thru 0x022646 (0x7068 bytes) 000164 1789403488.728004 mapwrite 0x02db5e thru 0x03ba2f (0xded2 bytes) 000165 1789403488.792449 fallocate 0x00e94a thru 0x017a2d (0x90e4 bytes) 000166 1789403488.799406 trunc from 0x03ba30 to 0x021d30 (0x19cff bytes) 000167 1789403488.824254 punch from 0x013470 to 0x01dd6e (0xa8ff bytes) 000168 1789403488.833268 read 0x01ea70 thru 0x01ee36 (0x3c7 bytes) 000169 1789403488.834156 mapread 0x007223 thru 0x016c9a (0xfa78 bytes) 000171 1789403488.845205 trunc from 0x01bd9a to 0x021d30 (0x5f97 bytes) 000172 1789403488.851444 write 0x006cea thru 0x00df02 (0x7219 bytes) 000173 1789403488.852923 fallocate 0x01c6d5 thru 0x01d6ce (0xffa bytes) 000174 1789403488.858864 trunc from 0x01e391 to 0x021d30 (0x39a0 bytes) 000175 1789403488.864479 read 0x00c564 thru 0x016577 (0xa014 bytes) 000176 1789403488.865469 trunc from 0x021d31 to 0x01be90 (0x5ea0 bytes) 000177 1789403488.882251 trunc from 0x01be91 to 0x00ba47 (0x10449 bytes) 000178 1789403488.897610 mapwrite 0x01a0ec thru 0x01fc35 (0x5b4a bytes) 000179 1789403488.969336 write 0x02f4a4 thru 0x037309 (0x7e66 bytes) 000180 1789403488.971017 trunc from 0x03730a to 0x02ba11 (0xb8f8 bytes) 000181 1789403488.989564 trunc from 0x02ba12 to 0x01f755 (0xc2bc bytes) 000182 1789403489.012562 fallocate 0x02e9bc thru 0x03cefe (0xe543 bytes) 000183 1789403489.019627 fallocate 0x01ced5 thru 0x024e3a (0x7f66 bytes) 000184 1789403489.026093 punch from 0x01be58 to 0x024e3a (0x8fe3 bytes) 000185 1789403489.033001 fallocate 0x00eff0 thru 0x01d8b9 (0xe8ca bytes) 000186 1789403489.038496 punch from 0x01cfb5 to 0x024e3a (0x7e86 bytes) 000187 1789403489.050146 read 0x0005e1 thru 0x002ad0 (0x24f0 bytes) 000188 1789403489.053022 punch from 0x01726c to 0x021b7a (0xa90f bytes) 000189 1789403489.066132 mapwrite 0x03a328 thru 0x03f182 (0x4e5b bytes) 000190 1789403489.131387 write 0x0374d3 thru 0x03ffff (0x8b2d bytes) 000191 1789403489.133163 trunc from 0x0021f8 to 0x00b875 (0x967e bytes) 000192 1789403489.141718 trunc from 0x040000 to 0x03e07a (0x1f85 bytes) 000194 1789403489.189817 MIRROR_RESYNC 2 mirrors 000195 1789403489.864730 write 0x03a1c0 thru 0x03ffff (0x5e40 bytes) 000196 1789403489.905500 trunc from 0x032a5c to 0x03ecbc (0xc261 bytes) 000197 1789403489.922286 mapread 0x0104be thru 0x01b265 (0xada8 bytes) 000198 1789403489.950122 read 0x01f822 thru 0x0275e7 (0x7dc6 bytes) 000199 1789403489.960003 read 0x013da6 thru 0x0203a1 (0xc5fc bytes) 000200 1789403489.962900 fallocate 0x018721 thru 0x01d68d (0x4f6d bytes) 000201 1789403489.980801 mapwrite 0x019da8 thru 0x0298fb (0xfb54 bytes) 000202 1789403490.034767 mapwrite 0x03d7cb thru 0x03ffff (0x2835 bytes) 000204 1789403490.075378 punch from 0x03be1c to 0x03ffff (0x41e4 bytes) 000205 1789403490.086750 fallocate 0x015a9f thru 0x023b07 (0xe069 bytes) 000206 1789403490.096875 read 0x02ef1f thru 0x03016f (0x1251 bytes) 000207 1789403490.097386 fallocate 0x036a41 thru 0x038748 (0x1d08 bytes) 000208 1789403490.118037 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5f839260 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x634dfed7 Mirror 5: 0x634dfed7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000209 1789403491.044175 trunc from 0x01b56e to 0x02339a (0x7e2d bytes) 000210 1789403491.089757 punch from 0x00c1aa to 0x01803b (0xbe92 bytes) 000213 1789403491.101108 punch from 0x01cac4 to 0x0266cd (0x9c0a bytes) 000214 1789403491.108374 mapread 0x035341 thru 0x03a318 (0x4fd8 bytes) 000215 1789403491.144393 mapread 0x0150a8 thru 0x01572e (0x687 bytes) 000216 1789403491.164079 trunc from 0x040000 to 0x026bdf (0x19420 bytes) 000217 1789403491.214299 write 0x03875f thru 0x03a89c (0x213e bytes) 000218 1789403491.215733 mapwrite 0x0364d3 thru 0x03bbbd (0x56eb bytes) 000219 1789403491.250069 trunc from 0x03ba82 to 0x03bbbd (0x13c bytes) 000220 1789403491.261055 read 0x009581 thru 0x00c5e7 (0x3067 bytes) 000222 1789403491.268272 write 0x002539 thru 0x00c397 (0x9e5f bytes) 000223 1789403491.274616 write 0x0335bf thru 0x03aa89 (0x74cb bytes) 000224 1789403491.276066 read 0x00e5e5 thru 0x00f6eb (0x1107 bytes) 000225 1789403491.277263 mapwrite 0x0006a5 thru 0x00f557 (0xeeb3 bytes) 000226 1789403491.304566 punch from 0x005213 to 0x00ba3d (0x682b bytes) 000227 1789403491.310332 trunc from 0x03bbbe to 0x02b77a (0x10443 bytes) 000228 1789403491.327503 mapread 0x004e63 thru 0x012158 (0xd2f6 bytes) 000229 1789403491.358480 read 0x006c61 thru 0x0149d5 (0xdd75 bytes) 000232 1789403491.360060 mapread 0x0213d9 thru 0x02471a (0x3342 bytes) 000233 1789403491.362407 punch from 0x022b77 to 0x029aa7 (0x6f31 bytes) 000234 1789403491.367436 trunc from 0x00782f to 0x011b0e (0xa2e0 bytes) 000235 1789403491.372946 fallocate 0x019edd thru 0x02940a (0xf52e bytes) 000236 1789403491.378559 trunc from 0x0034a1 to 0x00c768 (0x92c8 bytes) 000237 1789403491.384494 write 0x013af8 thru 0x0173f3 (0x38fc bytes) 000238 1789403491.385569 read 0x0281d0 thru 0x02b77a (0x35ab bytes) 000239 1789403491.407194 mapwrite 0x02c0f5 thru 0x039410 (0xd31c bytes) 000241 1789403491.495166 mapwrite 0x001352 thru 0x010eee (0xfb9d bytes) 000242 1789403491.532589 write 0x013eb0 thru 0x01ef07 (0xb058 bytes) 000243 1789403491.536020 fallocate 0x00cc42 thru 0x00ec0c (0x1fcb bytes) 000245 1789403491.547988 trunc from 0x039411 to 0x01bf3d (0x1d4d3 bytes) 000246 1789403491.569674 punch from 0x0119f8 to 0x01bf3d (0xa546 bytes) 000247 1789403491.588295 read 0x00f254 thru 0x01191a (0x26c7 bytes) 000249 1789403491.602468 write 0x03cb96 thru 0x03ffff (0x346a bytes) 000250 1789403491.604119 mapread 0x00c766 thru 0x0136d3 (0x6f6e bytes) 000251 1789403491.605572 mapread 0x015364 thru 0x01cecf (0x7b6c bytes) 000252 1789403491.622246 MIRROR_RESYNC 3 mirrors 000253 1789403492.518265 MIRROR_EXTEND to 4 mirrors 000254 1789403493.321714 MIRROR_RESYNC 4 mirrors 000255 1789403493.794721 trunc from 0x040000 to 0x0026a6 (0x3d959 bytes) 000256 1789403493.866774 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a7): Mirror 5: 0xcd4dbd04 (stale) Mirror 6: 0xcd4dbd04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a7): Mirror 5: 0xcd4dbd04 (stale) Mirror 6: 0xcd4dbd04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a7: skipped 000257 1789403494.177929 read 0x001294 thru 0x0026a6 (0x1413 bytes) 000258 1789403494.184701 trunc from 0x0026a7 to 0x03c1c5 (0x39b1f bytes) 000259 1789403494.211330 mapwrite 0x00b5c7 thru 0x01aa3d (0xf477 bytes) 000260 1789403494.261214 trunc from 0x03c1c6 to 0x039c17 (0x25ae bytes) 000261 1789403494.280422 trunc from 0x0003fd to 0x004f10 (0x4b14 bytes) 000262 1789403494.287902 fallocate 0x03fac2 thru 0x03ffff (0x53e bytes) 000263 1789403494.294103 mapread 0x008f8e thru 0x016f7a (0xdfed bytes) 000264 1789403494.296055 mapread 0x0016d7 thru 0x00d850 (0xc17a bytes) 000265 1789403494.306720 mapread 0x0134b6 thru 0x01ad6a (0x78b5 bytes) 000266 1789403494.307964 fallocate 0x01dc2f thru 0x0210f6 (0x34c8 bytes) 000267 1789403494.324141 fallocate 0x0070a4 thru 0x009044 (0x1fa1 bytes) 000268 1789403494.334107 write 0x0348b4 thru 0x03ffff (0xb74c bytes) 000270 1789403494.346447 trunc from 0x040000 to 0x020d1c (0x1f2e3 bytes) 000271 1789403494.366820 fallocate 0x01cc18 thru 0x028400 (0xb7e9 bytes) 000272 1789403494.380239 fallocate 0x02348b thru 0x02d8dd (0xa453 bytes) 000273 1789403494.384856 mapwrite 0x029da8 thru 0x02a80c (0xa65 bytes) 000274 1789403494.405267 trunc from 0x012ad9 to 0x01ce6e (0xa396 bytes) 000275 1789403494.410330 mapread 0x016527 thru 0x016d48 (0x822 bytes) 000276 1789403494.417842 read 0x0063ff thru 0x011d26 (0xb928 bytes) 000277 1789403494.419004 mapread 0x01bf3f thru 0x02b97c (0xfa3e bytes) 000278 1789403494.428773 write 0x030700 thru 0x0380f6 (0x79f7 bytes) 000279 1789403494.432195 read 0x01a200 thru 0x022a5f (0x8860 bytes) 000280 1789403494.433085 mapread 0x00d1ff thru 0x01402f (0x6e31 bytes) 000281 1789403494.434518 punch from 0x008545 to 0x011d48 (0x9804 bytes) 000282 1789403494.440436 fallocate 0x016043 thru 0x01e068 (0x8026 bytes) 000283 1789403494.445678 mapwrite 0x00b29d thru 0x01b0bc (0xfe20 bytes) 000284 1789403494.479537 mapwrite 0x03879b thru 0x03ffff (0x7865 bytes) 000285 1789403494.524183 mapread 0x036532 thru 0x0399a7 (0x3476 bytes) 000286 1789403494.526375 read 0x0025a2 thru 0x0028a1 (0x300 bytes) 000287 1789403494.527301 read 0x032307 thru 0x0326a7 (0x3a1 bytes) 000288 1789403494.532261 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x69a5e5c0 (stale) Mirror 6: 0x69a5e5c0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd6d8bef6 Mirror 7: 0xd6d8bef6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000289 1789403495.816684 mapread 0x03b26a thru 0x03ffff (0x4d96 bytes) 000290 1789403495.848782 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x69a5e5c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x69a5e5c0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd6d8bef6 Mirror 7: 0xd6d8bef6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000291 1789403496.562333 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000292 1789403497.002297 mapwrite 0x02b0c5 thru 0x03734e (0xc28a bytes) 000293 1789403497.118560 mapread 0x001aab thru 0x01045c (0xe9b2 bytes) 000294 1789403497.152203 fallocate 0x009974 thru 0x0123be (0x8a4b bytes) 000295 1789403497.163080 trunc from 0x0145e4 to 0x01c292 (0x7caf bytes) 000296 1789403497.183062 fallocate 0x0148ca thru 0x0246b9 (0xfdf0 bytes) 000298 1789403497.196229 punch from 0x03357c to 0x03b7ed (0x8272 bytes) 000299 1789403497.207964 fallocate 0x019cb2 thru 0x01d8ed (0x3c3c bytes) 000300 1789403497.217981 fallocate 0x0384aa thru 0x03ffff (0x7b56 bytes) 000301 1789403497.224001 mapwrite 0x03b10c thru 0x03ffff (0x4ef4 bytes) 000302 1789403497.250422 mapread 0x01e28f thru 0x027a93 (0x9805 bytes) 000303 1789403497.271431 punch from 0x03ddbd to 0x03ffff (0x2243 bytes) 000304 1789403497.280875 mapread 0x01160b thru 0x013217 (0x1c0d bytes) 000305 1789403497.281924 punch from 0x03d286 to 0x03f4e1 (0x225c bytes) 000306 1789403497.292088 punch from 0x02c739 to 0x02dc85 (0x154d bytes) 000307 1789403497.300814 write 0x014d53 thru 0x0208cd (0xbb7b bytes) 000308 1789403497.329904 trunc from 0x040000 to 0x03ca19 (0x35e6 bytes) 000309 1789403497.376443 read 0x02f999 thru 0x03ca19 (0xd081 bytes) 000310 1789403497.384912 mapread 0x032451 thru 0x03ca19 (0xa5c9 bytes) 000311 1789403497.387908 read 0x02d313 thru 0x02e31a (0x1008 bytes) 000312 1789403497.396753 mapread 0x010bff thru 0x020a51 (0xfe53 bytes) 000314 1789403497.403694 write 0x00f681 thru 0x01c38c (0xcd0c bytes) 000315 1789403497.406787 read 0x002b16 thru 0x00f3f7 (0xc8e2 bytes) 000316 1789403497.407898 mapwrite 0x01b646 thru 0x0204a0 (0x4e5b bytes) 000317 1789403497.479042 fallocate 0x022c6d thru 0x029ecb (0x725f bytes) 000318 1789403497.494812 trunc from 0x009154 to 0x015089 (0xbf36 bytes) 000319 1789403497.515758 write 0x009de5 thru 0x00ff66 (0x6182 bytes) 000321 1789403497.527616 read 0x006f0c thru 0x011e5a (0xaf4f bytes) 000322 1789403497.528318 write 0x018511 thru 0x024f7d (0xca6d bytes) 000323 1789403497.532513 mapread 0x007ab1 thru 0x0130e2 (0xb632 bytes) 000324 1789403497.534029 write 0x02b174 thru 0x0394ae (0xe33b bytes) 000325 1789403497.536180 fallocate 0x022e87 thru 0x0261df (0x3359 bytes) 000326 1789403497.554823 read 0x008331 thru 0x00c647 (0x4317 bytes) 000327 1789403497.555839 write 0x002af6 thru 0x00eb35 (0xc040 bytes) 000328 1789403497.561764 trunc from 0x03ca1a to 0x03f4dc (0x2ac3 bytes) 000330 1789403497.603968 trunc from 0x03f4dd to 0x030518 (0xefc4 bytes) 000331 1789403497.673943 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30519): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30519): Mirror 7: 0x148b7401 (stale) CRC-32 checksum value for chunk [0, 0x30519): Mirror 1: 0x96ad4c85 Mirror 8: 0x96ad4c85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30519: skipped 000332 1789403498.814462 trunc from 0x030519 to 0x03b2f3 (0xaddb bytes) 000333 1789403498.922316 write 0x02e5da thru 0x03ba6b (0xd492 bytes) 000334 1789403498.940170 read 0x006acf thru 0x00acf6 (0x4228 bytes) 000335 1789403498.960996 fallocate 0x01b8a4 thru 0x0218b2 (0x600f bytes) 000337 1789403498.975407 punch from 0x02954e to 0x02faca (0x657d bytes) 000338 1789403498.995500 fallocate 0x00761e thru 0x010d3d (0x9720 bytes) 000340 1789403499.007279 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ba6c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba6c): Mirror 7: 0x739f288b (stale) Mirror 8: 0xfc715319 (stale) CRC-32 checksum value for chunk [0, 0x3ba6c): Mirror 1: 0xef76d3b4 Mirror 9: 0xef76d3b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba6c: skipped 000341 1789403500.328806 trunc from 0x01f8bd to 0x02e1bb (0xe8ff bytes) 000342 1789403500.399744 write 0x019691 thru 0x021d56 (0x86c6 bytes) 000343 1789403500.436699 write 0x031651 thru 0x0399b7 (0x8367 bytes) 000344 1789403500.480843 punch from 0x00cf8f to 0x018f63 (0xbfd5 bytes) 000345 1789403500.490814 write 0x00beb4 thru 0x01546a (0x95b7 bytes) 000346 1789403500.514421 fallocate 0x025178 thru 0x02b87d (0x6706 bytes) 000347 1789403500.524922 mapwrite 0x02cd4b thru 0x0330a5 (0x635b bytes) 000348 1789403500.623331 write 0x0044fd thru 0x00f63a (0xb13e bytes) 000349 1789403500.633143 mapread 0x0109f0 thru 0x01b916 (0xaf27 bytes) 000350 1789403500.648952 fallocate 0x018490 thru 0x01885c (0x3cd bytes) 000351 1789403500.663001 punch from 0x023000 to 0x0272dd (0x42de bytes) 000352 1789403500.680322 trunc from 0x0202f4 to 0x02e307 (0xe014 bytes) 000354 1789403500.694887 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba6c): Mirror 7: 0x739f288b (stale) Mirror 8: 0xfc715319 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba6c): Mirror 7: 0x739f288b (stale) Mirror 8: 0xfc715319 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba6c: skipped 000355 1789403501.195621 fallocate 0x018042 thru 0x019719 (0x16d8 bytes) 000356 1789403501.215533 punch from 0x0216f5 to 0x02652c (0x4e38 bytes) 000357 1789403501.223576 read 0x00f3d8 thru 0x010efe (0x1b27 bytes) 000358 1789403501.231275 mapread 0x02882b thru 0x03116f (0x8945 bytes) 000359 1789403501.257854 write 0x020faf thru 0x02982c (0x887e bytes) 000360 1789403501.278038 trunc from 0x03ba6c to 0x015f49 (0x25b22 bytes) 000361 1789403501.310573 write 0x0181a3 thru 0x01ae78 (0x2cd6 bytes) 000362 1789403501.313974 fallocate 0x022e5d thru 0x02c622 (0x97c6 bytes) 000363 1789403501.339466 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c623): Mirror 7: 0xca95bc42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c623): Mirror 7: 0xca95bc42 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c623: skipped 000364 1789403501.846112 fallocate 0x0113f2 thru 0x016446 (0x5055 bytes) 000365 1789403501.853539 mapread 0x00b07a thru 0x013b20 (0x8aa7 bytes) 000367 1789403501.912287 mapwrite 0x02e7ac thru 0x039337 (0xab8c bytes) 000368 1789403502.017309 write 0x007547 thru 0x016834 (0xf2ee bytes) 000369 1789403502.037432 punch from 0x00d642 to 0x0116e7 (0x40a6 bytes) 000370 1789403502.082534 punch from 0x033c20 to 0x036904 (0x2ce5 bytes) 000371 1789403502.090874 trunc from 0x039338 to 0x00ba96 (0x2d8a1 bytes) 000373 1789403502.124473 write 0x01ad8a thru 0x01ae00 (0x77 bytes) 000374 1789403502.134427 trunc from 0x01ae01 to 0x0177db (0x3625 bytes) 000375 1789403502.180129 fallocate 0x0171a0 thru 0x0218df (0xa740 bytes) 000376 1789403502.189565 mapwrite 0x0234a1 thru 0x02d072 (0x9bd2 bytes) 000377 1789403502.284610 trunc from 0x02d073 to 0x013932 (0x19740 bytes) 000378 1789403502.319874 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x13933): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13933): Mirror 7: 0x84534e27 (stale) CRC-32 checksum value for chunk [0, 0x13933): Mirror 1: 0x9e9b8f43 Mirror 8: 0x9e9b8f43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13933: skipped 000379 1789403503.593877 trunc from 0x013933 to 0x006d9a (0xcb98 bytes) 000380 1789403503.666201 read 0x000f2a thru 0x00325f (0x2336 bytes) 000381 1789403503.676481 write 0x03faaf thru 0x03ffff (0x551 bytes) 000382 1789403503.680326 fallocate 0x02d3fa thru 0x02dae4 (0x6eb bytes) 000383 1789403503.688967 fallocate 0x02beb0 thru 0x02df2e (0x207f bytes) 000384 1789403503.700509 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd6d8bef6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd6d8bef6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000385 1789403504.095761 read 0x00a4d8 thru 0x015935 (0xb45e bytes) 000386 1789403504.107378 write 0x039275 thru 0x03ffff (0x6d8b bytes) 000387 1789403504.123981 mapwrite 0x0018d0 thru 0x01171c (0xfe4d bytes) 000388 1789403504.202042 fallocate 0x006086 thru 0x008640 (0x25bb bytes) 000389 1789403504.209138 write 0x03b728 thru 0x03ffff (0x48d8 bytes) 000390 1789403504.210247 read 0x037187 thru 0x03ffff (0x8e79 bytes) 000392 1789403504.232187 trunc from 0x0105ac to 0x013029 (0x2a7e bytes) 000393 1789403504.258206 punch from 0x03a51b to 0x03cf94 (0x2a7a bytes) 000394 1789403504.285930 mapwrite 0x0183bd thru 0x026e2a (0xea6e bytes) 000396 1789403504.320548 mapread 0x035df4 thru 0x03e86c (0x8a79 bytes) 000398 1789403504.351456 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000399 1789403504.501578 mapread 0x016b07 thru 0x01b8a0 (0x4d9a bytes) 000400 1789403504.543317 fallocate 0x033420 thru 0x03db75 (0xa756 bytes) 000401 1789403504.563322 punch from 0x0101e9 to 0x0193a3 (0x91bb bytes) 000403 1789403504.569447 read 0x0166d4 thru 0x018bd9 (0x2506 bytes) 000404 1789403504.575628 fallocate 0x023e81 thru 0x02e95f (0xaadf bytes) 000405 1789403504.584005 punch from 0x02bca5 to 0x02edc6 (0x3122 bytes) 000406 1789403504.591111 fallocate 0x00e2fe thru 0x014bbf (0x68c2 bytes) 000407 1789403504.598330 write 0x01e051 thru 0x01efd5 (0xf85 bytes) 000408 1789403504.605503 trunc from 0x03cd4b to 0x03ffff (0x32b5 bytes) 000409 1789403504.614498 mapwrite 0x009f61 thru 0x00ea9f (0x4b3f bytes) 000411 1789403504.664050 mapread 0x031563 thru 0x033247 (0x1ce5 bytes) 000412 1789403504.687818 punch from 0x012e99 to 0x01929f (0x6407 bytes) 000413 1789403504.696235 fallocate 0x005af2 thru 0x010628 (0xab37 bytes) 000414 1789403504.701865 trunc from 0x03a3a8 to 0x03ffff (0x5c58 bytes) 000415 1789403504.707865 fallocate 0x012d17 thru 0x019d33 (0x701d bytes) 000416 1789403504.713511 trunc from 0x040000 to 0x0055ff (0x3aa00 bytes) 000417 1789403504.737298 write 0x01cb85 thru 0x02ae2b (0xe2a7 bytes) 000418 1789403504.741255 punch from 0x011669 to 0x016600 (0x4f98 bytes) 000419 1789403504.751276 trunc from 0x02ae2c to 0x010f39 (0x19ef2 bytes) 000420 1789403504.773885 mapread 0x00dbf5 thru 0x010a7a (0x2e86 bytes) 000421 1789403504.796940 read 0x004a00 thru 0x008723 (0x3d24 bytes) 000422 1789403504.800017 mapwrite 0x031280 thru 0x03ffff (0xed80 bytes) 000423 1789403504.891601 MIRROR_EXTEND to 2 mirrors 000424 1789403505.672370 fallocate 0x01fc57 thru 0x02bffd (0xc3a7 bytes) 000426 1789403505.748591 write 0x00533b thru 0x0071b4 (0x1e7a bytes) 000427 1789403505.774711 mapread 0x030a35 thru 0x033efa (0x34c6 bytes) 000428 1789403505.809822 punch from 0x0295e4 to 0x032cdc (0x96f9 bytes) 000429 1789403505.823800 trunc from 0x03866d to 0x03ffff (0x7993 bytes) 000430 1789403505.833882 fallocate 0x0053ca thru 0x008597 (0x31ce bytes) 000431 1789403505.850917 mapwrite 0x003ca5 thru 0x0130a9 (0xf405 bytes) 000432 1789403505.921316 trunc from 0x006474 to 0x014c3c (0xe7c9 bytes) 000433 1789403505.926277 trunc from 0x02513e to 0x0277ab (0x266e bytes) 000434 1789403505.930885 trunc from 0x040000 to 0x024281 (0x1bd7e bytes) 000435 1789403505.950601 trunc from 0x024282 to 0x0064ab (0x1ddd6 bytes) 000436 1789403505.969393 mapread 0x000a41 thru 0x0064ab (0x5a6b bytes) 000437 1789403505.977443 mapwrite 0x005cf4 thru 0x00a32c (0x4639 bytes) 000438 1789403506.047402 fallocate 0x024954 thru 0x0327b3 (0xde60 bytes) 000439 1789403506.058042 trunc from 0x007ac2 to 0x00a32c (0x286b bytes) 000440 1789403506.067464 read 0x001d1f thru 0x00a32c (0x860e bytes) 000441 1789403506.075480 trunc from 0x00a32d to 0x005f03 (0x4429 bytes) 000442 1789403506.096612 trunc from 0x005ea8 to 0x005f03 (0x5c bytes) 000443 1789403506.104429 punch from 0x00379d to 0x005f03 (0x2767 bytes) 000444 1789403506.113155 trunc from 0x0058ee to 0x005f03 (0x616 bytes) 000445 1789403506.118160 trunc from 0x005f04 to 0x002eb5 (0x304e bytes) 000446 1789403506.143139 punch from 0x002aed to 0x002eb5 (0x3c9 bytes) 000447 1789403506.152953 mapread 0x000d42 thru 0x002eb5 (0x2174 bytes) 000448 1789403506.158919 fallocate 0x00a64d thru 0x00ee94 (0x4848 bytes) 000449 1789403506.164563 trunc from 0x000178 to 0x002eb5 (0x2d3e bytes) 000450 1789403506.174018 trunc from 0x002520 to 0x002eb5 (0x996 bytes) 000452 1789403506.179070 trunc from 0x002eb6 to 0x03507e (0x321c9 bytes) 000453 1789403506.192320 mapread 0x02456b thru 0x0309f7 (0xc48d bytes) 000454 1789403506.217816 fallocate 0x01ff9a thru 0x02d615 (0xd67c bytes) 000456 1789403506.223308 read 0x02ecf9 thru 0x03507e (0x6386 bytes) 000457 1789403506.224426 trunc from 0x007577 to 0x00cf67 (0x59f1 bytes) 000458 1789403506.229247 trunc from 0x03507f to 0x00a399 (0x2ace5 bytes) 000460 1789403506.243336 write 0x01c69f thru 0x020a44 (0x43a6 bytes) 000461 1789403506.250364 trunc from 0x020a45 to 0x0137b0 (0xd294 bytes) 000462 1789403506.268288 mapwrite 0x0021a1 thru 0x00eb4e (0xc9ae bytes) 000463 1789403506.299627 fallocate 0x003923 thru 0x009587 (0x5c65 bytes) 000464 1789403506.308076 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000465 1789403506.476020 write 0x029b2c thru 0x0397b6 (0xfc8b bytes) 000466 1789403506.494799 mapwrite 0x008e1f thru 0x01691e (0xdb00 bytes) 000468 1789403506.592549 write 0x0268bf thru 0x02b85c (0x4f9e bytes) 000469 1789403506.600110 mapread 0x029ad0 thru 0x02ca00 (0x2f31 bytes) 000470 1789403506.609872 MIRROR_EXTEND to 2 mirrors 000471 1789403507.626017 MIRROR_EXTEND to 3 mirrors 000472 1789403508.684938 mapread 0x0072c6 thru 0x00afbe (0x3cf9 bytes) 000473 1789403508.755336 read 0x030a9b thru 0x0397b6 (0x8d1c bytes) 000474 1789403508.783535 mapwrite 0x029ab5 thru 0x02fcd9 (0x6225 bytes) 000475 1789403508.957679 trunc from 0x013f65 to 0x01c5af (0x864b bytes) 000476 1789403508.966995 mapread 0x031379 thru 0x0397b6 (0x843e bytes) 000479 1789403508.986099 fallocate 0x01e653 thru 0x024257 (0x5c05 bytes) 000480 1789403508.995259 fallocate 0x039f72 thru 0x03ffff (0x608e bytes) 000481 1789403509.005144 trunc from 0x02c8b0 to 0x0373b2 (0xab03 bytes) 000482 1789403509.024431 punch from 0x00cd17 to 0x01975a (0xca44 bytes) 000483 1789403509.037782 trunc from 0x00f850 to 0x018671 (0x8e22 bytes) 000484 1789403509.045396 fallocate 0x019dcc thru 0x027f75 (0xe1aa bytes) 000485 1789403509.069404 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf563f401 (stale) Mirror 3: 0xf563f401 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x27edbf0 Mirror 4: 0x27edbf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000486 1789403510.369516 trunc from 0x040000 to 0x004529 (0x3bad6 bytes) 000487 1789403510.446839 mapwrite 0x038a80 thru 0x03ffff (0x7580 bytes) 000488 1789403510.549472 trunc from 0x024d47 to 0x027270 (0x252a bytes) 000489 1789403510.555602 mapwrite 0x03508e thru 0x039a04 (0x4977 bytes) 000490 1789403510.579971 trunc from 0x040000 to 0x030046 (0xffb9 bytes) 000493 1789403510.621127 mapwrite 0x035eb5 thru 0x03b82a (0x5976 bytes) 000495 1789403510.722240 fallocate 0x00cb69 thru 0x01a644 (0xdadc bytes) 000496 1789403510.738515 mapwrite 0x03b54e thru 0x03ffff (0x4ab2 bytes) 000497 1789403510.825831 trunc from 0x02a4aa to 0x0324a4 (0x7ffb bytes) 000498 1789403510.839826 fallocate 0x01c28c thru 0x01f80f (0x3584 bytes) 000499 1789403510.849441 trunc from 0x022f58 to 0x02b32f (0x83d8 bytes) 000500 1789403510.862069 mapwrite 0x001a2b thru 0x011081 (0xf657 bytes) 000501 1789403510.910307 punch from 0x011c87 to 0x0167b0 (0x4b2a bytes) 000502 1789403510.916295 mapread 0x032180 thru 0x03ffff (0xde80 bytes) 000503 1789403510.927067 punch from 0x023737 to 0x02b9b3 (0x827d bytes) 000504 1789403510.934228 punch from 0x02af71 to 0x031a23 (0x6ab3 bytes) 000505 1789403510.939603 fallocate 0x01d4e1 thru 0x01ec79 (0x1799 bytes) 000507 1789403510.946390 write 0x0204c3 thru 0x029c38 (0x9776 bytes) 000508 1789403510.956975 mapread 0x0222c8 thru 0x02c760 (0xa499 bytes) 000509 1789403510.967858 trunc from 0x040000 to 0x006e0e (0x391f1 bytes) 000510 1789403510.993137 fallocate 0x00d467 thru 0x01c0cd (0xec67 bytes) 000511 1789403511.001768 mapread 0x014673 thru 0x015b37 (0x14c5 bytes) 000512 1789403511.028078 mapread 0x0123f5 thru 0x01bfef (0x9bfb bytes) 000513 1789403511.030147 mapread 0x00d73f thru 0x017469 (0x9d2b bytes) 000514 1789403511.031447 trunc from 0x01c0ce to 0x032d01 (0x16c34 bytes) 000515 1789403511.057364 mapwrite 0x03c8cf thru 0x03e87c (0x1fae bytes) 000516 1789403511.171387 read 0x006f01 thru 0x00bbf0 (0x4cf0 bytes) 000517 1789403511.190580 trunc from 0x03e87d to 0x032fbd (0xb8bf bytes) 000518 1789403511.221618 write 0x006382 thru 0x006ed2 (0xb51 bytes) 000519 1789403511.236139 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32fbe): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32fbe): Mirror 2: 0x30df3715 (stale) Mirror 3: 0x30df3715 (stale) Mirror 4: 0xe629709b (stale) CRC-32 checksum value for chunk [0, 0x32fbe): Mirror 1: 0xeba2cdcc Mirror 5: 0xeba2cdcc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32fbe: skipped 000520 1789403512.445143 mapwrite 0x0259f3 thru 0x032e39 (0xd447 bytes) 000521 1789403512.617754 trunc from 0x032fbe to 0x02c4ad (0x6b10 bytes) 000522 1789403512.685543 fallocate 0x039d09 thru 0x03ffff (0x62f7 bytes) 000523 1789403512.695166 read 0x00cc46 thru 0x015746 (0x8b01 bytes) 000525 1789403512.710805 fallocate 0x022d4f thru 0x031877 (0xeb29 bytes) 000526 1789403512.722292 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf563f401 (stale) Mirror 3: 0xf563f401 (stale) Mirror 4: 0x27edbf0 (stale) Mirror 5: 0x4cdd679c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2359fa02 Mirror 6: 0x2359fa02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000527 1789403513.972510 mapwrite 0x009bdc thru 0x01697a (0xcd9f bytes) 000528 1789403514.241370 mapwrite 0x031287 thru 0x03bac5 (0xa83f bytes) 000529 1789403514.323493 punch from 0x001b34 to 0x004972 (0x2e3f bytes) 000530 1789403514.331157 fallocate 0x01359d thru 0x022874 (0xf2d8 bytes) 000532 1789403514.342848 mapwrite 0x01efc7 thru 0x0235de (0x4618 bytes) 000533 1789403514.413187 fallocate 0x00e3e1 thru 0x01422b (0x5e4b bytes) 000534 1789403514.422377 trunc from 0x040000 to 0x000487 (0x3fb78 bytes) 000535 1789403514.488421 trunc from 0x000302 to 0x000487 (0x186 bytes) 000536 1789403514.497392 mapread 0x0000d0 thru 0x000487 (0x3b8 bytes) 000537 1789403514.511496 mapwrite 0x00576e thru 0x005975 (0x208 bytes) 000538 1789403514.584903 mapwrite 0x019971 thru 0x0242a9 (0xa939 bytes) 000539 1789403514.717788 read 0x0064ec thru 0x014d08 (0xe81d bytes) 000540 1789403514.740711 trunc from 0x019fa7 to 0x0242a9 (0xa303 bytes) 000541 1789403514.748402 write 0x03a915 thru 0x03ffff (0x56eb bytes) 000542 1789403514.749743 trunc from 0x040000 to 0x012982 (0x2d67d bytes) 000543 1789403514.784781 MIRROR_RESYNC 6 mirrors 000544 1789403516.550421 trunc from 0x012983 to 0x02ba44 (0x190c2 bytes) 000546 1789403516.669086 trunc from 0x018b63 to 0x023386 (0xa824 bytes) 000547 1789403516.681017 write 0x021933 thru 0x02cd0e (0xb3dc bytes) 000548 1789403516.703806 read 0x000da8 thru 0x009bae (0x8e07 bytes) 000551 1789403516.724455 read 0x0179f9 thru 0x021b78 (0xa180 bytes) 000552 1789403516.744483 write 0x0259a2 thru 0x02af8d (0x55ec bytes) 000553 1789403516.753797 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd0f): Mirror 3: 0x37ccd8b6 (stale) Mirror 4: 0x37ccd8b6 (stale) Mirror 5: 0x37ccd8b6 (stale) Mirror 6: 0x37ccd8b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd0f): Mirror 3: 0x37ccd8b6 (stale) Mirror 4: 0x37ccd8b6 (stale) Mirror 5: 0x37ccd8b6 (stale) Mirror 6: 0x37ccd8b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd0f: skipped 000554 1789403517.119353 fallocate 0x017908 thru 0x01a453 (0x2b4c bytes) 000555 1789403517.126241 trunc from 0x01776d to 0x01f746 (0x7fda bytes) 000556 1789403517.131324 mapread 0x005b0c thru 0x013e02 (0xe2f7 bytes) 000557 1789403517.156951 fallocate 0x029b51 thru 0x0347c0 (0xac70 bytes) 000558 1789403517.161793 mapread 0x0109b3 thru 0x017ede (0x752c bytes) 000560 1789403517.164418 trunc from 0x02cd0f to 0x00c89e (0x20470 bytes) 000562 1789403517.187120 punch from 0x00398e to 0x003d7b (0x3ee bytes) 000563 1789403517.192382 read 0x0076d0 thru 0x00c89e (0x51cf bytes) 000564 1789403517.193202 mapwrite 0x0270a4 thru 0x02ebd4 (0x7b31 bytes) 000565 1789403517.237781 fallocate 0x01f655 thru 0x028959 (0x9305 bytes) 000566 1789403517.242526 write 0x03c113 thru 0x03cfb8 (0xea6 bytes) 000567 1789403517.244419 trunc from 0x01d4eb to 0x029be6 (0xc6fc bytes) 000568 1789403517.249333 punch from 0x00f532 to 0x01eb00 (0xf5cf bytes) 000569 1789403517.253787 fallocate 0x035155 thru 0x037755 (0x2601 bytes) 000570 1789403517.258406 mapwrite 0x019a94 thru 0x0252ce (0xb83b bytes) 000571 1789403517.299432 fallocate 0x0053f1 thru 0x008e38 (0x3a48 bytes) 000572 1789403517.304740 mapwrite 0x01e42e thru 0x02c329 (0xdefc bytes) 000573 1789403517.325609 read 0x0139c6 thru 0x013e13 (0x44e bytes) 000575 1789403517.326763 trunc from 0x03cfb9 to 0x01b09e (0x21f1a bytes) 000576 1789403517.343132 read 0x00c77f thru 0x018a9a (0xc31c bytes) 000577 1789403517.355210 punch from 0x002ffa to 0x0063ff (0x3406 bytes) 000578 1789403517.360611 write 0x02e893 thru 0x02f57e (0xcec bytes) 000579 1789403517.361687 write 0x03fb1e thru 0x03ffff (0x4e2 bytes) 000580 1789403517.366826 MIRROR_RESYNC 5 mirrors 000581 1789403518.890929 mapwrite 0x03bdae thru 0x03ffff (0x4252 bytes) 000583 1789403519.027111 trunc from 0x040000 to 0x002818 (0x3d7e7 bytes) 000584 1789403519.081130 mapread 0x001438 thru 0x002818 (0x13e1 bytes) 000585 1789403519.104537 trunc from 0x002819 to 0x029c97 (0x2747f bytes) 000586 1789403519.132895 trunc from 0x02817c to 0x029c97 (0x1b1c bytes) 000587 1789403519.138919 punch from 0x016395 to 0x021753 (0xb3bf bytes) 000588 1789403519.146167 write 0x02ad03 thru 0x03976b (0xea69 bytes) 000589 1789403519.149245 fallocate 0x03fed9 thru 0x03ffff (0x127 bytes) 000591 1789403519.156288 trunc from 0x0331d5 to 0x03976b (0x6597 bytes) 000592 1789403519.182306 write 0x02c71f thru 0x033348 (0x6c2a bytes) 000594 1789403519.191850 mapwrite 0x02bac4 thru 0x03b2a3 (0xf7e0 bytes) 000595 1789403519.271801 punch from 0x021542 to 0x02f964 (0xe423 bytes) 000596 1789403519.277374 trunc from 0x03b2a4 to 0x035b91 (0x5712 bytes) 000597 1789403519.307072 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35b92): Mirror 3: 0x885c883e (stale) Mirror 5: 0x885c883e (stale) Mirror 6: 0x885c883e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b92): Mirror 3: 0x885c883e (stale) Mirror 5: 0x885c883e (stale) Mirror 6: 0x885c883e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b92: skipped 000598 1789403519.795965 trunc from 0x0205d7 to 0x02e39c (0xddc6 bytes) 000599 1789403519.805518 write 0x01de75 thru 0x0253f8 (0x7584 bytes) 000601 1789403519.845928 trunc from 0x0172bb to 0x01bc05 (0x494b bytes) 000602 1789403519.856949 mapread 0x0315ba thru 0x035b91 (0x45d8 bytes) 000603 1789403519.890039 trunc from 0x00e3ec to 0x01241c (0x4031 bytes) 000604 1789403519.908504 fallocate 0x01f4d4 thru 0x020cf5 (0x1822 bytes) 000605 1789403519.920824 write 0x00cbe6 thru 0x0124a9 (0x58c4 bytes) 000606 1789403519.935735 trunc from 0x02c226 to 0x032521 (0x62fc bytes) 000607 1789403519.947473 trunc from 0x019797 to 0x02803d (0xe8a7 bytes) 000608 1789403519.987812 trunc from 0x0042eb to 0x007ad0 (0x37e6 bytes) 000609 1789403520.000373 trunc from 0x01e028 to 0x02851a (0xa4f3 bytes) 000610 1789403520.020859 MIRROR_RESYNC 4 mirrors 000613 1789403521.366808 trunc from 0x02b530 to 0x02c615 (0x10e6 bytes) 000614 1789403521.403789 trunc from 0x035b92 to 0x00a21c (0x2b975 bytes) 000615 1789403521.423170 mapread 0x004dbd thru 0x00a21c (0x5460 bytes) 000616 1789403521.439974 read 0x008ea5 thru 0x00a21c (0x1378 bytes) 000617 1789403521.440557 trunc from 0x00a21d to 0x023546 (0x1932a bytes) 000618 1789403521.455382 fallocate 0x0180ea thru 0x0217e0 (0x96f7 bytes) 000619 1789403521.465123 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23547): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23547): Mirror 3: 0x1283d166 (stale) Mirror 5: 0x1283d166 (stale) Mirror 6: 0x1283d166 (stale) CRC-32 checksum value for chunk [0, 0x23547): Mirror 1: 0x6edac598 Mirror 7: 0x6edac598 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23547: skipped 000620 1789403522.415059 fallocate 0x011bb7 thru 0x014bee (0x3038 bytes) 000621 1789403522.450042 trunc from 0x023547 to 0x03ad28 (0x177e2 bytes) 000622 1789403522.467690 read 0x016c2d thru 0x024232 (0xd606 bytes) 000623 1789403522.482055 mapwrite 0x03872b thru 0x038e6a (0x740 bytes) 000624 1789403522.498349 mapwrite 0x01920b thru 0x024b93 (0xb989 bytes) 000625 1789403522.516933 mapread 0x0178e8 thru 0x01ad74 (0x348d bytes) 000627 1789403522.518253 write 0x007e8b thru 0x01686c (0xe9e2 bytes) 000628 1789403522.524262 read 0x016d5a thru 0x020be3 (0x9e8a bytes) 000629 1789403522.524876 mapread 0x034e97 thru 0x03ad28 (0x5e92 bytes) 000630 1789403522.539806 fallocate 0x0283b2 thru 0x03132c (0x8f7b bytes) 000631 1789403522.544372 mapread 0x01ce7d thru 0x01eef4 (0x2078 bytes) 000632 1789403522.548835 MIRROR_RESYNC 5 mirrors 000634 1789403524.072138 mapwrite 0x00638b thru 0x01174b (0xb3c1 bytes) 000635 1789403524.213955 punch from 0x039e44 to 0x03ad28 (0xee5 bytes) 000636 1789403524.219782 punch from 0x02c0e3 to 0x034aec (0x8a0a bytes) 000637 1789403524.226860 mapread 0x0040da thru 0x01288c (0xe7b3 bytes) 000638 1789403524.229236 read 0x032145 thru 0x03220d (0xc9 bytes) 000639 1789403524.235535 fallocate 0x002d8c thru 0x00a4d0 (0x7745 bytes) 000640 1789403524.244901 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ad29): Mirror 3: 0x77a87f4f (stale) Mirror 5: 0x77a87f4f (stale) Mirror 6: 0x77a87f4f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad29): Mirror 3: 0x77a87f4f (stale) Mirror 5: 0x77a87f4f (stale) Mirror 6: 0x77a87f4f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad29: skipped 000641 1789403524.664421 trunc from 0x03ad29 to 0x006cc0 (0x34068 bytes) 000642 1789403524.694969 trunc from 0x0004c2 to 0x006cc0 (0x67ff bytes) 000643 1789403524.704047 read 0x006af7 thru 0x006cc0 (0x1ca bytes) 000644 1789403524.709484 read 0x000de4 thru 0x003e51 (0x306e bytes) 000645 1789403524.721449 fallocate 0x012e75 thru 0x020ed4 (0xe060 bytes) 000646 1789403524.726558 trunc from 0x020ed5 to 0x032d46 (0x11e72 bytes) 000647 1789403524.742007 mapwrite 0x0330a3 thru 0x03ffff (0xcf5d bytes) 000648 1789403524.820373 trunc from 0x03b0f9 to 0x03ef85 (0x3e8d bytes) 000649 1789403524.826851 read 0x00f285 thru 0x01ed25 (0xfaa1 bytes) 000650 1789403524.827745 read 0x03b1de thru 0x03e049 (0x2e6c bytes) 000651 1789403524.836009 write 0x00f391 thru 0x0111d5 (0x1e45 bytes) 000652 1789403524.837284 trunc from 0x02af2a to 0x034225 (0x92fc bytes) 000653 1789403524.842492 punch from 0x019c45 to 0x0262e1 (0xc69d bytes) 000654 1789403524.847578 punch from 0x01b704 to 0x029244 (0xdb41 bytes) 000655 1789403524.853004 read 0x02d71d thru 0x0311e4 (0x3ac8 bytes) 000656 1789403524.873052 read 0x013aa9 thru 0x01a61d (0x6b75 bytes) 000658 1789403524.874169 trunc from 0x0011be to 0x004b08 (0x394b bytes) 000659 1789403524.878465 read 0x0017b1 thru 0x00a902 (0x9152 bytes) 000660 1789403524.885697 mapwrite 0x0077e2 thru 0x00ee3d (0x765c bytes) 000661 1789403524.903145 mapwrite 0x0317fd thru 0x03ef89 (0xd78d bytes) 000663 1789403524.936727 read 0x01c31c thru 0x025c9e (0x9983 bytes) 000664 1789403524.937288 trunc from 0x0168cc to 0x01964e (0x2d83 bytes) 000665 1789403524.941825 write 0x0101d8 thru 0x012ce1 (0x2b0a bytes) 000666 1789403524.943342 read 0x02d973 thru 0x03d3e4 (0xfa72 bytes) 000668 1789403524.948712 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8c5b0002 (stale) Mirror 5: 0x8c5b0002 (stale) Mirror 6: 0x8c5b0002 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x634cbba1 Mirror 7: 0x634cbba1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000669 1789403526.110208 mapread 0x027f77 thru 0x0329d1 (0xaa5b bytes) 000671 1789403526.213543 fallocate 0x020e83 thru 0x028dc9 (0x7f47 bytes) 000672 1789403526.274027 write 0x03e313 thru 0x03ffff (0x1ced bytes) 000674 1789403526.287154 trunc from 0x02b5fb to 0x037186 (0xbb8c bytes) 000675 1789403526.303085 trunc from 0x040000 to 0x02ec25 (0x113da bytes) 000677 1789403526.340764 mapread 0x010b1a thru 0x01ba94 (0xaf7b bytes) 000678 1789403526.360305 read 0x01d9a5 thru 0x02a361 (0xc9bd bytes) 000679 1789403526.370453 trunc from 0x0191bc to 0x01a12d (0xf72 bytes) 000680 1789403526.382302 trunc from 0x02ec26 to 0x0163bf (0x18866 bytes) 000681 1789403526.408806 trunc from 0x0163c0 to 0x01fbd4 (0x9815 bytes) 000682 1789403526.437228 fallocate 0x0084fe thru 0x015312 (0xce15 bytes) 000683 1789403526.443375 mapwrite 0x0242f6 thru 0x0341fd (0xff08 bytes) 000684 1789403526.542043 fallocate 0x002acd thru 0x01233a (0xf86e bytes) 000685 1789403526.555262 fallocate 0x00ccc9 thru 0x014953 (0x7c8b bytes) 000686 1789403526.565355 trunc from 0x0341fe to 0x01e2a0 (0x15f5d bytes) 000687 1789403526.620632 write 0x000f07 thru 0x00298e (0x1a88 bytes) 000688 1789403526.636300 fallocate 0x01d667 thru 0x023ddd (0x6777 bytes) 000689 1789403526.651967 trunc from 0x017bbd to 0x020071 (0x84b5 bytes) 000690 1789403526.661181 mapwrite 0x0328ea thru 0x03517f (0x2896 bytes) 000691 1789403526.744789 write 0x039e57 thru 0x03eac6 (0x4c70 bytes) 000692 1789403526.764013 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3eac7): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eac7): Mirror 3: 0x7a61853 (stale) Mirror 5: 0x7a61853 (stale) Mirror 6: 0x7a61853 (stale) Mirror 7: 0x4b304c1d (stale) CRC-32 checksum value for chunk [0, 0x3eac7): Mirror 1: 0x2de26cc1 Mirror 8: 0x2de26cc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eac7: skipped 000693 1789403527.950130 mapread 0x02d379 thru 0x0344fc (0x7184 bytes) 000694 1789403527.988781 fallocate 0x03b1b8 thru 0x03ffff (0x4e48 bytes) 000695 1789403528.056816 trunc from 0x040000 to 0x004004 (0x3bffb bytes) 000696 1789403528.131486 mapread 0x003034 thru 0x004004 (0xfd1 bytes) 000697 1789403528.143281 mapwrite 0x03e5fb thru 0x03ffff (0x1a05 bytes) 000698 1789403528.238351 mapwrite 0x0131c7 thru 0x017e20 (0x4c5a bytes) 000699 1789403528.280181 read 0x03086d thru 0x03b606 (0xad9a bytes) 000700 1789403528.293108 trunc from 0x039cf4 to 0x03ffff (0x630c bytes) 000701 1789403528.298672 mapwrite 0x00ca01 thru 0x0171b1 (0xa7b1 bytes) 000702 1789403528.327833 MIRROR_RESYNC 6 mirrors 000703 1789403529.840962 write 0x000362 thru 0x008673 (0x8312 bytes) 000704 1789403529.897401 trunc from 0x040000 to 0x01de92 (0x2216d bytes) 000705 1789403529.940255 write 0x00234e thru 0x002542 (0x1f5 bytes) 000706 1789403529.941249 trunc from 0x01de93 to 0x0216ef (0x385d bytes) 000707 1789403529.966345 mapread 0x018db6 thru 0x01c1a1 (0x33ec bytes) 000708 1789403529.999306 mapwrite 0x0308e7 thru 0x03ae23 (0xa53d bytes) 000709 1789403530.062397 fallocate 0x005695 thru 0x0070c1 (0x1a2d bytes) 000710 1789403530.067094 trunc from 0x03ae24 to 0x006c7e (0x341a5 bytes) 000711 1789403530.083457 punch from 0x003839 to 0x006c7e (0x3446 bytes) 000712 1789403530.090058 read 0x002114 thru 0x006c7e (0x4b6b bytes) 000713 1789403530.105377 fallocate 0x02607b thru 0x027940 (0x18c6 bytes) 000714 1789403530.112416 fallocate 0x00d123 thru 0x01477f (0x765d bytes) 000715 1789403530.122000 trunc from 0x027941 to 0x01ac04 (0xcd3c bytes) 000716 1789403530.143774 mapread 0x00b1e7 thru 0x017141 (0xbf5b bytes) 000717 1789403530.183506 mapread 0x0032f6 thru 0x003ff0 (0xcfb bytes) 000718 1789403530.184991 punch from 0x019aae to 0x01a5b8 (0xb0b bytes) 000719 1789403530.190089 fallocate 0x02ea92 thru 0x036ccd (0x823c bytes) 000720 1789403530.195886 mapwrite 0x010914 thru 0x0186ba (0x7da7 bytes) 000722 1789403530.218832 trunc from 0x0073c4 to 0x009723 (0x2360 bytes) 000723 1789403530.223790 mapread 0x002b01 thru 0x006820 (0x3d20 bytes) 000724 1789403530.229491 punch from 0x012f1a to 0x01ac04 (0x7ceb bytes) 000725 1789403530.235984 mapwrite 0x0262f3 thru 0x0280bc (0x1dca bytes) 000726 1789403530.272473 trunc from 0x020cb0 to 0x021b4d (0xe9e bytes) 000727 1789403530.278904 punch from 0x009799 to 0x00c5b7 (0x2e1f bytes) 000728 1789403530.284481 read 0x01e1f6 thru 0x0280bc (0x9ec7 bytes) 000729 1789403530.300012 MIRROR_RESYNC 6 mirrors 000731 1789403532.191926 read 0x0061ff thru 0x009eb2 (0x3cb4 bytes) 000732 1789403532.197983 trunc from 0x026e80 to 0x027de2 (0xf63 bytes) 000733 1789403532.262094 write 0x0293d5 thru 0x03164e (0x827a bytes) 000735 1789403532.264175 mapwrite 0x02d37b thru 0x034cb3 (0x7939 bytes) 000736 1789403532.327508 mapread 0x004ab2 thru 0x00cb97 (0x80e6 bytes) 000737 1789403532.345231 mapread 0x00ff6c thru 0x01bca3 (0xbd38 bytes) 000738 1789403532.379934 punch from 0x02102f to 0x021f96 (0xf68 bytes) 000739 1789403532.391571 trunc from 0x034cb4 to 0x024b0c (0x101a7 bytes) 000741 1789403532.424084 trunc from 0x024b0d to 0x02cccc (0x81c0 bytes) 000742 1789403532.446166 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2cccd): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cccd): Mirror 3: 0x23e9a121 (stale) Mirror 5: 0x23e9a121 (stale) Mirror 6: 0x23e9a121 (stale) Mirror 7: 0x23e9a121 (stale) Mirror 8: 0x23e9a121 (stale) CRC-32 checksum value for chunk [0, 0x2cccd): Mirror 1: 0xbbb07582 Mirror 9: 0xbbb07582 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cccd: skipped 000743 1789403533.666475 write 0x015d32 thru 0x021a9e (0xbd6d bytes) 000744 1789403533.726284 write 0x00e214 thru 0x01a2f9 (0xc0e6 bytes) 000745 1789403533.734714 mapwrite 0x009875 thru 0x016007 (0xc793 bytes) 000746 1789403533.788282 mapwrite 0x026d35 thru 0x0272c1 (0x58d bytes) 000747 1789403533.810446 mapread 0x00e0a7 thru 0x01d255 (0xf1af bytes) 000748 1789403533.811872 mapread 0x012716 thru 0x02039a (0xdc85 bytes) 000749 1789403533.813806 trunc from 0x02cccd to 0x024519 (0x87b3 bytes) 000750 1789403533.846333 mapread 0x011f95 thru 0x016ecb (0x4f37 bytes) 000751 1789403533.849109 trunc from 0x02451a to 0x03dd62 (0x19849 bytes) 000752 1789403533.869389 punch from 0x0353a6 to 0x035968 (0x5c3 bytes) 000753 1789403533.874576 mapread 0x0178e7 thru 0x018b52 (0x126c bytes) 000754 1789403533.885218 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3dd63): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd63): Mirror 3: 0x31f6ada3 (stale) Mirror 5: 0x31f6ada3 (stale) Mirror 6: 0x31f6ada3 (stale) Mirror 7: 0x31f6ada3 (stale) Mirror 8: 0x31f6ada3 (stale) Mirror 9: 0x6b77ad2b (stale) CRC-32 checksum value for chunk [0, 0x3dd63): Mirror 1: 0xd690f5a4 Mirror 10: 0xd690f5a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd63: skipped 000755 1789403535.066442 read 0x00e203 thru 0x01226c (0x406a bytes) 000756 1789403535.084362 mapwrite 0x02acb9 thru 0x02d156 (0x249e bytes) 000757 1789403535.208880 read 0x016622 thru 0x02183d (0xb21c bytes) 000758 1789403535.233215 fallocate 0x02d5d1 thru 0x038c5b (0xb68b bytes) 000759 1789403535.242947 write 0x037568 thru 0x03df7a (0x6a13 bytes) 000760 1789403535.267025 trunc from 0x03df7b to 0x014709 (0x29871 bytes) 000761 1789403535.331390 punch from 0x01032b to 0x014709 (0x43df bytes) 000762 1789403535.338131 mapwrite 0x01862b thru 0x026461 (0xde37 bytes) 000763 1789403535.554309 MIRROR_RESYNC 8 mirrors 000766 1789403538.520387 fallocate 0x028fc1 thru 0x032165 (0x91a5 bytes) 000767 1789403538.558155 trunc from 0x026462 to 0x01d707 (0x8d5a bytes) 000769 1789403538.583233 trunc from 0x005787 to 0x00d4e3 (0x7d5d bytes) 000770 1789403538.589039 fallocate 0x012335 thru 0x0210d0 (0xed9c bytes) 000771 1789403538.594292 write 0x036ddb thru 0x039c28 (0x2e4e bytes) 000772 1789403538.596478 mapread 0x0275b9 thru 0x027e09 (0x851 bytes) 000773 1789403538.602304 mapread 0x016aa6 thru 0x02629a (0xf7f5 bytes) 000774 1789403538.622353 punch from 0x007a8c to 0x00c0b6 (0x462b bytes) 000775 1789403538.626956 fallocate 0x027aee thru 0x0339fd (0xbf10 bytes) 000777 1789403538.632111 trunc from 0x039c29 to 0x033cbc (0x5f6c bytes) 000778 1789403538.649377 mapwrite 0x00d067 thru 0x01bda2 (0xed3c bytes) 000779 1789403538.692924 mapwrite 0x00c5e8 thru 0x00e799 (0x21b2 bytes) 000780 1789403538.712737 trunc from 0x024a39 to 0x028843 (0x3e0b bytes) 000781 1789403538.723253 mapwrite 0x01c926 thru 0x01f722 (0x2dfd bytes) 000783 1789403538.757440 read 0x0161c9 thru 0x01ea20 (0x8858 bytes) 000784 1789403538.758152 punch from 0x021f0f to 0x022681 (0x773 bytes) 000785 1789403538.770062 fallocate 0x007dfe thru 0x0129f0 (0xabf3 bytes) 000786 1789403538.776837 trunc from 0x00a81a to 0x01433b (0x9b22 bytes) 000787 1789403538.790680 trunc from 0x007ba9 to 0x00f243 (0x769b bytes) 000788 1789403538.796595 mapread 0x023935 thru 0x02932d (0x59f9 bytes) 000789 1789403538.814221 mapwrite 0x02c42a thru 0x02ef31 (0x2b08 bytes) 000790 1789403538.864892 read 0x016153 thru 0x01b7f2 (0x56a0 bytes) 000791 1789403538.865859 mapread 0x01634e thru 0x01a54d (0x4200 bytes) 000792 1789403538.870849 write 0x00d34d thru 0x015bb9 (0x886d bytes) 000793 1789403538.898590 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33cbd): Mirror 3: 0x9f968148 (stale) Mirror 5: 0x9f968148 (stale) Mirror 6: 0x9f968148 (stale) Mirror 7: 0x9f968148 (stale) Mirror 9: 0x9f968148 (stale) Mirror 10: 0x9f968148 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cbd): Mirror 3: 0x9f968148 (stale) Mirror 5: 0x9f968148 (stale) Mirror 6: 0x9f968148 (stale) Mirror 7: 0x9f968148 (stale) Mirror 9: 0x9f968148 (stale) Mirror 10: 0x9f968148 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cbd: skipped 000796 1789403539.399586 mapread 0x01b18c thru 0x0260b7 (0xaf2c bytes) 000797 1789403539.472921 mapwrite 0x02af2c thru 0x02ffe0 (0x50b5 bytes) 000798 1789403539.500567 trunc from 0x033cbd to 0x028e92 (0xae2a bytes) 000799 1789403539.544154 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28e93): Mirror 3: 0xeea9dd35 (stale) Mirror 5: 0xeea9dd35 (stale) Mirror 7: 0xeea9dd35 (stale) Mirror 9: 0xeea9dd35 (stale) Mirror 10: 0xeea9dd35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28e93): Mirror 3: 0xeea9dd35 (stale) Mirror 5: 0xeea9dd35 (stale) Mirror 7: 0xeea9dd35 (stale) Mirror 9: 0xeea9dd35 (stale) Mirror 10: 0xeea9dd35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28e93: skipped 000800 1789403540.169966 write 0x0341f7 thru 0x03dd0f (0x9b19 bytes) 000802 1789403540.177238 trunc from 0x03dd10 to 0x02b34a (0x129c5 bytes) 000804 1789403540.197896 trunc from 0x02b34b to 0x016fda (0x14370 bytes) 000806 1789403540.217900 trunc from 0x0110aa to 0x016fda (0x5f31 bytes) 000807 1789403540.224407 mapwrite 0x0386f5 thru 0x03a721 (0x202d bytes) 000808 1789403540.278093 trunc from 0x03a722 to 0x01b298 (0x1f489 bytes) 000809 1789403540.301227 punch from 0x00f16d to 0x017427 (0x82bb bytes) 000810 1789403540.308234 trunc from 0x01b299 to 0x03190e (0x16676 bytes) 000811 1789403540.329517 fallocate 0x01445f thru 0x02379f (0xf341 bytes) 000812 1789403540.335158 trunc from 0x03190f to 0x02f53a (0x23d4 bytes) 000813 1789403540.353391 mapwrite 0x0259e2 thru 0x025be6 (0x205 bytes) 000815 1789403540.375206 fallocate 0x016325 thru 0x01cd77 (0x6a53 bytes) 000816 1789403540.380277 write 0x018fef thru 0x024126 (0xb138 bytes) 000818 1789403540.387916 punch from 0x00551a to 0x0055b2 (0x99 bytes) 000819 1789403540.393232 punch from 0x01b50e to 0x02a7cb (0xf2be bytes) 000820 1789403540.409316 fallocate 0x028187 thru 0x03503a (0xceb4 bytes) 000821 1789403540.414677 mapwrite 0x02ddf9 thru 0x03916a (0xb372 bytes) 000822 1789403540.492353 trunc from 0x03916b to 0x01c44e (0x1cd1c bytes) 000824 1789403540.528291 trunc from 0x01c44f to 0x031d0c (0x158be bytes) 000827 1789403540.557470 write 0x038258 thru 0x03ffff (0x7da8 bytes) 000828 1789403540.560404 fallocate 0x01eece thru 0x022cba (0x3ded bytes) 000829 1789403540.573843 fallocate 0x038700 thru 0x03ffff (0x7900 bytes) 000830 1789403540.601937 punch from 0x026d79 to 0x02fcb9 (0x8f41 bytes) 000831 1789403540.609105 mapread 0x0232b3 thru 0x031560 (0xe2ae bytes) 000832 1789403540.657631 write 0x0030dc thru 0x005708 (0x262d bytes) 000833 1789403540.682084 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x770b4474 (stale) Mirror 7: 0x770b4474 (stale) Mirror 9: 0x770b4474 (stale) Mirror 10: 0x770b4474 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x770b4474 (stale) Mirror 7: 0x770b4474 (stale) Mirror 9: 0x770b4474 (stale) Mirror 10: 0x770b4474 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000834 1789403541.109428 write 0x010056 thru 0x01d1d4 (0xd17f bytes) 000835 1789403541.133853 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x770b4474 (stale) Mirror 7: 0x770b4474 (stale) Mirror 10: 0x770b4474 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x770b4474 (stale) Mirror 7: 0x770b4474 (stale) Mirror 10: 0x770b4474 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000836 1789403541.544054 trunc from 0x040000 to 0x036d4e (0x92b1 bytes) 000837 1789403541.574490 read 0x02d044 thru 0x02d682 (0x63f bytes) 000838 1789403541.588901 fallocate 0x032c0a thru 0x03ffff (0xd3f6 bytes) 000841 1789403541.607274 read 0x02c924 thru 0x03aeb1 (0xe58e bytes) 000842 1789403541.628286 read 0x0305af thru 0x03d453 (0xcea5 bytes) 000844 1789403541.632768 trunc from 0x040000 to 0x013cc4 (0x2c33b bytes) 000845 1789403541.671241 punch from 0x00d362 to 0x013cc4 (0x6963 bytes) 000848 1789403541.700807 MIRROR_RESYNC 4 mirrors 000849 1789403542.925757 mapread 0x0051c6 thru 0x0136cb (0xe506 bytes) 000850 1789403542.972391 write 0x00ff36 thru 0x01b747 (0xb812 bytes) 000851 1789403543.010112 fallocate 0x006199 thru 0x015d5e (0xfbc6 bytes) 000853 1789403543.025280 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b748): Mirror 7: 0xd171891f (stale) Mirror 10: 0xd171891f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b748): Mirror 7: 0xd171891f (stale) Mirror 10: 0xd171891f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b748: skipped 000854 1789403543.470914 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b748): Mirror 10: 0xd171891f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b748): Mirror 10: 0xd171891f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b748: skipped 000855 1789403543.840424 MIRROR_RESYNC 2 mirrors 000856 1789403544.526906 read 0x00e012 thru 0x015862 (0x7851 bytes) 000858 1789403544.536901 write 0x00d188 thru 0x01597c (0x87f5 bytes) 000859 1789403544.595720 fallocate 0x01cb4b thru 0x024a7e (0x7f34 bytes) 000860 1789403544.605320 trunc from 0x024a7f to 0x016ade (0xdfa0 bytes) 000861 1789403544.637948 trunc from 0x016adf to 0x0011d8 (0x15906 bytes) 000862 1789403544.658631 write 0x01d20a thru 0x025ebf (0x8cb6 bytes) 000864 1789403544.665893 punch from 0x02093d to 0x025ebf (0x5583 bytes) 000865 1789403544.690969 trunc from 0x025ec0 to 0x013dc3 (0x120fc bytes) 000866 1789403544.720262 read 0x012ab8 thru 0x013dc3 (0x130c bytes) 000868 1789403544.731599 write 0x003751 thru 0x00ad4c (0x75fc bytes) 000869 1789403544.733319 read 0x00f5a7 thru 0x013dc3 (0x481d bytes) 000871 1789403544.734911 mapwrite 0x035fb6 thru 0x03f9af (0x99fa bytes) 000872 1789403544.796991 mapread 0x0337d9 thru 0x03c9b0 (0x91d8 bytes) 000874 1789403544.805295 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f9b0): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9b0): Mirror 10: 0x4fb81a2d (stale) CRC-32 checksum value for chunk [0, 0x3f9b0): Mirror 1: 0xdcde6e85 Mirror 11: 0xdcde6e85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9b0: skipped 000876 1789403545.965799 trunc from 0x03f9b0 to 0x0096d1 (0x362de bytes) 000877 1789403546.054692 mapwrite 0x03bda3 thru 0x03d32a (0x1588 bytes) 000878 1789403546.154957 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d32b): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d32b): Mirror 10: 0xfc3e31af (stale) Mirror 11: 0x47bda300 (stale) CRC-32 checksum value for chunk [0, 0x3d32b): Mirror 1: 0x71286189 Mirror 12: 0x71286189 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d32b: skipped 000880 1789403547.034296 MIRROR_RESYNC 4 mirrors 000881 1789403547.982418 trunc from 0x0090e3 to 0x00e471 (0x538f bytes) 000882 1789403548.060939 trunc from 0x03d32b to 0x00cdfe (0x3052c bytes) 000883 1789403548.100305 mapwrite 0x035dea thru 0x03ffff (0xa216 bytes) 000884 1789403548.204161 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa67f514 (stale) Mirror 11: 0xa67f514 (stale) Mirror 12: 0xa67f514 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdbaabf0f Mirror 13: 0xdbaabf0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000885 1789403549.286050 fallocate 0x000ee4 thru 0x00b46f (0xa58c bytes) 000886 1789403549.348219 read 0x00dd73 thru 0x01ca3b (0xecc9 bytes) 000887 1789403549.359441 mapread 0x01580c thru 0x01c20b (0x6a00 bytes) 000888 1789403549.360913 write 0x0027c6 thru 0x01172c (0xef67 bytes) 000889 1789403549.376834 trunc from 0x040000 to 0x01e80f (0x217f0 bytes) 000890 1789403549.441605 mapwrite 0x01674e thru 0x02655e (0xfe11 bytes) 000891 1789403549.576573 punch from 0x002615 to 0x0113c3 (0xedaf bytes) 000892 1789403549.588589 fallocate 0x0350a0 thru 0x03ffff (0xaf60 bytes) 000893 1789403549.603494 read 0x007ad5 thru 0x00b376 (0x38a2 bytes) 000894 1789403549.617145 punch from 0x020d3c to 0x02655e (0x5823 bytes) 000895 1789403549.629955 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2655f): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2655f): Mirror 10: 0xb5b69d5f (stale) Mirror 11: 0xb5b69d5f (stale) Mirror 12: 0xb5b69d5f (stale) Mirror 13: 0x8fc7cc4e (stale) CRC-32 checksum value for chunk [0, 0x2655f): Mirror 1: 0xfb425979 Mirror 14: 0xfb425979 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2655f: skipped 000896 1789403550.937194 read 0x012e0b thru 0x014abb (0x1cb1 bytes) 000897 1789403550.947449 trunc from 0x02655f to 0x030094 (0x9b36 bytes) 000898 1789403551.024870 punch from 0x02daaa to 0x030094 (0x25eb bytes) 000899 1789403551.034156 trunc from 0x030095 to 0x03072a (0x696 bytes) 000900 1789403551.068319 trunc from 0x004b78 to 0x00eb83 (0xa00c bytes) 000901 1789403551.077199 punch from 0x004254 to 0x00eaca (0xa877 bytes) 000902 1789403551.090163 write 0x01cabc thru 0x01e84f (0x1d94 bytes) 000903 1789403551.133121 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3072b): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3072b): Mirror 10: 0x5bd9ac72 (stale) Mirror 11: 0x5bd9ac72 (stale) Mirror 12: 0x5bd9ac72 (stale) Mirror 13: 0xc6774193 (stale) Mirror 14: 0xa597cbae (stale) CRC-32 checksum value for chunk [0, 0x3072b): Mirror 1: 0x428c694a Mirror 15: 0x428c694a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3072b: skipped 000906 1789403552.682405 mapwrite 0x02ee8d thru 0x03e4c7 (0xf63b bytes) 000907 1789403552.799708 trunc from 0x03e4c8 to 0x01c02f (0x22498 bytes) 000908 1789403552.824964 trunc from 0x00656c to 0x0072de (0xd73 bytes) 000909 1789403552.831055 fallocate 0x03dc92 thru 0x03ffff (0x236e bytes) 000910 1789403552.837141 trunc from 0x020ba7 to 0x03007b (0xf4d5 bytes) 000911 1789403552.842573 trunc from 0x004f37 to 0x009c14 (0x4cde bytes) 000912 1789403552.848533 fallocate 0x006564 thru 0x00a7fa (0x4297 bytes) 000914 1789403552.854603 trunc from 0x01367c to 0x01c31d (0x8ca2 bytes) 000915 1789403552.860413 trunc from 0x00f32e to 0x016566 (0x7239 bytes) 000916 1789403552.865821 fallocate 0x008825 thru 0x017205 (0xe9e1 bytes) 000917 1789403552.871127 trunc from 0x040000 to 0x01328f (0x2cd70 bytes) 000918 1789403552.892230 write 0x0251f6 thru 0x02da14 (0x881f bytes) 000919 1789403552.895927 write 0x034792 thru 0x03ffff (0xb86e bytes) 000920 1789403552.900240 fallocate 0x02306c thru 0x0281ed (0x5182 bytes) 000921 1789403552.909937 trunc from 0x00dd71 to 0x015415 (0x76a5 bytes) 000922 1789403552.916226 mapwrite 0x00fb5d thru 0x01767c (0x7b20 bytes) 000924 1789403552.955932 mapread 0x020813 thru 0x029978 (0x9166 bytes) 000925 1789403552.970754 punch from 0x02a859 to 0x031a6d (0x7215 bytes) 000926 1789403552.976452 punch from 0x02b4a5 to 0x032593 (0x70ef bytes) 000927 1789403552.981882 mapwrite 0x031c43 thru 0x0365bc (0x497a bytes) 000928 1789403553.023362 read 0x003a94 thru 0x006e4f (0x33bc bytes) 000931 1789403553.051508 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa67f514 (stale) Mirror 11: 0xa67f514 (stale) Mirror 13: 0xdbaabf0f (stale) Mirror 14: 0x52cf3ffc (stale) Mirror 15: 0x6814fe0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa67f514 (stale) Mirror 11: 0xa67f514 (stale) Mirror 13: 0xdbaabf0f (stale) Mirror 14: 0x52cf3ffc (stale) Mirror 15: 0x6814fe0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000932 1789403553.509846 trunc from 0x040000 to 0x00ecad (0x31352 bytes) 000933 1789403553.550344 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xecae): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xecae): Mirror 10: 0xa4c50f07 (stale) Mirror 11: 0xa4c50f07 (stale) Mirror 13: 0x29077341 (stale) Mirror 14: 0x6405417b (stale) Mirror 15: 0x6405417b (stale) CRC-32 checksum value for chunk [0, 0xecae): Mirror 1: 0x6405417b Mirror 16: 0x6405417b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xecae: skipped 000934 1789403554.946887 fallocate 0x03fccc thru 0x03ffff (0x334 bytes) 000935 1789403555.001851 mapread 0x008fe2 thru 0x00fb45 (0x6b64 bytes) 000936 1789403555.025891 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa67f514 (stale) Mirror 11: 0xa67f514 (stale) Mirror 13: 0xdbaabf0f (stale) Mirror 14: 0x52cf3ffc (stale) Mirror 15: 0x6814fe0f (stale) Mirror 16: 0x36e1252a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x36e1252a Mirror 17: 0x36e1252a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000937 1789403556.138512 mapread 0x03176e thru 0x03ffff (0xe892 bytes) 000938 1789403556.197439 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xa67f514 (stale) Mirror 11: 0xa67f514 (stale) Mirror 13: 0xdbaabf0f (stale) Mirror 14: 0x52cf3ffc (stale) Mirror 15: 0x6814fe0f (stale) Mirror 16: 0x36e1252a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x36e1252a Mirror 17: 0x36e1252a Mirror 18: 0x36e1252a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000939 1789403557.144500 punch from 0x01d855 to 0x0263e5 (0x8b91 bytes) 000940 1789403557.183406 write 0x011f98 thru 0x017ae5 (0x5b4e bytes) 000941 1789403557.197977 read 0x01174b thru 0x01cc77 (0xb52d bytes) 000942 1789403557.205958 read 0x00aaa3 thru 0x0141c0 (0x971e bytes) 000944 1789403557.217743 punch from 0x02db45 to 0x033dcf (0x628b bytes) 000945 1789403557.223418 mapwrite 0x0153dc thru 0x0157d2 (0x3f7 bytes) 000946 1789403557.245483 read 0x009921 thru 0x012608 (0x8ce8 bytes) 000947 1789403557.246594 trunc from 0x040000 to 0x013b96 (0x2c469 bytes) 000949 1789403557.280518 read 0x00a77d thru 0x013b96 (0x941a bytes) 000950 1789403557.282007 mapread 0x00c46c thru 0x00c7a0 (0x335 bytes) 000951 1789403557.283018 trunc from 0x013b97 to 0x01a870 (0x6cda bytes) 000952 1789403557.300983 trunc from 0x01a871 to 0x009890 (0x10fe0 bytes) 000954 1789403557.325215 write 0x038a51 thru 0x039fc4 (0x1574 bytes) 000955 1789403557.327356 punch from 0x021a11 to 0x02d061 (0xb651 bytes) 000956 1789403557.333165 mapwrite 0x01d702 thru 0x029b81 (0xc480 bytes) 000957 1789403557.363191 mapwrite 0x03519f thru 0x03d3ee (0x8250 bytes) 000958 1789403557.447395 read 0x01ca37 thru 0x02249f (0x5a69 bytes) 000959 1789403557.452467 MIRROR_RESYNC 9 mirrors 000960 1789403560.091224 mapread 0x020e0c thru 0x02795e (0x6b53 bytes) 000961 1789403560.126860 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) 000962 1789403560.879202 mapread 0x01da05 thru 0x02983b (0xbe37 bytes) 000963 1789403560.917179 MIRROR_EXTEND to 9 mirrors 000965 1789403562.006521 trunc from 0x03d3ef to 0x01926a (0x24184 bytes) 000966 1789403562.092155 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1926b): Mirror 13: 0xc7de2f79 (stale) Mirror 14: 0xc7de2f79 (stale) Mirror 15: 0xc7de2f79 (stale) Mirror 16: 0xc7de2f79 (stale) Mirror 17: 0xc7de2f79 (stale) Mirror 18: 0xc7de2f79 (stale) Mirror 19: 0xc7de2f79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1926b): Mirror 13: 0xc7de2f79 (stale) Mirror 14: 0xc7de2f79 (stale) Mirror 15: 0xc7de2f79 (stale) Mirror 16: 0xc7de2f79 (stale) Mirror 17: 0xc7de2f79 (stale) Mirror 18: 0xc7de2f79 (stale) Mirror 19: 0xc7de2f79 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1926b: skipped 000967 1789403562.709318 fallocate 0x0134b6 thru 0x01d749 (0xa294 bytes) 000968 1789403562.731974 mapread 0x016af1 thru 0x01c07e (0x558e bytes) 000969 1789403562.770430 fallocate 0x02936a thru 0x02cf00 (0x3b97 bytes) 000970 1789403562.776011 write 0x02d5cb thru 0x03cc94 (0xf6ca bytes) 000971 1789403562.786819 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cc95): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc95): Mirror 13: 0x619f007b (stale) Mirror 14: 0x619f007b (stale) Mirror 15: 0x619f007b (stale) Mirror 16: 0x619f007b (stale) Mirror 17: 0x619f007b (stale) Mirror 18: 0x619f007b (stale) Mirror 19: 0x619f007b (stale) CRC-32 checksum value for chunk [0, 0x3cc95): Mirror 1: 0xed74264e Mirror 20: 0xed74264e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc95: skipped 000973 1789403564.132309 fallocate 0x03cf27 thru 0x03ffff (0x30d9 bytes) 000974 1789403564.189808 trunc from 0x03cc95 to 0x035f00 (0x6d94 bytes) 000975 1789403564.222771 read 0x02cd50 thru 0x035ba3 (0x8e54 bytes) 000976 1789403564.238780 trunc from 0x035f01 to 0x02cb5a (0x93a6 bytes) 000977 1789403564.262414 write 0x003b13 thru 0x00d6a5 (0x9b93 bytes) 000978 1789403564.275925 write 0x029e3a thru 0x036f36 (0xd0fd bytes) 000979 1789403564.291587 punch from 0x007e83 to 0x00e7f8 (0x6976 bytes) 000980 1789403564.332027 punch from 0x02d0a4 to 0x036096 (0x8ff3 bytes) 000981 1789403564.340200 mapwrite 0x017473 thru 0x01dbaa (0x6738 bytes) 000982 1789403564.377875 read 0x009774 thru 0x00eb68 (0x53f5 bytes) 000983 1789403564.388026 punch from 0x0032c4 to 0x007d07 (0x4a44 bytes) 000984 1789403564.393799 read 0x020fb4 thru 0x02c1c5 (0xb212 bytes) 000985 1789403564.394794 trunc from 0x036f37 to 0x03b92c (0x49f6 bytes) 000986 1789403564.416347 mapwrite 0x03c715 thru 0x03ffff (0x38eb bytes) 000987 1789403564.479062 read 0x017429 thru 0x023636 (0xc20e bytes) 000988 1789403564.479930 fallocate 0x0148c3 thru 0x015b2b (0x1269 bytes) 000989 1789403564.486079 mapwrite 0x004c78 thru 0x00899c (0x3d25 bytes) 000990 1789403564.507257 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x1ca517af (stale) Mirror 14: 0x1ca517af (stale) Mirror 15: 0x1ca517af (stale) Mirror 16: 0x1ca517af (stale) Mirror 17: 0x1ca517af (stale) Mirror 18: 0x1ca517af (stale) Mirror 19: 0x1ca517af (stale) Mirror 20: 0xa189170d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd42ff270 Mirror 21: 0xd42ff270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000991 1789403565.732175 mapwrite 0x014881 thru 0x0235ee (0xed6e bytes) 000992 1789403565.867334 read 0x039193 thru 0x03c30e (0x317c bytes) 000993 1789403565.874399 trunc from 0x040000 to 0x00f232 (0x30dcd bytes) 000994 1789403565.940144 trunc from 0x00f233 to 0x01d026 (0xddf4 bytes) 000995 1789403565.969119 write 0x0324c0 thru 0x03ebba (0xc6fb bytes) 000997 1789403565.976143 trunc from 0x006c9d to 0x009668 (0x29cc bytes) 000998 1789403566.013627 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ebbb): Mirror 13: 0xed34605e (stale) Mirror 14: 0xed34605e (stale) Mirror 15: 0xed34605e (stale) Mirror 16: 0xed34605e (stale) Mirror 18: 0xed34605e (stale) Mirror 19: 0xed34605e (stale) Mirror 20: 0x3b33a952 (stale) Mirror 21: 0x30a19499 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ebbb): Mirror 13: 0xed34605e (stale) Mirror 14: 0xed34605e (stale) Mirror 15: 0xed34605e (stale) Mirror 16: 0xed34605e (stale) Mirror 18: 0xed34605e (stale) Mirror 19: 0xed34605e (stale) Mirror 20: 0x3b33a952 (stale) Mirror 21: 0x30a19499 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ebbb: skipped 000999 1789403566.670899 punch from 0x038166 to 0x03e07b (0x5f16 bytes) 001000 1789403566.679340 punch from 0x018f25 to 0x01ff24 (0x7000 bytes) All operations completed A-OK! PASS 70a (98s) == sanity-flr test 71: check mirror extend parent fid ==== 12:32:52 (1789403572) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1ca:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 12:33:03 (1789403583) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195761 s, 10.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799145 s, 13.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 12:33:11 (1789403591) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2926432 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 762176 .. failed Extending file size to 7141472 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5798208 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8787392 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8993824 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9118048 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9167936 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61322 61323 61326 61327 61328 done 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) 192.168.203.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:34:31 (1789403671) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5506144 .. Extending file size to 6639232 .. done Extending file size to 6785152 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7656768 .. Extending file size to 6972096 .. Extending file size to 7767712 .. failed Extending file size to 8798240 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9173536 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8834880 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62934 62935 62936 62938 62939 done 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) 192.168.203.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:35:50 (1789403750) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2396 1268908 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1269228 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17832 3589188 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25472 3581548 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43304 7170736 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2396 1268908 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2076 1269228 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17832 3589188 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25472 3581548 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43304 7170736 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107056 s, 19.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247627 s, 8.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xeb140b0c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:36:12 (1789403772) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173932 s, 12.1 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 12:36:19 (1789403779) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x75:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x144:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x145:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:36:28 (1789403788) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17211 s, 12.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x228:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x229:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:36:37 (1789403797) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131548 s, 15.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x229:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x229:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22c:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:36:45 (1789403805) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113882 s, 18.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x146:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x146:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:37:17 (1789403837) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792481 s, 13.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22c:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:38:03 (1789403883) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0673555 s, 31.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (43s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:38:46 (1789403926) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg306-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg306-server: Pool lustre.test_204f created Adding targets to pool oleg306-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg306-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131876 s, 15.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x147:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg306-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg306-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg306-server: Pool lustre.test_204f destroyed PASS 204f (67s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:39:54 (1789403994) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:40:00 (1789404000) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:40:09 (1789404009) oleg306-server: Pool lustre.test_206 created Adding targets to pool oleg306-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg306-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg306-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg306-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg306-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:40:37 (1789404037) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396871 s, 26.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687234 s, 15.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3548e622 Mirror 3: 0x3548e622 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:40:49 (1789404049) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:40:54 (1789404054) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:40:59 (1789404059) Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.107818 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2640000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x890000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x890000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x890000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x990000 Read this (4 bytes): BBBB 192.168.203.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg306-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:41:43 (1789404103) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0475335 s, 172 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:41:50 (1789404110) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722663 s, 14.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x92 lmm_fid: [0x200000405:0x92:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x92 lmm_fid: [0x200000405:0x92:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611282 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 12:41:52.000000000 -0400 Modify: 2026-09-14 12:41:52.000000000 -0400 Change: 2026-09-14 12:41:53.000000000 -0400 Birth: 2026-09-14 12:41:53.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:41:59 (1789404119) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477386 s, 22.0 MB/s oleg306-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg306-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg306-server: O/280000401/d18/152: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:42:21 (1789404141) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.211226 s, 194 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0368481 s, 111 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 14 12:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x97:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 14 12:42 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:42:30 (1789404150) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:42:37 (1789404157) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1863 sec =========== 12:42:45 (1789404165) === sanity-flr: start cleanup 12:42:47 (1789404167) === === sanity-flr: finish cleanup 12:42:56 (1789404176) ===