-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 01:14:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg349-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:14:24 (1787548464) === oleg349-client.virtnet: executing check_config_client /mnt/lustre oleg349-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg349-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff963689db4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff963689db4800.idle_timeout=debug disable quota as required oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:14:58 (1787548498) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7634] == sanity-flr test 0a: lfs mirror create with -N option == 01:15:00 (1787548500) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:15:20 (1787548520) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:15:23 (1787548523) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:15:26 (1787548526) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:15:53 (1787548553) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:15:57 (1787548557) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 01:16:00 (1787548560) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff963689db4800.stats=clear osc.lustre-OST0001-osc-ffff963689db4800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:16:08 (1787548568) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:16:20 (1787548580) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:16:29 (1787548589) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:16:32 (1787548592) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:16:41 (1787548601) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 01:16:50 (1787548610) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:16:57 (1787548617) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:17:00 (1787548620) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:17:08 (1787548628) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:17:19 (1787548639) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:17:26 (1787548646) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 1.03221 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199706 s, 5.3 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:17:36 (1787548656) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.470613 s, 37.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:17:45 (1787548665) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:17:55 (1787548675) Stopping /mnt/lustre-ost1 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff963689db4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff963689db4800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg349-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg349-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff963689db4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff963689db4800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg349-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (136s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:20:11 (1787548811) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.623678 s, 15.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff963689db4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff963689db4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg349-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:20:58 (1787548858) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff963689db4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff963689db4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg349-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff963689db4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff963689db4800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg349-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (109s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:22:47 (1787548967) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.66205 s, 11.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 01:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.058s user 0m0.000s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff963689db4800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff963689db4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.180s user 0m0.000s sys 0m0.217s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg349-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg349-server oleg349-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff963689db4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff963689db4800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.320s user 0m0.000s sys 0m0.184s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg349-server: oleg349-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg349-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff963689db4800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff963689db4800.ost_server_uuid in FULL state after 0 sec PASS 33b (109s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:24:37 (1787549077) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:24:41 (1787549081) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:24:44 (1787549084) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 01:24:48 (1787549088) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 01:24:56 (1787549096) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.572202 s, 22.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:25:08 (1787549108) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413754 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227314 s, 46.1 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:25:14 (1787549114) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:25:22 (1787549122) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00188902 s, 542 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00202583 s, 505 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104567 s, 979 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00130459 s, 785 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 01:25:31 (1787549131) create test files with size 15729348 .. 988d1c14b06608c2b7d36c1f300fc8a2 - eb41fbaeb99f11c3cddd3385ac42ad95 - 71a6c28108efe2171fd27a4aaf401c69 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg349-client.virtnet /mnt/lustre (opts:) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 01:25:54 (1787549154) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:26:11 (1787549171) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:26:17 (1787549177) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124062 s, 8.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382201 s, 27.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 01:26:26 (1787549186) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0899194 s, 23.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.360972 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.419336 s, 5.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 01:26:44 (1787549204) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 01:26:48 (1787549208) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:26:52 (1787549212) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220214 s, 47.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 01:26:59 (1787549219) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:27:02 (1787549222) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:27:10 (1787549230) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.2231 s, 47.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg349-client.virtnet /mnt/lustre (opts:) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:27:22 (1787549242) llite.lustre-ffff963689db4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.304381 s, 34.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg349-client.virtnet /mnt/lustre (opts:) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff963682c9c000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:27:35 (1787549255) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:28:15 (1787549295) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:28:54 (1787549334) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 01:28:56 (1787549336) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.262355 s, 12.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.106785 s, 29.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd10570bf (nosync,stale) Mirror 2: 0xd10570bf (nosync,stale) ** verify mirror 2 doesn't change original checksum: e384f3bbaee24922aa80fb841617a55e - mirror 1 checksum: 7b723d4443533d61e6ec71f90f332985 - mirror 2 checksum: e384f3bbaee24922aa80fb841617a55e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T01:28:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T01:28:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e384f3bbaee24922aa80fb841617a55e - mirror 1 checksum: 7b723d4443533d61e6ec71f90f332985 - mirror 2 checksum: 7b723d4443533d61e6ec71f90f332985 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T01:28:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T01:28:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:29:07 (1787549347) Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 01:29:12 (1787549352) shut down facet: mds1 facet_host: oleg349-server facet_failover_host: oleg349-server Failover mds1 to oleg349-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 01:29:48 (1787549388) targets are mounted 01:29:48 (1787549388) facet_failover done oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453412 s, 23.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:30:06 (1787549406) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:30:10 (1787549410) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:30:13 (1787549413) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0245757 s, 667 kB/s PASS 50c (13s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:30:26 (1787549426) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:30:31 (1787549431) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00410276 s, 998 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.170442 s, 24.0 kB/s PASS 60a (12s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:30:43 (1787549443) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:31:07 (1787549467) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:31:39 (1787549499) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:32:03 (1787549523) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:32:28 (1787549548) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 01:32:36 (1787549556) mirror create pid 54347 mirror split pid 54349 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54347 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54349 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 01:33:43 (1787549623) Seed set to 9625 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x8849 000001 1787549626.185972 trunc from 00000000 to 0x008848 (0x8849 bytes) truncating to largest ever: 0x31c8b 000002 1787549626.208556 trunc from 0x008849 to 0x031c8a (0x29442 bytes) 000003 1787549626.225298 trunc from 0x031c8b to 0x000c7b (0x3100f bytes) 000004 1787549626.253437 read 0x0004f8 thru 0x000c7b (0x784 bytes) 000005 1787549626.257386 trunc from 0x000c7c to 0x00d4c9 (0xc84e bytes) 000009 1787549626.276719 read 0x005138 thru 0x00d4c9 (0x8392 bytes) 000010 1787549626.290011 mapread 0x004e52 thru 0x00d4c9 (0x8678 bytes) 000012 1787549626.304172 MIRROR_EXTEND to 2 mirrors 000013 1787549627.291763 mapwrite 0x02e805 thru 0x032133 (0x392f bytes) 000016 1787549627.566359 mapwrite 0x03075f thru 0x0396dd (0x8f7f bytes) 000017 1787549627.655640 write 0x00c54b thru 0x016e6a (0xa920 bytes) 000018 1787549627.668372 trunc from 0x0396de to 0x014a06 (0x24cd7 bytes) 000019 1787549627.693436 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000020 1787549627.936774 mapwrite 0x00bb46 thru 0x00e7b6 (0x2c71 bytes) 000021 1787549628.085870 trunc from 0x014a07 to 0x00f2af (0x5757 bytes) 000024 1787549628.139167 read 0x00295f thru 0x00f2af (0xc951 bytes) truncating to largest ever: 0x3ed04 000029 1787549628.154298 trunc from 0x00f2b0 to 0x03ed03 (0x2fa54 bytes) 000031 1787549628.192069 read 0x028a23 thru 0x03808e (0xf66c bytes) 000032 1787549628.207485 write 0x0270c8 thru 0x02c68b (0x55c4 bytes) 000033 1787549628.221091 MIRROR_EXTEND to 2 mirrors 000034 1787549629.862899 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000040 1787549630.142164 mapread 0x0309eb thru 0x03e590 (0xdba6 bytes) 000042 1787549630.165245 mapwrite 0x00fad3 thru 0x01ab72 (0xb0a0 bytes) 000043 1787549630.253840 MIRROR_EXTEND to 2 mirrors 000044 1787549631.280956 mapread 0x03110e thru 0x033b78 (0x2a6b bytes) 000045 1787549631.304727 write 0x01ba15 thru 0x0211a3 (0x578f bytes) 000048 1787549631.380366 mapread 0x009775 thru 0x017aab (0xe337 bytes) 000049 1787549631.409231 trunc from 0x03ed04 to 0x02fbaa (0xf159 bytes) 000051 1787549631.466459 read 0x028cc8 thru 0x02fbaa (0x6ee3 bytes) 000052 1787549631.467148 write 0x01cfc7 thru 0x0294c7 (0xc501 bytes) 000054 1787549631.474774 read 0x01649d thru 0x01e31f (0x7e83 bytes) 000055 1787549631.477065 mapread 0x00955e thru 0x01545d (0xbf00 bytes) 000056 1787549631.479169 mapread 0x01a958 thru 0x01f9f2 (0x509b bytes) 000057 1787549631.484195 mapread 0x00b4cf thru 0x012650 (0x7182 bytes) 000058 1787549631.486771 read 0x020956 thru 0x02fbaa (0xf255 bytes) 000061 1787549631.496641 read 0x026439 thru 0x02fbaa (0x9772 bytes) 000062 1787549631.498718 write 0x032fc2 thru 0x03f743 (0xc782 bytes) 000063 1787549631.507214 read 0x0229db thru 0x025a35 (0x305b bytes) 000068 1787549631.512359 mapread 0x025b02 thru 0x03474e (0xec4d bytes) 000070 1787549631.547754 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000071 1787549631.727764 mapwrite 0x007749 thru 0x0101d1 (0x8a89 bytes) 000072 1787549631.832753 mapwrite 0x00c578 thru 0x012401 (0x5e8a bytes) 000073 1787549631.914355 read 0x00acf0 thru 0x0127c2 (0x7ad3 bytes) 000075 1787549631.916609 read 0x027c74 thru 0x02d457 (0x57e4 bytes) 000077 1787549631.932622 read 0x001c67 thru 0x008f13 (0x72ad bytes) 000081 1787549631.935700 mapwrite 0x02d2bf thru 0x032d84 (0x5ac6 bytes) 000082 1787549631.998784 read 0x00df23 thru 0x013305 (0x53e3 bytes) 000084 1787549632.000442 trunc from 0x03f744 to 0x000c82 (0x3eac1 bytes) 000085 1787549632.035773 trunc from 0x000c83 to 0x0328ba (0x31c38 bytes) 000087 1787549632.062185 read 0x031080 thru 0x0328ba (0x183b bytes) 000089 1787549632.079008 mapread 0x031552 thru 0x0328ba (0x1369 bytes) 000091 1787549632.082674 trunc from 0x0328bb to 0x021c6f (0x10c4b bytes) 000092 1787549632.108821 write 0x017729 thru 0x018599 (0xe71 bytes) 000098 1787549632.116400 mapwrite 0x0099b3 thru 0x00fa37 (0x6085 bytes) 000099 1787549632.268307 mapread 0x00a599 thru 0x00dbb1 (0x3619 bytes) 000101 1787549632.271406 write 0x017633 thru 0x01ea2e (0x73fc bytes) 000103 1787549632.282101 read 0x014e24 thru 0x01c3d7 (0x75b4 bytes) 000104 1787549632.284361 read 0x004a73 thru 0x00dab2 (0x9040 bytes) 000106 1787549632.294800 write 0x02d731 thru 0x03916a (0xba3a bytes) 000107 1787549632.296602 trunc from 0x03916b to 0x022703 (0x16a67 bytes) 000108 1787549632.345839 read 0x01cc24 thru 0x022703 (0x5ae0 bytes) 000109 1787549632.362389 trunc from 0x022704 to 0x03ddbc (0x1b6b9 bytes) 000110 1787549632.382764 mapread 0x022a03 thru 0x030377 (0xd975 bytes) 000111 1787549632.409413 mapwrite 0x015c3c thru 0x01797d (0x1d42 bytes) 000112 1787549632.512241 trunc from 0x03ddbd to 0x03b8e9 (0x24d3 bytes) 000113 1787549632.563165 trunc from 0x03b8ea to 0x02482c (0x170bd bytes) 000116 1787549632.610278 read 0x003361 thru 0x00b56b (0x820b bytes) 000117 1787549632.611424 trunc from 0x02482d to 0x01ba2f (0x8dfd bytes) 000118 1787549632.636064 mapwrite 0x00e2ee thru 0x00e343 (0x56 bytes) 000120 1787549632.695147 read 0x018879 thru 0x01ba2f (0x31b7 bytes) 000124 1787549632.699830 read 0x00d0c5 thru 0x01ba2f (0xe96b bytes) 000125 1787549632.700472 trunc from 0x01ba30 to 0x02ed23 (0x132f4 bytes) 000129 1787549632.729196 write 0x004dfc thru 0x006c1c (0x1e21 bytes) 000131 1787549632.733460 read 0x02b0b9 thru 0x02ed23 (0x3c6b bytes) 000132 1787549632.755361 read 0x01e67c thru 0x029518 (0xae9d bytes) 000134 1787549632.758291 write 0x017a2f thru 0x01d7ed (0x5dbf bytes) 000135 1787549632.759338 write 0x018be2 thru 0x01d15f (0x457e bytes) 000136 1787549632.760823 mapread 0x00aad4 thru 0x00d624 (0x2b51 bytes) 000137 1787549632.763391 trunc from 0x02ed24 to 0x00890c (0x26417 bytes) 000141 1787549632.824248 mapwrite 0x00a6f5 thru 0x018e40 (0xe74c bytes) 000142 1787549632.915311 trunc from 0x018e41 to 0x00ef3c (0x9f04 bytes) 000143 1787549632.930207 read 0x0024c1 thru 0x007f23 (0x5a63 bytes) 000146 1787549632.936473 MIRROR_EXTEND to 2 mirrors 000147 1787549633.536701 read 0x00b77d thru 0x00ef0b (0x378f bytes) 000149 1787549633.551039 write 0x01ad5f thru 0x01c63d (0x18df bytes) 000150 1787549633.641614 read 0x002af0 thru 0x003823 (0xd34 bytes) 000151 1787549633.668835 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c63e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c63e): Mirror 2: 0x3a5d691d (stale) CRC-32 checksum value for chunk [0, 0x1c63e): Mirror 1: 0x616e57e6 Mirror 3: 0x616e57e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c63e: skipped 000152 1787549635.556478 trunc from 0x01c63e to 0x0368bd (0x1a280 bytes) 000156 1787549635.718610 trunc from 0x0368be to 0x003ca0 (0x32c1d bytes) 000159 1787549635.753396 mapread 0x0026a7 thru 0x003ca0 (0x15fa bytes) 000161 1787549635.778067 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca1): Mirror 3: 0xb195af73 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca1): Mirror 3: 0xb195af73 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca1: skipped 000165 1787549636.171713 read 0x002119 thru 0x003ca0 (0x1b88 bytes) 000166 1787549636.178986 trunc from 0x003ca1 to 0x0388b7 (0x34c17 bytes) 000167 1787549636.197864 trunc from 0x0388b8 to 0x0277df (0x110d8 bytes) 000169 1787549636.226788 mapwrite 0x004f19 thru 0x010866 (0xb94e bytes) 000172 1787549636.296333 mapwrite 0x00047c thru 0x00e117 (0xdc9c bytes) 000175 1787549636.382322 mapread 0x0034ae thru 0x00f91f (0xc472 bytes) 000179 1787549636.386810 trunc from 0x0277e0 to 0x02c6c2 (0x4ee3 bytes) 000180 1787549636.421617 write 0x01154f thru 0x01c152 (0xac04 bytes) 000181 1787549636.434669 MIRROR_RESYNC 2 mirrors 000183 1787549637.449812 read 0x011a94 thru 0x01b241 (0x97ae bytes) 000191 1787549637.462299 write 0x0108bd thru 0x01a160 (0x98a4 bytes) 000194 1787549637.557933 mapwrite 0x03dd5d thru 0x03ffff (0x22a3 bytes) 000195 1787549637.657019 mapwrite 0x02982d thru 0x02e35a (0x4b2e bytes) 000196 1787549637.782547 MIRROR_RESYNC 2 mirrors 000198 1787549638.730483 read 0x02c676 thru 0x035714 (0x909f bytes) 000199 1787549638.741411 mapread 0x02953b thru 0x03175a (0x8220 bytes) 000200 1787549638.762076 write 0x03907b thru 0x03f4c4 (0x644a bytes) 000201 1787549638.851634 write 0x0068bc thru 0x009a2d (0x3172 bytes) 000203 1787549638.883818 trunc from 0x040000 to 0x02b34e (0x14cb1 bytes) 000204 1787549638.924939 mapread 0x027b7f thru 0x02b34e (0x37d0 bytes) 000205 1787549638.941324 write 0x039275 thru 0x03c25a (0x2fe6 bytes) 000206 1787549638.942933 write 0x021b9f thru 0x024117 (0x2579 bytes) 000207 1787549638.944339 mapwrite 0x03dbdc thru 0x03ffff (0x2424 bytes) 000208 1787549639.054202 write 0x0058fd thru 0x0123a8 (0xcaac bytes) 000212 1787549639.068301 mapread 0x02a3e3 thru 0x035f25 (0xbb43 bytes) 000214 1787549639.098562 read 0x023fdf thru 0x033d31 (0xfd53 bytes) 000219 1787549639.101691 read 0x035f30 thru 0x03ffff (0xa0d0 bytes) 000220 1787549639.102316 read 0x02683a thru 0x0349b2 (0xe179 bytes) 000222 1787549639.103595 read 0x03de92 thru 0x03ffff (0x216e bytes) 000223 1787549639.104158 write 0x01bde4 thru 0x0241c6 (0x83e3 bytes) 000224 1787549639.123896 trunc from 0x040000 to 0x03678f (0x9870 bytes) 000225 1787549639.144687 mapread 0x031a30 thru 0x03678f (0x4d60 bytes) 000226 1787549639.145772 trunc from 0x036790 to 0x031f69 (0x4826 bytes) 000227 1787549639.178905 trunc from 0x031f6a to 0x022fb0 (0xefb9 bytes) 000228 1787549639.203142 trunc from 0x022fb1 to 0x004828 (0x1e788 bytes) 000231 1787549639.230919 read 0x002498 thru 0x0043ec (0x1f55 bytes) 000235 1787549639.232876 write 0x007f8e thru 0x00a49e (0x2511 bytes) 000236 1787549639.234709 trunc from 0x00a49f to 0x005ea3 (0x45fb bytes) 000237 1787549639.259296 mapwrite 0x01e7cb thru 0x02668b (0x7ec1 bytes) 000239 1787549639.375870 MIRROR_RESYNC 2 mirrors 000240 1787549640.239874 MIRROR_EXTEND to 3 mirrors 000242 1787549641.242322 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000243 1787549641.601256 mapread 0x023670 thru 0x02372c (0xbd bytes) 000244 1787549641.607158 mapread 0x01535d thru 0x01d1ab (0x7e4f bytes) 000245 1787549641.633207 mapwrite 0x020f6d thru 0x0275c7 (0x665b bytes) 000246 1787549641.808880 read 0x01ad92 thru 0x01f675 (0x48e4 bytes) 000247 1787549641.810414 write 0x020199 thru 0x0249b9 (0x4821 bytes) 000249 1787549641.813285 mapread 0x018b9b thru 0x01986d (0xcd3 bytes) 000251 1787549641.828936 mapwrite 0x014f21 thru 0x01bef9 (0x6fd9 bytes) 000252 1787549641.909757 write 0x03ffef thru 0x03ffff (0x11 bytes) 000256 1787549641.917655 read 0x03cec1 thru 0x03ffff (0x313f bytes) 000257 1787549641.939641 write 0x039ecc thru 0x03a0db (0x210 bytes) 000258 1787549641.950648 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000261 1787549642.151376 mapread 0x01a37b thru 0x0211d9 (0x6e5f bytes) 000264 1787549642.172829 trunc from 0x040000 to 0x0278ab (0x18754 bytes) 000265 1787549642.198710 read 0x024b79 thru 0x0278ab (0x2d33 bytes) 000267 1787549642.208947 write 0x020dd0 thru 0x02597f (0x4bb0 bytes) 000268 1787549642.209940 trunc from 0x0278ac to 0x01ba02 (0xbea9 bytes) 000272 1787549642.248377 write 0x02b4c2 thru 0x039ed5 (0xea14 bytes) 000274 1787549642.250602 write 0x015215 thru 0x016f1e (0x1d0a bytes) 000276 1787549642.251937 mapread 0x020154 thru 0x02e3b8 (0xe265 bytes) 000277 1787549642.278420 write 0x008521 thru 0x0148d0 (0xc3b0 bytes) 000279 1787549642.287414 write 0x03a68d thru 0x03ffff (0x5973 bytes) 000280 1787549642.288403 mapread 0x021ce8 thru 0x02da15 (0xbd2e bytes) 000281 1787549642.289299 write 0x00c53e thru 0x015169 (0x8c2c bytes) 000282 1787549642.290218 write 0x01d887 thru 0x01fe58 (0x25d2 bytes) 000285 1787549642.293049 mapread 0x02307c thru 0x025a26 (0x29ab bytes) 000287 1787549642.295305 trunc from 0x040000 to 0x02c142 (0x13ebd bytes) 000288 1787549642.313431 trunc from 0x02c143 to 0x0183dd (0x13d65 bytes) 000289 1787549642.333929 mapread 0x014e7a thru 0x0183dd (0x3564 bytes) 000291 1787549642.336922 write 0x00d1ab thru 0x011807 (0x465d bytes) 000292 1787549642.338933 trunc from 0x0183de to 0x01eb9d (0x67c0 bytes) 000294 1787549642.358857 read 0x015821 thru 0x01eb9d (0x937d bytes) 000296 1787549642.378192 MIRROR_EXTEND to 2 mirrors 000298 1787549643.010257 trunc from 0x01eb9e to 0x035a7b (0x16ede bytes) 000301 1787549643.142260 write 0x01d45b thru 0x0256f9 (0x829f bytes) 000303 1787549643.174016 MIRROR_RESYNC 2 mirrors 000304 1787549644.101668 read 0x000678 thru 0x0094b1 (0x8e3a bytes) 000305 1787549644.126578 MIRROR_EXTEND to 3 mirrors 000309 1787549645.134889 write 0x03acf1 thru 0x03dd63 (0x3073 bytes) 000310 1787549645.264196 trunc from 0x03dd64 to 0x022c67 (0x1b0fc bytes) 000311 1787549645.295457 trunc from 0x022c68 to 0x00c7a9 (0x164be bytes) 000312 1787549645.319177 write 0x011731 thru 0x01c191 (0xaa61 bytes) 000313 1787549645.320642 trunc from 0x01c192 to 0x0022a2 (0x19eef bytes) 000314 1787549645.337929 read 0x000c80 thru 0x0022a2 (0x1623 bytes) 000323 1787549645.353017 mapread 0x000c68 thru 0x0022a2 (0x163b bytes) 000324 1787549645.357696 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22a3): Mirror 2: 0xa0d837f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22a3): Mirror 2: 0xa0d837f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22a3: skipped 000325 1787549645.658462 read 0x001954 thru 0x0022a2 (0x94f bytes) 000327 1787549645.669406 mapwrite 0x02a139 thru 0x031172 (0x703a bytes) 000328 1787549645.773928 mapwrite 0x020e98 thru 0x027033 (0x619c bytes) 000330 1787549645.847063 MIRROR_RESYNC 2 mirrors 000332 1787549646.685096 mapwrite 0x01df7d thru 0x02c426 (0xe4aa bytes) 000334 1787549646.993120 mapread 0x01453b thru 0x014f80 (0xa46 bytes) 000336 1787549647.007252 read 0x0069d9 thru 0x00d941 (0x6f69 bytes) 000337 1787549647.018282 mapread 0x01a8fe thru 0x02601a (0xb71d bytes) 000338 1787549647.019228 mapwrite 0x014a84 thru 0x01f7df (0xad5c bytes) 000341 1787549647.076323 read 0x015914 thru 0x019246 (0x3933 bytes) 000343 1787549647.078367 mapwrite 0x0300d0 thru 0x03df66 (0xde97 bytes) 000345 1787549647.177329 write 0x00c977 thru 0x0113d0 (0x4a5a bytes) 000346 1787549647.178704 mapwrite 0x021b1c thru 0x02beb9 (0xa39e bytes) 000347 1787549647.223825 read 0x0196e1 thru 0x01be0e (0x272e bytes) 000350 1787549647.226363 mapwrite 0x007b2b thru 0x009071 (0x1547 bytes) 000352 1787549647.278618 read 0x009d19 thru 0x014e86 (0xb16e bytes) 000356 1787549647.282365 mapwrite 0x01a4ce thru 0x02a179 (0xfcac bytes) 000357 1787549647.340156 mapread 0x00f094 thru 0x015a34 (0x69a1 bytes) 000358 1787549647.341352 trunc from 0x03df67 to 0x00286e (0x3b6f8 bytes) 000360 1787549647.356949 mapwrite 0x00378a thru 0x00e127 (0xa99e bytes) 000361 1787549647.422571 mapwrite 0x0392e6 thru 0x03ffff (0x6d1a bytes) 000362 1787549647.535219 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe84e05bc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x18bfe41a Mirror 3: 0x18bfe41a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000363 1787549648.533290 write 0x025c9e thru 0x02924a (0x35ad bytes) 000364 1787549648.638977 trunc from 0x040000 to 0x026ba7 (0x19458 bytes) 000365 1787549648.673429 MIRROR_RESYNC 3 mirrors 000367 1787549649.705058 read 0x00e3eb thru 0x01a5fb (0xc211 bytes) 000368 1787549649.714572 mapwrite 0x00837a thru 0x011d22 (0x99a9 bytes) 000371 1787549649.920381 read 0x025f1c thru 0x026ba7 (0xc8c bytes) 000372 1787549649.940162 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26ba8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ba8): Mirror 2: 0xb6c459af (stale) Mirror 3: 0xb6c459af (stale) CRC-32 checksum value for chunk [0, 0x26ba8): Mirror 1: 0x5a81591f Mirror 4: 0x5a81591f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ba8: skipped 000373 1787549650.940085 mapwrite 0x028b3a thru 0x02de26 (0x52ed bytes) 000375 1787549651.196264 mapread 0x00f8e5 thru 0x016a31 (0x714d bytes) 000376 1787549651.214352 mapread 0x0054fc thru 0x00b875 (0x637a bytes) 000377 1787549651.231668 trunc from 0x02de27 to 0x0351c4 (0x739e bytes) 000381 1787549651.280531 MIRROR_RESYNC 4 mirrors 000382 1787549653.002272 mapread 0x00e44f thru 0x010088 (0x1c3a bytes) 000383 1787549653.024154 mapread 0x0109e6 thru 0x012a1f (0x203a bytes) 000384 1787549653.025133 read 0x03512c thru 0x0351c4 (0x99 bytes) 000385 1787549653.030201 mapread 0x032ced thru 0x0351c4 (0x24d8 bytes) 000387 1787549653.042262 write 0x013496 thru 0x01516b (0x1cd6 bytes) truncating to largest ever: 0x3f47f 000388 1787549653.155087 trunc from 0x0351c5 to 0x03f47e (0xa2ba bytes) 000390 1787549653.193442 read 0x01da8e thru 0x0294b9 (0xba2c bytes) 000391 1787549653.220443 mapread 0x02d996 thru 0x03a4d4 (0xcb3f bytes) 000392 1787549653.221445 read 0x03cd04 thru 0x03f47e (0x277b bytes) 000393 1787549653.222347 write 0x033499 thru 0x03e37c (0xaee4 bytes) 000395 1787549653.227911 mapread 0x03b8eb thru 0x03e1d5 (0x28eb bytes) 000396 1787549653.229570 write 0x006d43 thru 0x0150df (0xe39d bytes) 000398 1787549653.233336 mapwrite 0x00974a thru 0x00a32b (0xbe2 bytes) 000399 1787549653.293970 trunc from 0x03f47f to 0x01afbe (0x244c0 bytes) 000400 1787549653.307131 mapread 0x00c67f thru 0x00e2b9 (0x1c3b bytes) 000402 1787549653.309133 mapwrite 0x00d77a thru 0x014739 (0x6fc0 bytes) 000404 1787549653.364203 mapread 0x016d14 thru 0x01afbe (0x42ab bytes) 000406 1787549653.365558 trunc from 0x01afbf to 0x023300 (0x8342 bytes) 000408 1787549653.385079 trunc from 0x023301 to 0x014985 (0xe97b bytes) 000411 1787549653.402858 read 0x014425 thru 0x014985 (0x561 bytes) 000414 1787549653.405228 mapread 0x001269 thru 0x00ae2b (0x9bc3 bytes) 000415 1787549653.406199 mapwrite 0x004ba8 thru 0x0105b3 (0xba0c bytes) 000416 1787549653.459140 mapwrite 0x011311 thru 0x0147bc (0x34ac bytes) 000417 1787549653.496397 mapread 0x013ba8 thru 0x014985 (0xdde bytes) 000418 1787549653.497232 mapread 0x009058 thru 0x00f154 (0x60fd bytes) 000419 1787549653.505218 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14986): Mirror 2: 0x1173e74f (stale) Mirror 4: 0x1173e74f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14986): Mirror 2: 0x1173e74f (stale) Mirror 4: 0x1173e74f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14986: skipped 000421 1787549653.854371 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x14986): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14986): Mirror 2: 0x1173e74f (stale) Mirror 4: 0x1173e74f (stale) CRC-32 checksum value for chunk [0, 0x14986): Mirror 1: 0x149d61dd Mirror 5: 0x149d61dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14986: skipped 000422 1787549654.797150 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x14986): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14986): Mirror 2: 0x1173e74f (stale) Mirror 4: 0x1173e74f (stale) CRC-32 checksum value for chunk [0, 0x14986): Mirror 1: 0x149d61dd Mirror 5: 0x149d61dd Mirror 6: 0x149d61dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14986: skipped 000423 1787549655.681225 trunc from 0x014986 to 0x02ac54 (0x162cf bytes) 000424 1787549655.829997 trunc from 0x02ac55 to 0x036b4e (0xbefa bytes) 000425 1787549655.859379 read 0x0289f0 thru 0x029a8f (0x10a0 bytes) 000428 1787549655.868447 read 0x01f7da thru 0x020b09 (0x1330 bytes) 000429 1787549655.883829 trunc from 0x036b4f to 0x021a82 (0x150cc bytes) 000430 1787549655.907623 read 0x009171 thru 0x00a800 (0x1690 bytes) 000431 1787549655.908244 trunc from 0x021a83 to 0x036ffd (0x1557b bytes) 000432 1787549655.927403 write 0x001c57 thru 0x007882 (0x5c2c bytes) 000433 1787549655.928652 mapwrite 0x00b768 thru 0x0199f8 (0xe291 bytes) 000434 1787549655.989312 mapwrite 0x028a2e thru 0x02c8c5 (0x3e98 bytes) 000436 1787549656.047918 mapwrite 0x02f55a thru 0x037720 (0x81c7 bytes) 000438 1787549656.133881 mapwrite 0x027d91 thru 0x0322e7 (0xa557 bytes) 000439 1787549656.175459 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37721): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37721): Mirror 2: 0x8ff44151 (stale) Mirror 4: 0x8ff44151 (stale) Mirror 5: 0x5a885d96 (stale) Mirror 6: 0x5a885d96 (stale) CRC-32 checksum value for chunk [0, 0x37721): Mirror 1: 0xf11a01e6 Mirror 7: 0xf11a01e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37721: skipped 000443 1787549656.922928 mapread 0x02a410 thru 0x03456e (0xa15f bytes) 000445 1787549656.951250 mapread 0x032982 thru 0x035f4b (0x35ca bytes) 000446 1787549656.963372 mapread 0x000d46 thru 0x00942f (0x86ea bytes) 000450 1787549656.993678 mapread 0x006752 thru 0x00e527 (0x7dd6 bytes) 000452 1787549657.020058 MIRROR_RESYNC 6 mirrors 000453 1787549658.124941 read 0x027614 thru 0x03597b (0xe368 bytes) 000455 1787549658.133138 mapread 0x011328 thru 0x016dc4 (0x5a9d bytes) 000457 1787549658.163777 mapread 0x009874 thru 0x009da0 (0x52d bytes) 000458 1787549658.179626 mapread 0x004e04 thru 0x010c43 (0xbe40 bytes) 000459 1787549658.180305 mapread 0x000cd4 thru 0x0011d5 (0x502 bytes) 000460 1787549658.186023 mapread 0x00577e thru 0x00b29e (0x5b21 bytes) 000462 1787549658.187032 mapread 0x01eb0c thru 0x020348 (0x183d bytes) 000463 1787549658.203171 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000465 1787549658.569054 trunc from 0x037721 to 0x03eab4 (0x7394 bytes) 000466 1787549658.720466 mapread 0x03d21f thru 0x03eab4 (0x1896 bytes) 000467 1787549658.744440 mapread 0x019dcf thru 0x01a9d8 (0xc0a bytes) 000468 1787549658.769810 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eab5): Mirror 5: 0xe0e367e5 (stale) Mirror 6: 0xe0e367e5 (stale) Mirror 7: 0xe0e367e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eab5): Mirror 5: 0xe0e367e5 (stale) Mirror 6: 0xe0e367e5 (stale) Mirror 7: 0xe0e367e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eab5: skipped 000472 1787549659.235013 MIRROR_RESYNC 4 mirrors 000473 1787549660.324729 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000476 1787549660.806588 trunc from 0x03eab5 to 0x000812 (0x3e2a2 bytes) 000477 1787549660.898608 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x813): Mirror 7: 0xe0df1ee5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x813): Mirror 7: 0xe0df1ee5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x813: skipped 000479 1787549661.262179 MIRROR_RESYNC 2 mirrors 000481 1787549662.036957 write 0x0268bf thru 0x028a8c (0x21ce bytes) 000486 1787549662.127588 read 0x025642 thru 0x028a8c (0x344b bytes) 000487 1787549662.147665 read 0x007b99 thru 0x0152d2 (0xd73a bytes) 000488 1787549662.179631 MIRROR_RESYNC 2 mirrors 000489 1787549663.085963 mapwrite 0x0395d0 thru 0x03ca44 (0x3475 bytes) 000490 1787549663.297021 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000493 1787549663.432998 write 0x01b516 thru 0x02a7cb (0xf2b6 bytes) 000494 1787549663.449672 read 0x039bb0 thru 0x03ca44 (0x2e95 bytes) 000495 1787549663.456995 trunc from 0x03ca45 to 0x03cace (0x8a bytes) 000497 1787549663.483381 read 0x01957a thru 0x02673a (0xd1c1 bytes) 000499 1787549663.499171 write 0x01d193 thru 0x01ef0f (0x1d7d bytes) 000500 1787549663.500325 write 0x0275c5 thru 0x03691f (0xf35b bytes) 000501 1787549663.502611 trunc from 0x03cacf to 0x00d86a (0x2f264 bytes) 000506 1787549663.535903 MIRROR_EXTEND to 2 mirrors 000507 1787549664.478980 mapread 0x009022 thru 0x00c76a (0x3749 bytes) 000508 1787549664.502446 mapwrite 0x038be0 thru 0x03ffff (0x7420 bytes) 000509 1787549664.697779 trunc from 0x040000 to 0x01fa2d (0x205d2 bytes) 000510 1787549664.737901 trunc from 0x01fa2e to 0x03e457 (0x1ea2a bytes) 000513 1787549664.761894 write 0x030fba thru 0x039197 (0x81de bytes) 000514 1787549664.774618 write 0x03410c thru 0x03632b (0x2220 bytes) 000516 1787549664.782174 MIRROR_RESYNC 2 mirrors 000517 1787549665.859872 mapread 0x015c83 thru 0x022b18 (0xce96 bytes) 000519 1787549665.927417 read 0x0267ea thru 0x032dd3 (0xc5ea bytes) 000521 1787549665.944147 read 0x00818d thru 0x0093f1 (0x1265 bytes) 000522 1787549665.977874 mapwrite 0x00f2f9 thru 0x0137d8 (0x44e0 bytes) 000523 1787549666.172622 write 0x02ecdd thru 0x0383e4 (0x9708 bytes) 000525 1787549666.186966 read 0x0183c0 thru 0x01ec2c (0x686d bytes) 000526 1787549666.201658 trunc from 0x03e458 to 0x023203 (0x1b254 bytes) 000529 1787549666.249465 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x23204): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23204): Mirror 2: 0x8e0e55da (stale) CRC-32 checksum value for chunk [0, 0x23204): Mirror 1: 0xcd34b187 Mirror 3: 0xcd34b187 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23204: skipped 000530 1787549667.573276 mapwrite 0x000e7d thru 0x0085ea (0x776e bytes) 000531 1787549667.836982 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23204): Mirror 2: 0x8e0e55da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23204): Mirror 2: 0x8e0e55da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23204: skipped 000533 1787549668.186023 read 0x011063 thru 0x016312 (0x52b0 bytes) 000534 1787549668.195133 read 0x021a50 thru 0x023203 (0x17b4 bytes) 000537 1787549668.209532 write 0x00d220 thru 0x00ea0d (0x17ee bytes) 000538 1787549668.216453 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000540 1787549668.415379 read 0x011b61 thru 0x01978d (0x7c2d bytes) 000543 1787549668.425785 mapwrite 0x00f040 thru 0x013b56 (0x4b17 bytes) 000545 1787549668.518090 mapread 0x014a8b thru 0x017e6d (0x33e3 bytes) 000546 1787549668.524113 MIRROR_EXTEND to 2 mirrors 000547 1787549669.466875 mapread 0x0012b0 thru 0x00644d (0x519e bytes) 000549 1787549669.490402 MIRROR_EXTEND to 3 mirrors 000550 1787549670.515929 read 0x001a70 thru 0x007cd5 (0x6266 bytes) 000551 1787549670.531036 mapread 0x020357 thru 0x021626 (0x12d0 bytes) 000552 1787549670.563038 mapwrite 0x02288f thru 0x023675 (0xde7 bytes) 000553 1787549670.727156 write 0x02916c thru 0x03893a (0xf7cf bytes) 000554 1787549670.729355 trunc from 0x03893b to 0x00df3b (0x2a9ff bytes) 000555 1787549670.759569 write 0x010c3c thru 0x011a27 (0xdec bytes) 000556 1787549670.764601 MIRROR_RESYNC 3 mirrors 000558 1787549671.837164 read 0x0044c4 thru 0x004f1d (0xa5a bytes) 000559 1787549671.842969 mapwrite 0x035663 thru 0x03d347 (0x7ce5 bytes) 000560 1787549671.999205 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d348): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d348): Mirror 2: 0xd41a4a94 (stale) Mirror 3: 0xd41a4a94 (stale) CRC-32 checksum value for chunk [0, 0x3d348): Mirror 1: 0xa7832e04 Mirror 4: 0xa7832e04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d348: skipped 000561 1787549672.963676 write 0x0288e1 thru 0x02be3b (0x355b bytes) 000562 1787549673.055153 MIRROR_RESYNC 4 mirrors 000564 1787549674.326309 write 0x036bb1 thru 0x03be4d (0x529d bytes) 000565 1787549674.447190 read 0x00c745 thru 0x01a028 (0xd8e4 bytes) 000566 1787549674.482867 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d348): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d348): Mirror 2: 0xf17b5d43 (stale) Mirror 3: 0xf17b5d43 (stale) Mirror 4: 0xf17b5d43 (stale) CRC-32 checksum value for chunk [0, 0x3d348): Mirror 1: 0x66edd43b Mirror 5: 0x66edd43b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d348: skipped 000567 1787549675.766134 write 0x0027dc thru 0x00363f (0xe64 bytes) 000570 1787549675.865989 mapwrite 0x019df2 thru 0x028dd9 (0xefe8 bytes) 000571 1787549676.054975 mapread 0x00250d thru 0x005c52 (0x3746 bytes) 000572 1787549676.064858 mapread 0x007b03 thru 0x01737c (0xf87a bytes) 000573 1787549676.071547 mapread 0x0397a8 thru 0x03d347 (0x3ba0 bytes) 000574 1787549676.083374 trunc from 0x03d348 to 0x03eab1 (0x176a bytes) 000578 1787549676.120117 mapread 0x038b7b thru 0x03eab1 (0x5f37 bytes) 000579 1787549676.131108 trunc from 0x03eab2 to 0x0035c3 (0x3b4ee bytes) 000580 1787549676.183701 write 0x03afcd thru 0x03ffff (0x5033 bytes) 000583 1787549676.197403 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7a942aa (stale) Mirror 3: 0x7a942aa (stale) Mirror 4: 0x7a942aa (stale) Mirror 5: 0xf1db0d09 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc19dc03c Mirror 6: 0xc19dc03c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000584 1787549677.539332 read 0x015591 thru 0x01d63a (0x80aa bytes) 000585 1787549677.558042 read 0x012608 thru 0x018503 (0x5efc bytes) 000586 1787549677.590001 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7a942aa (stale) Mirror 4: 0x7a942aa (stale) Mirror 5: 0xf1db0d09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7a942aa (stale) Mirror 4: 0x7a942aa (stale) Mirror 5: 0xf1db0d09 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc19dc03c Mirror 6: 0xc19dc03c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000587 1787549678.524977 MIRROR_RESYNC 5 mirrors 000588 1787549680.285617 MIRROR_RESYNC 5 mirrors 000590 1787549680.792199 trunc from 0x040000 to 0x003ef1 (0x3c10e bytes) 000592 1787549680.928359 trunc from 0x003ef2 to 0x00de56 (0x9f65 bytes) 000593 1787549680.963463 write 0x02c952 thru 0x02e992 (0x2041 bytes) 000596 1787549680.966634 mapwrite 0x02ab68 thru 0x039085 (0xe51e bytes) 000597 1787549681.094236 mapwrite 0x031c53 thru 0x035db7 (0x4165 bytes) 000598 1787549681.135399 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39086): Mirror 3: 0xdb756c11 (stale) Mirror 5: 0xdb756c11 (stale) Mirror 6: 0xdb756c11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39086): Mirror 3: 0xdb756c11 (stale) Mirror 5: 0xdb756c11 (stale) Mirror 6: 0xdb756c11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39086: skipped 000599 1787549681.591264 write 0x01ae26 thru 0x01dc96 (0x2e71 bytes) 000602 1787549681.605255 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39086): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39086): Mirror 3: 0xdb756c11 (stale) Mirror 5: 0xdb756c11 (stale) Mirror 6: 0xdb756c11 (stale) CRC-32 checksum value for chunk [0, 0x39086): Mirror 1: 0xe8470b16 Mirror 7: 0xe8470b16 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39086: skipped 000603 1787549682.867134 mapread 0x003c02 thru 0x00a4fc (0x68fb bytes) 000605 1787549682.918367 write 0x01ce54 thru 0x021310 (0x44bd bytes) 000606 1787549683.015829 trunc from 0x039086 to 0x018f1d (0x20168 bytes) 000608 1787549683.071193 trunc from 0x018f1e to 0x024fc1 (0xc0a4 bytes) 000610 1787549683.107419 MIRROR_RESYNC 5 mirrors 000613 1787549684.814997 mapwrite 0x019919 thru 0x0244ec (0xabd4 bytes) 000616 1787549685.047385 mapread 0x015fff thru 0x021a59 (0xba5b bytes) 000617 1787549685.048333 read 0x009642 thru 0x00f5d0 (0x5f8f bytes) 000618 1787549685.084732 MIRROR_RESYNC 5 mirrors 000619 1787549686.465925 trunc from 0x024fc2 to 0x002690 (0x22931 bytes) 000620 1787549686.615717 MIRROR_RESYNC 5 mirrors 000621 1787549687.870061 mapread 0x0015f5 thru 0x002690 (0x109c bytes) 000622 1787549687.888946 read 0x001038 thru 0x002690 (0x1659 bytes) 000624 1787549687.889599 write 0x0350ff thru 0x03ffff (0xaf01 bytes) 000626 1787549688.059199 mapread 0x0129f2 thru 0x017534 (0x4b43 bytes) 000627 1787549688.069043 mapwrite 0x03897e thru 0x03ffff (0x7682 bytes) 000628 1787549688.215449 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x327a75ce (stale) Mirror 6: 0x327a75ce (stale) Mirror 7: 0x327a75ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x327a75ce (stale) Mirror 6: 0x327a75ce (stale) Mirror 7: 0x327a75ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000630 1787549688.828352 MIRROR_RESYNC 4 mirrors 000631 1787549690.152959 write 0x0321df thru 0x03d108 (0xaf2a bytes) 000633 1787549690.231964 mapwrite 0x01ff2a thru 0x02f475 (0xf54c bytes) 000638 1787549690.333153 mapread 0x00edc1 thru 0x0166e5 (0x7925 bytes) 000639 1787549690.343447 read 0x03a56d thru 0x03f859 (0x52ed bytes) 000640 1787549690.356405 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd1990f5 (stale) Mirror 6: 0xd1990f5 (stale) Mirror 7: 0xd1990f5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd86636e Mirror 8: 0xd86636e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000642 1787549691.412979 MIRROR_RESYNC 5 mirrors 000646 1787549692.926102 write 0x020294 thru 0x021bec (0x1959 bytes) 000647 1787549693.009923 trunc from 0x040000 to 0x0256a1 (0x1a95e bytes) 000648 1787549693.064385 read 0x019275 thru 0x0256a1 (0xc42d bytes) 000649 1787549693.074778 mapread 0x00f92a thru 0x011b13 (0x21ea bytes) 000651 1787549693.095564 write 0x00c00a thru 0x010c20 (0x4c17 bytes) 000652 1787549693.096547 write 0x00e225 thru 0x01138a (0x3166 bytes) 000654 1787549693.097814 read 0x00199b thru 0x00a8a2 (0x8f08 bytes) 000655 1787549693.113093 trunc from 0x0256a2 to 0x014f05 (0x1079c bytes) 000656 1787549693.134036 mapwrite 0x00a0b8 thru 0x016e07 (0xcd50 bytes) 000657 1787549693.214994 trunc from 0x016e08 to 0x032fb5 (0x1c1ae bytes) 000661 1787549693.236301 mapwrite 0x003193 thru 0x00b474 (0x82e2 bytes) 000665 1787549693.316357 write 0x035936 thru 0x03e3be (0x8a89 bytes) 000666 1787549693.318247 mapwrite 0x00e1b0 thru 0x00f31f (0x1170 bytes) 000668 1787549693.387069 trunc from 0x03e3bf to 0x00eec0 (0x2f4fe bytes) 000669 1787549693.410181 mapwrite 0x01c6ab thru 0x01ecb1 (0x2607 bytes) 000671 1787549693.513009 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ecb2): Mirror 6: 0x9278d249 (stale) Mirror 7: 0x9278d249 (stale) Mirror 8: 0x9278d249 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ecb2): Mirror 6: 0x9278d249 (stale) Mirror 7: 0x9278d249 (stale) Mirror 8: 0x9278d249 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ecb2: skipped 000672 1787549693.869290 write 0x008354 thru 0x00ef38 (0x6be5 bytes) 000674 1787549693.888863 mapread 0x01a21c thru 0x01ecb1 (0x4a96 bytes) 000675 1787549693.913119 read 0x00fcd8 thru 0x0110e5 (0x140e bytes) 000678 1787549693.918934 mapwrite 0x021680 thru 0x02f9e0 (0xe361 bytes) 000679 1787549694.004826 read 0x02f364 thru 0x02f9e0 (0x67d bytes) 000682 1787549694.006324 read 0x00fab7 thru 0x010a4a (0xf94 bytes) 000683 1787549694.006865 mapread 0x004e0e thru 0x00dec2 (0x90b5 bytes) 000684 1787549694.021251 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f9e1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f9e1): Mirror 6: 0x65dc0134 (stale) Mirror 7: 0x65dc0134 (stale) Mirror 8: 0x65dc0134 (stale) CRC-32 checksum value for chunk [0, 0x2f9e1): Mirror 1: 0x192dc622 Mirror 9: 0x192dc622 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f9e1: skipped 000689 1787549695.218187 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f9e1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f9e1): Mirror 6: 0x65dc0134 (stale) Mirror 7: 0x65dc0134 (stale) Mirror 8: 0x65dc0134 (stale) CRC-32 checksum value for chunk [0, 0x2f9e1): Mirror 1: 0x192dc622 Mirror 9: 0x192dc622 Mirror 10: 0x192dc622 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f9e1: skipped 000691 1787549696.412910 read 0x02200e thru 0x02af0a (0x8efd bytes) 000692 1787549696.427843 mapread 0x01af8b thru 0x028182 (0xd1f8 bytes) 000694 1787549696.446438 read 0x017fe1 thru 0x02478b (0xc7ab bytes) 000695 1787549696.447097 write 0x023271 thru 0x02ece5 (0xba75 bytes) 000696 1787549696.588275 trunc from 0x02f9e1 to 0x00d8ef (0x220f1 bytes) 000697 1787549696.639910 MIRROR_RESYNC 6 mirrors 000698 1787549697.959390 mapread 0x00a631 thru 0x00d8ef (0x32bf bytes) 000700 1787549698.003961 read 0x009707 thru 0x00d8ef (0x41e9 bytes) 000702 1787549698.010912 MIRROR_EXTEND to 7 mirrors 000705 1787549699.039022 write 0x0212a8 thru 0x02844e (0x71a7 bytes) 000706 1787549699.137624 MIRROR_RESYNC 7 mirrors 000708 1787549701.480859 mapread 0x023bec thru 0x02844e (0x4863 bytes) 000711 1787549701.499389 trunc from 0x02844f to 0x01983c (0xec12 bytes) 000714 1787549701.572892 trunc from 0x01983d to 0x004bc2 (0x14c7a bytes) 000716 1787549701.599257 write 0x01a85c thru 0x024989 (0xa12e bytes) 000717 1787549701.602119 mapwrite 0x00616b thru 0x00732d (0x11c3 bytes) 000719 1787549701.663616 read 0x01e301 thru 0x024989 (0x6689 bytes) 000720 1787549701.664442 read 0x0236dc thru 0x024989 (0x12ae bytes) 000724 1787549701.671529 mapread 0x00da56 thru 0x01b729 (0xdcd4 bytes) 000725 1787549701.682870 mapread 0x00257c thru 0x011130 (0xebb5 bytes) 000726 1787549701.683897 mapread 0x00e95e thru 0x0165a1 (0x7c44 bytes) 000727 1787549701.684762 mapread 0x0072da thru 0x0077d3 (0x4fa bytes) 000729 1787549701.686693 mapwrite 0x01e306 thru 0x0249d1 (0x66cc bytes) 000730 1787549701.816322 trunc from 0x0249d2 to 0x00a42f (0x1a5a2 bytes) 000732 1787549701.839425 mapread 0x005121 thru 0x007139 (0x2019 bytes) 000735 1787549701.849047 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xa430): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa430): Mirror 6: 0xc1852a62 (stale) Mirror 7: 0xc1852a62 (stale) Mirror 8: 0xc1852a62 (stale) Mirror 9: 0xc1852a62 (stale) Mirror 10: 0xc1852a62 (stale) Mirror 11: 0xc1852a62 (stale) CRC-32 checksum value for chunk [0, 0xa430): Mirror 1: 0x4284e63d Mirror 12: 0x4284e63d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa430: skipped 000740 1787549703.162949 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xa430): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa430): Mirror 6: 0xc1852a62 (stale) Mirror 7: 0xc1852a62 (stale) Mirror 8: 0xc1852a62 (stale) Mirror 9: 0xc1852a62 (stale) Mirror 10: 0xc1852a62 (stale) Mirror 11: 0xc1852a62 (stale) CRC-32 checksum value for chunk [0, 0xa430): Mirror 1: 0x4284e63d Mirror 12: 0x4284e63d Mirror 13: 0x4284e63d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa430: skipped 000741 1787549704.489116 mapwrite 0x01e8ad thru 0x01f9d0 (0x1124 bytes) 000742 1787549704.746363 trunc from 0x01f9d1 to 0x01fffd (0x62d bytes) 000744 1787549704.794639 write 0x01121c thru 0x011a95 (0x87a bytes) 000745 1787549704.807458 write 0x0117e8 thru 0x0189a6 (0x71bf bytes) 000746 1787549704.817535 mapread 0x009daf thru 0x016486 (0xc6d8 bytes) 000747 1787549704.835109 mapread 0x004dcf thru 0x010dbc (0xbfee bytes) 000755 1787549704.859054 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fffe): Mirror 6: 0x8f41e256 (stale) Mirror 7: 0x8f41e256 (stale) Mirror 8: 0x8f41e256 (stale) Mirror 9: 0x8f41e256 (stale) Mirror 10: 0x8f41e256 (stale) Mirror 11: 0x8f41e256 (stale) Mirror 13: 0x5f79376c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fffe): Mirror 6: 0x8f41e256 (stale) Mirror 7: 0x8f41e256 (stale) Mirror 8: 0x8f41e256 (stale) Mirror 9: 0x8f41e256 (stale) Mirror 10: 0x8f41e256 (stale) Mirror 11: 0x8f41e256 (stale) Mirror 13: 0x5f79376c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fffe: skipped 000756 1787549705.619329 write 0x013df6 thru 0x01e561 (0xa76c bytes) 000757 1787549705.635543 trunc from 0x01fffe to 0x031859 (0x1185c bytes) 000758 1787549705.670447 write 0x009394 thru 0x0183f5 (0xf062 bytes) 000759 1787549705.688355 mapwrite 0x005286 thru 0x01151d (0xc298 bytes) 000760 1787549705.829219 write 0x02a808 thru 0x035cea (0xb4e3 bytes) 000761 1787549705.838601 MIRROR_RESYNC 8 mirrors 000762 1787549708.507710 read 0x030132 thru 0x035cea (0x5bb9 bytes) 000764 1787549708.517043 write 0x037cad thru 0x03e119 (0x646d bytes) 000765 1787549708.602196 mapread 0x00258f thru 0x010ae0 (0xe552 bytes) 000766 1787549708.624992 mapread 0x001618 thru 0x0030e0 (0x1ac9 bytes) 000768 1787549708.626849 trunc from 0x03e11a to 0x012e52 (0x2b2c7 bytes) 000770 1787549708.657098 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12e53): Mirror 6: 0x3efeb312 (stale) Mirror 7: 0x3efeb312 (stale) Mirror 8: 0x3efeb312 (stale) Mirror 9: 0x3efeb312 (stale) Mirror 10: 0x3efeb312 (stale) Mirror 13: 0x3efeb312 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12e53): Mirror 6: 0x3efeb312 (stale) Mirror 7: 0x3efeb312 (stale) Mirror 8: 0x3efeb312 (stale) Mirror 9: 0x3efeb312 (stale) Mirror 10: 0x3efeb312 (stale) Mirror 13: 0x3efeb312 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12e53: skipped 000773 1787549709.140411 mapread 0x000886 thru 0x00a284 (0x99ff bytes) 000774 1787549709.162215 trunc from 0x012e53 to 0x01963f (0x67ed bytes) 000775 1787549709.181129 mapwrite 0x005b07 thru 0x012cbd (0xd1b7 bytes) 000778 1787549709.280841 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x19640): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19640): Mirror 6: 0xc56c40d8 (stale) Mirror 7: 0xc56c40d8 (stale) Mirror 8: 0xc56c40d8 (stale) Mirror 9: 0xc56c40d8 (stale) Mirror 10: 0xc56c40d8 (stale) Mirror 13: 0xc56c40d8 (stale) CRC-32 checksum value for chunk [0, 0x19640): Mirror 1: 0xb8a9ac39 Mirror 14: 0xb8a9ac39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19640: skipped 000779 1787549710.331030 write 0x00828f thru 0x01058b (0x82fd bytes) 000780 1787549710.477302 mapwrite 0x005ea5 thru 0x012996 (0xcaf2 bytes) 000783 1787549710.610737 mapread 0x005807 thru 0x00b2a8 (0x5aa2 bytes) 000784 1787549710.611684 write 0x03be02 thru 0x03ffff (0x41fe bytes) 000785 1787549710.613603 trunc from 0x040000 to 0x014f02 (0x2b0fd bytes) 000786 1787549710.656853 read 0x00f47d thru 0x010709 (0x128d bytes) 000787 1787549710.657926 trunc from 0x014f03 to 0x002300 (0x12c02 bytes) 000791 1787549710.682191 MIRROR_RESYNC 8 mirrors 000794 1787549712.165265 trunc from 0x002301 to 0x012635 (0x10335 bytes) 000795 1787549712.298019 mapwrite 0x01d027 thru 0x02bd24 (0xecfe bytes) 000799 1787549712.435671 mapwrite 0x00df8c thru 0x011ae5 (0x3b5a bytes) 000801 1787549712.528602 mapread 0x013465 thru 0x016bac (0x3748 bytes) 000802 1787549712.536220 trunc from 0x02bd25 to 0x005051 (0x26cd3 bytes) 000804 1787549712.558108 mapwrite 0x02c42e thru 0x031153 (0x4d26 bytes) 000805 1787549712.643057 trunc from 0x031154 to 0x035e3f (0x4cec bytes) 000810 1787549712.666992 write 0x013564 thru 0x01777a (0x4217 bytes) 000813 1787549712.679178 mapread 0x01129e thru 0x01d6dc (0xc43f bytes) 000816 1787549712.733395 trunc from 0x035e40 to 0x01929d (0x1cba2 bytes) 000817 1787549712.774615 mapwrite 0x026349 thru 0x02e5bd (0x8275 bytes) 000818 1787549712.892758 mapwrite 0x00c9eb thru 0x010ce4 (0x42fa bytes) 000819 1787549712.968801 write 0x037a83 thru 0x03ffff (0x857d bytes) 000821 1787549712.983608 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x564bc131 (stale) Mirror 8: 0x564bc131 (stale) Mirror 9: 0x564bc131 (stale) Mirror 10: 0x564bc131 (stale) Mirror 13: 0x564bc131 (stale) Mirror 14: 0x564bc131 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x564bc131 (stale) Mirror 8: 0x564bc131 (stale) Mirror 9: 0x564bc131 (stale) Mirror 10: 0x564bc131 (stale) Mirror 13: 0x564bc131 (stale) Mirror 14: 0x564bc131 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000825 1787549713.429396 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x564bc131 (stale) Mirror 8: 0x564bc131 (stale) Mirror 9: 0x564bc131 (stale) Mirror 10: 0x564bc131 (stale) Mirror 13: 0x564bc131 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x564bc131 (stale) Mirror 8: 0x564bc131 (stale) Mirror 9: 0x564bc131 (stale) Mirror 10: 0x564bc131 (stale) Mirror 13: 0x564bc131 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000827 1787549713.931533 trunc from 0x040000 to 0x034ac0 (0xb53f bytes) 000829 1787549713.956403 mapwrite 0x03d905 thru 0x03ffff (0x26fb bytes) 000832 1787549714.080358 read 0x02178f thru 0x02ad2e (0x95a0 bytes) 000833 1787549714.113855 read 0x00543b thru 0x0098fa (0x44c0 bytes) 000836 1787549714.156065 MIRROR_RESYNC 6 mirrors 000837 1787549716.373796 MIRROR_RESYNC 6 mirrors 000840 1787549716.919789 write 0x039356 thru 0x03ffff (0x6caa bytes) 000841 1787549717.042327 mapwrite 0x032949 thru 0x03fa53 (0xd10b bytes) 000842 1787549717.183851 read 0x020527 thru 0x0294a4 (0x8f7e bytes) 000844 1787549717.195401 mapwrite 0x018c27 thru 0x01d26e (0x4648 bytes) 000845 1787549717.256262 mapwrite 0x01be31 thru 0x023d9e (0x7f6e bytes) 000846 1787549717.310075 write 0x0352a0 thru 0x03ffff (0xad60 bytes) 000847 1787549717.312277 mapwrite 0x0276af thru 0x030ca7 (0x95f9 bytes) 000848 1787549717.369430 mapwrite 0x023179 thru 0x027e88 (0x4d10 bytes) 000850 1787549717.436750 write 0x018430 thru 0x019149 (0xd1a bytes) 000851 1787549717.437750 write 0x00d2b9 thru 0x01542a (0x8172 bytes) 000854 1787549717.464914 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa9817d67 (stale) Mirror 8: 0xa9817d67 (stale) Mirror 9: 0xa9817d67 (stale) Mirror 10: 0xa9817d67 (stale) Mirror 13: 0xa9817d67 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8de9299e Mirror 14: 0x8de9299e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1787549718.893358 read 0x037a2b thru 0x03ffff (0x85d5 bytes) 000859 1787549718.994542 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa9817d67 (stale) Mirror 8: 0xa9817d67 (stale) Mirror 9: 0xa9817d67 (stale) Mirror 10: 0xa9817d67 (stale) Mirror 13: 0xa9817d67 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8de9299e Mirror 14: 0x8de9299e Mirror 15: 0x8de9299e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000860 1787549720.496149 trunc from 0x040000 to 0x023016 (0x1cfe9 bytes) 000863 1787549720.703561 read 0x017898 thru 0x0188bf (0x1028 bytes) 000864 1787549720.717078 write 0x033c7a thru 0x03752c (0x38b3 bytes) 000865 1787549720.718381 mapwrite 0x000159 thru 0x00450f (0x43b7 bytes) 000866 1787549720.913095 trunc from 0x03752d to 0x036cff (0x82d bytes) 000867 1787549720.942894 write 0x01e570 thru 0x02b940 (0xd3d1 bytes) 000868 1787549720.959061 write 0x03de4b thru 0x03ffff (0x21b5 bytes) 000874 1787549720.969957 read 0x027430 thru 0x035f3d (0xeb0e bytes) 000875 1787549720.983445 mapwrite 0x00d694 thru 0x0114d7 (0x3e44 bytes) 000876 1787549721.036771 trunc from 0x040000 to 0x033684 (0xc97b bytes) 000879 1787549721.066124 trunc from 0x033685 to 0x0278c3 (0xbdc1 bytes) 000880 1787549721.104796 write 0x035d05 thru 0x03af3f (0x523b bytes) 000881 1787549721.108774 read 0x02749d thru 0x034f4a (0xdaae bytes) 000882 1787549721.130473 mapwrite 0x002b88 thru 0x00fd00 (0xd179 bytes) 000884 1787549721.231586 mapread 0x033d44 thru 0x036554 (0x2811 bytes) 000885 1787549721.232547 mapread 0x026d5b thru 0x02a684 (0x392a bytes) 000888 1787549721.237181 mapwrite 0x01aedc thru 0x01fd1d (0x4e42 bytes) 000889 1787549721.318958 mapwrite 0x02d9a4 thru 0x033b3e (0x619b bytes) 000890 1787549721.373094 mapwrite 0x0154fd thru 0x021e26 (0xc92a bytes) 000891 1787549721.411361 trunc from 0x03af40 to 0x02c2a2 (0xec9d bytes) 000894 1787549721.432875 read 0x0185f3 thru 0x02211e (0x9b2c bytes) 000895 1787549721.444214 MIRROR_RESYNC 8 mirrors 000896 1787549723.246205 trunc from 0x02c2a3 to 0x00e953 (0x1d94f bytes) 000900 1787549723.445135 write 0x038900 thru 0x03ffff (0x7700 bytes) 000901 1787549723.447213 trunc from 0x040000 to 0x0140f2 (0x2bf0d bytes) 000902 1787549723.482919 MIRROR_RESYNC 8 mirrors 000904 1787549725.361000 MIRROR_RESYNC 8 mirrors 000905 1787549725.999164 MIRROR_EXTEND to 9 mirrors 000906 1787549727.032963 mapwrite 0x021f99 thru 0x027d4e (0x5db6 bytes) 000907 1787549727.228593 read 0x00fd6e thru 0x01748f (0x7722 bytes) 000909 1787549727.240226 write 0x02de67 thru 0x031514 (0x36ae bytes) 000910 1787549727.241931 read 0x00de66 thru 0x014f12 (0x70ad bytes) 000913 1787549727.253196 mapwrite 0x0347ae thru 0x03693f (0x2192 bytes) 000914 1787549727.338253 write 0x0387d7 thru 0x03ffff (0x7829 bytes) 000915 1787549727.339740 mapread 0x02d408 thru 0x02ea16 (0x160f bytes) 000916 1787549727.340538 trunc from 0x040000 to 0x022690 (0x1d96f bytes) 000918 1787549727.401884 write 0x02aadb thru 0x037fcc (0xd4f2 bytes) 000919 1787549727.404244 mapread 0x00ba81 thru 0x0171d4 (0xb754 bytes) 000920 1787549727.410663 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x37fcd): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fcd): Mirror 7: 0x89e43ed1 (stale) Mirror 8: 0x89e43ed1 (stale) Mirror 9: 0x89e43ed1 (stale) Mirror 10: 0x89e43ed1 (stale) Mirror 13: 0x89e43ed1 (stale) Mirror 14: 0x89e43ed1 (stale) Mirror 15: 0x89e43ed1 (stale) Mirror 16: 0x89e43ed1 (stale) CRC-32 checksum value for chunk [0, 0x37fcd): Mirror 1: 0x268aa52c Mirror 17: 0x268aa52c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fcd: skipped 000922 1787549728.883165 trunc from 0x037fcd to 0x016880 (0x2174c bytes) 000925 1787549728.984056 write 0x01279f thru 0x020c61 (0xe4c3 bytes) 000927 1787549728.994384 write 0x004b58 thru 0x013ee6 (0xf38f bytes) 000932 1787549729.007683 mapwrite 0x01f048 thru 0x023926 (0x48df bytes) 000935 1787549729.134569 write 0x005f85 thru 0x00ee95 (0x8f11 bytes) 000937 1787549729.138122 mapwrite 0x019de1 thru 0x029567 (0xf787 bytes) 000940 1787549729.241660 read 0x006f73 thru 0x011d9d (0xae2b bytes) 000942 1787549729.242699 trunc from 0x029568 to 0x00a025 (0x1f542 bytes) 000947 1787549729.266925 mapwrite 0x020811 thru 0x02a2de (0x9ace bytes) 000948 1787549729.364063 write 0x03f9a4 thru 0x03ffff (0x65c bytes) 000950 1787549729.367355 read 0x039482 thru 0x03ffff (0x6b7e bytes) 000952 1787549729.382971 write 0x020658 thru 0x022cab (0x2654 bytes) 000953 1787549729.384137 read 0x01e807 thru 0x02c3a0 (0xdb9a bytes) 000954 1787549729.390759 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb9819814 (stale) Mirror 8: 0xb9819814 (stale) Mirror 9: 0xb9819814 (stale) Mirror 10: 0xb9819814 (stale) Mirror 13: 0xb9819814 (stale) Mirror 14: 0xb9819814 (stale) Mirror 15: 0xb9819814 (stale) Mirror 16: 0xb9819814 (stale) Mirror 17: 0xf1106b69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaac0e4fb Mirror 18: 0xaac0e4fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000955 1787549730.747066 read 0x003cf2 thru 0x00d494 (0x97a3 bytes) 000958 1787549730.783638 mapwrite 0x022321 thru 0x02db06 (0xb7e6 bytes) 000960 1787549731.109375 read 0x026e99 thru 0x02fece (0x9036 bytes) 000961 1787549731.110070 write 0x01ec33 thru 0x028f98 (0xa366 bytes) 000962 1787549731.118016 mapwrite 0x01ca83 thru 0x01fbe6 (0x3164 bytes) 000963 1787549731.187916 mapread 0x002e3e thru 0x0088a1 (0x5a64 bytes) 000964 1787549731.205091 read 0x02a32e thru 0x02d261 (0x2f34 bytes) 000965 1787549731.207361 write 0x016058 thru 0x02234a (0xc2f3 bytes) 000967 1787549731.218988 trunc from 0x040000 to 0x006c11 (0x393ee bytes) 000968 1787549731.253428 mapwrite 0x021024 thru 0x0307b2 (0xf78f bytes) 000969 1787549731.381775 mapread 0x00f322 thru 0x0187d1 (0x94b0 bytes) 000971 1787549731.385238 read 0x01be9f thru 0x027eec (0xc04e bytes) 000976 1787549731.387930 mapwrite 0x03f7a8 thru 0x03ffff (0x858 bytes) 000978 1787549731.594785 mapwrite 0x027e83 thru 0x03628c (0xe40a bytes) 000981 1787549731.728957 read 0x03169c thru 0x039935 (0x829a bytes) 000982 1787549731.741050 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb9819814 (stale) Mirror 8: 0xb9819814 (stale) Mirror 9: 0xb9819814 (stale) Mirror 10: 0xb9819814 (stale) Mirror 13: 0xb9819814 (stale) Mirror 14: 0xb9819814 (stale) Mirror 15: 0xb9819814 (stale) Mirror 16: 0xb9819814 (stale) Mirror 17: 0xf1106b69 (stale) Mirror 18: 0xaac0e4fb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6d324337 Mirror 19: 0x6d324337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000986 1787549733.511402 trunc from 0x040000 to 0x017776 (0x28889 bytes) 000987 1787549733.622972 write 0x01e5d3 thru 0x02c3d7 (0xde05 bytes) 000990 1787549733.632770 mapread 0x01a5fa thru 0x02693a (0xc341 bytes) 000991 1787549733.657060 read 0x005668 thru 0x008ac4 (0x345d bytes) 000992 1787549733.665824 mapread 0x0137bc thru 0x015836 (0x207b bytes) 000994 1787549733.679696 mapwrite 0x00a40b thru 0x016784 (0xc37a bytes) 000996 1787549733.758902 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c3d8): Mirror 7: 0x6507b546 (stale) Mirror 9: 0x6507b546 (stale) Mirror 10: 0x6507b546 (stale) Mirror 13: 0x6507b546 (stale) Mirror 14: 0x6507b546 (stale) Mirror 15: 0x6507b546 (stale) Mirror 16: 0x6507b546 (stale) Mirror 17: 0xf0d28f06 (stale) Mirror 18: 0x2dd7ad3e (stale) Mirror 19: 0x7d0c7731 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3d8): Mirror 7: 0x6507b546 (stale) Mirror 9: 0x6507b546 (stale) Mirror 10: 0x6507b546 (stale) Mirror 13: 0x6507b546 (stale) Mirror 14: 0x6507b546 (stale) Mirror 15: 0x6507b546 (stale) Mirror 16: 0x6507b546 (stale) Mirror 17: 0xf0d28f06 (stale) Mirror 18: 0x2dd7ad3e (stale) Mirror 19: 0x7d0c7731 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3d8: skipped 000997 1787549734.665750 MIRROR_RESYNC 11 mirrors 000998 1787549737.224316 MIRROR_EXTEND to 12 mirrors 000999 1787549738.615147 mapread 0x0002bc thru 0x006dd4 (0x6b19 bytes) 001000 1787549738.673694 mapwrite 0x021488 thru 0x027a77 (0x65f0 bytes) All operations completed A-OK! PASS 70a (121s) == sanity-flr test 71: check mirror extend parent fid ==== 01:35:45 (1787549745) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 01:35:50 (1787549750) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218827 s, 9.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162116 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 01:36:00 (1787549760) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2550752 .. Extending file size to 5080864 .. done Extending file size to 6950624 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7494752 .. Extending file size to 8585344 .. Extending file size to 7115360 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7981120 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8207360 .. Extending file size to 8523456 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9400320 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57393 57394 57397 57398 57399 done 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) 192.168.203.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg349-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:37:22 (1787549842) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1506432 .. Extending file size to 5877824 .. failed Extending file size to 7076096 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8657632 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7500928 .. Extending file size to 8158304 .. Extending file size to 8180960 .. Extending file size to 8414016 .. Extending file size to 8895488 .. Extending file size to 8665312 .. Extending file size to 9044608 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58743 58745 58746 58747 58750 done 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) 192.168.203.149@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg349-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:38:45 (1787549925) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12972 s, 16.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.417075 s, 5.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc4b81b39 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:39:08 (1787549948) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131366 s, 16.0 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 01:39:18 (1787549958) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d9:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } PASS 203 (14s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:39:32 (1787549972) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.372693 s, 5.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (13s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:39:45 (1787549985) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261478 s, 8.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:39:55 (1787549995) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.636592 s, 3.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } PASS 204c (38s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:40:33 (1787550033) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg349-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886532 s, 11.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:41:34 (1787550094) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg349-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.295534 s, 7.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:42:32 (1787550152) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg349-server mds-ost sync done. Creating new pool test_204f oleg349-server: Pool lustre.test_204f created Adding targets to pool oleg349-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg349-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211636 s, 9.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg349-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg349-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg349-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:43:52 (1787550232) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:44:02 (1787550242) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:44:12 (1787550252) oleg349-server: Pool lustre.test_206 created Adding targets to pool oleg349-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg349-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg349-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg349-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg349-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:44:42 (1787550282) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401034 s, 26.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571491 s, 18.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc02aa605 Mirror 3: 0xc02aa605 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:44:53 (1787550293) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:44:58 (1787550298) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:45:02 (1787550302) Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0981727 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1df0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1db0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1db0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1db0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1880000 Read this (4 bytes): BBBB 192.168.203.149@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg349-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:45:41 (1787550341) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0239936 s, 341 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:45:47 (1787550347) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361344 s, 29.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fe lmm_fid: [0x200000404:0x1fe:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fe lmm_fid: [0x200000404:0x1fe:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834430 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 01:45:49.000000000 -0400 Modify: 2026-08-24 01:45:49.000000000 -0400 Change: 2026-08-24 01:45:50.000000000 -0400 Birth: 2026-08-24 01:45:48.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:45:55 (1787550355) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:45:59 (1787550359) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.563179 s, 72.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0684252 s, 59.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 01:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x200:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 01:46 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:46:07 (1787550367) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:46:16 (1787550376) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 1942 sec =========== 01:46:29 (1787550389) === sanity-flr: start cleanup 01:46:31 (1787550391) === === sanity-flr: finish cleanup 01:46:41 (1787550401) ===